I hope you all had a fantastic thanksgiving yesterday and are feeling extra full today.
It’s no surprise that today, Black Friday 2020 will be the biggest online shopping day, ever. Global lockdown, pent up consumer demand and the promise of discounts have created the perfect storm.
But, It could have been a very different story.
On Tuesday AWS announced the ‘US-EAST-1 region is suffering a “severely impaired” service’
What that actually meant was lots of AWS services are down and so were businesses. You’ve probably talked with you tech leaders about AWS once or twice. They’ll most likely have mentioned CloudWatch, DynamoDB, Lambda, Managed Blockchain etc.
Well, these services all went down.
The outage lasted for about 3 hours but access was throttled for the better part of a day.
The problem was worse than it looked.
You might think, well I could not use the US-EAST-1 region? Or thats fine, it only affects East Coast companies.
Not quite.
AWS defaults to US-EAST-1 when you don’t specify a region, which most people don’t. US-EAST-1 is, according to the AWS documentation, “the default Region for all API calls.”
How did even this happen and is it fixed?
It’s all to do with the Kinesis service. This service manages real-time data, such as telemetry from IoT devices. It’s how your kettle knows its time to boil or dishwasher knows you just opened it.
It seems to be fixed for now, but since AWS haven’t released a full statement yet the real cause and solution aren’t known.
This happening two days before Black Friday 2020 should have kept your CTO awake.
This was a close call.
Tech leaders should have taken note, resilience is the name of the game.
- How many of you have failover plans to other cloud providers just in case?
- If you do when was the last time you actually did a full failover test?
- Did this make you think about how dependent your businesses are on AWS?
What are you doing to do about it?
Thomas
You can sign up for more ‘unconsidered needs’ below.
For more ‘unconsidered needs’ sign up below.