zlacker

I hope this is only temporary. Where else will we discuss AWS outages when AWS goes down?

Not even a joke.

>>dralle+(OP)
The IP appears to be us-west-2, so we will still be able to discuss us-east-1 outages alright!

>>dralle+(OP)
If architected correctly, stuff deployed into aws stays up during all but the most extreme outages.

>>dralle+(OP)
It appears to be pointing at a bare EC2 instance, no doubt a lift-and-shift.

Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic.

>>f0e4c2+X
And then where do we discuss these extreme outages?

>>oars+o2
where did you discuss outages of HN's previous provider?

>>oars+o2
reddit.... shudders

>>dralle+(OP)
Twitter is nice for learning about outages...

>>cherio+h
but... the s3 buckets are in us-east-1, and postgres in ap-southeast-2. More regions better than one, for maximum impact with minimum effort.

>>shawnz+Y2
Less people are interested in outages of HN's previous provider.

>>oars+o2
If there's an extreme AWS outage, you're pretty much stuck to in-person or POTS.

>>oars+o2
HAM Radio

>>oars+o2
Better to discuss the next day when it's over than seeing bunch of upset comments realtime posted.

>>VoidWh+vd
People joke but from the top of a decent mountain I can reach 45 miles into the bay area from me on a $150 2m radio in my truck with rooftop antenna.

>>rubyis+b3
maybe not...

https://aws.amazon.com/solutions/case-studies/reddit-aurora-...

>>namech+Mg
That is so cool. If I had that (truck+radio) I'd be very tempted to scatter a few cheap relay nodes at good locations in the bay area. Monkeybrains+comcast is quite reliable.

>>ignora+k5
Is there at least one case when whole region went down ;)) ?

>>pojzon+Zt
At least once, in 2012? I remember because just two days before we had fully switched over to AWS and hadn’t done multi-region yet. We got our first downtime in 10 years and there was nothing we could do, unlike when we had the servers in a colo. We were at the mercy of Amazon. After that, we moved everything back to real physical servers.

>>f0e4c2+X
Of those "correctly" architected apps, most are not properly tested for the failovers and won't actually work as architected (because of your own bugs or because aws failover stuff has bugs and you can't even test it).

Eg, falls over due to steep traffic spikes caused by outages when autoscaling mechanisms get previously unseen levels of load increases and enter some yoyo oscillation pattern, whole AZ is overloaded because all the failovers from the other failing AZ triggering at once, hit circuit breakers, spin up too slowly to ever pass health checks etc. Or can't detect something becoming glacially slow but not outright failing.

See eg https://www.theverge.com/2021/12/22/22849780/amazon-aws-is-d... & https://www.theverge.com/2020/11/25/21719396/amazon-web-serv... etc (many more examples are out there)

>>namech+Mg
The problem with that is the only people you have to talk to is hams.

(Ex ham)

>>dralle+(OP)
/r/sysadmin

>>iasay+IA
Well same for computers & the internet :)

>>nwh5jg+fJ
Fair point!

>>ignora+k5
The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service.

Consider the extreme case where your service is scattered over every AWS region: here an outage of any AWS region is guaranteed to take down your service.

Compare that to the case where your service is bound to only one region: then the odds of a single region outage taking down your entire service is reduced to 1 out of however many regions AWS has (assuming each region has an equal chance of suffering an outage).

To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway).

>>banana+p1
> Even during the most extreme AWS events, my EC2 instances running dedicated servers kept seeing Internet traffic.

You were just lucky enough not to have been affected by AWS outages, but many others were.

You can get a lot of resilience to failure on AWS, but simply spinning up a dedicated EC2 instance is not nearly enough.

>>pmoria+nL
> The more regions a service is scattered over the greater the odds are that a single region outage somewhere in AWS will take down the whole service.

Agree. I think I should have suffixed a /s to my comment above.

> To guard against outages, the failover service has to be scattered over entirely different regions (or, even better, on an entirely different service provider... which is probably a good idea anyway).

Something, something... the greatest trick the devil (bigcloud) ever pulled...

>>mekste+Ed
If the AWS status wasn’t a static HTML page, I would agree.

>>pojzon+Zt
Per the spreadsheet here https://awsmaniac.com/aws-outages/ :

There seem to have been multiple "full" outages in 2011-12 in AWS' us-east-1 region, which, granted, is the oldest AWS region and likely has a bunch of legacy stuff. By "full" outages I mean that a few core services fell over but the entire region become inaccessible due to those core failures.

>>samspe+lV1
Its over 10 years ago tho. Are there any RECENT full region outages ?

Im forseeing a full downtime in Frankfurt this winter tho. Germany is in really bad position when it comes to electricity.