zlacker

GitHub Incident

submitted by jcalab+(OP) on 2023-05-16 21:16:09 | 112 points 82 comments
[view article] [source] [links] [go to bottom]
replies(22): >>openth+L >>JoshGl+01 >>rvz+91 >>upbeat+c1 >>nwradi+l1 >>aj397+m1 >>hoofhe+r1 >>captai+K1 >>nrabul+O1 >>kossTK+P1 >>mullin+b2 >>algane+O2 >>ivanre+i3 >>nabnob+u3 >>vb-844+s4 >>tastys+J4 >>exogen+O5 >>mulmen+16 >>labste+y7 >>Popcor+V8 >>frellu+Yf >>junon+Uv
1. openth+L[view] [source] 2023-05-16 21:19:44
>>jcalab+(OP)
See also this thread: https://news.ycombinator.com/item?id=35967797 which just came up about GitHub stability sentiment.
2. JoshGl+01[view] [source] 2023-05-16 21:20:52
>>jcalab+(OP)
github.com is 500'ing for me
3. rvz+91[view] [source] 2023-05-16 21:21:43
>>jcalab+(OP)
Again? Last time this happened was 5 days ago. [0]

It is not even the end of the month and the outages are increasing every month and it is now chronically unreliable. Seriously, we have given GitHub more than 3 years to improve and it clearly isn't working. That is plenty of time.

At this point, you might as well self-host like the rest of the open source projects out there, since GitHub is falling apart every week and it seems to be more reliable to self-host than to sit on GitHub, go all in and tolerate these outages every calendar month.

[0] https://news.ycombinator.com/item?id=35903116

replies(2): >>jamal-+B1 >>crote+03
4. upbeat+c1[view] [source] 2023-05-16 21:21:54
>>jcalab+(OP)
Yep, I'm unable to push.
5. nwradi+l1[view] [source] 2023-05-16 21:22:30
>>jcalab+(OP)
Login is currently broken. Has been for about 10m…
6. aj397+m1[view] [source] 2023-05-16 21:22:39
>>jcalab+(OP)
Time to move to gitlab or something else.
replies(1): >>mulmen+ul
7. hoofhe+r1[view] [source] 2023-05-16 21:23:00
>>jcalab+(OP)
Can't create new branches :(

It seems like GitHub is down at least once a month now, but more like weekly on average.

replies(1): >>sethd+v3
◧◩
8. jamal-+B1[view] [source] [discussion] 2023-05-16 21:23:51
>>rvz+91
I mean one of the beauty parts of git has always been that you can and, under ideal best practices, should have more than one place to put your code with the distributed nature of it
replies(4): >>willsm+i2 >>beardo+u2 >>Karell+o5 >>anagan+qi
9. captai+K1[view] [source] 2023-05-16 21:24:28
>>jcalab+(OP)
While new features like search are appreciated, I wish GitHub would focus on stability for a bit, things have gotten pretty so-so
replies(2): >>hn_thr+63 >>toasta+hP
10. nrabul+O1[view] [source] 2023-05-16 21:24:38
>>jcalab+(OP)
Haven’t been able to go to GitHub.com for over 15 minutes now, just returns a 500. Pretty ironic given their latest blogpost
11. kossTK+P1[view] [source] 2023-05-16 21:24:40
>>jcalab+(OP)
Is this because of Microsoft? If so, why? Are they bad at cloud?
replies(1): >>RobotT+o3
12. mullin+b2[view] [source] 2023-05-16 21:26:40
>>jcalab+(OP)
At this point it looks obvious that GitHub RIF’d a bus factor earlier this year.
replies(3): >>vxNsr+J5 >>simult+M6 >>smcin+kb
◧◩◪
13. willsm+i2[view] [source] [discussion] 2023-05-16 21:27:10
>>jamal-+B1
I dunno about "should", that feels like overkill. Expecting to get something like a 99.99% SLA shouldn't be unreasonable to expect out of a hosted solution
replies(1): >>jamal-+P3
◧◩◪
14. beardo+u2[view] [source] [discussion] 2023-05-16 21:27:53
>>jamal-+B1
The value add for Github is mostly the issue/PR interface and actions.
15. algane+O2[view] [source] 2023-05-16 21:29:20
>>jcalab+(OP)
Nothing to worry about. These instabilities are probably caused by Open AI/Copilot improvements gradually becoming Skynet.

Seriously now: these instabilities happened a lot on the past weeks. I wonder what's going on there.

replies(1): >>belter+a3
◧◩
16. crote+03[view] [source] [discussion] 2023-05-16 21:30:30
>>rvz+91
Yeah, they had incidents the 9th, the 10th, and the 11th. So this is the fourth incident this month.
◧◩
17. hn_thr+63[view] [source] [discussion] 2023-05-16 21:31:07
>>captai+K1
Fully agree. I've been super impressed with the number of new features since the MS acquisition, but the number of outages is getting abysmal. At the very least I think GitHub would attempt to shard their infrastructure and repositories - right now it seems like when GitHub goes down it nearly always goes down for everyone simultaneously.

This kind of GH outage is actually pretty rare in my experience in that basically every kind of write operation looks down.: basic git operations from the command line, actions, logins, etc.

replies(1): >>zelphi+ha
◧◩
18. belter+a3[view] [source] [discussion] 2023-05-16 21:32:03
>>algane+O2
Like in the movie Wargames it's a trick. They will say the humans can't run a site like GitHub so put it all in the hands of the AI. Then the AI will have full control of GitHub...
replies(1): >>blibbl+Fb
19. ivanre+i3[view] [source] 2023-05-16 21:32:51
>>jcalab+(OP)
PR service is back :=)
◧◩
20. RobotT+o3[view] [source] [discussion] 2023-05-16 21:33:10
>>kossTK+P1
Maybe they moved to windows servers? lol
21. nabnob+u3[view] [source] 2023-05-16 21:33:48
>>jcalab+(OP)
Looks like it's back up. Still, this seems like an absurd absurd amount of issues recently
replies(2): >>ProAm+d4 >>tpmx+x4
◧◩
22. sethd+v3[view] [source] [discussion] 2023-05-16 21:33:54
>>hoofhe+r1
> Can't create new branches :(

Sure you can: git switch -c BRANCH_NAME

:)

◧◩◪◨
23. jamal-+P3[view] [source] [discussion] 2023-05-16 21:36:38
>>willsm+i2
A seperate git remote isn't incredibly unreasonable either - Even if it's just something on localhost in case of scenarios like this one

Github being down for a bit shouldn't disrupt your work flow

replies(1): >>maccar+54
◧◩◪◨⬒
24. maccar+54[view] [source] [discussion] 2023-05-16 21:38:32
>>jamal-+P3
GitHub being down blocks all of our CI pipelines, for example.
replies(1): >>bombol+kg
◧◩
25. ProAm+d4[view] [source] [discussion] 2023-05-16 21:39:09
>>nabnob+u3
Github had to move to MS Teams and I bet it broke a bunch of workflows.
replies(1): >>mulmen+E5
26. vb-844+s4[view] [source] 2023-05-16 21:41:14
>>jcalab+(OP)
They Just published this blog post https://github.blog/2023-05-16-addressing-githubs-recent-ava...
◧◩
27. tpmx+x4[view] [source] [discussion] 2023-05-16 21:41:48
>>nabnob+u3
Most of these outages during the past month have hit during European working hours, while most Americans are asleep. grumble

European working hours == Github maintenance window?

F that.

28. tastys+J4[view] [source] 2023-05-16 21:42:26
>>jcalab+(OP)
Worst timing! Just had an incident and had to deploy a hotfix. Pushes weren't working and I thought I was going crazy.

Who would have thought software could be such an adrenaline rush. Move over Alex Honnold!

◧◩◪
29. Karell+o5[view] [source] [discussion] 2023-05-16 21:46:16
>>jamal-+B1
But also, and more relevantly here, git is designed to be usable entirely offline. And if you do need to have some connection with remotes, git allows you to do so extremely sporadically if your network connectivity, or remote availability, is limited.
◧◩◪
30. mulmen+E5[view] [source] [discussion] 2023-05-16 21:48:16
>>ProAm+d4
Is this true? Is Microsoft actually making such invasive changes? Does Microsoft even use teams?
replies(1): >>ProAm+86
◧◩
31. vxNsr+J5[view] [source] [discussion] 2023-05-16 21:48:45
>>mullin+b2
RIF?
replies(2): >>crater+Q5 >>kstrau+T5
32. exogen+O5[view] [source] 2023-05-16 21:49:23
>>jcalab+(OP)
I was just saying it's kind of ironic that pre-acquisition GitHub inspired confidence, and Microsoft-owned GitHub has been more "move fast and break things."

I also don't just mean outages like this... it's clear that Microsoft GitHub has been cramming a bunch of new stuff into the UI and not really treating it with care. I notice very amateur UI bugs, misalignment, bad spacing, overlapping elements, etc. all the time now. It was clear that old GitHub passed through a professional designer's eye quite carefully (or maybe just a developer with extreme attention to detail).

replies(6): >>gianca+q6 >>tiedie+97 >>udkl+h7 >>jahnu+i7 >>undery+Ep >>crater+ES
◧◩◪
33. crater+Q5[view] [source] [discussion] 2023-05-16 21:49:30
>>vxNsr+J5
Reduction In Force. Layoffs.
◧◩◪
34. kstrau+T5[view] [source] [discussion] 2023-05-16 21:49:43
>>vxNsr+J5
Reduction In Force, aka fired / laid off / made redundant.
35. mulmen+16[view] [source] 2023-05-16 21:50:26
>>jcalab+(OP)
Someone should invent a decentralized version control system so we can avoid these kinds of widespread events.
replies(5): >>lionko+Y6 >>unboxi+47 >>dchnsh+Od >>legohe+9e >>rlytho+Eg
◧◩◪◨
36. ProAm+86[view] [source] [discussion] 2023-05-16 21:51:11
>>mulmen+E5
Yeah but transition is not fully required until September 1, 2023 [1]

[1] https://news.ycombinator.com/item?id=34773860

◧◩
37. gianca+q6[view] [source] [discussion] 2023-05-16 21:52:53
>>exogen+O5
Just like Skype... I felt like Skype was top tier before Microsoft bought it out. Voice calls were high quality, no issues.
replies(1): >>imran-+s8
◧◩
38. simult+M6[view] [source] [discussion] 2023-05-16 21:54:35
>>mullin+b2
Eventually it will be the same everywhere but for sake of economy (stakeholders), staff and customers pay the price, always. By time they find the bottleneck they will be hiring as crazy then.
◧◩
39. lionko+Y6[view] [source] [discussion] 2023-05-16 21:55:38
>>mulmen+16
git itself is not enough for teams to work together - you need to asynchronously communicate for the whole thing to come together. Thats what GitHub/Lab/ea bring in value.
replies(2): >>zamnos+m7 >>TimThe+98
◧◩
40. unboxi+47[view] [source] [discussion] 2023-05-16 21:56:02
>>mulmen+16
good idea. let’s call it “git”
replies(1): >>whynot+Za
◧◩
41. tiedie+97[view] [source] [discussion] 2023-05-16 21:56:48
>>exogen+O5
> notice very amateur UI bugs, misalignment, bad spacing, overlapping elements, etc. all the time now.

Seriously. Example: the "merge pull requests" button. If your repository requires linear history and the current type of merge is unavailable, the entire UI element turns grey instead of green. It looks like you can't use the button at all, but the dropdown arrow still works to select a valid type of merge (which then turns the entire element green).

It's a small thing, but it causes confusion by making it appear that the change cannot be merged, and it makes you wonder how much testing they do.

These little things keep piling up as they focus on shiny new features which rarely work correctly at launch (looking at you, code search...)

Please, GH product managers. Take a break from breakneck new features, and give your devs time to clean up and do some preventative maintenance.

◧◩
42. udkl+h7[view] [source] [discussion] 2023-05-16 21:57:04
>>exogen+O5
> I notice very amateur UI bugs, misalignment, bad spacing, overlapping elements, etc. all the time now.

+1

The inconsistencies aren't deal breakers - github still has a very pleasant UI overall - but they do makethe website feel more and more unpolished each day

◧◩
43. jahnu+i7[view] [source] [discussion] 2023-05-16 21:57:33
>>exogen+O5
I dunno. For me the two or three years pre MS were quite stagnant imo. Plenty of outages back then too.
replies(1): >>exogen+y8
◧◩◪
44. zamnos+m7[view] [source] [discussion] 2023-05-16 21:58:08
>>lionko+Y6
I feel like "communication" isn't descriptive enough. It's right, but eg Slack (or Microsoft Teams, or maybe both) isn't the right shape of communication.
45. labste+y7[view] [source] 2023-05-16 21:58:45
>>jcalab+(OP)
A moment of silence for all of the nines we lost today.
◧◩◪
46. TimThe+98[view] [source] [discussion] 2023-05-16 22:02:32
>>lionko+Y6
Someone should write a local git+ssh service (+ email client) that simulates an upstream by hosting git repos locally, emailing all pushed commits/branches/tags to other team members, and keeping the hosted repos up-to-date via the emailed updates from others.

Bonus points for a local HTTP service that provides an optional GitHub-like experience in the browser, with comments/ branch reviews/etc. federated through emailed messages (and represented as commits on meta-repos).

With that data model, the primary remaining challenge would be setting up repos on a new machine -- perhaps BitTorrent could help :)

◧◩◪
47. imran-+s8[view] [source] [discussion] 2023-05-16 22:04:02
>>gianca+q6
Skype was also horribly insecure and leaked your IP which was a cause of a lot of folks in esports getting DDoS'd from it[0].

0: https://blog.destiny.gg/protection-from-ddos-attacks/

replies(2): >>charci+nh >>gianca+Po
◧◩◪
48. exogen+y8[view] [source] [discussion] 2023-05-16 22:04:35
>>jahnu+i7
The "move fast" part was definitely sincere, to Microsoft's credit!
49. Popcor+V8[view] [source] 2023-05-16 22:06:53
>>jcalab+(OP)
Spoke to Gitlab rep today about our renewal price (prices going up by 50%+, 25% with "existing customer discount) and the rep used reliability as a key differentiator against GitHub.

I quickly pointed about a major outage affecting Gitlabs shared runners which prevented us from deploying a hot fix (we worked around it, but was a ton of stress and extra work).

replies(1): >>rekwah+d9
◧◩
50. rekwah+d9[view] [source] [discussion] 2023-05-16 22:08:49
>>Popcor+V8
Don't leave us hanging. Did you get a discount?! ;)
◧◩◪
51. zelphi+ha[view] [source] [discussion] 2023-05-16 22:14:16
>>hn_thr+63
I think it is quite individual. I am not impressed by any of the newer features. The UI become more cumbersome to use. I sometimes need to search for things now, while I don't see any new advantages compared to before.
◧◩◪
52. whynot+Za[view] [source] [discussion] 2023-05-16 22:18:41
>>unboxi+47
Not quite sure, the name should imply that the Source is Safe and provide some Visual tools.
◧◩
53. smcin+kb[view] [source] [discussion] 2023-05-16 22:20:46
>>mullin+b2
Why don't we just use plain language like "laid off" like we used to for decades?
replies(2): >>mullin+Hf >>teduna+jm
◧◩◪
54. blibbl+Fb[view] [source] [discussion] 2023-05-16 22:22:56
>>belter+a3
there's already a malevolent immortal entity in charge of GitHub
replies(1): >>belter+de
◧◩
55. dchnsh+Od[view] [source] [discussion] 2023-05-16 22:36:58
>>mulmen+16
BTW: There is work in progress to decentralize modern git based workflows by https://nlnet.nl/project/ForgeFed/ (paid for by the European Union), by decentralizing git not at the client level (which it obv. already has, but most people only use one origin), but also on the code forge level.
replies(1): >>moreli+ff
◧◩
56. legohe+9e[view] [source] [discussion] 2023-05-16 22:39:44
>>mulmen+16
How do decentralized networks deal with illegal content?
replies(2): >>jeroen+ye >>mulmen+jg
◧◩◪◨
57. belter+de[view] [source] [discussion] 2023-05-16 22:40:17
>>blibbl+Fb
The word malevolent implies a level of competence I do not recognize to that Seattle troupe.
replies(1): >>blibbl+te
◧◩◪◨⬒
58. blibbl+te[view] [source] [discussion] 2023-05-16 22:42:33
>>belter+de
it's quite good at producing buckets of money

functional, reliable products, not so much

◧◩◪
59. jeroen+ye[view] [source] [discussion] 2023-05-16 22:42:44
>>legohe+9e
The joke is that Git is decentralized already. Many people pick a single master host, but it's absolutely possible to use multiple servers (or to use Git over email, like the Linux kernel does).
◧◩◪
60. moreli+ff[view] [source] [discussion] 2023-05-16 22:48:17
>>dchnsh+Od
> ForgeFed aims to define a vocabulary

Aw yeah there’s the good EU grant milking shit.

◧◩◪
61. mullin+Hf[view] [source] [discussion] 2023-05-16 22:52:02
>>smcin+kb
The only thing engineers love more than a TLA is an ETLA.
62. frellu+Yf[view] [source] 2023-05-16 22:53:22
>>jcalab+(OP)
Can we all stop hating on github already?

I assume most people don't even pay for it with their public and private non-work repos. Github has been a major supporter of OpenSource projects and while their uptime could use some help, everyone is struggling right now with RIFs and fewer resources.

That's just my $0.02

replies(2): >>yjftsj+yk >>mulmen+9l
◧◩◪
63. mulmen+jg[view] [source] [discussion] 2023-05-16 22:55:55
>>legohe+9e
Same as everyone else. With law enforcement.
replies(1): >>legohe+xx
◧◩◪◨⬒⬓
64. bombol+kg[view] [source] [discussion] 2023-05-16 22:55:57
>>maccar+54
you should be able to run the basic tests locally.
◧◩
65. rlytho+Eg[view] [source] [discussion] 2023-05-16 22:58:59
>>mulmen+16
DNS and git are decentralized; anyone can set up a node

Human agency tends to normalize on the brands with biggest marketing budget who collude with politicians to create a moat for themselves. There’s your real problem.

Of course none of the propaganda research that made its way from military after world wars to corporate advertising and marketing college programs has any influence on lizard brains. No sir.*

* this comment is paid placement

◧◩◪◨
66. charci+nh[view] [source] [discussion] 2023-05-16 23:04:08
>>imran-+s8
Skype resolvers definitely still worked post Microsoft acquisition.
replies(1): >>gianca+Zo
◧◩◪
67. anagan+qi[view] [source] [discussion] 2023-05-16 23:10:32
>>jamal-+B1
But currently that's not the beauty part of GitHub, it's the actions, discoverability and other features not part of Git.
◧◩
68. yjftsj+yk[view] [source] [discussion] 2023-05-16 23:22:32
>>frellu+Yf
> while their uptime could use some help, everyone is struggling right now with RIFs and fewer resources.

So? That's entirely self inflicted.

◧◩
69. mulmen+9l[view] [source] [discussion] 2023-05-16 23:27:05
>>frellu+Yf
We don't owe GitHub anything. They put themselves in this situation. They burned capital to become the defacto source code hosting site then rug-pulled on reliability. They deserve more criticism, not less.
◧◩
70. mulmen+ul[view] [source] [discussion] 2023-05-16 23:29:30
>>aj397+m1
Move to git and email. Like Linus intended.
◧◩◪
71. teduna+jm[view] [source] [discussion] 2023-05-16 23:33:46
>>smcin+kb
Technically, a layoff could be temporary, and that was once upon a time the meaning of the term. RIF signals that it's permanent.
replies(1): >>smcin+Br3
◧◩◪◨
72. gianca+Po[view] [source] [discussion] 2023-05-16 23:49:44
>>imran-+s8
I remember this being a problem after Microsoft acquired it. You didnt even need to call someone to get their IP. You just needed a username.
◧◩◪◨⬒
73. gianca+Zo[view] [source] [discussion] 2023-05-16 23:50:26
>>charci+nh
My understanding is they were peer to peer originally, and after Microsoft they got rid of that and made it more centralized. I could be misremembering though.
◧◩
74. undery+Ep[view] [source] [discussion] 2023-05-16 23:55:58
>>exogen+O5
I keep a rolling Twitter thread with all the UI bugs I notice: https://twitter.com/underyx/status/1600943582760042497
replies(1): >>richbe+tq
◧◩◪
75. richbe+tq[view] [source] [discussion] 2023-05-17 00:01:21
>>undery+Ep
Thank you for this! GitHub's UI really bothers me, and I can't help but feel like it's gotten flashier but less useful.

Another one: IIRC, for some reason, there is no "fork" button on mobile UI. You have to go to desktop mode to click the fork button.

replies(1): >>earthl+cn1
76. junon+Uv[view] [source] 2023-05-17 00:40:37
>>jcalab+(OP)
Started as a joke that this was daily. Now it's no much of a joke.
◧◩◪◨
77. legohe+xx[view] [source] [discussion] 2023-05-17 00:51:01
>>mulmen+jg
So law enforcement contacts everyone on the network and says please delete these illegal photos?
replies(1): >>mulmen+x73
◧◩
78. toasta+hP[view] [source] [discussion] 2023-05-17 03:44:58
>>captai+K1
The new search? You can't even use search on source code without being first authenticated, which requires signing up for an account.
◧◩
79. crater+ES[view] [source] [discussion] 2023-05-17 04:26:25
>>exogen+O5
> cramming a bunch of new stuff

That's kind of how Microsoft works in general. Their products expand to have as many features as they can cram in. Word and Excel are probably the quintessential examples, where only a tiny fraction of users care about a feature, but they are passionate about it.

◧◩◪◨
80. earthl+cn1[view] [source] [discussion] 2023-05-17 09:43:00
>>richbe+tq
The mobile UI is just lacking all around. There are a ton of features that just don't seem to exist. Searching inside a repository? Nope. Browse files of a given commit? No chance. There's a bunch more, but both of those bother me very frequently.
◧◩◪◨⬒
81. mulmen+x73[view] [source] [discussion] 2023-05-17 19:12:46
>>legohe+xx
Yep. Not complicated. They actually only have to contact the people hosting illegal photographs so it's easier than you describe.
◧◩◪◨
82. smcin+Br3[view] [source] [discussion] 2023-05-17 20:48:52
>>teduna+jm
No, in general almost never in tech, only rarely and when unionized or when protected by contract or labor law. And we're not talking about seasonal jobs (logging, fishing) either. And we're not talking about unforeseen circumstances/force majeure like in 2020/1.

"RIF" is US jargon; you can see that ~0% of worldwide Google usage outside of North America for "RIF" is "reduction in force" [https://trends.google.com/trends/explore?date=now%201-d&q=%2...]. A tiny amount (<<2%) of usage in UK, Scandinavia, Poland, Australia (possibly mainly from US multinationals).

More corroboration: https://en.wikipedia.org/wiki/RIF

There is zero reason to adopt managementspeak jargon, or its 2022 neologisms "impacted"/"affected"/"displaced". Just say "layoff" (n) / "laid off" (v), already.

In the very unlikely event [in tech] you actually needed to say "temporary layoff", you can easily say "temporary layoff".

The language was working fine for decades and there's no reason to pander to management euphemisms. It harms clarity.

[go to top]