zlacker

[parent] [thread] 28 comments
1. andsoi+(OP)[view] [source] 2023-07-02 00:36:24
> The people at Twitter who understood the system and could predict the side effects were all fired or left.

Are you saying the engineers who are now at Twitter don’t have the right skills?

replies(8): >>tadfis+D >>nights+n6 >>davidc+37 >>spamiz+K9 >>nborwa+cb >>95014_+zb >>scott_+qq >>tchaff+qy
2. tadfis+D[view] [source] 2023-07-02 00:42:30
>>andsoi+(OP)
That's a non-sequitir. You can be at the top of your field and not completely understand a complex system. Also, the right people may not have even been involved in the implementation of this "feature".
replies(4): >>andsoi+T4 >>wpietr+D5 >>hrisng+A8 >>Rimbo+8f
◧◩
3. andsoi+T4[view] [source] [discussion] 2023-07-02 01:19:09
>>tadfis+D
> You can be at the top of your field and not completely understand a complex system.

The tweet reads: "Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon's latest genius innovation is to block people from being able to read Twitter without logging in."

Does that strike you as complex? I mean, surely they had the context (need to log in) because it was all over the news

replies(2): >>fesoli+v6 >>BeefWe+1g
◧◩
4. wpietr+D5[view] [source] [discussion] 2023-07-02 01:26:52
>>tadfis+D
For sure. I'd add that even super-geniuses get overwhelmed when asked to do too many things at once. Kudos to whoever's still left at Twitter; the current erratic decay is better than I would have expected after this long. But when maintenance and improvement are deferred long enough, eventually you reach the point where on average solving 1 problem creates >1.0 problems.
5. nights+n6[view] [source] 2023-07-02 01:36:43
>>andsoi+(OP)
All the best people I know left.
◧◩◪
6. fesoli+v6[view] [source] [discussion] 2023-07-02 01:39:20
>>andsoi+T4
I don't know what kind of systems you worked on in your career, but even simpler systems with smaller userbases than twitter are quite complex if you are new to it.

Twitter serves their service to the entire world, with multiple layers of systems working in conjunction in order to make things work smoothly. A new engineer that has not been working on it for no more than a couple months would likely be unaware of how the different systems communicate and interact. A change like this will have have a lot of unintended consequences, and not having a senior engineer with lots of context leading the change will undoubtedly cause these kinds of issues.

replies(3): >>chris_+ec >>sangno+6f >>wruza+kl
7. davidc+37[view] [source] 2023-07-02 01:43:39
>>andsoi+(OP)
Are the very least they have terrible judgment considering they still work at Twitter (excluding people dealing with visa issues)
replies(1): >>dredmo+29
◧◩
8. hrisng+A8[view] [source] [discussion] 2023-07-02 02:00:01
>>tadfis+D
geohot was naively optimistic when he offered his so called "internship" to Musk. Ended up resigning in one month without delivering anything.
replies(1): >>buggle+ge
◧◩
9. dredmo+29[view] [source] [discussion] 2023-07-02 02:06:10
>>davidc+37
There might be numerous other reasons: <>>36556578 >
10. spamiz+K9[view] [source] 2023-07-02 02:15:10
>>andsoi+(OP)
They lack knowledge not skills. To my knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not)
replies(1): >>andsoi+Bc
11. nborwa+cb[view] [source] 2023-07-02 02:30:52
>>andsoi+(OP)
Alternately, if they have the “right skills” why is this happening? Clearly they don’t have some of the needed skills to prevent this. And a self-DDOS might be a good indication of that.
12. 95014_+zb[view] [source] 2023-07-02 02:34:05
>>andsoi+(OP)
William of Ockham would like a word.
replies(1): >>Sketch+Cg
◧◩◪◨
13. chris_+ec[view] [source] [discussion] 2023-07-02 02:41:19
>>fesoli+v6
Your argument is that it is a non sequitur to say that the right skilled people are not at twitter, using the argument that the right people with the available skills were not involved in this feature.
replies(1): >>Baseba+3q
◧◩
14. andsoi+Bc[view] [source] [discussion] 2023-07-02 02:44:55
>>spamiz+K9
> knowledge there is no "intuitively understand how my changes will impact a complex system without studying it" skill (unless that system is under a robust test suite, which Twitter is not

then how do they have confidence that anything works before they subject hundreds of millions of people around the world to system updates? that seems disrespectful to the user, if you asked me.

replies(2): >>jbeam+ke >>mypalm+Hx
◧◩◪
15. buggle+ge[view] [source] [discussion] 2023-07-02 03:00:06
>>hrisng+A8
Not true. He temporarily got the login nag modal removed. It was a nice few weeks.
replies(1): >>hrisng+Vq
◧◩◪
16. jbeam+ke[view] [source] [discussion] 2023-07-02 03:01:13
>>andsoi+Bc
They don’t have confidence. The site has been breaking down. It is indeed disrespectful to the user.
◧◩◪◨
17. sangno+6f[view] [source] [discussion] 2023-07-02 03:08:12
>>fesoli+v6
> A change like this will have have a lot of unintended consequences, and not having a senior engineer with lots of context leading the change will undoubtedly cause these kinds of issues.

Having a senior engineer with a lot of context is worthless if the work environment does not promote open communication. You don't want to be the senior engineer or leader who shows "poor judgement" by opposing the mercurial owner "for no reason" if you're overridden and the feature launch succeeds without a glitch; no one gets fired for implementing a request that came straight from the top.

This is why non-rushed, scaled roll-outs are essential for large system: had they tried this on 1% / 5% / 10% of random traffic first, they could have caught this. Yet again, if the directive to roll it out to production came from the very top, you set that gate to 100% immediately.

◧◩
18. Rimbo+8f[view] [source] [discussion] 2023-07-02 03:08:24
>>tadfis+D
Bingo. You can know a specific technology inside and out, but easily get lost in a large system built in that technology if you haven't worked with it before.

It's very easy to get caught in assumptions like, "Nobody would ever do things THIS way, so they must have built it THAT way," only to find out that, once upon a time, THIS way was the right way to do things, only for it to over time become less and less optimal, but the costs of changing things were too high to fix it. Once your system is old enough and large enough, you'll have several thousand things just like that.

◧◩◪
19. BeefWe+1g[view] [source] [discussion] 2023-07-02 03:16:51
>>andsoi+T4
It's not obvious. Per Mudge's whistleblowing, Twitter didn't have a test environment so it's entirely possible they had no idea this would happen.[1]

[1]: https://www.wired.com/story/mudge-twitter-whistleblower-secu...

replies(1): >>mips_r+y22
◧◩
20. Sketch+Cg[view] [source] [discussion] 2023-07-02 03:22:10
>>95014_+zb
Ol' Billy Razor?
◧◩◪◨
21. wruza+kl[view] [source] [discussion] 2023-07-02 04:27:36
>>fesoli+v6
“Trying to make a pigtail out of these unbrushed hair may unscrew the ears”. Sorry, yep, twitter is big. But if preventing tens of doomed requests before a login requires a senior engineer with lots of context, then the program was screwed up long before the layoffs.
replies(1): >>atoav+Tv
◧◩◪◨⬒
22. Baseba+3q[view] [source] [discussion] 2023-07-02 05:29:37
>>chris_+ec
You seem to be confusing knowledge and skill.

Twitter could be packed with extremely skillful senior engineers who don't understand the product well enough to predict complex outcomes of planned changes.

replies(1): >>shapef+0P
23. scott_+qq[view] [source] 2023-07-02 05:34:26
>>andsoi+(OP)
The quote doesn’t say that at all
◧◩◪◨
24. hrisng+Vq[view] [source] [discussion] 2023-07-02 05:41:15
>>buggle+ge
I just realize how poetic that is since you're now no longer able to view anything without logging in.
◧◩◪◨⬒
25. atoav+Tv[view] [source] [discussion] 2023-07-02 06:38:04
>>wruza+kl
I don't disagree, but: Twitter was big and it worked. Then someone created incentives for many people who know the code to leave.

The one's left don't know all the code (how could they?), but were forced to change many things about the site at a "just do it" basis. This error didn't happen because someone was too stupid to remove the code, it did happen because the connection to another thing was removed and the failsafe on the landing page doesn't have exponential backdown built in, not something you can necessarily know or investigate before, when an executive breaths down your neck and wants you to just do it.

This is about the new managment, not about engineers.

◧◩◪
26. mypalm+Hx[view] [source] [discussion] 2023-07-02 06:57:04
>>andsoi+Bc
A/B testing.
27. tchaff+qy[view] [source] 2023-07-02 07:05:43
>>andsoi+(OP)
Someone who has the right skills to be an air traffic controller doesn't have the right skills to be the only air traffic controller.
◧◩◪◨⬒⬓
28. shapef+0P[view] [source] [discussion] 2023-07-02 10:14:31
>>Baseba+3q
Likewise you can have a "poor quality" 1x employee who has knowledge of how everything within the stack is glued together; where there is chewingum and where there are steel beams.

They are potentially more vauable than the 100x engineer who has intimate knowledge of how googles shipping container datacentres work.

◧◩◪◨
29. mips_r+y22[view] [source] [discussion] 2023-07-02 19:22:42
>>BeefWe+1g
Seems Twitter has had a culture of rolling out changes straight into production without incremental testing, at least since well before Musk took over. Mudge was hired there in 2020 and what he found was a complete mess.
[go to top]