zlacker

[parent] [thread] 24 comments
1. molecu+(OP)[view] [source] 2018-05-24 23:43:06
2010: "At any moment, Justin Bieber uses 3% of our infrastructure. Racks of servers are dedicated to him"

https://gizmodo.com/5632095/justin-bieber-has-dedicated-serv...

replies(5): >>fizx+l >>freefl+M2 >>loopdo+8e >>firebo+wf >>liquid+Ov
2. fizx+l[view] [source] 2018-05-24 23:46:56
>>molecu+(OP)
Bieber-related conversation might reasonably have been 3%. Updating his friend count was a constant source of lock contention on a particular mysql.

There were never explicitly dedicated databases/servers, unless you count the hotspot caused by whatever shard he hashed into.

replies(2): >>lwansb+22 >>tzs+th
◧◩
3. lwansb+22[view] [source] [discussion] 2018-05-25 00:09:40
>>fizx+l
You would think a HyperLogLog would be approximately good enough when follower counts reach tens of millions.
replies(1): >>fizx+04
4. freefl+M2[view] [source] 2018-05-25 00:18:35
>>molecu+(OP)
Clicking that was already worth it for the lead picture, best laugh I had today, thank you!
◧◩◪
5. fizx+04[view] [source] [discussion] 2018-05-25 00:34:06
>>lwansb+22
Hyperloglog was only just invented, and hadn't really made its way into the popular developer consciousness.

Anyhow, the solution is easier than that (implement lock striping, i.e. https://stackoverflow.com/questions/16151606/need-simple-exp...), but I was reorg'd away before my code hit production, so I'm not sure what happened there.

replies(1): >>lorenz+27
◧◩◪◨
6. lorenz+27[view] [source] [discussion] 2018-05-25 01:24:38
>>fizx+04
Approximate counters are much much older than HyperLogLog. Here’s something from 1978: https://www.inf.ed.ac.uk/teaching/courses/exc/reading/morris.... Of course this solves a different problem because HyperLogLog is an, uh, interesting way to count followers (why would you need to do a count-distinct query? You can’t follow someone multiple times). In any case, the Flajolet Martin sketch dates back to 1985 and solves the same problem as HyperLogLog: https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
replies(1): >>aflam+Lj
7. loopdo+8e[view] [source] 2018-05-25 03:13:22
>>molecu+(OP)
A couple years ago, Bieber tweeted out a Twilio phone number for people to text. Luckily it happened on a Friday night, or it would have caused widespread outages.

Celebrity is an edge case that you have to be prepared for.

replies(1): >>mentat+je
◧◩
8. mentat+je[view] [source] [discussion] 2018-05-25 03:16:27
>>loopdo+8e
https://mashable.com/2015/10/30/justin-bieber-hotline-bling/... - It was an interesting experience.
9. firebo+wf[view] [source] 2018-05-25 03:41:44
>>molecu+(OP)
But, but, but...why does Twitter have so many engineers? I could write Twitter in a weekend!

--95% of anti-TWTR posters circa 2010-2016.

replies(4): >>spondy+Lh >>snovv_+Co >>scarfa+Kg1 >>tripzi+rr4
◧◩
10. tzs+th[view] [source] [discussion] 2018-05-25 04:16:14
>>fizx+l
Did they actually need to know Bieber's friend count both accurately and in real time?

I would expect that for top Twitter accounts like his, you could get a very good approximation by logging events that increment or decrement the friend count, and using the rates those events are occurring and the last known exact friend count to forecast the current friend count. That would not be exact, but I bet you could get it close enough that users would not see anything off about it.

replies(2): >>mrgord+2k >>segmon+Xa1
◧◩
11. spondy+Lh[view] [source] [discussion] 2018-05-25 04:18:51
>>firebo+wf
I think this post is relevant here: https://danluu.com/sounds-easy/
◧◩◪◨⬒
12. aflam+Lj[view] [source] [discussion] 2018-05-25 04:49:11
>>lorenz+27
Here is nice presentation covering those: (link: https://www.cs.princeton.edu/~rs/talks/AC11-Cardinality.pdf)
◧◩◪
13. mrgord+2k[view] [source] [discussion] 2018-05-25 04:53:38
>>tzs+th
Yes they made Storm for this
◧◩
14. snovv_+Co[view] [source] [discussion] 2018-05-25 06:02:57
>>firebo+wf
Before bring acquired, WhatsApp had what, 30 employees?

How did they do it? I know they used custom BSD servers so that a single box could keep close to 1M TCP connections open. I'm sure with a fixed target to aim for and all scope known upfront a small crack team of devs could do something similar for Twitter.

replies(2): >>akhilc+Wp >>zmb_+Ft
◧◩◪
15. akhilc+Wp[view] [source] [discussion] 2018-05-25 06:20:37
>>snovv_+Co
The beauty of Erlang/OTP!
replies(1): >>evanwe+5z
◧◩◪
16. zmb_+Ft[view] [source] [discussion] 2018-05-25 07:11:47
>>snovv_+Co
One-to-one vs. many-to-many messaging. The amount of work you need to do to deliver a WhatsApp message is constant and small -- just route the message to a single recipient's mailbox. The amount of work Twitter has to do to deliver a message grows as a function of followers. One celebrity tweeting another celebrity means you have to deliver the message to the mailboxes of the followers of both -- millions of times more work than WhatsApp per message. In addition, Twitter persists all the messages while WhatsApp doesn't.
replies(1): >>segmon+xa1
17. liquid+Ov[view] [source] 2018-05-25 07:42:52
>>molecu+(OP)
That was a period when the only people talking outside the company were the ones not trying to fix things inside of the company.

Twitter never had dedicated machines for individual users. Its just not how the infra ever worked (at least until I left). Requests landed on random boxes behind load balancers, those boxes talked to pools of memcache or mysql, etc. At no point was there ever "racks" or "machines" dedicated to specific individuals like the article claims.

That being said, some users created crazy hot shards when specific tweets went absolutely madhouse, especially Beiber and those like him.

Random twitter internals tidbit: We had a unit of measurement called a "MJ". Its the number of tweets per second that we had when the rumors of Michael Jacksons death were circling. It basically overloaded the system and had us running around on fire. It was 465 tweets a second. Within a year we had crossed a line where we never were below that number again. Hence "we are at about 12 MJ's" was jokingly used to compare "hair on fire" to every day a couple of years later. =)

replies(1): >>evanwe+Rx
◧◩
18. evanwe+Rx[view] [source] [discussion] 2018-05-25 08:13:44
>>liquid+Ov
Sorry, we just kept the Bieber box hidden from you in the "special closet". It was a SPARCstation 20 and we were afraid if ops found it they would shut it down.
replies(1): >>liquid+xy
◧◩◪
19. liquid+xy[view] [source] [discussion] 2018-05-25 08:22:08
>>evanwe+Rx
You know.. I found a special box (PowerMac Pro) in the "special closet" that had been moved like 5 times. It was Blane's old desktop and it had the twitter codebase checked out from like ~2007 era? Kind of nifty seeing just how much had changed since that got tucked away.
◧◩◪◨
20. evanwe+5z[view] [source] [discussion] 2018-05-25 08:29:45
>>akhilc+Wp
Especially if you completely rewrite Mnesia.
◧◩◪◨
21. segmon+xa1[view] [source] [discussion] 2018-05-25 14:24:02
>>zmb_+Ft
This! Everyone keep's saying it's BSD Erlang and jumping on the Erlang train. Fine tools btw, but WhatsApp is super simple compared to Twitter. rolls eyes
◧◩◪
22. segmon+Xa1[view] [source] [discussion] 2018-05-25 14:26:26
>>tzs+th
Ha, I always joke that Twitter is a good example of why one should avoid distributed systems. Every often, you see it showing the wrong number of friends, tweets, likes, etc. I don't care, I suppose some people care about such metrics.
replies(1): >>scarfa+3h1
◧◩
23. scarfa+Kg1[view] [source] [discussion] 2018-05-25 15:02:51
>>firebo+wf
Instagram has 13 employees when it was sold to FB.

Did any major companies have the technology growing pains that Twitter did - ie Facebook, Google, Amazon, YouTube, Netflix (streaming), Dropbox, etc.?

◧◩◪◨
24. scarfa+3h1[view] [source] [discussion] 2018-05-25 15:05:05
>>segmon+Xa1
That's the essence of the CAP theorem. You can have two of three - consistency, availability, and partition tolerance. Sometimes it makes sense to choose availability over consistency.
◧◩
25. tripzi+rr4[view] [source] [discussion] 2018-05-27 12:04:02
>>firebo+wf
I mean, obviously there must be a lot more going on inside twitter besides being a massive many-to-many pubsub messaging infrastructure. And frankly twitter can be improved by leaving a lot of stuff out and up to the client app, so it's not entirely unreasonabl?

> In one incident, he wrote, an error caused every user to log in as somebody else each time they refreshed the page.

Gotta admit, this story raised my eyebrow, this wasn't the dark ages of the web. What sort of crazy experimental authentication voodoo were they running?

[go to top]