https://gizmodo.com/5632095/justin-bieber-has-dedicated-serv...
There were never explicitly dedicated databases/servers, unless you count the hotspot caused by whatever shard he hashed into.
Anyhow, the solution is easier than that (implement lock striping, i.e. https://stackoverflow.com/questions/16151606/need-simple-exp...), but I was reorg'd away before my code hit production, so I'm not sure what happened there.
Celebrity is an edge case that you have to be prepared for.
--95% of anti-TWTR posters circa 2010-2016.
I would expect that for top Twitter accounts like his, you could get a very good approximation by logging events that increment or decrement the friend count, and using the rates those events are occurring and the last known exact friend count to forecast the current friend count. That would not be exact, but I bet you could get it close enough that users would not see anything off about it.
How did they do it? I know they used custom BSD servers so that a single box could keep close to 1M TCP connections open. I'm sure with a fixed target to aim for and all scope known upfront a small crack team of devs could do something similar for Twitter.
Twitter never had dedicated machines for individual users. Its just not how the infra ever worked (at least until I left). Requests landed on random boxes behind load balancers, those boxes talked to pools of memcache or mysql, etc. At no point was there ever "racks" or "machines" dedicated to specific individuals like the article claims.
That being said, some users created crazy hot shards when specific tweets went absolutely madhouse, especially Beiber and those like him.
Random twitter internals tidbit: We had a unit of measurement called a "MJ". Its the number of tweets per second that we had when the rumors of Michael Jacksons death were circling. It basically overloaded the system and had us running around on fire. It was 465 tweets a second. Within a year we had crossed a line where we never were below that number again. Hence "we are at about 12 MJ's" was jokingly used to compare "hair on fire" to every day a couple of years later. =)
Did any major companies have the technology growing pains that Twitter did - ie Facebook, Google, Amazon, YouTube, Netflix (streaming), Dropbox, etc.?
> In one incident, he wrote, an error caused every user to log in as somebody else each time they refreshed the page.
Gotta admit, this story raised my eyebrow, this wasn't the dark ages of the web. What sort of crazy experimental authentication voodoo were they running?