zlacker

What frameworks would you use to handle such an steep growth curve? Most startups I know of start of with rails or the like - and obviously they couldn't handle the strain. So what would you use?

>>lunchb+(OP)
The one my team knows and / or likes best. When you hit really big scale, architecture is more of a differentiator than language, but you can always move inefficiencies into their own services written in a more performant technology. The thing you need most when starting is the ability to produce, test, and iterate quickly.

>>lunchb+(OP)
I would use Rails, but I wouldn't do things like try to write an asynchronous message queue in Ruby (like Twitter did).

I mean sites like Shopify, Hulu, YellowPages.com, etc. "handle the strain" just fine.

>>lunchb+(OP)
The first question should be: do I need a framework at all. There's almost always overhead, you have to ask if it's worth it

>>lunchb+(OP)
The web part of it is pretty easy to scale. You simply add more web servers. The problem is in the storage/db layer. I imagine the feed was the primary challenge scaling Twitter.

Other components such as the search were probably also quite tricky. One thing i've never figured out is how Facebook handles search on your timeline. That a seriously complex problem.

Linkedin Recently published a bunch of papers about their feed tech:

https://engineering.linkedin.com/blog/2016/03/followfeed--li...

And Stream's stackshare is also interesting: https://stackshare.io/stream/stream-and-go-news-feeds-for-ov...

>>lunchb+(OP)
The JVM. I know this is the boring answer but the tooling and performance combination is tough to beat. It's why Twitter themselves switched to the JVM (a mix of Scala and Java).

>>lunchb+(OP)
Very few sites have had and will have that kind of growth curve. Moreover, the cloud and the workloads you can run on it are substantially different now than they were 10+ years ago. So what would I use? Probably the most productive framework I could find, not necessarily the most performant one, because odds are that my app would never attain the scale I'm preparing for, but that preparation overhead may be the handicap that prevents that level of success in the first place.

>>lunchb+(OP)
It's a much bigger mistake to overengineer for the highly unlikely event of such a steep growth curve than it is to get to that part of the curve and flail about semi-ineffectually for a couple of years like twitter did at the time.

>>lunchb+(OP)
WhatsApp was a pretty well known case for Erlang[1], given that I'd think Phoenix would be a great fit.

[1] https://www.wired.com/2015/09/whatsapp-serves-900-million-us...

>>tschel+H1
>One thing i've never figured out is how Facebook handles search on your timeline. That a seriously complex problem.

From a user's point of view, they don't. I search for things a lot (or at least, I used to before I realized how bad it is) and things that I know I saw yesterday don't come up - even if I put in a very specific search. Sometimes it'll say "no results found", but I'll find it in a tab I hadn't closed yet.

>>lunchb+(OP)
No framework will save you, it's not about language or framework at that point, they necessarily won't help you, but they can sure hurt you. It's more about your architecture.

>>vvande+65
Erlang won't have helped twitter. Erlang has a very simple architecture, once they sent you a message and know it has reached your device, they no longer care about it. Twitter has to keep all the tweets, likes, retweets, show it all followers in real time. 100x more demanding than WhatsApp simple deliver and forget.

>>phaedr+11
How much of the content in those sites change as fast as Twitter does? Mainly static content can be displayed using about any framework/language. Moreso when you include caching. I don’t think Rails was simply a good choice for the workload Twitter needed to support (not that they knew it at the time).

>>nkozyr+A1
You're getting downvoted, but this is one of the arguments for going with unikernels.

>>vvande+65
Phoenix will be much better than Rails out of the box for things that need a certain level of concurrency. However Twitter's problems were about scaling dependent writes, this is a data-store / architecture problem that really has nothing to do with the chosen language.

>>tschel+H1
What seems difficult about facebook search ?

>>lunchb+(OP)
obviously they couldn't handle the strain

They could handle the strain, within what their audience considered acceptable. That's the important thing. The key to scaling a startup is to always be just ahead of where your audience gives up. Anything less and your precious users will abandon ship and you'll fail; anything more you're burning too much cash, you'll run out of runway and you'll fail.