Twitter timeline and search
Design the Twitter timeline: a user posts a tweet, and their followers see it in a home timeline. Also design search over all tweets.
- 100 million active users a day
- 500 million tweets a day
- Fan-out to an average of 100 followers
- Deliver in under a second
- Read-heavy timeline
- Search must be fast
- High availability
The core use cases are posting a tweet, viewing the home timeline, viewing the user timeline, and searching keywords.
500 million tweets a day is about 5,800 tweets a second, with peaks far above that. Each tweet fans out to 100 followers on average, so the write amplification, not the tweet volume, sets the load.
Gather requirements and scope the problem. Ask questions to clarify use cases and constraints. Discuss assumptions.
Without an interviewer to address clarifying questions, we'll define some use cases and constraints.
Use cases
We'll scope the problem to handle only the following use cases
- User posts a tweet
- Service pushes tweets to followers, sending push notifications and emails
- User views the user timeline (activity from the user)
- User views the home timeline (activity from people the user is following)
- User searches keywords
- Service has high availability
Out of scope
- Service pushes tweets to the Twitter Firehose and other streams
- Service strips out tweets based on users' visibility settings
- Hide @reply if the user is not also following the person being replied to
- Respect 'hide retweets' setting
- Analytics
Constraints and assumptions
State assumptions
General
- Traffic is not evenly distributed
- Posting a tweet should be fast
- Fanning out a tweet to all of your followers should be fast, unless you have millions of followers
- 100 million active users
- 500 million tweets per day or 15 billion tweets per month
- Each tweet averages a fanout of 10 deliveries
- 5 billion total tweets delivered on fanout per day
- 150 billion tweets delivered on fanout per month
- 250 billion read requests per month
- 10 billion searches per month
Timeline
- Viewing the timeline should be fast
- Twitter is more read heavy than write heavy
- Optimize for fast reads of tweets
- Ingesting tweets is write heavy
Search
- Searching should be fast
- Search is read-heavy
Calculate usage
Clarify with your interviewer if you should run back-of-the-envelope usage calculations.
- Size per tweet:
tweet_id- 8 bytesuser_id- 32 bytestext- 140 bytesmedia- 10 KB average- Total: ~10 KB
- 150 TB of new tweet content per month
- 10 KB per tweet * 500 million tweets per day * 30 days per month
- 5.4 PB of new tweet content in 3 years
- 100 thousand read requests per second
- 250 billion read requests per month * (400 requests per second / 1 billion requests per month)
- 6,000 tweets per second
- 15 billion tweets per month * (400 requests per second / 1 billion requests per month)
- 60 thousand tweets delivered on fanout per second
- 150 billion tweets delivered on fanout per month * (400 requests per second / 1 billion requests per month)
- 4,000 search requests per second
- 10 billion searches per month * (400 requests per second / 1 billion requests per month)
Handy conversion guide:
- 2.5 million seconds per month
- 1 request per second = 2.5 million requests per month
- 40 requests per second = 100 million requests per month
- 400 requests per second = 1 billion requests per month