System Design Part 2: Estimation, Consistent Hashing, Rate Limiters, CAP & Unique IDs

In Part 1 we grew one tiny server into something that could handle millions of users. That was the "what" of system design.

This time we cover the "how" and the "how much": five ideas that come up right after you draw your first architecture diagram. No code, just clear explanations.

1. Back-of-the-Envelope Estimation

When planning a wedding, the first question is not "which flowers?" It is "how many guests?" Fifty guests and five thousand guests need completely different plans.

System design works the same way. Before drawing any boxes, get a rough feel for the traffic and storage you are dealing with. You do not need to be exact, just roughly right.

Let's try it on a Twitter-like app. Assume 150 million daily users, each posting 2 times and opening their feed 100 times a day. A day has about 100,000 seconds (the real number is 86,400, but we round).

• Writes: 150M × 2 ÷ 100,000 ≈ 3,000 posts per second
• Reads: 150M × 100 ÷ 100,000 ≈ 150,000 reads per second

People read about 50 times more than they write. That one fact tells you to invest in caching, read replicas and a CDN, which is exactly the toolkit from Part 1.

Storage tells a similar story. Text posts are tiny, around 90 GB a day, so a sharded database handles them fine. But if 1 in 10 posts has a 1 MB photo or video, that is roughly 30 TB a day. Media clearly does not belong in the database. It goes into object storage like S3, served through a CDN.

We did not guess. The numbers made the design decision for us.

2. Consistent Hashing

In Part 1 we split a database across 4 shards using user_id % 4. It is simple, and quietly dangerous.

The problem

Picture a restaurant with 4 waiters. The rule: divide the table number by 4, and the remainder picks the waiter. Now one waiter calls in sick, so the rule becomes "divide by 3". Suddenly almost every table has a different waiter, and nobody knows anyone's order.

That is what happens to a cache cluster when a server disappears. Nearly every key maps to a new server that does not have the data, so all requests fall through to your database at once, and it collapses. This is called a cache stampede.

The idea

Consistent hashing fixes this. Imagine a circular clock. Place every server on the clock, and place every piece of data on it too. To find the owner of a piece of data, walk clockwise until you hit a server.

If Server B leaves, only B's data moves, to the next server clockwise. Everyone else stays put. Adding a server works the same way in reverse. Instead of reshuffling nearly everything, only a small slice moves.

Virtual nodes

With only a few servers, one can end up owning half the clock. The fix is to place each server on the clock many times (100 or so spots). These are called virtual nodes.

Now each server owns lots of tiny pieces, the data spreads evenly, and when a server leaves, its load is shared across all the others instead of crushing one neighbor.

This idea quietly powers databases like DynamoDB and Cassandra, cache clusters, and CDNs.

3. Designing a Rate Limiter

Think of a bouncer at a popular club. They are not there to be rude, only to keep the room from getting so crowded that nobody has a good time.

A rate limiter is that bouncer for your API. It limits how many requests one client can send in a given time, and says "slow down" to anyone over the limit. It protects you from abuse, from buggy scripts stuck in a loop, from surprise bills on paid services, and from password brute-forcing.

It usually sits at the front door, in an API gateway, so blocked requests never reach your real servers. Those clients get back a 429 Too Many Requests response.

The two approaches worth knowing

Token bucket. A bucket slowly fills with tokens, say one per second, up to a maximum. Each request spends a token, and if the bucket is empty you wait. It allows short bursts, since a quiet client has saved up tokens. Many big companies use this one.

Sliding window. Count requests over the last 60 seconds from right now, instead of resetting at fixed minute marks. This avoids the flaw of fixed windows, where someone sends a full allowance at 11:59 and another at 12:00 and gets double the intended rate.

The "many servers" problem

You rarely have one gateway. If each gateway keeps its own private count, a client hitting 10 gateways gets 10 times the allowance.

The fix is to keep the counters in one shared, fast place, usually Redis, and make the check-and-update a single unbreakable step so two gateways cannot both approve the last request.

4. The CAP Theorem

Imagine you and a friend in different cities each keep a copy of a shared notebook. One day the phone lines go down, and someone asks you a question. You have two choices:

1. Refuse to answer until you can confirm with your friend. Your notes are never wrong, but you were unhelpful.
2. Answer with what you have. You were helpful, but your friend may have updated something you have not seen.

That is the whole theorem. When a network failure splits a distributed system, you must choose between being consistent (correct) and available (always answering). You cannot have both at that moment. Since network failures will happen, the real question is what you protect when they do.

The right choice depends on your product:

• Bank balance? Choose consistency. Showing a wrong number, or letting someone spend the same money twice, is a disaster. Better to say "try again in a moment."
• Like counter? Choose availability. If one person sees 10,482 likes and another sees 10,479 for a few seconds, nobody is harmed.

Systems that favor availability promise eventual consistency: if no new changes arrive, all copies settle on the same value, usually within milliseconds.

So it is not about which database is "better". It is about what your product can tolerate.

5. Generating Unique IDs

On a single database, IDs are easy: just count up, 1, 2, 3. But once the database is split into shards, each shard starts counting from 1, and soon five different users are all #42.

We need IDs that are unique across all machines without everyone calling a central office for a number. We also want them roughly sorted by time, which keeps pagination and database indexes happy.

The common options:

• UUIDs are long random codes generated on any machine, with no coordination. The downsides are that they are large and unordered. Newer versions like UUIDv7 and ULID add a timestamp at the front to fix that.
• A ticket server is one database that only hands out numbers. Simple, but if it goes down, nothing can be created.
• Snowflake, invented at Twitter, builds each ID from meaningful pieces, like a phone number:

The timestamp keeps IDs sortable, the datacenter and machine parts guarantee two machines never produce the same ID, and the counter handles several IDs created in the same millisecond. Every machine works alone, yet each one can produce millions of unique IDs per second.

And remember: for a small app on one database, plain auto-increment is perfectly fine. Pick the simplest thing that survives your scale.

Quick Recap

1. Estimate first. Know your reads, writes and storage before you design.
2. Use consistent hashing instead of hash % N when servers come and go.
3. Rate limit at the edge with token bucket or sliding window, with counters shared in Redis.
4. Pick consistency or availability per feature. Money needs the first, feeds can live with the second.
5. Use time-sortable IDs like Snowflake, UUIDv7 or ULID when you shard.

In Part 3, we will put it all to work by designing real systems from start to finish: a URL shortener, a notification service, and a news feed.

Until then, keep asking "how many?" before "which tool?"

References & Recommended Resources