System Design:
the decisions that make a system grow

CHAPTER 01

When the app grows

Imagine that you created a delivery app. The code is clean, the tests pass, and the first ten people use it without any problems. One hundred users? It still works. Then the app appears in a viral video — and suddenly ten thousand orders arrive at the same time.

The problem may not be in the business rule. It often appears when resource limits and defects invisible at low load — race conditions, memory leaks, N+1 queries, locks, misconfigured pools or expensive algorithms — begin to surface. Queries wait for connections, locks and processing capacity; retries can make everything worse.

In the demo, CPU and database connections saturate first. In practice, these are common — not universal — bottlenecks. Network, disk, memory, garbage collection, locks, thread pools, external APIs or provider limits may be the first ceiling. Measure before choosing the solution.

CHAPTER 02

What is System Design

Programming is solving problems within a component. System Design is deciding how components make up a system: where data lives, how services talk to each other, what happens when something fails, and how you find out that it has failed.

In practice, every design decision answers one of these seven questions:

• DATAWhere is each piece of information stored — and who is the source of truth?
• COMMUNICATIONDo services talk synchronously (waiting for a response) or asynchronously (via queues)?
• SCALEWhen demand doubles, will the system grow by changing machines or adding machines?
• FAILUREWhat happens when — not “if” — a component goes down?
• SECURITYWho can access what, and how does the system protect itself from abuse?
• OBSERVABILITYHow do you see what is happening inside, in production?
• OPERATIONS & EVOLUTIONHow do you deploy, migrate, test, roll back and change the system without disrupting users?

None of these questions has a universal “right” answer. System Design is the discipline of trade-offs: every choice buys a benefit at a cost, and a good designer knows which cost the product can afford at that moment.

CHAPTER 03

The simple beginning

Every famous architecture started the same: one client, one server, one database. The user makes a request, the server processes it, consults the database, and the response comes back via the same path.

It's worth understanding this path carefully, because everything that comes after — caches, queues and replicas — is a variation of it. Press the button and follow the request:

This architecture has a somewhat unfair name: “too simple”. For many early-stage products with moderate load, it is often the right decision. User count alone does not decide: usage frequency, operation cost, data volume and availability requirements matter.

CHAPTER 04

Scalability: one bigger machine or multiple machines?

When a component saturates, there are two classic ways to increase capacity. Vertical scaling: use a more powerful machine. Horizontal scaling: add instances and distribute traffic. Before either, there is a cheaper question: can we do less work by fixing queries, indexes, algorithms, batching or redundant calls?

Vertical scaling is simple: nothing changes in the code; you just pay more. But it has a ceiling (the biggest machine in the world is still one machine) and preserves a single point of failure — if it goes down, everything goes down. Horizontal scaling is elastic and resilient, but it comes at a price: your code must handle multiple instances running at the same time, which makes sessions, uploads and local state more complex.

In practice, many systems use sufficiently powerful machines and multiple instances behind a load balancer. Horizontal scaling reduces dependence on one instance, but does not remove single points in the load balancer, database, cache, authentication, region or provider. Redundancy must span the architecture.

CHAPTER 05

Latency & throughput: fast is not the same thing as handling volume

Two metrics dominate any performance conversation, and they measure different things. Latency is the time it takes a request to be responded to — a user's experience. Throughput is how many requests the system processes per second — the capacity of the whole.

A system can have great latency with few people and collapse with volume. The classic pattern: latency remains stable as long as there is spare capacity and explodes non-linearly as the system approaches saturation. Drag the slider and observe the knee of the curve:

Concurrency is how many operations are in progress; offered throughput is what arrives; processed throughput is what completes. Ten concurrent users do not imply ten requests per second. A p95 is the value at or below which 95% of observations fall; the remaining 5% had equal or higher latency. p99 makes the same cut at 99%.

CHAPTER 06

Data, cache & CDN: the bottleneck just moved

You scaled horizontally, put five servers behind the load balancer… and the system is still slow. Why? Because all five hit the same database. Scaling the application layer only pushes the bottleneck downstream.

The first tool against this is a cache: a fast in-memory layer (Redis, Memcached) that stores the results of frequent queries. A restaurant's menu doesn't change every second — so why query the database 12,000 times per minute to answer the same question?

Cache is not magic. TTL is only one strategy; event-driven invalidation, cache-aside, write-through, write-behind and versioning also exist. Each can introduce stale data, hot keys, stampedes, expiration avalanches and cold starts. A cache does not fix a bad query; it only avoids running it sometimes.

Before or alongside caching, the database can gain indexes, better queries, pooling, batching, materialized views and denormalization. When one node is no longer enough, fundamental decisions appear:

• REPLICATIONCopy data for fault tolerance and read scale; asynchronous replicas may serve stale reads.
• PARTITIONINGSplit data across nodes for storage and write scale, paying complexity in queries and rebalancing.
• CONSISTENCYDecide when a write becomes visible: strong, eventual or guarantees such as read-your-writes.
• CDN & OBJECTSServe images, menus and files at the edge so this work never reaches the origin API.

CHAPTER 07

Asynchronous communication: not everything needs to happen now

When the user taps “place order”, the system can record the request and return “received” or “processing”. Payment, stock reservation and confirmation may happen asynchronously — as long as the interface does not promise an outcome that has not happened. “Order received” differs from “order confirmed.”

This is where messaging and streaming systems such as RabbitMQ, SQS or Kafka help, each with different models and guarantees. Kafka is a distributed event-streaming platform with retention, replay, partitions and consumer groups — not simply a traditional queue.

The cost is designing delivery and processing guarantees: at-most-once, at-least-once and exactly-once only in bounded contexts; plus duplication, ordering, retries, dead-letter queues, poison messages, partitioning and backpressure. Usually the message itself does not “fail”: delivery is not acknowledged or the consumer cannot process it.

CHAPTER 08

Resilience: failures will happen — plan for recovery

At scale, failure is no exception, it is routine: a disk dies, a deploy breaks, an entire datacenter goes down. A resilient system is not something that never fails — it is something that continues to serve while it fails.

The kit starts with a timeout: without a limit, a call may remain stuck indefinitely. Then come retries with exponential backoff and jitter, circuit breakers, idempotency, bulkhead isolation, graceful degradation and failover. Retries should target transient failures, with attempt limits and budgets; otherwise they multiply traffic during overload.

Idempotency means equivalent executions do not create additional effects in observable state; it often requires a key, deduplication, a unique constraint and a transaction. Regional failover is not instant by definition: it depends on replication, routing, secondary capacity and tests. RPO defines acceptable data loss; RTO, acceptable downtime.

During overload, resilience also means protecting capacity: rate limits, quotas, backpressure, load shedding and priority can preserve order creation and payment while recommendations and reports degrade.

CHAPTER 09

Observability: find the problem before the user does

“The app is slow” is the worst bug report in the world — and it is what you will get without instrumentation. Observability is the ability to investigate internal state through emitted signals. Logs, metrics and traces matter, but only help with context, correlation, trace IDs, controlled cardinality, queries, retention, sampling and actionable alerts.

Logs tell you what happened, event by event. Metrics show aggregate behavior over time (CPU, latency, error rate). Traces follow a single request across all services — and show exactly where it spent time. Use the trace below to find the culprit of a slowdown:

The trace shows that 2.4 seconds are concentrated in the payment service. To claim an external gateway is the cause, we would need a child span for that call; without it, the delay may still be a lock, database, network, garbage collection or internal processing. SLIs measure behavior such as latency and success; SLOs define objectives such as “99.9% of valid requests will succeed.”

CHAPTER 10

The complete architecture — and when not to use it

Put everything together and one possible architecture emerges: a client using a CDN for static content, WAF and rate limiting protecting the entry point, a load balancer distributing traffic, stateless APIs, cache, replicated or partitioned data and messaging. Security, observability, configuration and deployment are not isolated boxes: they span every component.

Now, the most important lesson of the article: this diagram is not a goal, it is a repertoire. Each box exists to solve a specific problem that you may not already have. Starting a new product with Kafka, five microservices and three regions is as wrong as serving a million users with just one server.

System Design done well is this: knowing the tools, understanding the trade-offs, and having the discipline to apply the right solution to the current size of the problem — leaving the path open to evolve when the problem grows.