# System Design Evolution: Building a URL Shortener from MVP to Planet-Scale

> *“Every simple system starts as a toy and evolves into infrastructure. The art lies in knowing when to evolve.”*

---

## 🧭 Why This Article Exists

Most system design discussions about URL shorteners stop at *“put a DB behind an API.”*  
This piece goes all the way — from **single-binary MVP** to **Meta-level, multi-region infrastructure** — showing how and why each evolution happens.

It’s meant to be both:

* A **reference document** for **system design interviews**
    
* A **guide** for engineers who actually want to *build* a scalable URL shortener in Go (TDD + production-grade)
    

---

## 📑 Table of Contents

1. Stage 1 – MVP (Proof of Concept)
    
2. Stage 2 – Single-Instance Production
    
3. Stage 3 – Database Persistence
    
4. Stage 4 – Caching Layer
    
5. Stage 5 – Distributed System
    
6. Stage 6 – Enterprise Scale
    
7. Architectural Decisions
    
8. SLOs and Failure Design
    
9. Interview Framework: How to Use This
    
10. Key Takeaways
    
11. Further Reading
    

---

## 🧩 Stage 1 — MVP (Proof of Concept)

```mermaid
flowchart LR
  C[Client] -->|HTTP| A[Go Server<br/>Single Binary]
  A --> M[(In-Memory Map<br/>key → long_url)]
  A --> MET[In-Memory Metrics]
```

| Aspect | Value |
| --- | --- |
| Deployment | Single binary |
| Storage | Go `map` + RWMutex |
| Scalability | ~1K requests/sec |
| Cost | &lt;$50/mo |
| Reliability | Data lost on restart ❌ |

### Example (Go)

```go
type URLShortener struct {
  mu   sync.RWMutex
  data map[string]string
}
```

✅ Fast, tiny, deployable in minutes  
❌ No persistence, single point of failure

**When to stop here:**  
Hackathon projects, demos, coding challenges.

---

## ⚙️ Stage 2 — Single-Instance Production

```mermaid
flowchart LR
  C[Client] -->|HTTP| A[Go App :8080]
  subgraph App Internals
    A --> CFG[Env Config]
    A --> LOG[Structured Logs]
    A --> H[Healthz /metrics]
    A --> KG[Base62 KeyGen]
    A --> VAL[URL Validation]
    A --> M[(In-Memory Map)]
  end
```

### New Concepts

* Graceful shutdown
    
* Environment-based config (`BASE_URL`, `PORT`)
    
* URL canonicalization
    
* Collision retry loop (insert-if-absent)
    

✅ Production-grade hygiene  
❌ Still no durability or scale

---

## 🗄️ Stage 3 — Database Persistence

### Architecture

```mermaid
flowchart LR
  C[Client] --> A[Server]
  A -->|Read/Write| PG[(PostgreSQL)]
  A --> MET[Metrics Exporter]
  subgraph PostgreSQL
    PG --> T[(urls table)]
  end
```

**Schema**

```sql
CREATE TABLE urls (
  key CHAR(8) PRIMARY KEY,
  long_url TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW(),
  click_count INT DEFAULT 0
);
CREATE INDEX idx_long_url ON urls(long_url);
```

### Why PostgreSQL?

* Easy to reason about consistency
    
* Native unique constraints for idempotency
    
* Works well with Go `pgxpool`
    
* Trivial to scale reads via replicas
    

| Metric | Value |
| --- | --- |
| Latency | 20–50 ms per read |
| Throughput | ~1K req/s |
| Data durability | ✅ |
| Horizontal scaling | 🚧 Needs more |

✅ Durable and correct  
❌ All reads hit DB  
❌ Writes limited by one primary

---

## ⚡ Stage 4 — Add a Caching Layer

### Architecture

```mermaid
flowchart LR
  C[Client] --> A[Go App]
  A -->|GET key| R[Redis Cluster]
  R -->|Hit: long_url| A
  R -->|Miss| PG[(PostgreSQL)]
  PG -->|long_url| A
```

**Read path**

```mermaid
sequenceDiagram
  participant Client
  participant App
  participant Redis
  participant Postgres

  Client->>App: GET /{key}
  App->>Redis: GET key
  alt Cache Hit
    Redis-->>App: long_url
    App-->>Client: 302 Location: long_url
  else Cache Miss
    App->>Postgres: SELECT long_url WHERE key=?
    Postgres-->>App: long_url
    App->>Redis: SET key=long_url EX 86400
    App-->>Client: 302 Location: long_url
  end
```

**Write path**

```mermaid
sequenceDiagram
  participant Client
  participant App
  participant Postgres
  participant Redis

  Client->>App: POST /api/shorten {url}
  App->>Postgres: INSERT (key, long_url)\nON CONFLICT RETRY
  Postgres-->>App: OK
  App->>Redis: DEL key  (invalidate)
  App-->>Client: 201 Created {key, short_url}
```

**Cache Policy**

* **LRU**: only top 1–5 % URLs
    
* **TTL**: 24 hours
    
* **Hit rate**: 80–90 % (Zipfian workloads)
    

| Metric | Value |
| --- | --- |
| Avg latency | ~5 ms |
| Peak throughput | 100K req/s |
| Cache memory | few GB |
| DB load reduction | ~90 % |

✅ Orders-of-magnitude faster  
✅ DB load drops drastically  
❌ Need cache invalidation strategy  
❌ Cache consistency issues possible

---

## 🌐 Stage 5 — Distributed System

### Sharded, Multi-Region Design

```mermaid

graph TB
    subgraph Internet["Internet"]
        Client1["Client 1"]
        Client2["Client 2"]
        ClientN["Client N"]
    end

    subgraph Global["Global Infrastructure"]
        GeoDNS["GeoDNS<br/>Global Load Balancer"]
    end

    subgraph US["🇺🇸 US Region"]
        USAPI["API Gateway<br/>Rate Limiter"]
        USAPP1["App Pod 1"]
        USAPP2["App Pod 2"]
        USAPP3["App Pod N"]
        USRED["Redis Cluster<br/>100 Shards"]
        USPG["PostgreSQL Cluster<br/>3,844 Shards<br/>Primary + 3 Replicas"]
    end

    subgraph EU["🇪🇺 EU Region"]
        EUAPI["API Gateway<br/>Rate Limiter"]
        EUAPP1["App Pod 1"]
        EUAPP2["App Pod 2"]
        EUAPP3["App Pod N"]
        EURED["Redis Cluster<br/>100 Shards"]
        EUPG["PostgreSQL Cluster<br/>3,844 Shards<br/>Primary + 3 Replicas"]
    end

    subgraph Analytics["📊 Analytics & Events"]
        Kafka["Kafka<br/>Event Stream"]
        Prometheus["Prometheus<br/>Metrics"]
        ELK["ELK Stack<br/>Logging"]
    end

    Client1 --> GeoDNS
    Client2 --> GeoDNS
    ClientN --> GeoDNS

    GeoDNS -->|Route to nearest| USAPI
    GeoDNS -->|Route to nearest| EUAPI

    USAPI --> USAPP1
    USAPI --> USAPP2
    USAPI --> USAPP3

    EUAPI --> EUAPP1
    EUAPI --> EUAPP2
    EUAPI --> EUAPP3

    USAPP1 --> USRED
    USAPP2 --> USRED
    USAPP3 --> USRED

    EUAPP1 --> EURED
    EUAPP2 --> EURED
    EUAPP3 --> EURED

    USAPP1 --> USPG
    USAPP2 --> USPG
    USAPP3 --> USPG

    EUAPP1 --> EUPG
    EUAPP2 --> EUPG
    EUAPP3 --> EUPG

    USAPP1 -.->|Async| Kafka
    EUAPP1 -.->|Async| Kafka

    USPG -.->|Metrics| Prometheus
    EUPG -.->|Metrics| Prometheus

    USAPP1 -.->|Logs| ELK
    EUAPP1 -.->|Logs| ELK

    USPG -->|Replication| EUPG
```

**Key generation**

* 8-char Base62 key = 62⁸ ≈ 2.18 × 10¹⁴ combinations
    
* First 2 chars = shard prefix (3,844 shards)
    
* Predictable, uniform distribution
    

**Replication Model**

| Role | Responsibility |
| --- | --- |
| Primary | Writes |
| Replicas | Reads |
| Sync replication | Strong consistency |
| Async cross-region | DR + read latency |

**Performance**

* ~500K req/s reads
    
* p99 latency ≈ 20 ms
    
* Availability ≈ 99.99 %
    

✅ Scale by adding shards  
✅ Regional isolation  
❌ Operational overhead (failover, backups)

---

## 🏢 Stage 6 — Enterprise / Meta-Scale

### Global Topology

```mermaid
flowchart TB
  GDNS[GeoDNS / Anycast]
  APIGW[API Gateway / Global Rate Limit]

  subgraph Region A
    AAPP[K8s App Pods]
    ARED[Redis Cluster]
    APG[(Postgres Shards)]
    AKF[(Kafka)]
  end

  subgraph Region B
    BAPP[K8s App Pods]
    BRED[Redis Cluster]
    BPG[(Postgres Shards)]
    BKF[(Kafka)]
  end

  subgraph Observability
    PM[Prometheus]
    GR[Grafana]
    ELK[ELK / Logs]
    TR[Jaeger / Tracing]
  end

  GDNS --> APIGW
  APIGW --> AAPP
  APIGW --> BAPP

  AAPP --> ARED
  AAPP --> APG
  AAPP --> AKF

  BAPP --> BRED
  BAPP --> BPG
  BAPP --> BKF

  AAPP -. metrics .-> PM
  BAPP -. metrics .-> PM
  PM --> GR
  AAPP -. logs .-> ELK
  BAPP -. logs .-> ELK
  AAPP -. traces .-> TR
  BAPP -. traces .-> TR
```

**Supporting Infrastructure**

* **Redis Cluster**: 300 shards (multi-region replication)
    
* **Postgres Shards**: 3,844 per region × 3 replicas
    
* **Eventing**: Kafka for click streams
    
* **Observability**: Prometheus, Grafana, ELK, Jaeger
    
* **Disaster Recovery**: RPO &lt; 10 s, RTO &lt; 1 min
    

**Numbers (illustrative)**

| Metric | Value |
| --- | --- |
| Read throughput | 5 M req/s |
| Write throughput | 100 K req/s |
| p99 latency | 20 ms |
| Cache hit rate | 85 % |
| Availability | 99.999 % |
| Annual volume | 150 T requests/year |

✅ Planet-scale durability  
✅ Real-time analytics  
✅ Regional failover  
❌ Huge operational complexity

---

## 🧮 Architectural Decisions

| Decision | Choice | Reasoning |
| --- | --- | --- |
| **Key length** | 8 chars | ~218 T unique keys, negligible collision probability |
| **Sharding** | Prefix (first 2 chars) | Uniform distribution, easy routing |
| **Cache strategy** | Hot 1–5 %, LRU + TTL 24h | High hit-rate, bounded memory |
| **Metrics** | In-mem counters + periodic DB flush | 500 ns updates, durability via batch |
| **Storage API** | Interface-based (`Store` interface) | Enables swap: memory ↔ Redis ↔ Postgres |
| **Security** | Validate URLs, block private IPs, rate-limit writes, sign aliases | Prevent SSRF, spam, and abuse |
| **Observability** | RED metrics, structured logs, tracing | Enable SLO monitoring & debugging |

---

## 📈 SLOs and Failure Design

| Tier | Target | Degradation Strategy |
| --- | --- | --- |
| **Availability** | 99.99 % | Serve from cache → read replicas → failover region |
| **Latency** | p50 ≤ 10 ms, p99 ≤ 50 ms | Prefer cached reads; fallback replicas |
| **Durability** | RPO ≤ 10 s, RTO ≤ 1 min | Promote replica, replay Kafka logs |
| **Scalability** | Linear ≈ 1M req/s per region | Horizontal app + cache scaling |

---

## 🧠 Interview Framework: How to Use This

When asked to *“Design a URL shortener”* in interviews, use this 5-minute outline:

| Step | What to Say | Key Points |
| --- | --- | --- |
| **1\. Requirements** | Shorten, redirect, delete, analytics | Functional + non-functional (scale, latency) |
| **2\. MVP** | One Go server + in-memory map | Start small |
| **3\. Persistence** | Add Postgres for durability | Read/write paths |
| **4\. Scale reads** | Add Redis read-through cache | Explain cache invalidation |
| **5\. Scale writes** | Shard DB by key prefix | 62² ≈ 3.8K shards |
| **6\. Geo-scale** | Multi-region replicas, CDN, rate limiting | Fault tolerance |
| **7\. Observability** | Metrics, tracing, logging | SLOs & debugging |
| **8\. Trade-offs** | Latency vs Durability vs Cost | Articulate why |

💡 *Hint:* Keep numbers handy — 8-char key ≈ 2 × 10¹⁴, Redis hit ~ 1 ms, DB read ~ 50 ms.

---

## 🔍 Key Takeaways

1. **Scale evolves.** Each layer solves a previous bottleneck.
    
2. **Most traffic is read-heavy.** Optimize 80 % cached reads first.
    
3. **Simplicity beats premature scale.** Don’t shard until you must.
    
4. **Measure relentlessly.** Latency histograms, p99, error budgets.
    
5. **Design for failure.** Everything breaks; plan for graceful degradation.
    

---

## 📚 Further Reading

* [Designing Data-Intensive Applications – Martin Kleppmann](https://dataintensive.net/)
    
* [Google SRE Book – Chapter 6 (Monitoring)](https://sre.google/sre-book/monitoring-distributed-systems/)
    
* [Base62 and Collision Probability](https://en.wikipedia.org/wiki/Birthday_problem)
    
* [Postgres Sharding](https://www.percona.com/blog/an-overview-of-sharding-in-postgresql-and-how-it-relates-to-mongodbs/)
    

---

## 🏁 Closing Thoughts

Building a URL shortener seems trivial — until you have to do it for billions of users.

Start with a single process.  
Measure.  
Then **add layers only when you must**.

That’s not just how you scale systems — it’s how you scale engineering judgment.

---

## 🏁 Code

> All source code, tests, and diagrams are available here → [https://github.com/AkshayContributes/url-shortener](https://github.com/AkshayContributes/url-shortener)

---

*Written by Akshay Thakur (2025)*  
*Staff-level System Design Reference | Go + Distributed Systems + Scale Engineering*

---