Designing a Scalable URL Shortener Handling 100M urls Per Month

URL shorteners are essential in modern web applications, allowing users to create compact links for sharing and tracking. But what happens when your service scales to 100 million urls per month? This article explores the architecture, storage, latency, and scalability considerations required to handle this level of traffic.

Key Features:

One of the core challenges in building a scalable URL shortening system is ensuring that each generated short URL is unique, even in a distributed environment with multiple replicas.

Assumptions:

1. Assume the short URL will be 7 digits at maximum.
2. Assume our application will be distributed in multiple servers.
3. Assume 100 million new unique URL shortenings per month.
4. Assume the short url expiration is per month.

Traffic and System Capacity Assumptions

Assumptions:

• 100M short URLs generated per month
• 200:1 read/write ratio (for every new short URL created, it is accessed 200 times)
• Traffic distribution: Uniform over time
• Storage required per URL: 100 bytes (URL metadata, analytics, etc.)

Breakdown of Requests Per Second (RPS)

URL Creations:

Total writes per month: 100.000.000 / (30 x 24 x 3600) = 40 urls/second

URL Reads:

Total reads per second: 40 x 200 = 8.000 reads/second

Storage Estimation

Each URL entry consists of:

• Short URL: 7–10 characters (Base62 encoding)
• Original URL: 100–300 characters (avg. 150 bytes)
• Metadata: User ID, timestamp, analytics (~50 bytes)

Each entry size: ~100 bytes Total storage per month: 100M x 100B = 10GB/month

Assuming 3 years of data retention: 10 GB x 36 = 360GB

This is manageable with distributed databases or cloud storage solutions.

System Architecture

Architecture Diagram

1. URL Generation Strategy

To generate unique short URLs, we have a few options:

UUIDv7 (7-digit base62)

Highly random, no sequence predictability Higher collision probability with truncation.

function generateShortUrl(): string {
  const uuid = uuidv7().replace(/-/g, ""); // Remove dashes

  // Extract the last 7 hexadecimal characters (higher randomness part of UUID)
  const uuidSubstring = uuid.slice(-7);

  // Convert hex substring to a numeric value
  const numericValue = parseInt(uuidSubstring, 16);

  // Encode in Base62 to get a short URL
  return base62Encode(numericValue);
}

• 7-character Base62 UUIDv7 is unsafe for large-scale URL shortening (e.g., 100M+ URLs/month).
• If you keep only a small portion (e.g., the last 7 characters), you’re no longer using all the entropy.

import random
import hashlib

def generate_uuid7():
    """Simulates a UUIDv7-like structure with random values."""
    random_part = random.getrandbits(64)  # Last 64 bits have high randomness
    return f"{random_part:016x}"  # Convert to a 16-character hex string

def extract_last_7_chars(uuid_str):
    """Extracts the last 7 characters from a UUID."""
    return uuid_str[-7:]

def base62_encode(hex_str):
    """Encodes a hex string into Base62."""
    BASE62_CHARSET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
    num = int(hex_str, 16)
    encoded = ""
    base = len(BASE62_CHARSET)
    while num > 0:
        encoded = BASE62_CHARSET[num % base] + encoded
        num //= base
    return encoded or "0"

def simulate_collisions(num_samples):
    """Simulates the collision probability when truncating UUIDv7 to 7-character Base62."""
    seen = set()
    collisions = 0

    for _ in range(num_samples):
        uuid7 = generate_uuid7()
        short_key = base62_encode(extract_last_7_chars(uuid7))

        if short_key in seen:
            collisions += 1
        seen.add(short_key)

    return collisions, num_samples

# Run simulation for 1 million URLs
num_samples = 1_000_000
collisions, total = simulate_collisions(num_samples)

# Output results
collision_probability = collisions / total
collision_probability

the simulation colision probability result is 0.001778. The simulation for 1 million URLs shows a collision probability of ~0.178% (1.78 in 1000 URLs) when truncating UUIDv7 to 7 Base62 characters.

• At 100M URLs/month (~1.2B per year), the probability of at least one collision becomes effectively 100%.
• For 100 million URLs/month, there is an estimated 76.1% probability of at least one collision using a 7-character Base62 truncated UUIDv7.
• Collisions will increase exponentially as more URLs are generated.

MD5 (7 digit Base62)

Consistent hash from original URLPossible hash collisions, needs a uniqueness check. MD5 produces a 128-bit hash (32 hexadecimal characters). A 7-character substring (assuming Base62) provides log₂(6²⁷) ≈ 41.7 bits of entropy.

function generateShortUrl(originalUrl: string): string {
  // Generate an MD5 hash of the original URL
  const md5Hash = crypto.createHash("md5").update(originalUrl).digest("hex");

  // Extract the first 7 hexadecimal characters
  const md5Substring = md5Hash.substring(0, 7);

  // Convert the substring into a numeric value
  const numericValue = parseInt(md5Substring, 16);

  // Encode in Base62 to get a short URL
  return base62Encode(numericValue);
}

• At 100M URLs per month (~1.2B per year), the probability of at least one collision is effectively ~100%, based on Birthday Paradox formula.
• MD5 is do not maintain order so this prevents efficient database indexing strategies that leverage ordering (e.g., time-based partitioning).
• Alternatively need a uniqueness check for every generation of short url if we want to extract the first 7 digit of MD5 that caused to database latency problem and overhead

Sequential (Redis Counter 7 digit base62)

Simple, efficient, low overhead Predictable, susceptible to enumeration attacks

const baseNumber = 100000000000 //100.000.000.000 billion
const redisCtr = await getRedisCtr();
const base62Key = convertToBase62(baseNumber + redisCtr);
await incrementCounter(baseNumber + redisCtr);

//output 100000000000 + 1 => 1l9Zo9o
return base62Key;

With 7 digits, our short URL can hold up to 3.5 trillion unique URLs with zero collisions. However, it may be vulnerable to enumeration attacks because the URL is predictable due to the sequential counter.

1-digit Base62: 0-61

2-digit Base62: 62^2 = 3,844

3-digit Base62: 62^3 = 238,328

4-digit Base62: 62^4 = 14,776,336

5-digit Base62: 62^5 = 916,132,832

6-digit Base62: 62^6 = 56,800,235,584

7-digit Base62: 62^7 = 3,521,614,606,208

✅ Base62 (7 digits) can handle up to ~3.52 trillion unique URLs.
✅ At 100M URLs/month, it lasts over 2,900 years.
✅ If you ever exceed 3.52 trillion, Base62 just moves to 8 digits automatically.

Use a Time-Based Component + Sequential (Redis Counter 7 digit base62)

To solve Sequential (Redis Counter base62) enumeration attack we will use this approach

const uniqueValue = (Date.now() << 10) | (counter & 0x3FF);
const shortUrl = base62Encode(uniqueValue);

//output
uniqueValue = (1709491200000 << 10) | (123 & 0x3FF);

//convert to base62
1750475417600123 (decimal) → "B8A7zXG"

• Date.now() (milliseconds timestamp) → Ensures time-based uniqueness.
• (<< 10) shift left by 10 bits → Allocates 10 bits for the counter.
• Redis counter (counter & 0x3FF) → Provides 1024 possible values (0–1023) per millisecond.
• Base62 encoding → Converts the large integer into a short URL string.

✅ For 100M URLs/month (40 URLs/sec), there is practically no collision risk.
✅ Even at 10,000 URLs/sec peak load, the collision probability is <1%.

From the code above, we eliminate the possibility of an enumeration attack and ensure zero collisions in the uniqueness of short URLs by using a sequential counter, even in a distributed system with multiple servers.

2. Database Choice

• High-read, low-write workload → Use NoSQL (e.g., Redis, mongodb, documentDB, DynamoDB, or Cassandra)
• Ensuring uniqueness → Primary key constraints or distributed counters
• Fast retrieval → Redis caching for hot URLs

Database schema (NoSql) Example:

{
 "short_url": "https://our-system.com/hBxNjgU",
 "original_url": "https://example.com/very-long-url",
 "created_at": "2025–03–03T12:00:00Z",
 "updated_at": "2025–03–03T12:00:00Z",
 "deleted_at": null,
 "is_active": true,
 "meta_data": {
    //any meta data that you want to save
 },
}

Assume we choose aws documentdb for this case for early start.

Scaling Plan

Since it has 8,000 reads/sec and a 200:1 read/write ratio, optimizing queries & indexing is crucial to ensure low latency and prevent bottlenecks. Utilization of compound index is important in here:

db.urls.createIndex({ is_active: 1, short_url: 1 })

3. Caching Strategy

To reduce database reads, implement:

According to the Pareto Principle (80/20 rule), about 20% of short URLs will account for 80% of traffic. This means we should focus our caching strategy on frequently accessed URLs to maximize efficiency.

Redis Memory Estimation

• Hot URLs (20%) = 0.2 x 100M = 20M entries
• Each entry in Redis: Short URL (key) → Original URL (value)
• Average short URL size: 10 bytes
• Average original URL size: 150 bytes
• Total Redis memory required: 20M x (10B + 150B) = 3.2GB

With an LRU eviction policy, Redis can efficiently manage this cache size, keeping only the most popular URLs in memory for fast redirection.

4. Load Balancing & API Gateway

• Nginx / Envoy / AWS API Gateway to distribute traffic
• Rate limiting & abuse detection

5. Expiration URL strategy

• API-Based Expiration Check when fetching a short URL, check created_at If expired, set is_active: false and return 404 Not Found.
• Set Expiration on Creation (1-Month TTL) (Time-To-Live) per URL.
• Batch Cleanup via AWS EventBridge + ECS a cron job (EventBridge Rule) triggers a task on AWS ECS Fargate every month to set is_active: false on each row.

Conclusion:

Building a scalable URL shortener that handles 100M URLs per month requires careful architectural decisions to balance uniqueness, storage efficiency, performance, and scalability.

✅ Short URL Generation: We explored multiple approaches, including UUIDv7, MD5 hashing, sequential counters, and time-based + counter strategies, ultimately choosing a time-based sequential Base62 encoding to ensure uniqueness while preventing enumeration attacks.

✅ Storage & Database Optimization: Given the high-read, low-write workload, we leverage NoSQL databases (Redis, MongoDB, DynamoDB) with caching strategies to improve retrieval speeds while ensuring uniqueness constraints at scale.

✅ Caching for Performance: Applying the Pareto Principle (80/20 rule) ensures that hot URLs are prioritized in Redis caching, significantly reducing database read pressure.

✅ Expiration Strategy for Storage Efficiency: Instead of immediate deletion, URLs are marked inactive upon expiration, and a batch cleanup process using AWS EventBridge + ECS Fargate ensures efficient database maintenance.

By implementing distributed architecture, caching, efficient URL generation, and scheduled cleanups, this design ensures the system can scale seamlessly, minimize latency, and handle billions of requests per year — all while keeping storage and operational costs manageable.