Distributed ID Generation

Introduction

Every system needs identifiers.

Examples:

At small scale, generating IDs is simple:

Auto Increment: 1, 2, 3, 4, 5

But in distributed systems, things become much harder.

Modern platforms like:

• Twitter
• Instagram
• Uber

generate millions of IDs per second across thousands of servers.

The challenge becomes:

Generating such IDs reliably is the problem of Distributed ID Generation.

Why ID Generation Becomes Hard in Distributed Systems

Imagine a system with 100 application servers.

If every server generates IDs locally:

Server A → ID 1
Server B → ID 1
Server C → ID 1

Now collisions happen.

Multiple servers generate duplicate IDs.

This creates severe issues:

• Data corruption
• Wrong records
• Inconsistent references
• Broken relationships between entities

Distributed System Architecture Problem

Each server must generate IDs without coordination bottlenecks.

Key Requirements of Distributed ID Generators

Approaches to Distributed ID Generation

Several strategies exist.

Let's explore them deeply.

1. Database Auto-Increment IDs

How It Works

The database generates IDs automatically.

CREATE TABLE users (
   id BIGINT AUTO_INCREMENT,
   name VARCHAR(100)
);

Insert query:

INSERT INTO users(name) VALUES ('Alice');

Generated IDs:

1
2
3
4

Architecture

Problems

For large-scale systems, this approach does not scale.

2. UUID (Universally Unique Identifier)

UUID is a 128-bit identifier designed to be globally unique.

Example:

550e8400-e29b-41d4-a716-446655440000

UUID Generation

Generated locally without coordination.

import { randomUUID } from "crypto"
 
const id = randomUUID()
console.log(id)

Structure (UUID v4)

Advantages

Problems

For high-scale databases, UUIDs fragment indexes heavily.

3. Centralized ID Generation Service

Another approach is creating a dedicated ID generation service.

Architecture:

Each request:

GET /generate-id

Response:

983482394

Advantages

Problems

Large systems avoid this design.

4. Twitter Snowflake ID Generator

One of the most famous distributed ID generators.

Developed by:

• Twitter

Key Idea

Generate IDs using:

Timestamp + Machine ID + Sequence

This ensures:

• uniqueness
• ordering
• scalability

Snowflake ID Structure

64-bit integer:

Visual Structure

Example ID

154742918274918

Internally contains:

timestamp = creation time
machine id = server
sequence = request counter

How Snowflake Generates IDs

Steps:

1. Get current timestamp
2. Use server's machine ID
3. Increment sequence number
4. Combine bits

Snowflake Generation Flow

Benefits

Capacity

Snowflake supports:

Real-World Systems Using Snowflake-like IDs

Many companies built similar systems.

5. Database Sequence with Sharding

Another approach:

Split sequences across shards.

Example:

Server1 generates:

1,4,7,10

Server2 generates:

2,5,8,11

Server3 generates:

3,6,9,12

Architecture

Trade-offs

ID Ordering and Database Indexing

Why ordering matters.

Databases store indexes as B-trees.

Sequential IDs:

1 → 2 → 3 → 4

Work efficiently.

Random IDs:

A23 → 7FF → 91A

Cause:

• page splits
• index fragmentation
• slower inserts

Snowflake solves this because IDs are time ordered.

Global ID Generation Architecture

Typical distributed architecture:

Each server generates IDs locally.

Failure Handling

Edge cases:

Clock Drift

If system clock moves backward:

Snowflake may generate duplicate IDs.

Solutions:

• NTP synchronization
• wait until time catches up

Machine ID Conflicts

If two machines use the same ID.

Solution:

• assign IDs via configuration
• service discovery

Comparison of ID Generation Strategies

Snowflake-style generators are the industry standard today.

Best Practices

Prefer 64-bit IDs

Efficient storage and indexing.

Avoid Random UUIDs in Databases

They cause index fragmentation.

Use Snowflake-like systems

For:

• distributed microservices
• high throughput platforms

Ensure Clock Synchronization

Critical for time-based IDs.

Final Architecture Summary

Every node generates unique, ordered IDs without coordination.

Key Takeaways

Distributed ID generation is a fundamental building block of scalable architectures, enabling large platforms to create billions of records reliably without coordination bottlenecks.