Key Principle
All difficulty in replication lies in handling changes to replicated data (Ch. 5). Static copies are trivial; propagating mutations across independently-failing nodes produces every consistency anomaly the chapter explores. The three replication architectures -- leader-based, multi-leader, and leaderless -- form a spectrum trading ease of reasoning for fault tolerance and availability. Replication is orthogonal to partitioning: replication copies the same data across nodes for redundancy and read throughput; partitioning splits data into subsets for write throughput and dataset size.
Why This Matters
Replication serves four purposes: high availability, disconnected operation, latency reduction via geographic proximity, and read scalability. Every production database uses replication, and the choice of architecture determines which failure modes your application must handle.
The synchronous/asynchronous spectrum is the foundational trade-off. Synchronous replication guarantees follower currency but "any one node outage would cause the whole system to grind to a halt" (Ch. 5). Asynchronous replication is "widely used, especially if there are many followers or if they are geographically distributed" but risks silent data loss -- "any writes that have not yet been replicated to followers are lost" on leader failure (Ch. 5). Semi-synchronous (one sync follower, rest async) is the pragmatic middle ground most systems actually run: "This guarantees that you have an up-to-date copy of the data on at least two nodes" (Ch. 5).
Replication lag has no upper bound: "The term 'eventually' is deliberately vague: in general, there is no limit to how far a replica can fall behind" (Ch. 5). This unbounded lag drives the three consistency guarantees -- read-after-write, monotonic reads, consistent prefix reads -- each addressing a specific user-visible anomaly.
Good Examples
GitHub failover incident (ref [13]): A promoted follower reused autoincrementing primary keys already assigned by the old leader, causing MySQL-Redis inconsistency and private data disclosure. Demonstrates that failover dangers multiply when external systems depend on database-generated identifiers (Ch. 5).
Amazon shopping cart bug: Per-row conflict resolution in a multi-leader system preserved added items but dropped removals, causing deleted items to reappear. Shows how per-row resolution breaks the atomicity a transaction was supposed to provide (Ch. 5).
Offline-client-as-datacenter: Any app that must work offline is implicitly doing multi-leader replication with extreme lag. "Each device is a 'datacenter,' and the network connection between them is extremely unreliable" (Ch. 5). Calendar sync, note-taking apps, and CouchDB all face every multi-leader conflict problem at maximum severity.
Logical (row-based) replication logs: Decouple replication from storage format by describing row-level changes (column values for inserts, PK for deletes). Enable backward compatibility across versions, different storage engines on replicas, and change data capture. Industry trend favors this over WAL shipping, which tightly couples leader and follower storage format and blocks zero-downtime rolling upgrades (Ch. 5).
Counterpoints
Assuming lag stays small: "Pretending that replication is synchronous when in fact it is asynchronous is a recipe for problems down the line" (Ch. 5). Stress-test by imagining lag of minutes or hours. If users would see broken behavior, the system needs an explicit guarantee, not a hope.
Trusting quorums for strong consistency: Six edge cases produce stale reads even with w + r > n: sloppy quorums, concurrent writes, concurrent read/write, failed partial writes, stale-backup restoration, and LWW clock skew. Quorum parameters adjust the probability of staleness, not eliminate it (Ch. 5).
Last Write Wins as safe default: LWW is "dangerously prone to data loss" because it silently discards acknowledged writes. Combined with unreliable clocks (Ch. 8), it can discard even causally later writes. The only safe LWW pattern is immutable, write-once keys (Ch. 5).
Sloppy quorums as consistency mechanism: "A sloppy quorum actually isn't a quorum at all in the traditional sense. It's only an assurance of durability" (Ch. 5). You get durability but stale reads until hinted handoff completes.
Key Quotes
"All of the difficulty in replication lies in handling changes to replicated data." -- Martin Kleppmann, Chapter 5
"Multi-leader replication is often considered dangerous territory that should be avoided if possible." -- Martin Kleppmann, Chapter 5
"If losing data is not acceptable, LWW is a poor choice for conflict resolution." -- Martin Kleppmann, Chapter 5
"If you want to avoid losing data, you -- the application developer -- need to know a lot about the internals of your database's conflict handling." -- Martin Kleppmann, Chapter 5
Rules of Thumb
- Default to single-leader replication unless you have a concrete availability or latency requirement demanding multi-leader/leaderless
- Use semi-synchronous replication to guarantee data on at least two nodes without blocking on all followers
- Pin user reads to one replica (e.g., hash user ID) to prevent monotonic-read violations
- Prefer logical (row-based) replication logs over WAL shipping for version independence and zero-downtime upgrades
- Treat conflict avoidance (routing all writes for a record through one leader) as the pragmatic default for multi-leader setups
- Use version vectors, not timestamps, for causal ordering in leaderless systems
- Concurrency is about knowledge gaps, not wall-clock overlap: "Two operations are concurrent if neither happens before the other" (Ch. 5)
- For deletions in version-vector merge, use tombstones -- naive union-merge causes removed items to reappear
Related References
- Partitioning - replication copies data across nodes; partitioning splits it -- orthogonal axes that compose
- Transactions - transactions remedy the consistency anomalies that replication lag introduces
- Distributed Systems Problems - clock unreliability makes LWW dangerous; consensus formalizes leader election