Concepts

Agreeing under failure

Consistency

CAP Theorem

What a distributed system does when the network splits in two — the only moment CAP actually applies.

Partitions are not optional, so the real choice is which promise to break when one happens.

partitionsCPAPavailability

Try it

Move the dials — the sentence under the picture changes.
When partitioned, prefer
Replica Av1clients hereReplica Bv1clients here tooreplicationRecent operationsnothing yet — try writing to A, then reading from B
With the network intact there is no choice to make — both replicas agree and both answer. CAP only bites during a partition, which is why “pick two of three” is a misleading summary: partition tolerance is not optional, so the real choice is what to do when one happens. Cut the network to see it.

What it actually says

Given a network partition — the network splits and the two halves cannot talk — a distributed system must choose between remaining consistent and remaining available. That is the whole theorem, and it is much narrower than the way it is usually quoted.

"Pick two of three" is a misleading summary. Partition tolerance is not a property you choose — networks fail, and if your system has more than one machine it will eventually be split. So there is no choice among three options. There is a single question: when the split happens, do you refuse to answer, or do you answer without being sure?

The widget makes this concrete. With the link intact, both replicas agree and both serve, and there is no decision to make at all. Cut the network and the two modes finally behave differently.

CP — refuse rather than be wrong

A CP system requires a quorum (a majority of the replicas) before answering. On the minority side of a partition, reads and writes fail. Nothing is ever stale and nothing diverges. The users on that side are looking at an error page.

This is the right trade whenever a wrong answer costs more than no answer: account balances, inventory you will actually ship, configuration that controls how other systems behave. etcd, ZooKeeper and Spanner sit here.

AP — answer with what you have

An AP system keeps serving on both sides. Reads may be stale, and writes accepted on both sides produce two versions of the truth that must be reconciled once the partition heals.

This is the right trade whenever unavailability is the expensive failure: shopping carts, timelines, presence, session data, metrics. Dynamo-style stores, Cassandra at low quorums, and essentially every cache sit here.

Reconciliation is the real work

Choosing AP is choosing to write a merge rule.

  • Last-write-wins by timestamp is simple, and it silently discards real writes whenever clocks disagree.
  • Vector clocks (a version stamp per replica) correctly distinguish concurrent updates from ordered ones and hand both versions to the application — which now has to know what to do with them.
  • CRDTs merge deterministically by construction. Excellent when your value fits one; not general.

Beyond CAP: PACELC

CAP only describes behaviour during a partition, which is rare. PACELC adds the part you live with every day: else, when the system is healthy, you are still trading latency against consistency. A quorum read is slower than a local one whether or not anything is broken. In practice this second trade shapes far more of your system than the first one ever will.

Take this with you

  • The one idea: partitions are not optional. The only choice is what to do during one: refuse, or answer without being sure.
  • In an interview, do not say "pick two". Say which data is CP and which is AP in your design, and why — and mention the everyday latency trade (PACELC).
  • At work, most of your data is AP already (caches, sessions, feeds). Make sure the few things that must be CP — money, inventory, config — actually are.