post-mortem · nexus_cluster · Python, Raft · 3 pages
A Cluster That Could Not Keep a Leader
i copied the election timeout from the raft paper along with its assumption that the network is the slow part. on one machine there is no network, and gc pauses were wider than the timeout.
abstract
A cluster that could not hold a leader, with no network, no partition and no packet loss, because every node was on one machine. The election timeouts I took from the Raft paper were shorter than the pauses my own runtime could produce, so the cluster spent its time deposing leaders that were merely busy.
the symptom
Leadership churned. A leader would be elected correctly and replaced seconds later with nothing having failed. Term numbers climbed steadily and throughput was poor because the cluster was always mid-election. Every node was following the protocol exactly; it was correctly responding to a false signal.
root cause
The leader was not failing, it was pausing, and the timeout was smaller than the pause. Serializing a batch of log entries runs under the interpreter lock, and garbage collection stops the process, and a Raft node under write load produces a lot of short-lived garbage. With every node sharing one CPU, those pauses correlate across the cluster, which is the worst case for a failure detector.
the fix
Measure the delivery gap distribution, then set the timeout above the observed tail rather than the average, keeping the randomization. Shrink the work done inline on the heartbeat path. The general form: a timeout constant carries assumptions about its environment, and copying the number without copying the environment imports a belief you have not checked.