Clocks That Know Their Own Uncertainty: Google Spanner, TrueTime, and Mastering Time in the Cloud

The Hook: A Database That Claims the Impossible

For decades, distributed computing held an article of faith: a database can be globally distributed, strictly consistent, and highly available — but never all three at once. Replicate data across continents and you must accept either inconsistency or downtime. That, roughly, was the lesson distilled from the CAP theorem. And then, in October 2012, Google published a paper at the OSDI conference about a system called Spanner that seemed to shatter that expectation: a relational database, spread across the entire globe, with genuine ACID transactions, external consistency — and availability of more than five nines.

The most remarkable part was not the sheer scale but how Google solved the consistency problem. Not through a clever consensus algorithm alone, not through more communication between data centers — but through time itself. More precisely: through a clock that does something ordinary clocks cannot. It does not pretend to know the exact time. Instead, it returns an interval and guarantees: "The true time lies somewhere in here." This clock is called TrueTime, and its central invention is the honest disclosure of its own uncertainty.

Why should this interest you — beyond intellectual elegance? Because the problem Spanner solves lurks in every distributed architecture you deal with professionally. The moment two machines in different places must jointly decide on a truth — which transaction came first, which write wins, which state is current — you run into the very question Spanner works on: What does "simultaneous" even mean when there is no shared clock? The answer TrueTime gives is not just an engineering feat. It is a lesson in how to handle uncertainty: not by talking it away, but by measuring it and planning around it.

This article takes you the full distance: from the fundamental problem of time in distributed systems, through Lamport's logical clocks and the limits of classical time synchronization, to TrueTime's core trick — known uncertainty — the "commit wait" built on top of it, lock-free consistent snapshots, and finally the often-misunderstood question of whether Spanner "breaks" the CAP theorem or not. At the end stands a practical mental tool that reaches well beyond databases.

Part 1: The Problem – Why There Is No "Now" in the Cloud

The Illusion of a Shared Present

On a single computer, the ordering of events is unproblematic. There is one CPU, one clock, one instruction stream; whatever happened first simply happened first. But the instant two machines in different places are involved, that self-evidence falls apart. Machine A in Frankfurt and Machine B in Iowa each have their own quartz clock, and these clocks inevitably diverge — they drift. A typical server quartz clock can deviate from true time by several seconds per day. There is no physically privileged "shared present" that both machines could simply invoke.

Worse still: even if you could synchronize the clocks perfectly, physics imposes a hard limit. Information propagates at most at the speed of light. A message from Europe to North America realistically needs tens of milliseconds over fiber. In his CAP paper, Brewer does the arithmetic: a distance of about 1,000 miles is roughly five million feet, and at about half a foot per nanosecond in the medium, that yields a minimum travel time of about 10 milliseconds — for the bare distance alone. The question "What is the state of the system right now?" simply has no unique, immediately available answer across large distances. This fact is not fixable; it is anchored deep in the structure of space and time (an echo of relativity that also plays a role in Spooky Action at a Distance: Quantum Entanglement from Einstein to the Quantum Internet: there is no universal simultaneity).

Why "Just Use NTP" Is Not Enough

A practitioner's obvious answer is: "Just synchronize the clocks with NTP." The Network Time Protocol aligns machine clocks against time servers and typically keeps them within a few milliseconds, in good cases below that. For log files and cron jobs, that is entirely sufficient.

For a strictly consistent database it is not — for a subtle but decisive reason: NTP does not tell you how wrong it currently is. You get a number, a timestamp that pretends to be exact. In reality it may be off by a few milliseconds — but you do not know by how many. If two machines query their local, slightly diverging clocks and both assign a timestamp to a transaction, it can happen that a transaction which occurred later in the real world receives an earlier timestamp. Causal ordering reverses. For a banking system, an inventory, or an authorization database, that is a corruption risk: a read could see a state that does not include an already-committed change.

So the real poison is not the error itself — every clock has one — but the unknown error. A timestamp without an error estimate is like a measurement without error bars: it suggests a precision it does not have. It is precisely this deceptive false precision that TrueTime later eliminates. (Anyone thinking about error bars and their pitfalls will find a kindred story in The Cosmic Tension: Why the Universe Seems to Have Two Rates of Expansion — there too, the honest handling of uncertainty decides the entire interpretation.)

Part 2: Logical Time – Lamport's Ingenious Workaround and Its Limit

Ordering Events Without Trusting Clocks

Long before anyone thought of globally synchronized atomic clocks, Leslie Lamport solved the ordering problem in 1978 in a completely different, almost philosophical way. His famous paper Time, Clocks, and the Ordering of Events in a Distributed System does not ask "What time is it really?" but rather "Which events must have happened before which others?"

Lamport's key concept is the happened-before relation. It defines a partial ordering purely from the logic of causality: within a process, the order is clear; and if process A sends a message that process B receives, then the sending necessarily happened before the receiving. From these two rules Lamport builds "logical clocks" — simple counters that are incremented at every event and passed along with every message. The result is a consistent ordering of causally connected events, entirely without physical time. This idea is one of the foundations of all of distributed computing; vector clocks and similar constructs build on it.

Lamport's approach has a compelling property: it is always correct, because it relies on no clock at all. But it has an equally fundamental limit.

What Logical Clocks Cannot Do: the Outside World

Logical clocks order only events that are causally connected through a chain of messages. Events with no such chain between them remain incomparable — the ordering is partial, not total. Above all, logical clocks do not know the real time of the outside world. That becomes the problem the moment observers come into play who stand outside the system.

Imagine you book a flight online (transaction T1), confirm the booking, then call your partner, who books the adjacent seat on the same website (transaction T2). In real life, T1 demonstrably happened before T2 — you both know it, because a phone call lay between the two bookings, that is, a communication channel outside the database. A correct database must respect this external ordering: no one may ever see a state in which T2 took effect before T1. This strong property is called external consistency (closely related to linearizability): if T2 begins to commit in the real world after T1 has finished committing, then the timestamp of T2 must be greater than that of T1 — even if the two transactions run on opposite sides of the earth.

Logical clocks alone cannot guarantee this, because they do not see the externally running channel (the phone call). External consistency requires a reference to physical time. And with that we are back at the clock problem — except that now it must inescapably be solved. This is exactly where TrueTime comes in.

Part 3: TrueTime – the Clock That Knows Its Own Uncertainty

An Interval Instead of a Point

The central, almost defiantly simple idea of TrueTime: an honest clock returns not a point in time but a time interval. The core method TT.now() returns a pair of values — earliest and latest — and formally guarantees: the absolute real time at the moment of the call lies, with certainty, within this interval. In the notation of the Spanner paper, for tt = TT.now() it always holds that tt.earliest ≤ t_abs(e_now) ≤ tt.latest.

Half the width of this interval is the instantaneous error bound, usually denoted by the Greek letter ε (epsilon). Epsilon is thus the measure of admitted uncertainty. A perfect clock would have ε = 0; an honest real clock has a small but nonzero ε and — here is the trick — it knows this ε and discloses it. Two additional convenience methods follow immediately: TT.before(t) is true exactly when t is guaranteed to lie in the future (t < tt.earliest), and TT.after(t) is true exactly when t is guaranteed to lie in the past (t > tt.latest).

From this follows a plain, powerful rule: if two TrueTime intervals do not overlap, then the real ordering of the two calls is unambiguously known. If they overlap, you do not know — and must be careful. TrueTime thus makes the boundary of the knowable explicit instead of obscuring it.

How Hardware Becomes Trust: GPS and Atomic Clocks

To keep the interval narrow, Google operates time-master machines in every data center and a timeslave daemon on every ordinary machine. The masters draw their time from two deliberately different physical sources, because these have complementary failure modes:

The majority of masters have GPS receivers with their own antennas. Via the satellites' atomic clocks, GPS delivers very accurate time, but it is vulnerable to antenna and receiver failures, local radio interference, spoofing, and systematic faults (such as incorrect leap-second handling). A smaller group of masters — half-jokingly called "Armageddon masters" in the paper — is instead equipped with atomic clocks. These are surprisingly affordable (of the same order of magnitude as a GPS master) and fail in a way that is uncorrelated with GPS: they drift slowly due to frequency error, but are immune to GPS disruptions. By combining both sources, the system survives the failure of either world.

Every daemon regularly polls a variety of masters — nearby and distant GPS masters as well as some Armageddon masters — and applies a variant of Marzullo's algorithm to detect and reject "liars" (faulty or deviating masters) and to synchronize to the agreeing majority. Machines whose local clock deviates more than the specified worst-case bound are taken out of service.

The Magnitude: Epsilon as a Sawtooth

How large is ε in practice? Here the mechanism becomes tangible. Between two synchronizations, the daemon lets its uncertainty grow, because the local clock slowly drifts without a fresh alignment. Google conservatively assumes a worst-case drift rate of 200 microseconds per second. The poll interval — the gap between two synchronizations — was 30 seconds at the time of the paper. In this rhythm, ε follows a sawtooth curve: right after synchronization the uncertainty is small, then it grows linearly with the drift until the next synchronization pulls it back down. In Google's production environment at the time, ε thus oscillated between about 1 and 7 milliseconds, averaging roughly 4 milliseconds.

Consider what is happening here: instead of claiming the clock is exact, the system continuously measures how uncertain it currently is and writes that uncertainty into every timestamp. The question of whether a machine "really knows" the correct time or merely happens to be right is thereby shifted from luck to a dependable guarantee — a technical cousin of the epistemological question from Three Pages Against 2,000 Years: The Gettier Problem and What Knowledge Really Is: when is a belief genuine knowledge and not just a lucky hit?

Part 4: Commit Wait – "Wait Out the Uncertainty"

The Simplest Conceivable Trick, and Why It Works

Now comes the moment when the honest clock turns into real external consistency — and the solution is of almost cheeky simplicity. It is called commit wait.

When Spanner commits a write transaction, the coordinating leader assigns a commit timestamp s. It chooses s to be at least as large as TT.now().latest at the time of the commit — that is, as safely not too early. And then the system does something counterintuitive: it waits. More precisely, the leader waits until TT.after(s) is true — that is, until the entire TrueTime clock has guaranteed moved past s, until s therefore lies indisputably in the past. Only then are the locks released and the transaction made visible as committed.

The wait is on average about 2ε (you have to wait from now to latest and then out the uncertainty), so with ε ≈ 4 ms, a few milliseconds. Crucially: this wait runs in parallel with the internally required communication of the two-phase commit, so in practice it adds barely any latency. You "pay" for the uncertainty, but usually out of a budget you had to spend anyway.

Why This Guarantees the Ordering

The proof behind it is short and beautiful. Consider two transactions T1 and T2, where T2 only begins to commit in the real world after T1 has finished committing. Then a small chain of inequalities holds:

• By commit wait, the timestamp of T1 is guaranteed to be before the absolute real commit time of T1: s1 < t_abs(commit of T1).
• By assumption, T2 began only after that point: t_abs(commit of T1) < t_abs(start of T2).
• By causality and the choice of s2 = TT.now().latest at the start of T2, s2 is no smaller than the real start time of T2.

Putting it together, s1 < s2 follows necessarily. The timestamps thus respect the real, externally observable ordering — exactly what external consistency demands. The whole magic consists of not ignoring the uncertainty but waiting it out. Brewer phrases it as a general pattern: "waiting out the uncertainty" is a recurring, robust technique in distributed systems.

It is worth pausing to appreciate the beauty of this solution. The hardest problem of distributed databases — a global ordering without a global clock — is solved not by more communication but by patience, backed by a clock that is honest about its limits. Instead of asking "Do we agree on what time it is?", each node merely says: "I will wait until, even in the worst case, no confusion is possible."

Part 5: The Real Payoff – Consistent Snapshots Without Locks

As elegant as commit wait is, Brewer explicitly points out that the greatest practical value of TrueTime lies elsewhere: in consistent reads without locks.

Spanner is a multi-version system (Multi-Version Concurrency Control): it retains old versions of every datum along with their timestamps. Because every write carries a monotonically increasing, real-anchored timestamp, you can read the entire database at an exact point in the past — a clean snapshot. Such snapshot reads need no locks at all and do not block ongoing writes. Any sufficiently up-to-date replica node can answer them locally, which makes reads fast and cheap.

So that a replica knows whether it may already serve a read at time t, it maintains a so-called safe time t_safe: the highest timestamp up to which it is guaranteed to be up to date. It may answer a read at t exactly when t ≤ t_safe. This is a remarkably precise form of "knowing": the replica does not claim to know everything, but knows exactly how far its knowledge reaches.

The practical benefit is enormous. An analytics query over the whole database — say a large MapReduce run — can pick a single, precise timestamp and obtain reproducible, internally consistent results, exactly as if the world had been paused for a moment. In older systems such as Bigtable, the notion of time across data shards was "jagged," which made results unpredictable especially for the very recent past. This same consistency is also why clean backups and restores can be built from Spanner snapshots: a snapshot always lies between two transactions and never contains a half-applied transaction that would violate an invariant — and that holds even when you do not explicitly know the invariants.

TrueTime even allows snapshots to be taken consistently across multiple independent systems, as long as all use monotonically increasing TrueTime timestamps and agree on a snapshot point. And you can agree not only on the past but on the future: Spanner supports atomic schema changes by setting a future point in time at which all replicas switch to the new schema simultaneously. The whole idea of reconstructing state from an ordered sequence of timestamped events is at its core the same one behind The Logbook of Truth: Understanding Event Sourcing and CQRS — Spanner, in a sense, supplies the globally reliable clock that makes such a log unambiguous across continents in the first place.

Part 6: Does Spanner Break the CAP Theorem? A Common Misunderstanding

What CAP Really Says

No article about Spanner can avoid the question that comes up in every other discussion: "Doesn't Spanner defeat the CAP theorem?" The short answer is: no — and the longer one is more instructive than the legend.

The CAP theorem goes back to a conjecture by Eric Brewer (around 2000), which Seth Gilbert and Nancy Lynch proved formally in 2002. Simplified, it states: of the three properties consistency (C, here thought of as serializability), availability (A, 100% — every request receives a response), and partition tolerance (P, the system keeps working despite broken network connections), you cannot have all three at once without restriction. Over wide-area networks, partitions are considered inevitable; once you accept that, you must choose between C and A during a partition. Important — and often overlooked — is Brewer's nuance: you have to give it up only during an actual partition, and the actual theorem speaks of 100% availability, while the interesting practice is about realistically high availability.

Spanner's Honest Answer: "Effectively CA," Technically CP

Brewer himself devoted a paper to this in 2017 — Spanner, TrueTime and the CAP Theorem — and there cleared up the legend. His position is refreshingly sober: Spanner is technically a CP system. When a partition occurs, Spanner chooses consistency and, in case of doubt, sacrifices availability. This is due to concrete design decisions: updates run over Paxos groups that need a majority (quorum); if a leader loses the quorum, writes stall. And the two-phase commit for cross-group transactions can likewise block if the participants are separated.

And yet users may treat Spanner in practice like a CA system — Brewer calls this "effectively CA." The reason is not TrueTime but something far more down-to-earth: Google's own private wide-area network. Every Spanner packet flows exclusively over Google-controlled routers and links; every data center hangs on the global network via at least three independent fiber paths. This makes real partitions so rare that they practically vanish as a cause of unavailability. In Brewer's analysis of internal incidents, fewer than 8% fell into the "network" category — and in not a single case was a large cluster set cut off from another, nor did a Spanner quorum ever end up on the minority side of a partition. Spanner thereby achieves more than five nines of availability internally. But if a real partition does strike, Spanner clings uncompromisingly to consistency.

Here lies the truly important insight, and it is often told backwards: TrueTime does not "circumvent" CAP. Brewer emphasizes explicitly that TrueTime contributes little to availability. TrueTime's contribution is consistency — external consistency, lock-free consistent reads, and consistent snapshots. Availability comes from the network and from years of operational hardening. The merit of known uncertainty, then, is not to break a theorem but to deliver a stronger consistency guarantee without paying the usual latency and availability penalty for it.

A Map of the Approaches

To make the classification easier, here is an overview of the strategies by which distributed systems handle time and ordering:

The TrueTime row is the actual point: it buys consistency not through more talking but through a better clock and a short, planned wait.

The Central Takeaway

The central, transferable insight behind Spanner and TrueTime is not "Google built an expensive clock." It is far more general and practical:

Uncertainty does not vanish when you ignore it — but it becomes manageable as soon as you measure it and plan it into your approach. An ordinary clock lies through false precision: it states a number and conceals its error. TrueTime's breakthrough is to ship the error openly alongside — and on that honesty you can build a hard guarantee, by simply waiting, in case of doubt, until no confusion is possible anymore.

Transfer this to your own work. Everywhere you rely on a value — a cache entry, a replicated status, the result of a distributed job, an external API timestamp — the TrueTime question pays off: How large is my ε here, really? And do I plan for that uncertainty, or do I pretend it does not exist? A system that knows its own ignorance and treats it conservatively is more robust than one that blindly trusts apparent precision. That holds for databases as much as for estimates in a project plan, for monitoring thresholds, or for the question of whether two events in your logs really happened in the order their timestamps suggest. "Wait out the uncertainty" is more than a database trick — it is a stance toward incomplete knowledge.

Reflection Question

Think of a place in your architecture where you rely on timestamps or on "the current state" of a remote system — a synchronization, a distributed lock, an ordering assumption between two services. Now imagine each of these clocks returned to you not a point in time but an interval with its honest uncertainty. How large would that interval really be — microseconds, milliseconds, seconds? And which of your current assumptions about "before" and "after" would shatter if the intervals overlapped? What would you have to measure or wait out to turn a risky assumption into a dependable guarantee?

Cross-References in the Vault

• The Logbook of Truth: Understanding Event Sourcing and CQRS – Both topics revolve around state as an ordered sequence of timestamped events; TrueTime provides the globally unambiguous ordering that makes such a log reliable across continents in the first place.
• Spooky Action at a Distance: Quantum Entanglement from Einstein to the Quantum Internet – The absence of universal simultaneity and the speed of light as a hard limit on information connect relativity and distributed systems directly.
• The Cosmic Tension: Why the Universe Seems to Have Two Rates of Expansion – A prime example that the honest treatment of error bounds (ε) decides the entire interpretation — in cosmology as in databases.
• Three Pages Against 2,000 Years: The Gettier Problem and What Knowledge Really Is – When does a replica "know" the time, and when is it merely right by luck? The safe-time guarantee is the technical answer to a deeply epistemological question.

Sources

• James C. Corbett et al. (2012): Spanner: Google's Globally-Distributed Database. Proceedings of OSDI '12. Original PDF: https://research.google.com/archive/spanner-osdi2012.pdf
• Eric A. Brewer (2017): Spanner, TrueTime and the CAP Theorem. Google Research. PDF: https://research.google.com/pubs/archive/45855.pdf
• Google Cloud: Spanner – TrueTime and external consistency (official documentation): https://docs.cloud.google.com/spanner/docs/true-time-external-consistency
• Leslie Lamport (1978): Time, Clocks, and the Ordering of Events in a Distributed System. Communications of the ACM 21(7), pp. 558–565: https://lamport.azurewebsites.net/pubs/time-clocks.pdf
• Seth Gilbert & Nancy Lynch (2002): Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. ACM SIGACT News 33(2): https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf
• Barbara Liskov (1991): Practical Uses of Synchronized Clocks in Distributed Systems. ACM PODC. Summary via the Spanner / Brewer references (see above).

← All articles