The Backdoor at the Heart of Linux: The XZ Attack and the Anatomy of a Supply-Chain Compromise

The Hook: The 500 Milliseconds That Saved the World

On 28 March 2024, a software engineer named Andres Freund sent an unremarkable email to the security mailing list of the Openwall project. Freund worked on PostgreSQL at Microsoft and spent his spare time on performance optimization — that peculiar discipline in which people get worked up over milliseconds no ordinary human would ever notice. This very quirk would turn out to be one of the luckiest accidents in the history of IT security.

Freund had noticed that SSH logins on one of his Debian test systems were taking about 500 milliseconds instead of the usual 100. Half a second. For most people the blink of an eye; for Freund an anomaly that would not let him rest. He also noticed that the sshd process was consuming unusually much CPU time, and that Valgrind — a tool for error analysis — was emitting strange messages pointing to the library liblzma. liblzma is part of XZ Utils, a compression tool installed on virtually every Linux system in the world. What did a compression library have to do with the speed of SSH logins?

Freund dug deeper. And what he found was not a bug, a memory leak, or a careless section of code. It was a carefully engineered, masterfully camouflaged backdoor — one that would have allowed an attacker with the right cryptographic key to execute arbitrary code on the target system without any login whatsoever. And this backdoor was already circulating in the pre-release versions of the major Linux distributions, on a direct path into the stable releases that, a few weeks later, would have been rolled out to millions of servers worldwide.

For anyone working in IT security, this is one of the most instructive stories of recent years. Because the XZ backdoor, cataloged as CVE-2024-3094 with the maximum CVSS score of 10.0, was no ordinary attack. It was the product of a campaign patiently conducted over nearly three years, in which an attacker did not hack into a system but earned the trust of an open-source community. It targeted not a technical weakness but the human and social infrastructure on which all modern software rests. And it very nearly succeeded — stopped not by a security product, not by an audit, not by a government agency, but by a single curious engineer annoyed by half a second.

This article takes you along the full route: from the question of why an obscure compression library holds so much power, through the anatomy of the years-long social-engineering campaign, to the precise technical workings of the backdoor and the fundamental lessons this near-catastrophe holds for how we think about software supply chains.

Part 1: How a Compression Library Can Rule the World

The Invisible Foundation

To grasp the force of this attack, one must first understand a basic principle of modern software that almost no one outside the field knows: practically no software today is written from scratch. Every program stands on the shoulders of hundreds, often thousands, of external libraries — reusable building blocks that other people wrote, published, and maintain. A modern server is less a single edifice than a termite mound of tens of thousands of nested dependencies.

XZ Utils is a particularly deep-lying block in this mound. It implements the LZMA compression algorithm and provides both the command-line tool xz and the software library liblzma. Compression is one of those quiet base functions needed everywhere: in package management, in handling compressed files, in kernels, in bootloaders. liblzma is so fundamental that it is present on nearly every Linux installation in the world and is loaded in the background by countless other programs.

And here a chain of circumstances comes into play that made the attack possible in the first place — and so dangerous. sshd, the OpenSSH server daemon that provides encrypted remote access on almost every Linux server and is thus the crown jewel of any system's defenses, does not actually load liblzma directly. OpenSSH itself has nothing to do with LZMA compression. But several major distributions — among them Debian and the Red Hat family — apply their own patch to sshd that links it to libsystemd, so the service can cleanly register and deregister with the init system systemd (specifically to notify it of readiness). And libsystemd, in turn, pulls in liblzma so it can process compressed log data.

The result of this chain is astonishing: by way of the detour sshd → libsystemd → liblzma, the code of an obscure compression library lands in the address space of a server's most privileged, internet-facing service. Whoever controls liblzma potentially controls sshd. The attacker understood this chain precisely and exploited it deliberately.

The One-Person Problem

There is a second circumstance without which the story would be inconceivable, and it is human rather than technical in nature. XZ Utils was, over many years, essentially developed and maintained by a single person: the Finnish programmer Lasse Collin, known online as Larhzu. A tool running on billions of devices hung on the volunteer commitment of a single human being who cared for it in his spare time, without pay.

This constellation is frighteningly common in the world of open-source software. The cartoonist Randall Munroe captured it in a famous xkcd comic: a huge, rickety structure of blocks whose entire weight rests on a single thin rod at the very bottom — labeled "a project some random person in Nebraska has been thanklessly maintaining since 2003." The entire digital infrastructure of the modern world is riddled with such rods. XZ Utils was one of them.

And such a rod has a vulnerable property: the person carrying it grows tired. Lasse Collin had suffered from psychological strain and exhaustion over the years, which he himself publicly hinted at. A single volunteer bearing the weight of a globally used project alone, overburdened and longing for relief — that is not a technical weakness, but it is a social attack surface. And it was precisely this attack surface the attacker targeted with surgical precision.

Part 2: The Long Preparation — Anatomy of a Social-Engineering Campaign

The Entrance of "Jia Tan"

On 29 October 2021, a new contributor named Jia Tan (username JiaT75) appeared on the development platform and submitted a first contribution to XZ Utils — a harmless patch for a configuration file. The first independently authored change followed on 21 January 2022. In the months that followed, Jia Tan increasingly took part in the project: code reviews, translations, maintenance of the continuous-integration infrastructure, minor improvements. An impeccable, helpful, competent contributor — exactly the kind of participant an overburdened maintainer dreams of.

I am of the opinion that the cold patience of this approach is the truly frightening thing about the whole story. Whoever was behind "Jia Tan" was not playing for a quick coup but investing years in the patient construction of a reputation. It was an attack not on code but on trust — and trust cannot be hacked; it can only be earned deceptively.

The Pincer Attack of the Sock Puppets

The actual maneuver began in the spring and summer of 2022, and it reveals a remarkable psychological sophistication. On the project's mailing list, several new voices suddenly appeared — users with names like Jigar Kumar and Dennis Ens — who exerted massive pressure on Lasse Collin. They complained about sluggish development, about releases being too rare, about responses to requests being too slow. "Patches spend years on this mailing list," wrote one. "Is the maintainer still around? Or is the project dead?" needled another. The tone was demanding, occasionally contemptuous, always aimed at portraying the maintainer as overwhelmed and the project as neglected.

These accounts had one thing in common: minimal activity elsewhere, suspicious creation dates, all established only after 2021. Later analysis suggests they were sock puppets — fake identities of the same actor or group. Their function was that of a pincer attack: while they pressured Collin and poisoned the atmosphere, Jia Tan positioned himself as the calm, diligent, reliable savior — the one who was already helping and to whom one could, surely, finally hand responsibility to placate the others.

The calculation paid off. An exhausted, embattled volunteer maintainer, presented with a seemingly flawless helper, hands over responsibility. Over the course of 2022, Collin gradually granted Jia Tan more rights. By 2023 Jia Tan effectively held the role of a co-maintainer, with the authority to commit changes and prepare releases. The attacker had not stolen the keys to the project — they had been handed to him.

Preparing the Ground

Before the actual payload was planted, Jia Tan spent months inconspicuously preparing the ground. In June 2023 he introduced so-called IFUNC resolvers into the code (with names like crc32_resolve and crc64_resolve), disguised as a legitimate performance optimization. IFUNC (indirect function) is a mechanism of the GNU toolchain that lets a program decide at runtime, very early during process startup, which of several variants of a function it will use — for instance depending on the capabilities of the specific CPU. In itself this is a perfectly common and sensible technique. But here it became a tool: because IFUNC resolvers run very early in initialization, they offered the ideal, inconspicuous foothold for later manipulating foreign functions at runtime.

At the same time, Jia Tan created further preconditions: he arranged for a certain compiler safety feature (which would have surfaced errors early) to be silently disabled in the build, and he got the project to switch off, in oss-fuzz — an automated bug-hunting service — a check that could have exposed the later malicious code. Each of these steps was harmless and plausibly justified on its own. Only in combination did they set the stage for the final act.

Part 3: The Backdoor — How the Attack Worked Technically

The Art of Hiding Outside of Git

Now comes one of the most elegant and, at the same time, most insidious aspects of the entire attack, and it rewards close attention because it lays bare an important truth about software supply chains. The malicious code was not to be found in the publicly viewable source-code repository. Anyone who looked at the Git repository of XZ Utils saw clean, harmless code.

The trick lies in a distinction that many users are not even aware of: between the source code in the version-control system (Git) and the so-called release tarball — that downloaded, packaged archive from which the distributions actually build the software. For historical and practical reasons, the two are not identical. The tarball is generated when a release is created and contains additional generated files, in particular the scripts of the build system (such as the Autotools script configure and helper files like build-to-host.m4). It was precisely into these generated, virtually never human-inspected build files that the attacker smuggled his trigger — into the tarball, but not into Git.

That is the crux: humans review the source code. Machines build the tarball. When the two differ, a gap opens between them in which an attacker can hide. Jia Tan, as maintainer, was the one who produced and signed the official tarballs — and could, at this step, insert the manipulated version that no one compared against the clean Git state.

The Camouflaged Payload in the Test Files

But where does one hide the actual malicious code without it standing out? The answer is downright brazen: in the test files. A compression tool must, of course, also be tested with malformed, corrupted, and unusual input data. The XZ project therefore contained a collection of deliberately "broken" compressed test files with names like bad-3-corrupt_lzma2.xz and good-large_compressed.lzma. What could be more natural than to fill a file that by definition is supposed to look like random, corrupted binary junk with actually encrypted malicious code? To a human eye — and to most automated checks — the two are indistinguishable.

The attack then unfolded in two stages:

Stage 1 — during configuration. When building the software, the system first runs the configure script. The smuggled-in code in the build system (build-to-host.m4) recognized, based on certain conditions, whether this was a suitable target system (Linux, x86-64, a build within a distribution's package build, using gcc and the GNU toolchain), and in that case extracted a hidden script from one of the doctored test files. If the conditions were not met — for instance on an ordinary developer's machine without the full package-build context — nothing happened, and the backdoor stayed dormant. This targeting was another camouflage mechanism: the attack activated only where it was needed and stayed hidden where curious eyes might have discovered it.

Stage 2 — during compilation. The unpacked script then decrypted the actual payload from a further test file, using several layers of obfuscation (among them an RC4-like stream cipher and various text transformations with tools like head and tr), and injected the malicious object code into the emerging liblzma library. At the end of this multi-stage, nested process stood a finished, seemingly regular compression library that secretly carried a backdoor.

The Grab for the Heart: RSA_public_decrypt

So what did this backdoor do once it had arrived in the address space of sshd? Here the circle closes with the IFUNC resolvers introduced earlier. The malicious code used the function-resolution mechanism to covertly redirect a pointer at runtime: it replaced the OpenSSH function RSA_public_decrypt with its own, malicious variant. This function is called during SSH public-key authentication — so it sits at exactly the point where a client proves its identity.

The manipulated function checked, at every login, whether the data presented by the client contained a certain structure signed with a secret Ed448 key. Ed448 is a method of elliptic-curve cryptography. The clincher: the backdoor contained only the attacker's public key. Only someone in possession of the corresponding private key — the attacker himself — could generate valid commands that the backdoor would accept. Even if a security researcher discovered the backdoor and fully understood its mechanism, he could not abuse it himself; the cryptography secured its exclusive use by its creator.

If the signature was valid, the function extracted a command from the transmitted structure and passed it directly to the system function system() for execution — and did so before actual authentication. The result is the most severe class of vulnerability there is: a pre-authentication remote code execution. An attacker could have gained full control over the server with a single correctly signed connection attempt, without any valid password and without any legitimate key. And because the access ran over the regular SSH port and the regular RSA processing, it would have left barely a trace in the logs.

Let us summarize the chain in its full length:

Part 4: The Discovery — and How Close It Was

An Accident Foretold

Now back to Andres Freund and his 500 milliseconds. It is worth appreciating how fragile the chain of coincidences was that caused the attack to fail. Freund was not looking for a backdoor. He was not doing security research. He had merely noticed, while running benchmarks, an unusual CPU load and a perceptible delay in SSH logins — a side effect of the additional, covertly running code of the backdoor. Because he possessed the rare combination of curiosity, deep system knowledge, and the stubbornness to chase a seemingly trivial anomaly to its root, he pulled the thread further: Valgrind messages about liblzma, a look at the recent changes, tracing the build process — and finally the realization that something here was fundamentally, deliberately, and maliciously rotten.

On 28 March 2024 he reported his finding. His email began with the remarkably sober words that, after observing "some strange symptoms," he had investigated the origin and concluded that the upstream XZ repository and the tarballs had been compromised. Within hours the security community grasped the magnitude. The U.S. agency CISA issued a warning, distributions pulled the affected versions, and CVE-2024-3094 received the maximum severity of 10.0.

How Near the Abyss

The truly disturbing thing about this story is not that it happened, but how narrowly it turned out well. The affected versions 5.6.0 (late February 2024) and 5.6.1 (March 2024) had not yet reached the stable branches of the major distributions. They sat in the testing and development branches: in Debian unstable/sid and the experimental branches, in Fedora Rawhide and the betas of Fedora 40 and 41, in openSUSE Tumbleweed, in Kali Linux, in Arch Linux (there, however, without the decisive systemd detour to sshd). In the weeks before the discovery, Jia Tan and the sock puppets had actively pressured the distributions to adopt the new version quickly into the stable releases.

Had Freund's curiosity struck only a few weeks later — or not at all — then the backdoor would have been rolled out with the next stable releases onto millions of production servers: in cloud data centers, in corporate infrastructures, in critical systems. A single actor would then have held, through a cryptographically secured master key, access to a vast portion of the world's Linux server landscape. The field was quick to agree: this was one of the most ambitious and best-executed supply-chain attacks ever publicly known — and it failed on half a second and one human being who could not look away.

The Response and the Open Questions

Lasse Collin, the true maintainer, suddenly found himself at the center of a global security incident. He revoked Jia Tan's access, worked through the incident, and published a detailed account of the malicious changes. On 29 May 2024, version 5.6.2 appeared, removing the backdoor.

Many questions remain open to this day. Who "Jia Tan" really was is unknown. The evidence — the patience spanning years, the considerable resources, the operational discipline, the working-hours pattern that points more to an Eastern European or Western Asian time zone than to the claimed Far Eastern identity, the deliberate obfuscation of metadata — points, by broad expert assessment, to a state-sponsored actor (an APT). I am of the opinion that a robust, publicly proven attribution to a specific state is so far lacking and that caution is warranted here; the burden of proof for concrete accusations is high, and circumstantial evidence is not certainty. Only one thing is sure: this was not the weekend work of a bored lone individual but a strategic, long-planned operation.

Part 5: The Lessons — What We Must Learn from the Near-Disaster

The XZ incident is so valuable precisely because it makes an entire bundle of structural problems visible as if under a magnifying glass. For everyone who deals with software security, it is worth drawing the lessons carefully.

The first and most fundamental lesson concerns the human dimension of the supply chain. We secure our systems against technical attacks — firewalls, encryption, intrusion detection — but the software supply chain ultimately rests on trust relationships between people. The attacker circumvented every technical defense by making himself a trusted insider. No firewall ruleset in the world would have stopped this attack, because the attack came from within, with legitimate privileges. Trust is the real attack surface, and trust cannot be patched.

The second lesson is the fragility of the maintenance structure. XZ hung on a single, overburdened volunteer. This constellation — critical infrastructure carried by underfunded, overworked individuals — is the rule in the open-source ecosystem, not the exception. The sock puppets' pressure campaign worked precisely because it exploited a real distress: that of a person who needed help. The sober consequence is that the maintenance of critical base software must not be a hobby left to the chance of volunteerism, but a societal task that deserves reliable resources.

The third lesson is technical in nature: the gap between what humans review and what machines build. The backdoor survived because it hid in exactly that gap — in the generated build artifacts of the tarball, which no one compared against the clean Git state. The consequence, widely discussed and partly implemented since, is called reproducible builds: procedures by which anyone can independently produce, bit-for-bit, the very same finished artifact from the viewable source code. If the officially delivered binary deviates by even a single bit, it stands out. What cannot be deterministically reproduced from the source code is ultimately not truly verifiable.

The fourth lesson concerns the depth of dependencies. Hardly any security officer ever had liblzma on the list of their critical components — and yet, two links removed, it sat at the heart of sshd. You can only protect what you know. Tools like a software bill of materials (SBOM), which discloses the complete dependency tree of an application, are not bureaucratic ends in themselves but the precondition for knowing at all what your own system actually stands on. The XZ incident gave enormous momentum to the already growing movement toward SBOMs and supply-chain security (for instance within frameworks such as SLSA).

And finally the fifth, most uncomfortable lesson, which at the same time holds a timid consolation. The attack was stopped not by a multimillion-dollar security product, not by an AI, not by a government agency, but by a single attentive human who did not ignore an anomaly. That is both encouraging and alarming. Encouraging, because it shows that careful, curious, competent people are the last and sometimes only line of defense. Alarming, because a defense that rests on the lucky accident of a single attentive engineer is no robust defense at all. How many such campaigns may have run — and be running — with more operational discipline, without the telltale 500 milliseconds, never to be discovered? That question should keep anyone who thinks about security wide awake.

The Central Takeaway

Above all, the XZ backdoor teaches one thing: the most dangerous attacks target not your technology but your trust. An attacker patient enough will not breach your firewall — he will become a valued, reliable member of your community, your team, your supply chain, and you will open the door for him yourself.

For practical work, this means a concrete shift of perspective. Ask not only which software you deploy, but who actually maintains it and how resilient that maintenance is. A critical tool hanging on a single exhausted person is a risk — regardless of how clean the code looks. For one central element of your own stack, take the time to actually examine the chain of dependencies and the people behind it. Keep a software bill of materials for your most important systems. And cultivate the attitude of Andres Freund: do not ignore the 500 milliseconds. The small, inexplicable anomaly that seems to bother no one is sometimes the only visible end of a very long, very carefully concealed thread.

Reflection Question

If the defense against the most ambitious publicly known supply-chain attack to date ultimately hinged on chance — on a single engineer who could not ignore half a second — what does that say about the robustness of the digital infrastructure on which our entire civilization rests? And what responsibility follows from it for you, who are part of this supply chain — as a consumer, as a contributor, perhaps yourself as that lonely rod at the very bottom of the structure?

Cross-References in the Vault

• Harvest Now, Decrypt Later: Post-Quantum Cryptography and the Race Against the Quantum Computer — on the cryptographic methods (such as elliptic-curve signatures) on which the XZ backdoor, too, built its exclusive usability.
• When the Processor Guesses Too Much: Spectre, Meltdown, and the Sin of Speculative Execution — another case in which a deep-lying, taken-for-granted layer of the system became an attack surface.
• NIS2 and What It Really Means for Mid-Market IT Consulting Firms in Germany — on the regulatory response to supply-chain risks and the duty to secure one's own supply chain.
• When IT Is Not Allowed to Fail: DORA and Digital Operational Resilience in Finance — on third-party and concentration risk as the regulatorily addressed core of modern resilience.
• How Machines Come to Agree: Distributed Consensus from FLP to Paxos to Raft — on trust between components in distributed systems, a related facet of the same fundamental problem.

Sources

• Wikipedia: XZ Utils backdoor — https://en.wikipedia.org/wiki/XZ_Utils_backdoor
• Russ Cox (research!rsc): Timeline of the xz open source attack — https://research.swtch.com/xz-timeline
• Sam James (thesamesam), GitHub Gist: xz-utils backdoor situation (CVE-2024-3094) — https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27
• Akamai Security Research: XZ Utils Backdoor — Everything You Need to Know — https://www.akamai.com/blog/security-research/critical-linux-backdoor-xz-utils-discovered-what-to-know
• Datadog Security Labs: The XZ Utils backdoor (CVE-2024-3094) — https://securitylabs.datadoghq.com/articles/xz-backdoor-cve-2024-3094/
• LWN.net: How the XZ backdoor works — https://lwn.net/Articles/967192/
• NIST National Vulnerability Database: CVE-2024-3094 Detail — https://nvd.nist.gov/vuln/detail/cve-2024-3094

← All articles