Betting on a Stranger's Code: Firecracker microVMs and the End of the Container-versus-VM Dilemma

The Hook: Millions of Strangers on One Machine

Every time a serverless function fires somewhere in the world – a tiny snippet of code that resizes an uploaded image, records an order, or answers a chat message – something happens in the data center that, on closer inspection, is alarmingly bold. The operator takes foreign code it knows nothing about, whose intentions are opaque, and runs it on the very same physical machine on which, in the neighboring slot, the code of another equally unknown customer runs. Two potential attackers, one shared piece of silicon. And the operator promises both of them – and a regulator that will levy million-euro fines if things go wrong – that neither can spy on, disturb, or take over the other.

This is not a fringe phenomenon; it is the economic foundation of the entire serverless cloud. Its decisive trick is called multitenancy: a server is not rented to a single customer but shared simultaneously among hundreds or thousands, each paying only for the milliseconds in which their code actually computes. It is precisely this bundling of many small, sporadic loads onto little hardware that makes serverless services cheap. But it works only if the walls between tenants are absolutely tight – tight against actively malicious code, not merely against accidental misbehavior.

For a long time it seemed one had to choose between two evils. On one side stood Linux containers (Docker, LXC): feather-light, started in milliseconds, packable at extreme density – but with an unsettling property. All containers on a host share the same operating-system kernel. A single bug in that kernel, exploited by a single malicious container, can tear down the partition and hand over control of the whole machine. On the other side stood classical virtual machines: each with its own kernel, cleanly separated by hardware virtualization, a proven bulwark – but heavy, slow to start, and carrying an overhead that destroys density and with it the entire economic model of the serverless cloud.

The received wisdom was: you can have strong security or low overhead, not both. For a public cloud provider that accepts arbitrary foreign code, this tradeoff is unacceptable. Both properties are non-negotiable. This is exactly where Firecracker enters – an open-source Virtual Machine Monitor from Amazon Web Services, presented at the USENIX NSDI conference in 2020. Firecracker's thesis is as simple as it is consequential: the dilemma is false. You can have the hard hardware isolation of a real virtual machine and the lightness of a container – if you are willing to throw away everything that makes a virtual machine heavy and keep only what makes it secure.

For someone like Sven, who stands with one foot in cloud architecture and one in IT security, Firecracker is a textbook case of how an uncompromising re-examination of the trust boundary can shift a seemingly iron tradeoff curve. This article takes you the full distance: from the question of what isolation even means, through Firecracker's radical minimalism and its hard numbers, to the statistical bet of oversubscription, the snapshot revolution against the cold-start problem, and the broader ecosystem of competing isolation techniques.

Part 1: The Dilemma – Container versus Virtual Machine

To understand why Firecracker had to be built at all, one must cleanly dissect the problem AWS had with its own service. When AWS Lambda launched in 2014, it chose Linux containers as its isolation mechanism. Containers are not a single feature but a bundle of kernel mechanisms: cgroups cap resource consumption (CPU, memory), namespaces hide parts of the system from one another (private process IDs, private filesystem, private network), and seccomp-bpf can restrict which system calls a process is even allowed to make to the kernel.

These tools are powerful and efficient. Their problem is structural: the container process still calls directly into the shared host kernel. The Linux kernel is a vast piece of software with an enormous attack surface – hundreds of system calls, filesystems, network stacks, drivers. Every exploitable bug in this shared kernel is a potential escape hatch: if malicious code inside a container manages to compromise the kernel, all isolation collapses, because the kernel sits above every container. The security community sums it up in a phrase: your container is not a sandbox. Containers were originally conceived to separate cooperating workloads of the same operator from one another, not to imprison hostile code from foreign customers.

The virtual machine solves precisely this problem – at the cost of weight. With hardware virtualization via KVM (the Kernel-based Virtual Machine inside the Linux kernel), each guest gets its own virtual hardware, its own page tables, and above all its own operating-system kernel. The processor itself, with special virtualization instructions, enforces the separation between host and guest. The guest kernel is treated as untrusted; even if malicious code fully takes it over, it is still trapped inside its virtual machine. The attack surface facing the customer shrinks from "the entire Linux kernel" to "the narrow interface the Virtual Machine Monitor offers the guest."

The catch was traditionally the software that provides this virtual hardware: the Virtual Machine Monitor (VMM), almost always QEMU. QEMU is a marvel of flexibility – it can boot arbitrary operating systems, emulate countless devices, mimic foreign processor architectures. That universality has a price: at the time of the Firecracker work, QEMU comprised more than 1.4 million lines of code. This means slow startup (seconds pass before the first useful computation), noticeable memory and CPU overhead per virtual machine, and – ironically – a substantial attack surface within the very program meant to guarantee security. For a service that wants to pack thousands of tiny, constantly changing functions onto each machine, this is untenable.

The core question, then, was not "container or virtual machine?" but: Can we keep the hard isolation boundary of virtualization while shedding the weight that stems solely from QEMU's universality and is not needed at all for the serverless use case?

Part 2: What Isolation Really Means

Before we come to Firecracker's answer, a brief but precise look at three concepts that decide everything in this debate: the trust boundary, the trusted computing base, and the side channel.

The trust boundary is the imagined line along which a system decides: from here on, I no longer trust this code. In a container, this line runs at the edge of the process – but everything beyond it shares the same kernel, so a kernel breach leaps across the boundary. In a virtual machine, the line runs around the entire guest, enforced by hardware. The difference is not one of degree but of kind: in the first case an attacker must find a software flaw, in the second a flaw in the separation enforced by the CPU itself.

The trusted computing base (TCB) is the totality of all hardware and software one must trust for the security promise to hold. The smaller the TCB, the less code can harbor a fatal flaw. Here lies Firecracker's central design philosophy: if the VMM is itself part of the TCB – and it is, since it provides the virtual hardware and separates the guests – then a VMM of 50,000 lines is massively safer than one of 1.4 million. Less code means less attack surface. Minimalism here is not aesthetics but a security strategy.

The side channel, finally, is the uncomfortable reminder that even perfect logical isolation does not suffice as long as tenants share the same physical hardware. Attacks like Spectre and Meltdown showed in 2018 that a process can siphon information out of another security domain through shared caches, through the processor's speculative execution, or through shared compute cores – without ever breaching the logical boundary (see When the Processor Guesses Too Much: Spectre, Meltdown, and the Sin of Speculative Execution). This is why the virtualization boundary alone is not enough: Firecracker additionally requires the host to enable countermeasures – such as disabling Simultaneous Multithreading (Hyper-Threading), so that two foreign tenants never share the same physical core, together with Kernel Page-Table Isolation and the relevant branch-predictor protections. Isolation is thus a layered edifice of hardware virtualization, minimal software, and microarchitectural hygiene – not a single switch.

With these three concepts in hand, Firecracker's design becomes legible as a single, consistent answer: keep the trust boundary of virtualization, shrink the TCB radically, and close the side channels through operational rules.

Part 3: Firecracker's Bet – Keep KVM, Throw Away QEMU

Firecracker's decisive insight is a clean decomposition. The security of virtualization does not come from QEMU but from KVM, the virtualization substrate in the Linux kernel that enforces the hardware separation. QEMU supplies "only" the emulation of the virtual devices and the surrounding machinery. Firecracker's bet, therefore, is: keep KVM. Throw QEMU away entirely and replace it with a tiny, purpose-built VMM.

The result is a program of roughly 50,000 lines of Rust code – about 96 percent fewer than QEMU. Two things about this sentence are significant. First, the size: 50,000 instead of 1,400,000 lines means a dramatically smaller TCB and attack surface. Second, the language: Rust is a memory-safe systems language that eliminates an entire class of classic security vulnerabilities – buffer overflows, use-after-free, dangling pointers – already at compile time. For a program whose sole reason to exist is security, choosing precisely the language that structurally prevents the most common security bugs is consistent. Firecracker, incidentally, shares its Rust substrate through the collaborative rust-vmm project with other minimal VMMs, such as crosvm, the Virtual Machine Monitor of Chrome OS.

Firecracker is, as the authors themselves stress, most notable for what it cannot do. It offers no BIOS. It cannot boot arbitrary kernels and would not, without significant changes, start something like Microsoft Windows. It emulates no legacy devices and no PCI. At the time of publication it supported no live migration of virtual machines. Each of these omissions is a deliberate choice: a VMM that need only serve modern Linux (and OSv) guests for container and function workloads needs none of this – and every omitted feature is code that need not be written, need not be maintained, and above all cannot be attacked. Firecracker thus replaces QEMU not by trimming QEMU down but through an uncompromising rebuild for a sharply defined purpose.

The model for security isolation is beguilingly simple: one Firecracker process runs per microVM. Every function, every container sits in its own microVM, guarded by its own Firecracker process. If a function fails, one need only kill that single process. The attack surface of a tenant is confined to its own virtual hardware, behind which KVM and the host kernel lie hidden.

Part 4: The Minimalist Blueprint

Let us look at what this minimalism concretely looks like, for here lies the real engineering.

The device model. Instead of a hundred emulated devices, Firecracker offers only a handful, and these via virtio – a lean, standardized mechanism for virtual input/output in which the guest knows it is virtualized and can therefore dispense with expensive hardware mimicry. Firecracker essentially provides a virtio block device (for the disk), a virtio network device, and a serial console. Just how radically lean this is shows in the line counts from the paper: the actual block-device driver in the guest is around 50 lines of Rust, the serial driver about 250, and the entire virtio block device including data structures comes in at around 1,400 lines. Where QEMU is flexible and complex, Firecracker is tight and simple.

The Jailer. Even the tiny VMM could, in theory, harbor a bug. Firecracker therefore wraps another defensive layer around it: the Jailer. It is a wrapper program that places the Firecracker process into a restrictive sandbox before it boots the guest. Should a guest, against expectation, manage to break out of the virtual machine into the VMM process, that process already sits in a tightly cinched cage – with reduced privileges, its own namespaces, and pared-down access to the host. Defense in depth: not one perfect wall, but several staggered ones, so that breaking through one still grants no access to the whole.

seccomp-bpf. Additionally, Firecracker uses seccomp-bpf to restrict which system calls the VMM process itself may even make to the host kernel. This narrows the interface between VMM and host – the last remaining attack surface – to a minimum.

Rate limiters. Finally, Firecracker builds configurable rate limiters into its virtio devices. They cap both throughput (bandwidth) and the rate of operations (input/output operations per second for the disk, packets per second for the network). This serves not security in the narrow sense but performance isolation: on a densely packed host, a single greedy tenant must not be able to starve the disk or the network for everyone else. One detail is interesting: you can permit a one-time "burst" to accelerate booting and then enforce the steady-state limit – a pragmatic compromise between fast startup and fair sustained use. Crucially, these limits are enforced in the VMM, because the guest is not trusted to observe them itself.

In sum: Firecracker relies on KVM and the hardware for hard separation, on the host kernel's process scheduler and memory manager for memory and CPU fairness, and layers the Jailer, seccomp-bpf, and rate limiters on top. It is precisely this division of labor – leaving the heavy, proven work to the Linux kernel and implementing only the necessary, minimally, itself – that makes the low overhead possible.

Part 5: The Numbers That Make the Difference

A design is only as good as its measurable properties. Firecracker's key figures are the reason an elegant idea became a load-bearing pillar of the world's cloud infrastructure. Three numbers from the original paper stand out:

First, the memory overhead of under 5 MB per microVM (with the provided minimal Linux guest-kernel configuration). This is the amount of host memory the VMM and its surroundings consume before the guest does any useful work at all. For Lambda functions that make do with as little as 128 MB of memory, it is the difference between "a laughable percentage" and "ruins the economics." Less than 5 MB means one can house thousands of microVMs on a single machine without wasting memory on pure overhead.

Second, the boot time of under 125 milliseconds to application code. That is the time from the start command to the moment the function actually computes. An eighth of a second – fast enough that a serverless platform can spin up a microVM on demand without the calling user perceiving a delay. By comparison: a classical QEMU-based virtual machine with BIOS and a full device set typically needs many seconds for the same.

Third, the creation rate of up to 150 microVMs per second per host. Because serverless loads change constantly – functions come and go by the second – the platform must not only start microVMs quickly one at a time but many of them in rapid succession. 150 fresh starts per second per machine give the orchestration enough headroom to keep pace with the churn of the workload.

These numbers are no laboratory dreams. Firecracker has been running in production in AWS Lambda since 2018, carrying millions of workloads; it also underlies AWS Fargate, the service for serverless containers. Remarkably, AWS was able to migrate existing Lambda customers to the new isolation technology seamlessly, without their having to change any code – proof that Firecracker meets its compatibility goal: it runs arbitrary unmodified Linux binaries. And Firecracker is open source under the Apache 2.0 license, so the technology is not confined to AWS but has become a building block for an entire generation of sandbox platforms.

Part 6: Density and Oversubscription – The Statistical Bet

Why is low overhead so existential? Because the entire business model rests on density and oversubscription – and both are, at their core, a bet against statistics.

Serverless functions are tiny and sporadic. A typical function computes for perhaps a few dozen milliseconds and then sleeps for minutes. If one permanently assigned each function a reserved resource, the machine would lie idle most of the time. The way out is oversubscription: one sells more capacity than physically exists, trusting that never will all tenants call up their full quota at once. A slot whose function is currently asleep can briefly serve another. The authors call this outright a statistical bet: the platform must keep resources as busy as possible while ensuring that every tenant, when the need arises, does get what it requires without perceptible displacement by its neighbors.

For this bet to pay off, two things are needed, both of which Firecracker delivers. First, real performance isolation: a sleeping neighbor must not, upon waking, suddenly be starved by a greedy co-tenant – this is exactly what the rate limiters and the delegation to the host kernel's scheduler are for. Second, high density per machine: only if thousands of microVMs with minimal overhead fit onto a host is the law of large numbers on the operator's side – the more independent, sporadic loads on a machine, the smoother and more predictable the aggregate utilization, the safer the bet. Every megabyte of overhead saved increases the number of tenants over which the risk is averaged.

Thus the circle closes back to security: because one wants to pack many foreign tenants densely, the isolation between them must be hard; and because the isolation runs through individual, lightweight microVMs, one can pack them densely in the first place. Firecracker is the point where the security requirement and the economic imperative coincide, rather than contradicting one another.

Part 7: The Cold Start and the Snapshot Revolution

As fast as 125 milliseconds sounds, for some workloads even that is too much. The problem is called the cold start. When a function is invoked for which no warm microVM stands ready, the platform must spin up a new one – boot the microVM, start the runtime, load dependencies, run initialization code. With heavy runtimes like the Java Virtual Machine, initialization alone – loading classes, firing up the just-in-time compiler – can consume seconds and dwarf the microVM boot. For latency-sensitive services, this is the sore point of the serverless cloud.

The elegant answer is snapshotting. The idea: boot a microVM once, let it run through all its expensive initialization, and then freeze its complete state – working memory and virtual device states – as a snapshot. When the function is later needed, one does not start from zero but restores the snapshot and resumes exactly where the fully initialized machine stood. Firecracker supports precisely this, and the restore is astonishingly fast: in favorable cases a microVM snapshot can be restored in as little as around 4 milliseconds, and for a decently sized Linux system one reliably stays under ten milliseconds. "Seconds of cold start" become milliseconds.

On this technique rests AWS Lambda SnapStart, introduced in November 2022, initially for Java: when a function version is published, the function is warmed up once, all initialization code is run, and then an encrypted Firecracker snapshot of the fully initialized state is persisted and cached. Every later cold start loads this snapshot instead of booting anew – seconds become milliseconds.

But snapshots open a subtle, security-critical Pandora's box that has occupied its own piece of research ("Restoring Uniqueness in MicroVM Snapshots"). If one snapshot serves as a template from which many identical microVMs are cloned, then all these clones start with the same internal state – including the same store of randomness (entropy). Two freshly cloned machines drawing the same "random numbers" can generate identical cryptographic keys or session identifiers – a potentially catastrophic breach. The cold-start trick therefore demands additional measures to restore the uniqueness of each resumed machine despite the shared template. A fine illustration that every optimization in distributed systems exacts a price elsewhere – here, that "identically cloned" and "cryptographically unique" stand in tension.

Part 8: The Ecosystem – gVisor, Kata Containers, Cloud Hypervisor

Firecracker is not the only answer to the isolation problem, and it is worth locating its place in the spectrum of approaches. Roughly, three families can be distinguished, arrayed along the axis of "strength of isolation versus closeness to native performance."

At one end stand the classical containers (Docker, LXC): shared kernel, least isolation, but practically no overhead and full speed. Suitable for trusted, cooperating loads – unsuitable as a sandbox for hostile foreign code.

In the middle sits gVisor from Google. Its approach is idiosyncratic: instead of spinning up a hardware VM, gVisor interposes a user-space kernel between the guest and the real host kernel. This intermediate kernel intercepts the container's system calls and services them itself, so the container never directly touches the real host kernel. This shrinks the attack surface – the dangerous real kernel is masked – but offers no hardware isolation; the separation is again software-based, though with a much narrower interface. The price is a performance cost on every system call that must be intercepted and re-created.

At the other end lie the microVM approaches with true hardware virtualization: Firecracker itself, the closely related Cloud Hypervisor (also built on rust-vmm), and Kata Containers. Kata is not a competing VMM but an integration layer: it fits into the container standards and into Kubernetes and transparently spins up, for each container, a lightweight virtual machine – optionally even with Firecracker as the engine. The common rule of thumb is: Firecracker dominates serverless platforms and the new AI sandboxes, in which potentially risky code generated by AI agents must be executed safely; Kata owns multi-tenant Kubernetes; and most organizations are best served using Firecracker through Kata rather than building a microVM orchestration themselves – for Firecracker is a VMM, not a complete orchestration solution.

The crux for the architecture decision: hardware virtualization (Firecracker, Kata) stops kernel exploits at the root, because the attacker would have to defeat the boundary enforced by the CPU. A user-space kernel (gVisor) narrows the attack surface but still relies on software. A shared kernel (classical containers) leaves every tenant vulnerable to every kernel bug. Whoever must isolate hostile foreign code lands almost inevitably at some form of hardware virtualization – and Firecracker has shown that this need no longer be heavy.

Limits and Shadows

For all its elegance, Firecracker is no panacea, and intellectual honesty demands naming the edges.

First, Firecracker is specialized, not universal. What makes it secure and light – no BIOS, no PCI, no legacy devices, only modern Linux guests – simultaneously makes it useless for anything that needs those missing features. Anyone who needs Windows, exotic operating systems, or demanding device passthrough (say, GPUs for heavy compute) is poorly served by Firecracker. It is a precision tool for a specific purpose, not a replacement for the Swiss-army-knife character of QEMU.

Second, Firecracker shifts the trust boundary; it does not abolish it. The hard separation rests on KVM, the host kernel, and ultimately the correctness of processor virtualization. A bug in KVM or in hardware virtualization itself would still be devastating – the TCB is smaller, but not zero. And as stressed above, side channels remain a distinct problem that the virtualization boundary does not solve and that can only be hedged through operational discipline (Hyper-Threading off, kernel mitigations on, no physical cores shared between tenants). Firecracker makes these requirements explicit, but it does not waive them.

Third, the network and input/output performance of virtio-based microVMs is not in every case on par with that of a mere container solution; the authors themselves concede that their approach does not deliver the optimum for workloads with the highest throughput demands. Firecracker is optimized for thousands of ephemeral, small VMs, not for maximum single-instance performance.

I am of the opinion that precisely this honest specialization is Firecracker's real lesson: the breakthrough came not through yet another universal tool but through the willingness to simplify radically for a sharply drawn use case and to sacrifice everything superfluous.

A Framework to Take Away: The Isolation Spectrum

The following table condenses the landscape. It is deliberately simplified – real performance depends heavily on workload and configuration – but it orders the approaches along the decisive axes.

The guideline for practice: the more hostile the code you give a stage to, the further down this table you should place yourself. And the glad tidings Firecracker brought are that "further down" no longer automatically means "much more expensive and slower."

The Central Takeaway

Firecracker's real lesson is not one about virtualization but about engineering thought. A seemingly iron dilemma – security or lightness – turned out to be an artifact of an ill-fitting choice of tool. It was not virtualization that was heavy, but QEMU's universality. As soon as the use case was drawn sharply enough ("only modern Linux guests, only container and function loads"), almost the entire weight fell away as simply superfluous – and with it the attack surface.

For one's own work this translates as: when you face a stubborn tradeoff, ask not only "How do I find the best compromise on this curve?" but "Is the curve itself an artifact of an assumption I hold to be immovable?" Firecracker shifted the curve by dropping the assumption that a Virtual Machine Monitor must be universal. The practical prompt for Sven and every cloud architect is therefore concrete: choose your isolation technique deliberately according to the hostility of the code you run – trusted loads in containers, risky or foreign code in microVMs – and treat minimalism not as sacrifice but as an active security strategy: every line of code you do not write is a line that cannot be attacked.

A Question to Reflect On

Firecracker shrank the trusted computing base radically by sacrificing all unneeded universality. Where in your own systems – in your architecture, your code, your dependencies – do you carry "universality" you do not actually need, yet whose attack surface and maintenance burden you silently pay? And what would it cost to cut it away as uncompromisingly as Firecracker cut away QEMU?

Cross-References in the Vault

• When the Processor Guesses Too Much: Spectre, Meltdown, and the Sin of Speculative Execution – why the virtualization boundary alone does not close side channels and Firecracker demands additional microarchitectural countermeasures.
• The Ring That Shares the Load: Consistent Hashing and the Art of Moving Gracefully – the orchestration layer that decides which microVM lands on which host.
• Eleven Nines: Erasure Coding, Reed-Solomon, and How the Cloud Makes Data Practically Unlosable – the same thinking (redundancy and reliability out of unreliable parts) at the storage layer.
• Clocks That Know Their Own Uncertainty: Google Spanner, TrueTime, and Mastering Time in the Cloud – another example of a deep technical foundation beneath a seemingly simple cloud promise.
• How Machines Come to Agree: Distributed Consensus from FLP to Paxos to Raft – how the many hosts of a serverless fleet arrive at shared decisions at all.

Sources

1. Agache, A., Brooker, M., Florescu, A., Iordache, A., Liguori, A., Neugebauer, R., Piwonka, P., Popa, D. M.: Firecracker: Lightweight Virtualization for Serverless Applications. USENIX NSDI 2020. — https://www.usenix.org/conference/nsdi20/presentation/agache (Full text: https://www.usenix.org/system/files/nsdi20-paper-agache.pdf)
2. USENIX NSDI '20 presentation page for Firecracker. — https://www.usenix.org/conference/nsdi20/presentation/agache
3. "Restoring Uniqueness in MicroVM Snapshots." arXiv:2102.12892. — https://arxiv.org/abs/2102.12892
4. Brooker, M.: Lambda SnapStart, and snapshots as a tool for system builders. — https://brooker.co.za/blog/2022/11/29/snapstart.html
5. Google gVisor – Container runtime sandbox (project page). — https://github.com/google/gvisor
6. Kata Containers – project documentation. — https://katacontainers.io/

Note on the figures: the key numbers (< 5 MB overhead, < 125 ms boot, up to 150 microVMs/s/host, ~50,000 lines of Rust, QEMU > 1.4M lines) come from the peer-reviewed original NSDI 2020 paper and refer to the configuration described there; real-world values vary with guest kernel, workload, and hardware.

← All articles