Under the Hood

Imp turns Kubernetes objects into Firecracker microVMs. Its operator decides what should exist. Its node agent does the work that must happen on the selected Linux node.

This page describes the current source implementation. It does not claim that every path has been proven in every cluster.

Current implementation

1. One desired state, two execution roles

flowchart LR
  user[User or controller] --> api[Kubernetes API]
  api --> operator[Imp operator]
  operator --> vm[ImpVM desired state]
  vm --> agent[Node agent]
  agent --> fc[Firecracker microVM]
  agent --> status[ImpVM status]
  status --> api

Plain explanation: the operator makes the cluster-wide decision. The node agent performs the node-local work: disks, networking, Firecracker, and guest communication.

Evidence in the repository:

• API contracts
• Pool reconciliation
• Node-local runtime work
• Firecracker and VSOCK path

2. A VM is built from layers, then started

flowchart LR
  base[Base OCI image] --> rootfs[Ext4 root filesystem]
  workload[Workload OCI layer] --> rootfs
  agentbin[Injected guest agent] --> rootfs
  rootfs --> disk[VM disk]
  disk --> fc[Firecracker]
  fc --> guest[Guest agent over VSOCK]

Plain explanation: Imp does not run a container as the workload. It turns OCI image layers into a VM filesystem, starts a microVM, and talks to a small guest agent over VSOCK, a host-to-guest channel.

When the guest agent is enabled, Imp writes a small /.imp/init wrapper into the VM disk. The wrapper starts the guest agent, then runs the image's normal /sbin/init. Workload launch happens later through VSOCK.

Evidence in the repository:

• Root filesystem assembly
• Guest init injection
• Runner launch boundary

3. An ImpVM is not a Kubernetes Pod

The Imp agent is a Kubernetes DaemonSet Pod. Kubernetes schedules that Pod onto a Linux node. The agent container mounts the host's /dev/kvm, the Firecracker binary, and the guest kernel.

An ImpVM is different. It is not another Pod created for the workload. Once an ImpVM is assigned to a node, the agent starts Firecracker as a Linux process on that node. Firecracker uses KVM through /dev/kvm to boot a separate guest kernel and the ext4 root filesystem Imp built from OCI layers.

flowchart TD
  k8s[Kubernetes schedules one agent Pod per enabled node] --> agent[Linux Imp agent Pod]
  agent --> kvm[Host /dev/kvm]
  agent --> kernel[Guest kernel image]
  agent --> rootfs[Ext4 root filesystem]
  kvm --> fc[Firecracker process]
  kernel --> fc
  rootfs --> fc
  fc --> guest[Guest Linux system]

Plain explanation: a Pod runs container processes that share the node's Linux kernel. An ImpVM is a guest Linux system with its own kernel. Kubernetes knows the agent Pod; Imp records the VM as an ImpVM custom resource and starts its Firecracker process on the selected node.

	Agent Pod	ImpVM
Kubernetes object	DaemonSet Pod	ImpVM custom resource
Who selects the node	Kubernetes scheduler	Imp's VM scheduler
What starts the work	Kubelet starts the agent container	Agent starts a Firecracker process
Kernel seen by workload	Node's shared Linux kernel	A separate guest Linux kernel
Image use	Container image becomes the process filesystem	OCI layers become an ext4 VM disk
Main runtime record	Pod status	ImpVM.status.runtimePID and phase

Evidence in the repository:

• Linux-only Firecracker driver
• Agent DaemonSet and host mounts
• VM runtime status

GitHub runners

4. GitHub App to one runner

sequenceDiagram
  participant O as Pool controller
  participant G as GitHub API
  participant S as VM-owned Secret
  participant A as Node agent
  participant V as Guest VM

  O->>G: App JWT
  G-->>O: Installation token
  O->>G: JIT runner setup
  O->>S: Store one-time setup
  A->>V: VSOCK execution environment
  V-->>A: Runner exits
  A->>S: Delete setup

Plain explanation: the GitHub App key stays with the operator. The one-time setup reaches the guest through the VSOCK execution environment. It is not baked into the VM disk or placed in command arguments.

Evidence in the repository:

• App JWT and installation-token exchange
• JIT configuration contract
• VM-owned JIT Secret
• VSOCK handoff and deletion

For the full paths, read how a runner is configured and how a queued job reaches a runner.

Runner image boundary

Runner images use two OCI layers so the operating-system base and CI software can change independently.

flowchart TD
  base[Debian base] --> compose[Rootfs composer]
  layer[Actions runner layer] --> compose
  compose --> disk[Cached ext4 disk]
  disk --> boot[Firecracker boot]
  boot --> wrapper[Runner wrapper]
  wrapper --> jit[JIT setup over VSOCK]
  jit --> job[Register and run one job]

The base supplies the guest userspace, certificates, .NET runtime libraries, and an unprivileged runner user. The runner layer supplies /home/runner/actions-runner and the wrapper executable. Firecracker attaches the guest NIC; the image and wrapper bring up eth0 and DNS before registration. Alpine and musl images are not the default contract because compatibility with the official Actions runner is not established.

The JIT value is not part of either image, the VM manifest, the root filesystem cache key, or a command-line argument. The operator creates a VM-owned one-time Secret. The node agent reads that exact Secret, and guest-agent execution passes the value through the process environment.

Runner lifecycle implementation

The pool controller creates an ephemeral VM, creates or recovers its VM-owned JIT Secret, and removes the new VM if setup cannot be minted or persisted. The agent hands the one-time setup to the guest and removes that Secret after the runner exits. Terminal VMs are deleted and the next reconciliation restores capacity within the pool limits.

flowchart TD
  reconcile[Reconcile pool capacity] --> vm[Create ephemeral VM]
  vm --> mint[Create JIT Secret]
  mint --> handoff[Deliver one-time setup]
  handoff --> exit[Runner exits]
  exit --> cleanup[Remove Secret and VM]
  cleanup --> replace[Restore pool capacity]
  mint -. setup failure .-> remove[Remove new VM]

Plain explanation: one VM owns one setup Secret. If setup cannot be made safely, the controller removes that VM instead of letting it boot without a runner. When the runner finishes, the temporary setup and terminal VM are removed. The pool can then replace capacity.

Evidence in the repository:

• Pool lifecycle and Secret ownership
• Credential validation
• Guest handoff and cleanup

Security invariants

Must happen	Must not happen
Keep App credentials in the operator-side credentials Secret	Put App keys or JIT data in ImpVM.spec.env
Send the one-time setup through the guest execution environment	Bake JIT data into the root filesystem or its cache key
Keep the JIT Secret owned by exactly one VM	Print tokens, private keys, or JIT payloads in logs
Delete the JIT Secret after runner exit or bounded cleanup	Treat request acceptance as proof that the runner completed

Read the system in order

• Architecture for the operator and node-agent split.
• Runner pool for capacity and job execution.
• Runner configuration for GitHub App and JIT setup.
• Job scheduling for capacity, node placement, and GitHub job assignment.
• Lifecycle for VM lifecycle concepts.