VM Management

VM lifecycle and management

Compute providers manage virtual machines through the vm.Manager interface. The implementation the node wires up uses Lima (Linux virtual machines via QEMU) to create isolated Ubuntu 24.04 VMs with SSH access. A VM is provisioned before the lease_accept block is committed — if provisioning fails the lease is not accepted — and torn down on settlement. Boot time is approximately 21 seconds from the start of provisioning to a running VM with SSH.

Manager interface

type Manager interface {
    Provision(leaseHash string, res Resources, accessPubKey string) (*Info, error)
    Teardown(leaseHash string) error
    DialSSH(leaseHash string) (net.Conn, error)
    Get(leaseHash string) (*Info, error)
    List() []*Info
}

Method

Description

Provision

Create and start a VM with the specified resources. Injects the consumer's SSH public key via the Lima provision script. Returns VM info.

Teardown

Stop and delete the VM via limactl stop then limactl delete --force. Called automatically on lease settlement.

DialSSH

Open a TCP connection to the VM's SSH port on localhost. Used by the tunnel protocol to proxy SSH sessions from remote consumers.

Get

Retrieve the current state of a VM by lease hash from the in-memory map.

List

Return all VMs managed by this provider.

There is no Exec method: the provider does not run commands inside a consumer's VM on request. Access is SSH only, through the tunnel protocol.

Lima implementation

The LimaManager (vm/lima_manager.go) manages QEMU-based VMs using the limactl CLI tool from the Lima project.

Architecture

xe node process (provider)
    │
    ├── LimaManager
    │     ├── limactl create (from YAML template)
    │     ├── limactl start (boots QEMU VM)
    │     ├── limactl list --json (discover SSH port)
    │     └── limactl stop + delete --force (teardown)
    │
    └── QEMU processes (one per active lease)
          ├── xe-<leaseHash[:12]> → Ubuntu 24.04 VM
          └── xe-<leaseHash[:12]> → Ubuntu 24.04 VM

Initialisation

When a provider node starts with --provide, the LimaManager is created:

mgr, err := vm.NewLimaManager(cfg.DataDir, cfg.LimactlPath)
  • LIMA_HOME is set to {dataDir}/lima — all VM state, disk images, and sockets live here
  • Template directory is {dataDir}/lima-templates — YAML templates are written here before limactl create
  • Images directory is {dataDir}/images — the Ubuntu cloud image is cached here
  • On startup, limactl --version is executed to verify the tool is available; if it is not, provider startup fails
  • If the Ubuntu cloud image is not present locally, it is downloaded automatically from the upstream Ubuntu cloud images server (~600 MB)
  • Orphaned Lima instances left behind by a previous run are cleaned up

VM naming convention

VMs are named xe-{leaseHash[:12]} — the first 12 hex characters of the lease block hash. This keeps names short while avoiding collisions. For example, lease hash 6226fb52501a9052b533... creates a VM named xe-6226fb52501a.

Provision flow

When Provision(leaseHash, resources, accessPubKey) is called:

  1. Convert access key — the accessPubKey (ed25519 public key as 64-char hex string) is converted to SSH authorized_keys format. This is the consumer's key for SSH access to the VM. The rendered key must match ^ssh-[a-z0-9-]+ [A-Za-z0-9+/=]+$, since it is interpolated into a shell script.

  2. Render template — a Lima YAML template is generated referencing the local Ubuntu 24.04 cloud image with the requested resources and SSH key:

    vmType: "qemu"
images:
  - location: "file:///var/lib/xe-node/images/ubuntu-24.04-x86_64.img"
    arch: "x86_64"
cpus: 1
memory: "512MiB"
disk: "5GiB"
mounts: []
containerd:
  system: false
  user: false
provision:
  - mode: system
    script: |
      #!/bin/bash
      mkdir -p /root/.ssh && chmod 700 /root/.ssh
      echo '<ssh-ed25519 key>' >> /root/.ssh/authorized_keys
      chmod 600 /root/.ssh/authorized_keys
      sed -i 's/#PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
      sed -i 's/#PubkeyAuthentication.*/PubkeyAuthentication yes/' /etc/ssh/sshd_config
      sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
      systemctl restart sshd

    The image is referenced via file:// from the local cache at {dataDir}/images/. SSH is pre-installed in the Ubuntu cloud image — the provision script only injects the consumer's key and restarts sshd.

  3. Write template — saved to {dataDir}/lima-templates/{vmName}.yaml

  4. Create VM — limactl create --tty=false --name={vmName} {templatePath} uses the cached Ubuntu cloud image as a copy-on-write backing file (instant, no download). Timeout: 2 minutes.

  5. Start VM — limactl start --tty=false {vmName} boots QEMU with KVM acceleration and runs the provision script, which injects the SSH key. Total time to a reachable VM is approximately 21 seconds; the timeout is 5 minutes. On failure the manager captures the tail of ha.stderr.log and serial.log for diagnostics, then deletes the instance.

  6. Wait for SSH — polls limactl list --json every 2 seconds for up to 2 minutes until the VM reports an SSH port and that port accepts a TCP connection. This port is on 127.0.0.1 and is the tunnel target.

  7. Store state — the VM info (lease hash, SSH port, status, resources) is stored in the in-memory vms map

Resource constraints

Resource

Minimum

Default

Description

vCPUs

1

From lease

QEMU -smp parameter

Memory

512 MiB

From lease

QEMU -m parameter

Disk

5 GiB

From lease

QCOW2 disk image size

If the lease requests less than the minimum, the minimum is used.

Teardown

When Teardown(leaseHash) is called (triggered by lease settlement):

  1. Execute limactl stop {vmName} then limactl delete --force {vmName} — this stops the QEMU process and removes all VM artifacts (disk, sockets, logs)
  2. Delete the template file from {dataDir}/lima-templates/
  3. Remove the entry from the in-memory vms map

SSH connection

DialSSH(leaseHash) opens a raw TCP connection to the VM's SSH port:

net.Dial("tcp", fmt.Sprintf("127.0.0.1:%d", vm.sshPort))

This is used by the tunnel protocol to proxy SSH sessions from remote consumers through libp2p streams to the VM's local SSH server.

Data types

Resources

type Resources struct {
    VCPUs    uint64 `json:"vcpus"`
    MemoryMB uint64 `json:"memory_mb"`
    DiskGB   uint64 `json:"disk_gb"`
}

Info

type Info struct {
    LeaseHash   string       `json:"lease_hash"`
    Status      string       `json:"status"`
    Resources   *Resources   `json:"resources,omitempty"`
    Credentials *Credentials `json:"credentials,omitempty"`
    CreatedAt   int64        `json:"created_at"`
    Error       string       `json:"error,omitempty"`
}

type Credentials struct {
    Username string `json:"username"`
    Password string `json:"password"`
}

The Lima manager provisions SSH-key access only and leaves Credentials unset, so in practice the field is absent. It exists for managers that issue a username and password; such a manager would send them to the consumer over the vm_credentials direct message, and the vm_status relay a non-provider node uses to answer GET /vms/{lease} returns only what the provider's VM manager reports.

VM status values

Status

Description

provisioning

VM is being created (limactl create/start in progress)

running

VM is active, SSH port is available, accepting commands and tunnel connections

stopped

VM has been torn down after lease settlement

error

VM encountered a fatal error during provisioning or execution

VM lifecycle

lease_accept confirmed
        │
        ▼
  ┌──────────────┐    vm_credentials     ┌──────────┐
  │ Provision    │ ──────────────────────►│ Consumer │
  │ (limactl     │    (direct msg)       └──────────┘
  │  create+start)│
  └──────────────┘
        │
        ▼
  ┌──────────────┐
  │ Running      │ ◄── SSH tunnel from consumer (via libp2p)
  │ (Ubuntu VM)  │
  └──────────────┘
        │
        │  lease expires + settle
        ▼
  ┌──────────────┐
  │ Teardown     │
  │ (limactl     │
  │  stop+delete)│
  └──────────────┘
  1. Provision — triggered by autoAcceptLease() before the lease_accept block is committed (~21 seconds). Lima creates an Ubuntu 24.04 VM from the cached cloud image with the consumer's SSH key injected by the provision script. If provisioning fails, the lease is not accepted.
  2. Credential delivery — the provider sends a vm_credentials direct message to the consumer with the VM info.
  3. Running — the consumer accesses the VM over SSH, through the SSH gateway and the libp2p tunnel protocol. There is no remote command-execution path.
  4. Teardown — triggered by settleLease() after the lease_settle block is committed. The VM is deleted and its resources freed. A terminal lease whose teardown failed is retried by the settle loop's orphan sweep.

VM access

The SSH gateway provides transparent remote access to VMs. The consumer connects via standard SSH:

xe ssh <leaseHash>

which is equivalent to connecting with the lease hash as the SSH username:

ssh -i <lease key> -p 2222 <leaseHash>@<gateway-host>

The gateway authenticates using the AccessPubKey from the lease block, then tunnels the session through libp2p to the provider node, which proxies to the VM's local SSH server. See SSH Gateway & Tunnel Protocol for full details.

Message handlers

The node registers two VM-related message handlers:

Message

Direction

Description

vm_credentials

Provider → Consumer

Delivers VM info after provisioning. Accepted only from the lease's provider.

vm_status

Any → Provider

Returns the provider's current VM info for a lease. Any peer may query, which lets a bootstrap node proxy GET /vms/{lease}.

Provider node flags

Providers enable compute leasing with the following xe node flags:

Flag

Type

Default

Description

--provide

bool

false

Enable provider mode

--vcpus

uint

2

Number of vCPUs available for leasing

--memory

uint

2048

Memory in MB available for leasing

--disk

uint

20

Disk in GB available for leasing

--price-multiplier

uint

1000

Price multiplier ×1000 stamped onto the performance certificate. Non-baseline values are simulation-only (#297).

--ssh-port

int

0

SSH gateway listen port (0 = disabled)

--limactl-path

string

(empty → limactl)

Path to the limactl binary

The following flags shape the local auto-accept policy (#229). A zero/empty value means "no limit", and the zero-value policy is fully permissive:

Flag

Type

Description

--min-lease-duration

duration

Reject leases shorter than this (e.g. 1h)

--max-lease-duration

duration

Reject leases longer than this (e.g. 720h)

--min-lease-cost

uint

Reject leases below this cost

--max-lease-cost

uint

Reject leases above this cost

--max-concurrent-leases

uint

Cap on active leases (running + provisioning)

xe node --provide --vcpus 4 --memory 8192 --disk 100 --ssh-port 2222

When --provide is set, the node:

  1. Requires sys.timekeepers in the state chain — it refuses to start without it.
  2. Initialises the LimaManager with the configured limactl path.
  3. Sets up the tunnel protocol handler for incoming SSH tunnel requests.
  4. Runs the performance benchmark and publishes a signed certificate.
  5. Advertises available resources via marketplace gossip every 5 minutes.
  6. Watches for unaccepted lease blocks every 5 seconds and automatically accepts them.
  7. Runs the settleLoop() every 10 seconds to settle, force-settle, archive and clean up.

Marketplace discovery

Providers and consumers find each other through the marketplace gossip topic (xe/marketplace):

  1. Consumer publishes a ResourceRequest with desired vCPUs, memory, disk, and duration.
  2. Provider receives the request, checks available resources, and replies with a signed ResourceOffer carrying the computed total_cost and the certificate_hash that cost was derived from.
  3. Consumer receives the offer and creates the lease block, referencing the same certificate hash so the price is locked.

The entire negotiation happens over gossip before any on-chain blocks are created.

Infrastructure requirements

Lima uses QEMU as its virtualisation backend. The host must have:

  • QEMU installed (qemu-system-x86_64)

  • Lima installed (limactl v1.0.6+)

  • KVM support — Lima requires /dev/kvm (hardware virtualisation extensions: Intel VT-x or AMD-V). Hosts without KVM (most cloud VPS) cannot run Lima VMs.

  • KVM permissions — the xe user must be able to access /dev/kvm. Since pm2's uid/gid setting drops supplementary groups, add a udev rule:

    # /etc/udev/rules.d/99-kvm.rules
KERNEL=="kvm", MODE="0666"

  • Non-root execution — Lima refuses to run as root. The node process must run as a non-root user (e.g., the xe service user).

  • Disk space — the Ubuntu cloud image is ~600 MB. Each running VM uses additional disk for its QCOW2 overlay (copy-on-write from the shared base image).

[!WARNING] KVM requirement Standard cloud VPS instances typically do not expose hardware virtualisation. Bare-metal servers or VPS with nested virtualisation enabled are required for Lima VM support. Without KVM, the lease lifecycle (creation, acceptance, attestation, settlement) still works — only the actual VM boot fails.

Manager interfaceLima implementationArchitectureInitialisationVM naming conventionProvision flowResource constraintsTeardownSSH connectionData typesResourcesInfoVM status valuesVM lifecycleVM accessMessage handlersProvider node flagsMarketplace discoveryInfrastructure requirements