Skip to content
NebulaCtrlInstall

Internals

View as Markdown

0008. Optional build servers and OCI registry distribution

  • Status: accepted
  • Date: 2026-10-27
  • Amends: 0004 (building on the target stays the default; this adds an optional second path), 0006 (the managed builder is reused on build servers)

Context#

ADR 0004 builds every git service on the server that runs it and has no registry. That is the right zero-configuration path, and it stays. It has three costs that owners of larger fleets asked to remove:

  • a build competes for CPU, memory and disk with the running service on the production host;
  • untrusted Dockerfiles (and fork pull-request previews) execute next to production data and runtime secrets;
  • a build exists on one host only, so a rollback outside the host's imageRetention window, a move to another server and a reconcile redeploy all need a rebuild.

Coolify and Dokploy solve this with a build server that pushes to a registry and deployment servers that pull. Both SSH into every host, so "build here, run there" needs only credentials on both ends. NebulaCtrl agents only dial out (ADR 0002): there is no agent-to-agent path, and the control plane is the only rendezvous. Their registry handling is also their main bug source (an ambient ~/.docker/config.json that differs between users, credentials that "log in" but cannot push, a mutable tag that the target does not re-pull), and neither product retains or collects registry storage.

The owner decided (phase 8, 2026-10-27):

  • support both a bring-your-own (BYO) registry and a managed "Set up for me" registry now;
  • the managed registry is CNCF Distribution registry:3.1.2, pinned by index digest, as an explicit exception to the "stable for two years" library rule (the project is 12 years old, the v3 line 19 months);
  • always fall back to the target when no build server or no usable registry exists, with a visible event, never a blocked deploy;
  • one BuildKit builder per project on a build server, and previews never build on a shared build server unless the owner opts in.

Decision#

A build server builds and pushes to an OCI registry; the target pulls by digest.

  1. Where a build runs is a placement decision, resolved service → project → team, with target as the default at every level. A registry is needed only when something resolves to build_server. Compose stacks and image services never build, so they are unaffected.
  2. Server roles are two orthogonal flags. accept_workloads (existing) decides placement of workloads; the new build_enabled decides build eligibility. Deployments (today's default) is accept_workloads on and build_enabled off, Builds is the reverse, Both sets both. Build eligibility never reads accept_workloads. Each build server has build_max_concurrent (default 2), build_allow_emulation and build_allow_previews (both off).
  3. Two durable commands per deployment. The control plane writes the release first, sends a build_image command to the build server (clone the pinned commit, docker buildx build --push, return the manifest digest), records the pushed image, and then sends the ordinary deploy command to the target with Image{Ref: repo@sha256:…}. The target engine already pulls images by digest with send-time registry credentials, so everything after the pull (architecture guard, create, health, route, verify, retire) is the existing engine. A new step push maps to the building status; no new deployment status exists.
  4. Registry. registries are team objects of two kinds: external (any OCI registry with a basic-auth push credential and an optional read-only pull credential: GHCR, Docker Hub, GitLab, Harbor, self-run) and managed (a registry platform service, platform services). ECR, Google Artifact Registry and Azure ACR need token refresh and are not supported in this version.
  5. Image identity. The repository is <host>/<namespace>/<project-slug>-<service-slug> (an empty namespace is allowed), the tag is r-<release8>, and the release records the registry manifest digest and registryRef = repo@sha256:…. Targets deploy the digest, never the tag. Each pushed image is recorded in registry_images with its own repository, so renaming a service never breaks a rollback. Provenance and SBOM attestations are off (--provenance=false --sbom=false), so the pushed result is a single manifest rather than an index.
  6. One platform per build. v1 builds one single-platform image for the target's architecture. A build server of another architecture is eligible only with build_allow_emulation on and an agent that reports the platform in facts.emulation. Multi-arch manifests are not built.
  7. Sticky selection with a concurrency cap. An explicit pin wins; otherwise rendezvous hashing over eligible servers with the service id as the key keeps the BuildKit cache warm. Each build server has its own cap; further builds queue on the control plane and the deployment stays preparing with "Waiting for a build slot on <server>".
  8. Isolation. The agent creates one BuildKit builder per project, nebulactrl-buildkit-<project8>, with its own cache and garbage-collection limits. Preview builds go to the target unless a build server sets build_allow_previews. Runtime environment variables never reach a build server; only the git credential, build variables and the registry push credential do.
  9. Credentials are send-time only and never ambient. The push credential reaches the build agent in the command frame's secrets and is written to a mode-0700 temporary DOCKER_CONFIG that exists for one build and is always removed; pulls pass the credential per request. agent_commands.payload carries only booleans. For the managed registry the control plane is the token server (/registry/token, Docker token authentication): each command receives an expiring credential scoped to one repository and the needed actions (15 minutes by default, longer for a build, at most 6 hours), which the registry exchanges for a JWT valid at most 15 minutes. No long-lived registry credential rests on any agent host.
  10. Rollback needs no build server. Rolling back a registry release is an ordinary deploy of repo@digest on the current target. image_pruned is decided from registry_images.state before dispatch; for a BYO registry, where NebulaCtrl deletes nothing, an image removed out of band shows up as a failed pull. "Rebuild commit" uses the current effective build location.
  11. Retention. Targets keep imageRetention images locally, as before (fast rollback, works while the registry is down). The control plane owns the live set (current release, releases of deployments in flight, the newest imageRetention succeeded releases of each service). For a registry with managed retention (the managed registry) it marks other registry_images rows expired, deletes those explicit digests through the registry API and runs registry garbage-collect (without --delete-untagged) inside a read-only window. For BYO registries NebulaCtrl records images but never marks them expired and never deletes.
  12. Failure handling is by situation, in build servers: troubleshooting: a build server offline at dispatch falls back to the target; one that goes offline after acknowledging the command never times out into a terminal state (the rule for every acknowledged command, servers); a registry that cannot be reached fails fast in a pre-build check; a pull failure on the target leaves the old release serving.

Every frame that belongs to a deployment (events, results, logs, source scans, cancel) is now accepted from any server that owns a command of that deployment, looked up through agent_commands, instead of only from deployments.server_id. Cancel targets whichever command is active.

Consequences#

  • A build server never runs production workloads (when set to Builds), so a build can no longer starve the service it is building, and an untrusted Dockerfile no longer executes on the host that holds runtime secrets. The cost is a second privileged host (the builder is privileged BuildKit) that holds the git credential, build variables and a push credential for all projects of the team; it is inside the documented trusted-team boundary (threat model). Per-project builders isolate caches, not the kernel.
  • A rollback of a registry release works with the build server offline or deleted and no longer depends on the target's local image cache. It depends on the registry being up and on the digest still existing there.
  • The registry is a new availability dependency for build-server deployments only: a registry outage blocks the push and the pull, but a fallback keeps deploys working when none is configured.
  • Two WAN hops per deploy replace zero (build server → registry, registry → target), mitigated by layer deduplication in the OCI protocol.
  • The managed registry holds every image of the team, so its host is a high-value target; it needs a public hostname, port 443 and an ACME email (Docker daemons require TLS for non-loopback registries). Traefik's default 60 s entrypoint readTimeout can cut a large layer upload over a slow uplink, so the registry's route is given a larger timeout.
  • BYO registries cannot be garbage-collected by NebulaCtrl. A lifecycle rule that deletes untagged images, or only keeps a few tags, can destroy rollback targets; the docs say what to keep.
  • Old releases keep working: a release with a local imageId keeps the target-built semantics; a release with a registryRef takes precedence. Both representations coexist.
  • Agents that predate the build.push capability are never chosen as build servers; the deployment falls back to the target.
  • Multi-arch manifests, ECR/GAR/ACR token refresh, registry cache export and a registry-free transport are explicitly not included.

Alternatives considered#

  • Relay transport: docker save | docker load through the control-plane WebSocket. The only registry-free option that works with outbound-only agents, and its one real merit. Rejected for the first version: the control plane sits in the image data path (it would see every layer in clear); there is no layer deduplication (docker save ships every parent layer, so a 300 MB image with 30 MB of changed layers moves ~600 MB through the control plane instead of ~60 MB with a registry); both agents must be online at once and a drop restarts the whole transfer; the bulk bytes would share the one socket that carries heartbeats, logs and commands; and rollback would still need each target's local image. It stays a possible later fallback for installs that cannot run a registry host.
  • Remote BuildKit driven by the target (docker buildx create --driver remote tcp://…). Needs an inbound listener on the build server; agents never accept inbound connections. It still would need the source to flow to the builder, so it has no advantage over the builder the build agent drives locally (which is what this ADR does).
  • zot instead of registry:3. Inline garbage collection and a retention policy remove the read-only window, and it has dedupe and scrub. Its default garbage collection deletes untagged manifests after a delay, which is a footgun for digest-pinned rollback targets unless every retained release keeps a tag, and it is a larger surface. registry:3 is the reference implementation, smaller, and its token authentication fits a control plane that every agent can already reach. Both speak the OCI API, so swapping is a catalog change; the unique r-<release8> tag already makes zot safe.
  • Native multi-arch. One buildx invocation cannot span two hosts without an inbound path, so native multi-arch would be two builds on two build servers plus docker buildx imagetools create. Valuable only for fleets that move one release between architectures; recorded as the extension path.
  • A registry credential stored in ~/.docker/config.json on each server (Coolify, Dokploy). Rejected: a global docker login leaves long-lived credentials on a shared builder and fails when the build and the pull run under different Docker configs.
  • Always random build-server choice. Fragments the cache (1/N hit chance) and was a Coolify complaint; replaced by sticky selection.
  • Hosted builders (Docker Build Cloud and similar). Contradicts the self-hosted, no-SaaS goal.