Build servers and registries

By default a git service is built on the server that runs it (builds, ADR 0004): nothing to set up, no registry. A build server is an optional second path: one enrolled server builds the image, pushes it to an OCI registry, and the server that runs the service pulls it by digest. Use it when builds should not compete with production for CPU and disk, when untrusted Dockerfiles should not run next to production data, or when a release must be deployable on any server without a rebuild. The design and the alternatives are in ADR 0008.

Nothing changes until something resolves to Build server (below). Services that use an image, Compose stacks and databases never build, so they are unaffected.

How a deployment works#

1. The control plane resolves the commit, picks a build server and a registry, and writes the release.
2. The build server clones the pinned commit, builds one image for the target's architecture with BuildKit, pushes it as <registry>/<namespace>/<project>-<service>:r-<release8>, and returns the manifest digest.
3. The control plane records the image (registryRef on the release, a row in the registry's image list) and sends the normal deploy command to the target, which pulls repo@sha256:… and runs the usual steps (health check, routing, retire the old container).

The deployment's timeline shows both servers (Built on X, deployed on Y) and a push step. The build log is the build server's output; the pull and deploy lines come from the target. Build, push and pull all count as the building status.

What goes where:

Secrets are attached to the command when it is sent and are never stored in the command row (threat model).

Server roles#

Open the server's page, Settings → Role:

accept_workloads keeps its meaning (placement of services, drains, templates, platform services); build eligibility never reads it. A Builds server therefore cannot be chosen for a service, a drain or a template, and a Both server builds on the same Docker daemon that runs its workloads.

More settings on a build server:

Changing the role is audited (server.build_role). Use a dedicated host for Builds: the BuildKit builder is privileged, so a compromised build host exposes the git credentials, build variables and push credentials of every project that builds there. Give it the agent and Docker only, and revoke it like any server if it is suspect (servers).

A server needs a current agent to be a build server (the build.push capability). An older agent is never chosen; the deployment falls back to the target.

Where builds run#

Placement is resolved service → project → team. The first level that is not inherit wins; the team default is Target.

The pinned server and the registry follow the same order (service, then project). The CLI sets the service override with --build-location and --build-server on nebulactrl services create and services update.

The deploy plan shows the result before you deploy: the build server, the registry, the platform and every warning (fallback, emulation, a cold cache).

Choosing the server. A pin always wins. Otherwise NebulaCtrl hashes the service over the eligible servers, so the same service keeps landing on the same build server and its BuildKit cache stays warm. A server is eligible when it is build_enabled, online and not revoked, has the build.push capability, matches the target's architecture (or may emulate it), is allowed to build previews if this is a preview, and the registry exists and is healthy. A server at its concurrency cap is still eligible: the build queues.

Reusing a build. If a live image already exists for the same commit, build configuration, build variables (compared by a keyed hash, never stored), builder version and platform, the build is skipped: the deployment's timeline says "Reusing the image already built for commit <sha7> (no build needed)" and it deploys that digest. This is also why a redeploy that only changes a runtime variable is fast.

Rebuild commit (offered when a rollback target is gone) uses the current effective placement, not the one the old release was built with.

Registries#

A registry is a team object (Settings → Registries). A build server needs one; there are two kinds.

Bring your own registry#

Any OCI/Docker v2 registry with basic authentication. Fields:

Test registry (per server, POST /registries/{id}/check?serverId=) asks that server to reach the registry and try a push and a pull with the stored credentials, and shows the registry's own error. Run it for the build server and for each target before the first deploy. The same check runs automatically at the start of every build, before minutes are spent building.

Recipes (credentials and names are the provider's; check its current console):

GitHub Container Registry (GHCR)

• Host ghcr.io, namespace your user or organisation in lowercase.
• GHCR accepts classic personal access tokens only. Push: a token with write:packages. Pull: a second token with read:packages. The username is your GitHub login.
• A classic token is account-wide, not limited to one repository; create a dedicated bot account for it if that matters. New packages are private by default and linked to the account, not to a repository, so set the package access for the organisation if other members need it.

Docker Hub

• Host docker.io, namespace your Docker Hub user or organisation.
• Create an access token (push: Read & Write; pull: Read-only) and use your Docker Hub username with it.
• NebulaCtrl creates one repository per service (<project>-<service>); check your plan's private-repository limit and the pull rate limits.

GitLab

• Host registry.gitlab.com (or your instance's registry host), namespace <group>/<project>: the images live under that GitLab project.
• Push: a deploy token with write_registry (or a personal or project access token with write_registry). Pull: a deploy token with read_registry. The username is the deploy token's username.

Harbor

• Host your Harbor hostname, namespace the Harbor project name. The Harbor project must exist first: Harbor does not create projects on push.
• Create a robot account for the project: push and pull for the push credential, pull only for the pull credential. Use the robot's full name (it contains a $) as the username.

Other self-run registries (registry:3, Gitea, Forgejo, a cloud provider's registry with a static credential) work the same way. A registry on plain HTTP is refused unless it is on a loopback address, because Docker daemons only accept plain HTTP for loopback (the integration test uses one).

Not supported: AWS ECR, Google Artifact Registry and Azure ACR hand out tokens that expire within hours; a stored password does not work with them and NebulaCtrl does not refresh tokens (limits).

Set up for me (managed registry)#

The registry is a platform service of kind registry: CNCF Distribution registry:3.1.2, pinned by digest and run unmodified, on a server you choose, with a data volume and a public hostname. Setting it up creates the registry object for you.

• Needs a public hostname, port 443 on that server's proxy and the server's ACME email: Docker daemons require TLS for any registry that is not on loopback, so the certificate has to be real (a .localhost name is the development exception). Put it on the build server (the push stays local and only pulls cross the network) or on the control-plane server.
• Credentials are short-lived and scoped. The control plane is the registry's token server (/registry/token, Docker token authentication). Every build and every pull gets an expiring credential limited to one repository and the actions needed (15 minutes by default, longer for a build), which the registry exchanges for a token valid at most 15 minutes. No long-lived registry password exists on any host, and a revoked server cannot get a new one.
• Large uploads. The proxy's default 60 second request-read timeout can cut a big layer upload over a slow link, so the registry's domain has large uploads on, which raises it to 30 minutes on that server (the proxy container is recreated once).
• Retention is managed (below).
• Back it up? The images can be rebuilt from their commits, so it is optional; if you want a copy, back up its volume like any service (volume backups).

Cross-architecture builds and emulation#

A deployment has exactly one target server and so one CPU architecture. v1 builds one single-platform image for that architecture (for example linux/arm64). There are no multi-arch manifests.

• A build server of the same architecture is always eligible.
• A build server of another architecture is eligible only if Allow emulation is on for it and it reports that platform in its facts (facts.emulation, detected from the kernel's binfmt_misc QEMU handlers). Otherwise it is skipped, and with no eligible server the deployment falls back to the target.
• Emulation is much slower than native builds, especially for compilation. Railpack builds one platform per build and has no cross-compile shortcut, so every install and build step runs emulated.
• To enable it on an amd64 build host: docker run --privileged --rm tonistiigi/binfmt --install all, then restart the agent so it reports the platform.
• A faster alternative: build natively on a server of the target's architecture, or cross-compile in your Dockerfile with FROM --platform=$BUILDPLATFORM … and TARGETPLATFORM/TARGETARCH, so the slow part runs natively even on the amd64 builder.
• Placement only considers build servers whose architecture matches the target (or that may emulate it), so an image for the wrong architecture is not built in the first place. A release records the platform it was built for (platform in its source); do not move a release's service to a server of the other architecture without rebuilding the commit, or the container fails with exec format error.

Per-project builders and preview isolation#

A build server runs one BuildKit builder per project, named nebulactrl-buildkit-<project8>, each with its own cache. Cache mounts and layer caches are therefore not shared between projects, so one project's build cannot read or poison another's cache. It costs disk: the same base layers are cached once per project. Garbage-collection limits are set on each builder, and an agent-level daily prune stays as a backstop (Docker cleanup). Remove a builder and its cache with docker buildx rm -f nebulactrl-buildkit-<project8>; the next build recreates it.

A per-project builder isolates caches, not the kernel: the builder is privileged and a container escape on the build host reaches every project that builds there. That is inside the trusted-team boundary; do not put mutually untrusting teams' projects on one build server.

Pull-request previews never build on a shared build server. They build on their target, as before, unless a build server has Allow previews on. Fork pull-request code is untrusted (previews); keep Allow previews off on any build server that holds credentials you care about, or dedicate one build server to previews.

Fallback to the target#

A deployment never blocks because there is no build server. When nothing usable exists, the build runs on the target, exactly as without this feature, the timeline shows "No build server available; building on <target> (<reason>)" (a placement event), and the deploy plan carries the same warning. This happens when:

• no server is build_enabled, online, and capable (build.push);
• the architecture does not match and emulation is off or not reported;
• the build server was offline when the deployment was dispatched (nothing had been sent yet);
• no registry is configured, or a managed registry is not healthy;
• the deployment is a preview and no build server allows previews.

Fallback is decided before the build command is sent. A build server that goes offline after it acknowledged the command is a different case: the true outcome is unknown, so the deployment waits ("Waiting for build server <name>", with the time it went offline) and never times out into a terminal state (servers). When it reconnects it replays the result. Cancel the deployment, or deploy again after switching the service to Target, if the host is gone.

Rollback and retention#

A rollback of a registry release needs no build server. It is an ordinary deploy of repo@digest on the current target: the build server can be offline, removed or revoked, and the target does not need the image locally. It does need the registry up and the digest still present in it. The target also keeps its newest imageRetention images locally (default 5), so a rollback inside that window works without the registry too.

If the digest is gone, the rollback does not start the new release and the running one keeps serving: for a managed registry NebulaCtrl knows beforehand from its image list and refuses with 409 image_pruned; for a BYO registry it finds out when the target's pull fails (the deployment fails at the pull step, "Previous release is still serving", with the registry's own message). Either way, Rebuild commit <sha7> deploys the commit again.

What is kept. For every service the live set is the current release, the releases of deployments in flight, and the newest imageRetention succeeded releases (the ones Roll back to this release can pick). Each image NebulaCtrl pushes is recorded with its own repository and digest in the registry's image list (Settings → Registries → Images, GET /registries/{id}/images), whose state is live, expired or deleted.

• Managed registry: an hourly sweep deletes the digests of images already marked expired (those outside the live set) through the registry API, one explicit digest at a time; it never touches a live image or a digest it did not record, so a push whose result was not yet stored is safe. Blobs nothing references are then reclaimed by registry garbage-collect (never with --delete-untagged) in a weekly window that starts between 02:00 and 05:00 UTC: builds for this registry queue, running pushes get up to 2 hours to finish, the registry is set read-only (a short redeploy), the collection runs, and the registry returns to read-write. Pulls work during the window; pushes get 405.
• BYO registry: NebulaCtrl never deletes anything and does not mark anything expired: its images stay live in the list for good, and the registry keeps growing until you set a lifecycle rule.

What to keep in a BYO registry

• Do not enable "delete untagged" rules. A digest-pinned rollback target can look untagged to a registry that only counts tags.
• Keep at least imageRetention recent tags per repository, plus a margin: every release has a unique, never-moved tag r-<release8>, so rules that keep "the newest N tags" keep the newest releases. Ten is a comfortable number for the default of 5.
• Never delete the digest of the current release: a reconcile redeploy, a drain or a move to another server pulls it again.
• Delete by age only if the age exceeds how long you want rollbacks to work.

Lifecycle rules that fit:

Local images. A build server pushes with --push and nothing is loaded into its Docker image store, so there is no image keep-list on a build-only server; only BuildKit cache, the pinned BuildKit and Railpack images and clone workdirs remain (Docker cleanup). A target keeps the images of registry releases in its keep list (Docker cleanup).

Troubleshooting#

The build log of a registry deployment lists both servers' steps. Use Deployments → the deployment → Logs and read the build and push lines for the build server and the pull and deploy lines for the target.

Limits#

• One platform per build, no multi-arch manifests. A release runs on servers of one architecture; moving it across architectures means a rebuild. The extension path is native builds on two build servers joined with docker buildx imagetools create.
• No ECR, Google Artifact Registry or Azure ACR token refresh. Only registries with a static basic credential work. A long-lived access key through a registry that supports static credentials works; tokens that expire in hours do not.
• No registry-free transport. There is no docker save | load relay through the control plane; a build server needs a registry that both it and the targets can reach. Without a registry, build on the target.
• BYO registries are never cleaned up by NebulaCtrl, and a lifecycle rule can delete a rollback target.
• GHCR credentials are account-wide (classic tokens only); other registries scope as finely as their credentials allow.
• The build server is a shared privileged host for every project of the team; per-project builders isolate caches only (threat model).
• The managed registry needs a public hostname with a real certificate, and is one per team like every platform service (platform services).
• Provenance and SBOM attestations are off, so the registry holds a single plain manifest per release.
• Stacks, image services, databases and platform services do not use build servers.
• Plain-HTTP registries are accepted on loopback addresses only.
• Registry cache export (--cache-to type=registry) is not offered; the persistent per-project builder is what keeps builds fast.

API and audit#

Audit actions: registry.create, registry.update, registry.delete, registry.check, server.build_role, and deployment.build_server (which build server built a release). The deployment resource carries the build server's id next to the target's.