# NebulaCtrl documentation
> NebulaCtrl is a self-hosted, Docker-native platform-as-a-service. You connect your own Linux servers, deploy from Git or a container image, and manage variables, domains, databases, backups, logs and rollbacks from a canvas, a CLI or a REST API. A Go control plane with a Postgres database talks to a small agent on each Docker host; Traefik on each host routes traffic.
Every page of the NebulaCtrl docs in one file, in reading order. Each page starts with its title and its source URL. The API endpoint reference is not repeated here: it is generated from https://nebulactrl.com/api/v1/openapi.json.
---
# What is NebulaCtrl?
Source: https://nebulactrl.com/docs/getting-started/what-is-nebulactrl
NebulaCtrl is a self-hosted platform for running your apps on servers you own. You connect a Linux machine with Docker, point NebulaCtrl at a Git repository or a container image, and it builds, starts, routes and watches the result. There is no Kubernetes, no Swarm and no hosted service in the middle: one control plane, one small agent per server, and plain Docker underneath.
If you have used Heroku, Railway or Coolify, the shape is familiar. The difference is that everything runs on machines you control, and the interface is a canvas where you see your services and the lines between them.
NebulaCtrl is free to use and runs on your own servers. The source is coming soon, under the AGPL-3.0; until it is published, the installer, the container images and the CLI are all downloadable without an account: the installer from `nebulactrl.com`, the images from the public registry `ghcr.io/nebulactrl/nebulactrl`, the CLI from your own control plane.
## What you get
- **Deploys from Git or an image.** Push to a branch and deploy, or pull `ghcr.io/you/app:1.4`. A repository without a Dockerfile is built with Railpack. See [builds](https://nebulactrl.com/docs/operations/builds).
- **A deploy you can trust.** A new version is started and health-checked *before* traffic moves to it, and a failed deploy leaves the old one serving. See [deployments](https://nebulactrl.com/docs/operations/deployments).
- **Domains and HTTPS.** Every server runs a proxy that gets Let's Encrypt certificates for you. See [domains and proxy routes](https://nebulactrl.com/docs/operations/proxy-routes).
- **Variables and secrets** that are encrypted at rest, can refer to each other (`${{db.DATABASE_URL}}`), and are never shown again once saved. See [variables and secrets](https://nebulactrl.com/docs/operations/variables).
- **Databases and backups**: PostgreSQL, MySQL, MariaDB, Valkey and more, with scheduled backups. See [managed databases](https://nebulactrl.com/docs/operations/databases).
- **Previews, scaling, scheduled tasks, a web terminal**, and a catalogue of one-click [templates](https://nebulactrl.com/docs/operations/templates).
- **Several servers** in one place, including machines behind a home router: agents dial out, so no inbound port is needed for the control plane.
- **A CLI and an API** that do everything the dashboard does. See the [CLI reference](https://nebulactrl.com/docs/reference/cli) and the [API reference](https://nebulactrl.com/docs/reference/api).
## How the pieces fit
```
you ──► dashboard / CLI / API ──► control plane ──► Postgres
▲
outbound WebSocket from each agent
│
server A: agent ─ proxy ─ your containers
server B: agent ─ proxy ─ your containers
```
- The **control plane** is one Go program with a Postgres database. It holds your projects, settings and history, and tells agents what to run.
- An **agent** runs on each server. It builds images, starts containers, runs health checks and reports back. It connects *out* to the control plane, so servers behind NAT work.
- A **proxy** (Traefik) on each server terminates HTTPS and routes each domain to the right container. The agent manages it; you never edit its config.
The control plane can go down without stopping your apps. Containers keep running; you only lose the ability to change things until it is back.
## What it is not
NebulaCtrl is for a trusted team running its own infrastructure. A NebulaCtrl administrator is a root administrator of every connected server, because Docker access is root. It is not a way to host strangers' code safely. Read the [threat model](https://nebulactrl.com/docs/security/threat-model) and the [known limitations](https://nebulactrl.com/docs/operations/limitations) before you rely on it.
## Where next
1. [Learn the vocabulary](https://nebulactrl.com/docs/getting-started/concepts): projects, environments, services, servers and staged changes.
2. [Follow the quickstart](https://nebulactrl.com/docs/getting-started/quickstart): from a bare machine to a running app with a domain and a database.
---
# Concepts
Source: https://nebulactrl.com/docs/getting-started/concepts
Six ideas explain almost everything you see in NebulaCtrl.
## Team
A **team** is the top level: it owns the projects, servers, API tokens, integrations and audit log. When you set up NebulaCtrl you create the first team and its owner. People join a team by invitation and get a role, which decides what they can change. See [teams, roles and invites](https://nebulactrl.com/docs/operations/teams).
## Project
A **project** groups the services of one product, for example "Shop" with its web app, worker and database. Projects are what you switch between in the top bar.
## Environment
Every project has **environments** such as `production` and `staging`. An environment is a separate copy of the project's services, with its own variables, domains and data. Each one has its own private network, so `staging` cannot reach `production` by accident. The environment's name is in the top bar, and production is marked so you notice where you are. See [networking](https://nebulactrl.com/docs/operations/networking).
## Service
A **service** is one thing that runs: a web app, a worker, a database or a scheduled job. Its **source** is a Git repository (built into an image) or a ready-made Docker image. Services appear as cards on the **canvas**, and clicking one opens the **inspector** with tabs for Overview, Deployments, Variables, Backups, Metrics, Analytics, Logs and Settings.
The lines between cards are connections: they show which service talks to which, and connecting two cards proposes the right variable (for example `DATABASE_URL=${{pg.DATABASE_URL}}`) for you to confirm.
A database is a service too, created from **Create → Database**, with its credentials generated and its data kept on a volume. See [managed databases](https://nebulactrl.com/docs/operations/databases).
## Server
A **server** is a Linux machine with Docker and the NebulaCtrl agent. Each service is placed on one server. A server's card shows whether it is online, its CPU, memory and disk, and the state of its proxy. The machine you install NebulaCtrl on is your first server. See [servers and agents](https://nebulactrl.com/docs/operations/servers).
## Staged changes
Changing a service's settings, variables, domains or volumes does **not** restart it. The change is *staged*: the card shows a **Staged** chip, and a bar says "N changes to apply". Nothing about the running app changes until you deploy.
From that bar you can **Review** the exact differences (secret values stay hidden), **Discard** them, or **Deploy** to apply them together. Staging is what lets you change five things and restart once.
## Deployments and releases
A **deployment** is one attempt to get a service to its configured state. It moves through *preparing, building, releasing and verifying* and ends as `succeeded`, `failed`, `canceled`, `rolled_back` or `rollback_failed`. The words say what happened, and a failed one says at which step.
Each deployment is made from a **release**: a frozen record of the source (image digest or Git commit), the settings and the variables. You can **roll back** to any earlier release.
> [!NOTE]
> A rollback restores the application image and its configuration only. Databases, volumes and external services are not rolled back.
---
# Quickstart
Source: https://nebulactrl.com/docs/getting-started/quickstart
This walks you from nothing to an app on the internet with its own domain and a PostgreSQL database. Allow about twenty minutes, most of it waiting for downloads.
You need:
- A Linux machine you can SSH into as root or with `sudo`: Debian 12/13, Ubuntu 22.04, 24.04 or 26.04, or RHEL, Rocky or AlmaLinux 9, on amd64 or arm64. A small VPS with 2 GB of RAM is plenty. See [requirements](https://nebulactrl.com/docs/installing/requirements).
- Ports 80 and 443 free on it.
- Optional but recommended: a domain you control, with an A record pointing at the machine.
## 1. Install
On the machine, run the installer. With a domain for the dashboard:
```sh
curl -fsSL https://nebulactrl.com/install.sh | sudo bash -s -- --domain nebula.example.com --email you@example.com
```
Without a domain, run it plain and the dashboard is served over HTTP on port 8000, which is fine for trying things out:
```sh
curl -fsSL https://nebulactrl.com/install.sh | sudo bash
```
The installer is published at `https://nebulactrl.com/install.sh`, and every control plane serves the same script at its own `/install.sh`; see [install NebulaCtrl](https://nebulactrl.com/docs/installing). The installer sets up Docker if it is missing, generates every secret, starts the database and the control plane, and prints two things at the end: the **dashboard URL** and a **setup URL** with a one-time token. The control-plane image is public (`ghcr.io/nebulactrl/nebulactrl`), so nothing on your machine needs a GitHub or registry login.
> [!WARNING]
> Store `NEBULACTRL_SECRET_KEY` somewhere other than the machine. Without it, a restored backup cannot decrypt your variables. The installer prints the command to read it: `sudo grep NEBULACTRL_SECRET_KEY /opt/nebulactrl/.env`.
Open the setup URL, choose a team name, and create your account. You land on the projects page, and the machine you installed on is already listed under **Servers** as your first server.
## 2. Add another server (optional)
Skip this if one machine is enough: the first server can run your apps. To add a second host:
1. Open **Servers** and choose **Add server**.
2. Give it a name. If you know them, add its public IP, a base domain and an email for Let's Encrypt. The base domain is what generated domains hang off.
3. Choose **Create install command**, copy it, and run it on the new host. The host needs Docker 24 or newer already installed.
The dialog waits for the agent. When it connects, the server shows as online with its CPU, memory and disk. Details and troubleshooting are in [add a server](https://nebulactrl.com/docs/installing/servers).
## 3. Create a project
On the projects page choose **New project**, name it, and open it. A project starts with a `production` environment and an empty canvas that says "Create your first service".
## 4. Deploy your first app
We will use `traefik/whoami`, a tiny web server that prints information about the request it received. It is a good first app because it has nothing to configure.
1. On the canvas choose **Create**, then **Docker image**.
2. Enter `traefik/whoami:latest` as the image. The name fills in as `whoami`.
3. Pick the server to run it on, and set the port to `80`.
4. Choose **Create service**.
The card appears with a **Staged** chip, and the toast says nothing runs until you deploy. In production the toast offers **Review & deploy**; in other environments it offers **Deploy now**. Either way, the **Deploy · 1** button in the top bar applies your staged changes.
Watch the card: it goes through *Preparing*, *Releasing* and *Verifying*, and settles on a deployed state when the health check passes. Open the service and its **Deployments** tab to see each step and the log.
To deploy from Git instead, choose **Create → Git repository**. A repository with a Dockerfile uses it; one without gets a build plan from Railpack. See [builds](https://nebulactrl.com/docs/operations/builds) and [Git providers](https://nebulactrl.com/docs/operations/git-providers).
## 5. Give it a domain
1. Open the service, then **Settings → Networking**.
2. If you set a base domain or public IP on the server, choose **Generate domain**. It creates a hostname like `whoami-production-myproject. --wait` fits in a CI job. Your control plane serves it: `curl -fsSL https:///cli.sh | sh` installs the build that matches it (also under **Settings → API tokens**), then `nebulactrl login --url https:// --token `.
---
# Install NebulaCtrl
Source: https://nebulactrl.com/docs/installing
One command installs the control plane on a fresh machine, and the machine is the first server your apps can run on:
```sh
curl -fsSL https:///install.sh | sudo bash -s -- --domain nebula.example.com --email you@example.com
```
`` is where the installer is published: `https://nebulactrl.com/install.sh`, or `/install.sh` of any running control plane (the repository is private, so the script is not fetched from it).
Without `--domain` the dashboard is served on `http://:8000` (plain HTTP, with a warning) so you can try NebulaCtrl on a machine that has no domain yet:
```sh
curl -fsSL https:///install.sh | sudo bash
```
When it finishes it prints the dashboard URL and a **setup URL** with a one-time token. Open that, create the admin account, and deploy something.
Supported hosts: Debian 12/13, Ubuntu 22.04, 24.04 and 26.04, RHEL/Rocky/AlmaLinux 9; amd64 and arm64.
## What the installer does
It always does the following, in this order:
1. Checks the system (distribution, systemd, CPU, RAM, free ports) and installs the base packages it needs.
2. Installs **Docker Engine and the Compose plugin from Docker's official repository** if Docker is missing.
3. Applies **safe Docker daemon defaults**: log rotation, `live-restore`, `userland-proxy: false` and a default address pool that does not overlap the agent's.
4. **Generates every secret** (database password, the instance secret key, the setup token, the local agent's enrollment token) and keeps them in `/opt/nebulactrl/.env`, mode 0600.
5. Starts **Postgres 17 on a private, internal network** (SCRAM authentication, tuned to the machine's RAM, no published port) and the control plane in a hardened container.
6. **Serves the dashboard**: over HTTPS with a Let's Encrypt certificate when you give a domain, otherwise over plain HTTP on port 8000.
7. Schedules a **daily control-plane backup** to a local directory and reminds you to add an off-site destination.
8. Prints the setup URL.
These steps are also on by default; each has a flag to turn it off:
| Step | Default | Off with |
|---|---|---|
| Enroll this machine as the **first server** (an agent on the host, "deploy apps here") | on | `--no-local-server` |
| Firewall: SSH (detected port), 80, 443/tcp and 443/udp only (ufw, firewalld on RHEL) | on | `--no-firewall` |
| Automatic security updates (unattended-upgrades, dnf-automatic) | on | `--no-auto-updates` |
| fail2ban for SSH | on | `--no-fail2ban` |
| [Lynis](https://nebulactrl.com/docs/security/host-hardening) audit and a curated set of safe fixes | on | `--no-hardening` |
| Daily control-plane backup | on | `--no-backup` |
Every step is **idempotent**: re-run the installer to upgrade or repair. It never regenerates a secret that exists, and it **never changes your SSH configuration** in a way that could lock you out.
## Pages
1. [The installer](https://nebulactrl.com/docs/installing/installer): every option, every step, the files and ports it touches, HTTPS and the dashboard route, re-running and recovering
2. [Requirements and supported hosts](https://nebulactrl.com/docs/installing/requirements)
3. [The control plane](https://nebulactrl.com/docs/installing/control-plane): how it runs, the layout under `/opt/nebulactrl`, manual Compose install, first run
4. [Add more servers](https://nebulactrl.com/docs/installing/servers): the first server, enrollment tokens, the agent installer, "Accept workloads"
5. [Upgrade and roll back](https://nebulactrl.com/docs/installing/upgrade)
6. [Uninstall](https://nebulactrl.com/docs/installing/uninstall)
7. [Migrate from Coolify](https://nebulactrl.com/docs/installing/migrate-from-coolify): install beside Coolify on the same host, copy the data, cut over, roll back
Next, read [`docs/operations`](https://nebulactrl.com/docs/operations) (backups first) and [`docs/security`](https://nebulactrl.com/docs/security).
For a development setup (hot reload, dev agent, test database) see [`CONTRIBUTING.md`]() instead.
---
# Requirements
Source: https://nebulactrl.com/docs/installing/requirements
## Every host (control plane and servers)
| | Supported |
|---|---|
| OS | **Debian 12 and 13, Ubuntu 22.04, 24.04 and 26.04, RHEL, Rocky Linux and AlmaLinux 9** (systemd required) |
| CPU | **amd64** and **arm64** (`uname -m` = `x86_64` or `aarch64`) |
| Docker | **Docker Engine 24 or newer** from Docker's official repository; the control-plane installer installs it |
| Docker Compose plugin | **`docker compose` 2.24 or newer** (`docker-compose-plugin` from Docker's repository). The installer installs it with Docker, and installs or upgrades it when Docker is already there without it. Compose stacks need 2.24; servers without it refuse stack deploys |
| Access | root (the agent runs as root because the Docker socket is root-equivalent) |
| Network | Outbound HTTPS (Docker's repository, registries, git hosts, Let's Encrypt, and for servers the control plane) |
| Ports | **80/tcp, 443/tcp and 443/udp free** for the host's Traefik proxy, plus **9000 on 127.0.0.1** for its admin entrypoint (never exposed). If 80/443 are taken, the server shows a proxy error and adding domains fails with `proxy_unavailable`. `--proxy-http-port`, `--proxy-https-port` and `--proxy-admin-port` move them (agent installer) |
| Disk | Images, build cache and volumes live in Docker's data root. A build needs space for the clone, cache and the image. The server card warns when the root filesystem crosses its threshold (default 80 %) and the scheduled Docker cleanup keeps images and cache in check |
The installer refuses an unsupported distribution (`--force-unsupported` tries a Debian or RHEL family system anyway). macOS and Windows hosts are not supported; Docker Desktop is for development only.
Nothing needs to listen on a *server* for the control plane: the agent connects outbound, so hosts behind NAT or a home router work. Only the proxy's 80/443 must be reachable by your visitors (and by Let's Encrypt).
## The control-plane host
It is also a server (its agent runs the proxy that serves the dashboard), so everything above applies, plus:
| | Needed |
|---|---|
| CPU / RAM | 1 vCPU and **1 GB** for the control plane alone; plan for the apps you will deploy there too (turn **Accept workloads** off to keep the host for NebulaCtrl). Postgres is tuned to the RAM it finds |
| Disk | Postgres data and the local backups (`/var/backups/nebulactrl`); deployment logs are capped at 10,000 lines each and metrics are pruned ([monitoring](https://nebulactrl.com/docs/operations/monitoring)) |
| Network | Reachable by every agent and by the browsers that use the UI |
| DNS | For HTTPS: a hostname whose **A record points at this machine** before you run the installer with `--domain`. Without a domain the dashboard is plain HTTP on port 8000 |
| Time | NTP: enrollment tokens, sessions and the 30 second handshake window are time based |
The control plane is a **single instance**: no clustering, no high availability ([limitations](https://nebulactrl.com/docs/operations/limitations)).
## Domains for your services
- A wildcard or per-service DNS A record pointing at the *server's* public IP (the UI shows exactly what to create).
- Or no DNS at all: set the server's public IP and use **Generate domain**, which creates `--..sslip.io`.
- `*.localhost`, `*.local`, bare names and IP addresses get Traefik's default self-signed certificate (browsers warn; use `curl -k`).
- Certificates for other names come from Let's Encrypt via Traefik (HTTP-01 on port 80), or DNS-01 with a provider; set an ACME email on the server (the installer does for the first one).
---
# The installer
Source: https://nebulactrl.com/docs/installing/installer
`deploy/install/install.sh` installs the control plane on one machine and makes that machine the first server. It is a single bash script. The control plane serves a self-contained copy (libraries and files inlined) at `/install.sh`, so this works from any running instance too:
```sh
curl -fsSL https:///install.sh | sudo bash -s -- [options]
sudo bash deploy/install/install.sh [options] # from a checkout
```
Supported: **Debian 12/13, Ubuntu 22.04, 24.04 and 26.04, RHEL/Rocky/AlmaLinux 9**; amd64 and arm64; systemd; root. Anything else stops with a message (`--force-unsupported` tries a Debian or RHEL family system anyway).
`--check` runs the system and port checks, prints the plan and changes nothing.
## Two ways to serve the dashboard
| | With `--domain nebula.example.com --email you@example.com` | Without a domain |
|---|---|---|
| URL | `https://nebula.example.com`, Let's Encrypt certificate | `http://:8000` (plain HTTP) |
| Served by | the host's own proxy (see below) | the control-plane container, published on `0.0.0.0:8000` |
| Control-plane container binds | `127.0.0.1:8080` (only the host's agent uses it) | `0.0.0.0:8000` |
| Before you start | the domain's **A record must point at this machine** and 80/443 must be reachable from the internet (Let's Encrypt HTTP-01) | nothing |
Without a domain the installer ends with a notice: passwords and sessions travel unencrypted, and servers you add from other machines need `--insecure-dev`. Use it on a trusted network and switch soon. Switching is a re-run with `--domain`: secrets, data and the agent identity are kept, the dashboard moves behind the proxy, and port 8000 stops being published.
The installer warns when the domain does not resolve yet, or resolves to another address than this machine's. The certificate is issued as soon as DNS is right. `--public-ip` overrides the detected address (it is also what **Generate domain** uses, `--..sslip.io`).
### Single proxy on the control-plane host
The control-plane host always runs an agent, and **that agent's Traefik is the only proxy on the machine**. The compose file has no Traefik of its own. In domain mode the control plane adds one *system route* to the route table of that server: the dashboard's hostname, automatic Let's Encrypt (the server's ACME email is the one you gave), to `nebulactrl:8080` on the `nebulactrl-edge` network. It is part of every route push, deploy and reconcile of that server, so no deployment can displace it, and no service can take the dashboard's hostname (`409 domain_conflict`). It appears in the server's proxy route table as "NebulaCtrl dashboard".
Because the host's agent enrolls **before** the admin account exists (the installer creates the first team and hands the agent its token; **Setup** then adopts that team, and you become its owner), the route and the certificate are in place before you open the setup link, so the setup page already loads over HTTPS.
If the dashboard route breaks, the dashboard is still reachable on `http://127.0.0.1:8080` from the machine itself (for example through `ssh -L 8080:127.0.0.1:8080`).
With `--no-local-server` there is no agent and no proxy: in domain mode the container stays on `127.0.0.1:8080` and you provide the HTTPS proxy (it must pass WebSocket upgrades on `/agent/v1/connect` and must not buffer `text/event-stream`).
### Accept workloads
The first server has a setting **Accept workloads** (Servers → the server → Settings → Scheduling; `acceptWorkloads` in the API), **on by default**: you can deploy apps next to NebulaCtrl. Turn it off to keep the machine for NebulaCtrl alone. Off refuses new services and moves onto it (`422 server_not_accepting_workloads`, also in the drain planner, template and Compose-import checks and the placement pickers) and leaves what already runs untouched. Every server has the setting; the installer's host is the one marked "control plane".
### Dashboard access: allow list and DNS-01
In domain mode the dashboard is one route on the host's proxy, and two options shape it:
```sh
... --domain nebula.example.com --email you@example.com \
--dashboard-allow 100.64.0.0/10,fd7a:115c:a1e0::/48 \
--dashboard-dns01 cloudflare --dns-token-file /root/cf-token
```
- `--dashboard-allow ` puts an **IP allow list** on the dashboard route (Traefik `ipAllowList`, matched against the address of the TCP peer, which is the visitor when nothing sits in front of the proxy). The example above is "tailnet only". Addresses and CIDRs, IPv4 and IPv6, at most 200 entries; `off` removes the list. Everything else gets `403` from the proxy, including the setup link, so open the setup URL from an allowed address.
- `--dashboard-dns01 cloudflare` issues the dashboard's certificate with **DNS-01**, which needs no inbound connection: it works when the name resolves to a tailnet or LAN address, or when another proxy holds port 80 (see [side by side](#side-by-side-with-another-proxy)). `--dns-token-file ` names a file that holds the Cloudflare API token (Zone:Read and DNS:Edit on the zone). The token is **read from the file, never from the command line, and never printed**; the installer keeps it in `.env` (0600) and the control plane seals it into the DNS-01 credential store (the same one as Servers → Proxy → DNS-01) the first time it starts. Delete the file afterwards; rotate the token later in the DNS credential, not by re-running the installer.
- Both options need `--domain` (without a domain there is no HTTPS route; use `--bind` below) and the local server.
- They **seed** the settings once. Afterwards an instance admin edits them in **Settings → Instance → Dashboard access**: the allow list (the dashboard refuses to save a list that excludes your current address unless you confirm) and the certificate method. Changes are audited. Re-running the installer with different values does not overwrite what you changed there.
`--dashboard-dns01` supports Cloudflare in the installer. For another provider, add its credential and resolver under Servers → Proxy → DNS-01 after the install, then choose it in Dashboard access.
### Binding the dashboard to one address
Without a domain, `--bind ` publishes the dashboard port on that local address only, for example the Tailscale address: `--bind 100.122.182.103 --port 8100`. The dashboard URL becomes `http://:`. The host's own agent still connects over `127.0.0.1`: the installer adds `docker-compose.loopback.yml` (merged by `COMPOSE_FILE` in `.env`) that publishes the same port on loopback. The address is remembered in `install.conf`; an address you put into `.env` by hand is adopted on the next run instead of being reset to `0.0.0.0`. `--bind 0.0.0.0` goes back to every address. `--bind` must be an address of this machine; with `--domain` it is ignored (the container listens on `127.0.0.1` behind the proxy).
Docker-published ports bypass ufw/firewalld, so the bind address is the protection, not the firewall.
### Side by side with another proxy
NebulaCtrl's Traefik normally owns ports 80 and 443. To run it next to another proxy (the typical case: Coolify's Traefik while you migrate, see [migrate from Coolify](https://nebulactrl.com/docs/installing/migrate-from-coolify)), give it other ports:
```sh
... --port 8100 --proxy-http-port 8080 --proxy-https-port 8443 [--proxy-admin-port 9100]
```
- `--proxy-http-port`, `--proxy-https-port` and `--proxy-admin-port` (loopback only, default 9000) go to the local agent's installer, which writes them into the agent's systemd unit. Apps answer on `https://:8443` until the proxy moves.
- With a domain the dashboard URL carries the HTTPS port (`https://nebula.example.com:8443`). `--port` is then the control plane's loopback port (default 8080; the installer picks 8081, and so on, when the proxy sits on 8080), and `--port` is the dashboard port in IP mode. Coolify's dashboard uses 8000, so pass `--port 8100` there.
- **Certificates: use DNS-01.** Let's Encrypt HTTP-01 needs port 80 on the internet, which the other proxy holds. The installer warns when the dashboard would use HTTP-01 with the proxy off port 80; use `--dashboard-dns01`. The same is checked for app domains (the plan and the domain list warn "HTTP-01 cannot validate"): give them a DNS-01 resolver or a wildcard certificate under Servers → Proxy.
- The firewall step opens the configured ports. On a host that runs another proxy, pass `--no-firewall` (a first ufw activation would close the ports the other proxy needs).
**Moving the proxy back to 80/443.** Stop the other proxy first (`docker stop coolify-proxy`), then either
1. re-run the installer with the default ports: `curl -fsSL http://127.0.0.1:8100/install.sh | sudo bash -s -- --proxy-http-port 80 --proxy-https-port 443`. The agent's unit is rewritten, the agent restarts, and Traefik is recreated on the new ports (about a second of dropped connections). The installer checks that 80/443 are free first and refuses otherwise, then waits for the proxy to publish them; or
2. as an instance admin, in **Servers → the server → Proxy → Proxy ports**: review the plan, confirm to stage it, then **Deploy proxy** applies it by recreating Traefik ([proxy ports](https://nebulactrl.com/docs/operations/proxy-ports)). Use Reset to hand control back to the agent's own flags.
A port set in Proxy ports **overrides** the installer's flags, so after using it, change ports there (or press Reset) rather than by re-running the installer. In domain mode the dashboard URL (`NEBULACTRL_PUBLIC_URL`) includes the HTTPS port, so after moving back re-run the installer (step 1) to rewrite it, or use the old URL until you do: the route is the same.
## Options
| Option | Meaning |
|---|---|
| `--domain ` | Serve over HTTPS with Let's Encrypt |
| `--email ` | Let's Encrypt contact; required with `--domain` |
| `--public-ip ` | This machine's public IPv4 address (detected when omitted) |
| `--port ` | Dashboard port without a domain (default 8000). With a domain: the control plane's loopback port for the host's agent (default 8080) |
| `--bind ` | Without a domain: publish the dashboard on this local address only (for example the Tailscale IP). Remembered |
| `--proxy-http-port `, `--proxy-https-port `, `--proxy-admin-port ` | Host ports of the local server's proxy (default 80, 443, 9000), to run beside another proxy. Remembered |
| `--dashboard-allow ` | Allow list on the dashboard route (needs `--domain`); `off` removes it |
| `--dashboard-dns01 cloudflare`, `--dns-token-file ` | DNS-01 certificate for the dashboard; the token is read from the file and never printed |
| `--version `, `--image [` | Image to run. The default is the version of the control plane that served the script (`curl ]/install.sh`), or `latest` when the script was fetched from nebulactrl.com: `ghcr.io/nebulactrl/nebulactrl:` (a public image; nothing needs `docker login`) |
| `--image-archive ` | `docker load` the image from a `docker save` archive instead of pulling it |
| `--no-docker-install` | Fail instead of installing Docker |
| `--no-daemon-defaults` | Leave `/etc/docker/daemon.json` alone |
| `--no-local-server` | No agent on this machine |
| `--no-backup` | No daily control-plane backup |
| `--no-firewall` / `--no-fail2ban` / `--no-auto-updates` / `--no-hardening` | Skip that step |
| `--restart-docker` | Restart Docker to apply `daemon.json` even when containers would stop |
| `--dir`, `--backup-dir` | `/opt/nebulactrl`, `/var/backups/nebulactrl` |
| `--force-unsupported`, `--check` | see above |
| `--force-upgrade` | go on with an upgrade although the control-plane backup taken first failed (without it a failed backup stops the installer before anything is changed) |
| `--uninstall …` | run the uninstaller ([uninstall](https://nebulactrl.com/docs/installing/uninstall)) |
Options are remembered in `/opt/nebulactrl/install.conf`, so a plain re-run keeps your choices; flags override them.
> [!NOTE]
> Every release publishes the control-plane image to `ghcr.io/nebulactrl/nebulactrl` (public) for linux/amd64 and linux/arm64, tagged `X.Y.Z`, `X.Y` and `latest`. Before the first release exists, build one from a checkout (`docker build -t nebulactrl:local .`) and use `--image nebulactrl:local` (same machine) or `docker save nebulactrl:local -o nebulactrl.tar` and `--image nebulactrl:local --image-archive nebulactrl.tar` (another machine). The installer's own tests do exactly this.
## What each step does
Every step is idempotent and safe to run again. The installer prints `ok`, `warn` and what it did; warnings are repeated at the end. A failure stops the run and says in which step; fix the cause and run it again.
1. **System checks.** Root, Linux, supported distribution, systemd, architecture, RAM (warns below 900 MB), free disk (warns below 3 GB), and that the proxy's ports (80/443 unless `--proxy-http-port`/`--proxy-https-port` say otherwise) and the dashboard port are free (unless NebulaCtrl itself holds them; on a re-run only ports the proxy does not publish yet are checked). Every option is validated here: CIDR lists, ports, `--bind` (an address of this machine), the token file. Missing `curl`, `ss`, `base32`, CA certificates and `gpg` are installed.
2. **Docker.** If `docker` is missing, Docker Engine, the Compose plugin and buildx are installed from Docker's official repository (`download.docker.com`, signed key in `/etc/apt/keyrings` or the `.repo` file for dnf; Rocky and Alma use Docker's CentOS repository). Distribution packages that conflict (`docker.io`, `podman-docker`, …) stop the run with an instruction rather than being removed. An existing Docker is kept: it must be **24 or newer** and have the **Compose plugin** (installed from Docker's repository when only that is missing).
3. **Daemon defaults** (`/etc/docker/daemon.json`): `log-driver json-file` with `max-size 10m` and `max-file 3`; `live-restore: true`; `userland-proxy: false`; `default-address-pools` `10.212.0.0/16` in /24s (the agent allocates its environment networks from `10.213.0.0/16`, so the two never collide). **Existing keys always win**: a key you already set is kept and reported, only missing keys are added, and the original is saved as `daemon.json.nebulactrl-bak`. `userland-proxy` and the address pool need a Docker restart: on a fresh Docker it just happens; when containers already run **without** live-restore it is **not** done (they would stop) and the installer tells you to restart Docker in a maintenance window, or pass `--restart-docker`.
4. **Edge network.** `nebulactrl-edge` is created with the label the agent uses (the agent would create it, but the control plane joins it).
5. **Files and secrets** in `/opt/nebulactrl`: `docker-compose.yml`, `.env` (0600), `install.conf`, `uninstall.sh`, `state/`. Generated once, never regenerated: the Postgres password (48 hex), `NEBULACTRL_SECRET_KEY` (32 random bytes, base64), the setup token, and the local agent's enrollment token. If a database volume from an earlier install exists but `.env` is gone, the installer **refuses** to invent a new key and says so. Postgres settings (`PG_*`) are recomputed from the RAM on every run; unchanged RAM means unchanged file means no restart.
6. **Control plane and Postgres.** Postgres 17 runs on an **internal Docker network** (no route out, no published port, reachable only by the control plane), initialised with SCRAM-SHA-256 for local and host connections and data checksums, capabilities dropped. Tuning from RAM: `shared_buffers` 15 % (128 MB–2 GB, the host also runs apps), `effective_cache_size` 40 %, `work_mem` and `maintenance_work_mem` scaled, `random_page_cost` 1.1 on SSD/NVMe, WAL compression, an idle-in-transaction timeout. The control plane runs read-only, with all capabilities dropped, `no-new-privileges` and without the Docker socket. The installer waits for both to be healthy and for `/readyz`.
7. **Daily control-plane backup.** The instance backup under Settings → Instance is switched on: daily 03:30 UTC, 14 kept, to the **local directory** `/var/backups/nebulactrl/control-plane` (owner uid 65532, mode 0700), with a first run right away. It is only seeded while nobody configured it. The secret key is never in a backup. A local copy is not off-site: the installer and the UI both say so, add an S3-compatible bucket.
8. **Local agent.** The agent binary comes from the control plane (`/install-agent.sh`, the same installer other servers use) and runs as the systemd unit `nebulactrl-agent`, dialing `http://127.0.0.1:`. (The agent accepts plain HTTP only for loopback addresses.) A drop-in sets `RestartSec=12` so that early restarts, before the control plane accepts the token, stay under the enrollment rate limit. Re-runs upgrade the binary and keep the identity.
9. **Firewall.** SSH first, then 80/tcp, 443/tcp, 443/udp, and the dashboard port in IP mode. ufw on Debian/Ubuntu (default deny incoming, allow outgoing, only set when ufw was inactive), firewalld on RHEL family. The SSH ports are the ones `sshd -T` reports, the one your session came in on (`$SSH_CONNECTION`) and anything sshd listens on; rules are added **before** the firewall is switched on. Where the kernel has no firewall support (some containers) the step warns and continues. **Docker-published ports bypass ufw/firewalld rules**; NebulaCtrl never publishes service ports on its own, and TCP/UDP routes you create carry mandatory allowlists.
10. **fail2ban** for sshd (`/etc/fail2ban/jail.d/nebulactrl-sshd.local`: 5 failures in 10 minutes, 1 hour ban, systemd journal backend; EPEL on RHEL family).
11. **Automatic security updates**: `unattended-upgrades` (`20auto-upgrades`) or `dnf-automatic` set to `upgrade_type = security`, `apply_updates = yes`. No automatic reboot. Docker's packages are not security-origin, so Docker is not upgraded behind your back.
12. **Lynis hardening.** `lib/hardening.sh` runs a Lynis audit, applies a curated set of **safe** remediations, re-audits, and prints the hardening index with what remains; the agent repeats it weekly and the server's **Security** tab shows the result. It never touches SSH authentication, root login or firewall rules. It only warns when it fails. See [host hardening](https://nebulactrl.com/docs/security/host-hardening).
13. **Summary:** dashboard URL, the setup URL with the one-time token, where the files and backups are, the secret-key reminder (`sudo grep NEBULACTRL_SECRET_KEY /opt/nebulactrl/.env`: store it apart from the backups), the off-site backup reminder and every warning.
**SSH is never modified.** The installer reads sshd's configuration to find the ports to allow and changes nothing in it; the test suite checks that `sshd_config*` is byte-identical after an install, a re-run and an uninstall.
## First run
Open the **setup URL** the installer printed (`/setup?token=…`). The page needs the token because the installer sets `NEBULACTRL_SETUP_TOKEN`: without it nobody who finds the dashboard first can claim the admin account (`403 setup_token_invalid`; attempts are rate limited). The token is removed from the address bar after the page reads it and stops mattering when setup is done. It is stored in `/opt/nebulactrl/.env` and `INSTALL-SUMMARY.txt`. Choose your team name, create the account, and the first server (this machine) is already online.
## Files, ports and containers
| | |
|---|---|
| `/opt/nebulactrl/` | `docker-compose.yml`, `.env` (every secret, 0600), `install.conf`, `uninstall.sh`, `state/`, `INSTALL-SUMMARY.txt` |
| `/var/backups/nebulactrl/control-plane/` | local control-plane backups |
| `/usr/local/bin/nebulactrl-agent`, `nebulactrl-agent.service` (+ `.service.d/10-installer.conf`), `/etc/nebulactrl-agent.env` | the local agent |
| `/var/lib/nebulactrl-agent/` | agent identity, build workspaces, log history |
| `/etc/docker/daemon.json` (+ `.nebulactrl-bak`), `/etc/apt/keyrings/docker.asc`, `docker.list` / `docker-ce.repo` | Docker |
| containers `nebulactrl-nebulactrl-1`, `nebulactrl-postgres-1`; networks `nebulactrl_backend` (internal), `nebulactrl-edge`; volume `nebulactrl_postgres-data` | the control plane |
| containers `nebulactrl-traefik`, `nebulactrl-pages`, volume `nebulactrl-traefik-acme` | the agent's proxy |
| Listening on the host | 80, 443 (tcp, udp) the proxy, or the `--proxy-http-port`/`--proxy-https-port` you chose; `127.0.0.1:9000` the proxy's admin; `127.0.0.1:8080` (domain mode, `--port` changes it) or `0.0.0.0:8000` (IP mode, `--bind` narrows it) the dashboard; SSH. **Nothing else: Postgres is not published.** |
## Re-running, upgrading, recovering
- **On an existing installation the installer backs the control plane up first** (`nebulactrl-server pre-upgrade-backup` in the running container; a `pg_dump` to `/var/backups/nebulactrl/pre-upgrade-.dump` when that control plane is too old to have the command) and stops before changing anything if the backup failed, unless `--force-upgrade`. See [upgrades](https://nebulactrl.com/docs/operations/upgrades#backup-before-a-control-plane-upgrade).
- `curl … | sudo bash` again, or `sudo bash /opt/nebulactrl/…` with the same options: nothing that exists is regenerated; changed settings are applied; containers are only recreated when their definition changed (a new image tag, new RAM tuning); the agent binary is upgraded to the control plane's version. See [upgrade](https://nebulactrl.com/docs/installing/upgrade).
- A step failed: fix the cause and run again. A power loss or a killed run leaves nothing a re-run does not repair.
- `.env` lost but the database volume remains: restore `.env` from your backup (the installer will not invent a key), or [uninstall with `--purge`](https://nebulactrl.com/docs/installing/uninstall) and restore the database from a backup.
- **Locked out** (lost password or lost 2FA of the only Owner): `cd /opt/nebulactrl && docker compose exec nebulactrl nebulactrl-server reset-password you@example.com` prints a one-time reset link (30 minutes); `reset-2fa you@example.com` removes the authenticator app, passkeys and recovery codes after you type the email. Both run on the host against the database, end the account's sessions and are audited as system actions. If setup was never completed there is no account to reset: use the setup URL instead. See [Troubleshooting → Locked out](https://nebulactrl.com/docs/operations/troubleshooting#locked-out).
- The host's agent identity from an earlier installation is moved aside (`/var/lib/nebulactrl-agent.old-`) when a fresh control plane is installed over it.
## Tested
`tests/install/run.sh` runs the real installer in disposable systemd machines (OrbStack) for Debian 12, Ubuntu 22.04, 24.04 and 26.04, and Rocky 9: install through the piped, self-contained script, secrets absent from its output, Postgres private and SCRAM-only, the daily backup written, firewall, fail2ban and updates configured; a re-run that changes nothing; setup with and without the token and enrollment of the host's agent before the admin exists; the switch to a domain with the dashboard served through the host proxy (HTTP→HTTPS redirect, WebSocket upgrade, login through the proxy); an app deployed on the control-plane host and **Accept workloads** off refusing the next one while the first keeps serving; uninstall (data kept), reinstall over the kept data, and `--purge`. See [`tests/install/README.md`](). The audit/hardening step is exercised when `NC_TEST_HARDENING=1`.
---
# Migrate from Coolify
Source: https://nebulactrl.com/docs/installing/migrate-from-coolify
This page moves **one host** from Coolify 4.x to NebulaCtrl without a second server. Both run side by side on the host while you rebuild and rehearse; Coolify keeps serving the whole time. A short **cutover window** (minutes) then moves ports 80/443 to NebulaCtrl, and Coolify stays stopped but intact for a soak period, which is also your rollback.
If you can afford a second machine, installing NebulaCtrl there, copying the data and switching DNS is simpler and needs no downtime. This page is for the single-host case.
## What changes hands
| Coolify | NebulaCtrl |
|---|---|
| `coolify-proxy` (Traefik) on 80/443, dashboard on `:8000` | NebulaCtrl's own Traefik (run by its agent), first on spare ports and then on 80/443; dashboard on a port you pick (`--port 8100` below) or behind its own domain |
| Applications (Dockerfile, image, Git) | Services; see the [Coolify importer](#2-recreate-the-projects) |
| Databases | Managed databases (Postgres, PostGIS, Valkey and others); see [restoring a dump](#3-rehearse-copy-the-data-and-test-on-the-spare-ports) |
| Per-domain rules (IP allow lists, compression) | Per-domain access and compress options |
| Daily database backup | Scheduled backups to S3, off the host's disk |
| Dynamic Traefik files (an external upstream such as Plex on a tailnet node) | An external-service route; by hand |
## 0. Before you start
- A way to **log in over the network that survives the cutover**: SSH is untouched by NebulaCtrl's installer, and Tailscale (if you use it) is separate software.
- List what holds the ports you care about: `ss -ltnp | grep -E ':(80|443|8000|8100|8080|8443)\b'`. Coolify uses 80/443 (proxy), 8000 (dashboard) and 6001/6002 (realtime). The commands below use **8100** for NebulaCtrl's dashboard and **8080/8443** for its proxy.
- Cloudflare: while HTTP-01 cannot work (see below), every certificate is issued with **DNS-01**. Create an API token with **Zone:Read** and **DNS:Edit** on the zones of your domains, and keep it in a file readable by root only. If your DNS is Cloudflare-proxied, also connect Cloudflare in NebulaCtrl (Settings, Integrations) so it owns the records and can lock the origin; see [Cloudflare](https://nebulactrl.com/docs/operations/cloudflare).
- Keep the Coolify host's **Coolify API token** (read, and read-sensitive if you want the secrets imported) for the importer.
- Know your cutover window: the apps are down for the time between stopping Coolify and the first successful request through NebulaCtrl (plan 5 to 10 minutes).
## 1. Install NebulaCtrl side by side
Run this on the Coolify host. It does not touch ufw, fail2ban or the hardening settings, and it leaves ports 80, 443 and 8000 to Coolify:
```sh
echo -n '' | sudo install -m 0600 /dev/stdin /root/cf-token
curl -fsSL https://nebulactrl.com/install.sh | sudo bash -s -- \
--no-firewall --no-fail2ban --no-hardening \
--port 8100 --proxy-http-port 8080 --proxy-https-port 8443 \
--domain nebula.example.com --email you@example.com \
--dashboard-allow 100.64.0.0/10,fd7a:115c:a1e0::/48 \
--dashboard-dns01 cloudflare --dns-token-file /root/cf-token
```
(`/install.sh` of any running NebulaCtrl control plane serves the same script. Add `--no-auto-updates` too if you do not want unattended upgrades on this host.)
What each part does:
| Option | Why |
|---|---|
| `--no-firewall --no-fail2ban --no-hardening` | Keep the host as it is. Without `--no-firewall` the installer would switch on ufw, which also closes ports Coolify needs (6001/6002, 8000) because it has no tailnet rules |
| `--port 8100` | With a domain this is the control plane's **loopback** port that the host's agent uses (it is 8080 by default, which is now the proxy's). Without a domain it is the dashboard's port; Coolify's dashboard holds 8000 |
| `--proxy-http-port 8080 --proxy-https-port 8443` | NebulaCtrl's Traefik runs next to Coolify's. Apps answer on `https://:8443` until you move it |
| `--domain`, `--email` | The dashboard's own HTTPS name. Its URL carries the port: `https://nebula.example.com:8443` |
| `--dashboard-allow ` | Only these addresses may open the dashboard. `100.64.0.0/10` and `fd7a:115c:a1e0::/48` are the tailnet ranges |
| `--dashboard-dns01 cloudflare --dns-token-file …` | The dashboard certificate comes from DNS-01. It works while the name resolves to a tailnet address and while another proxy holds port 80. The token is read from the file, never printed, and sealed into NebulaCtrl's credential store |
**Without a domain**, tailnet only: replace the domain, allow list and DNS-01 options with `--bind --port 8100`. The dashboard is then `http://:8100` (plain HTTP over the tailnet), published on that address only and remembered by later runs. See [the installer](https://nebulactrl.com/docs/installing/installer#binding-the-dashboard-to-one-address).
Point `nebula.example.com` at the host's **tailnet** address (A record). Open the setup link the installer prints from a tailnet device; the allow list also blocks the setup page from everywhere else. The installer warns about HTTP-01 where it could not work, and the first server (this host) is already enrolled: Servers, the host, **Proxy** shows its ports as 8080/8443.
Check that nothing of Coolify's moved: `curl -sI http://127.0.0.1:80/`, your sites, and Coolify's dashboard on `:8000` all answer as before.
## 2. Recreate the projects
Pre-issue certificates **before** any DNS change: for each domain set the DNS-01 resolver (`cloudflare`) or add a wildcard certificate under Servers, Proxy, DNS-01. HTTP-01 cannot validate while Coolify owns port 80; NebulaCtrl warns on every domain that would use it ("HTTP-01 cannot validate"). See [DNS-01 and wildcard certificates](https://nebulactrl.com/docs/operations/dns01-wildcard-certificates).
### Importer
Import projects, applications, variables, domains and databases from the Coolify API. The **control plane** (not your laptop) must reach the Coolify URL; a tailnet address is fine. The token needs the Coolify abilities `read` and `read:sensitive` (the second one is what lets secrets come across). Review first, then stage; nothing is deployed unless you pass `--deploy`:
```sh
export COOLIFY_TOKEN=
nebulactrl import coolify --url https://coolify.example.com --plan # review (alias --dry-run; exit 1 on blockers; --json)
nebulactrl import coolify --url https://coolify.example.com --server \
[--registry-user ghcr.io= --registry-token-env ghcr.io=] [--git-connection ] \
[--set-env .=]
```
It stages projects and environments, image and Git services, managed databases of the same engine and version, **every** domain of each application, health checks, limits, Docker options, pre- and post-deploy commands, volumes (empty) and variables with secrets marked and database references rewritten to `${{db.*}}`. **Stays manual:** the data of databases and volumes (the importer prints the `databases restore` command to use), registry credentials unless you pass them, private-repository access (a GitHub App or deploy key cannot be exported), locked variable values unless you supply them with `--set-env`, DNS, Traefik labels and dynamic files (an external Plex route, per-domain IP allow lists, redirects), scheduled tasks, one-click services and host port mappings. More flags (`--exclude`, `--rename`, `--domain`, `--secret`, `--skip-var`, ...) and the full mapping are on the importer's own page: [import from Coolify](https://nebulactrl.com/docs/operations/import-coolify).
Then: review every staged service in the dashboard (variables marked secret, health checks, the `entrypoint` and pre-deploy command overrides your Coolify apps used), create the managed databases **empty**, and add an S3 backup destination and schedule.
## 3. Rehearse: copy the data and test on the spare ports
Copy each database with `pg_dump` from Coolify's container into the new managed database, then deploy the apps on the spare ports and test with `curl --resolve` so no DNS has to change:
```sh
curl -sk --resolve app.example.com:8443:127.0.0.1 https://app.example.com:8443/
```
### Restoring a dump
Dump from Coolify's database container (the apps' writers stopped for the final one) and restore into a managed database, by default as a **new sibling database** so the original stays untouched:
```sh
docker exec pg_dump -U -Fc > db.pgdump
nebulactrl databases restore --file db.pgdump
```
The upload streams through the control plane to the agent, with no temporary copy; `--no-owner` and `--no-acl` are applied for you. In the dashboard: the database, Backups, "Restore from a dump file". Details, limits and `--in-place` are in [restore from a dump file](https://nebulactrl.com/docs/operations/database-backups#restore-from-a-dump-file) (that page is the reference; this section is short on purpose). Do the rehearsal restore as often as you like; only the final one matters.
Cache and queue stores such as Valkey/Redis usually start **empty** after the queue drains; copy them only when you know they hold state you cannot rebuild.
Rehearse everything: logins, background jobs, scheduled tasks, uploads, third-party webhooks that would hit a different host name, the external routes.
## 4. The cutover window
Pick the window when traffic is lowest and tell people. Have a second SSH session open.
1. **Stop the writers.** Stop the queue workers and let the queues drain, then stop the Coolify applications (Coolify, or `docker stop `).
2. **Final data copy.** Take a last `pg_dump` and restore it into the managed database (as above, with the apps stopped). Point the new services at the new databases if they are not already.
3. **Stop Coolify's proxy.** Stop Coolify itself first, because it can start its proxy again, then the proxy: `docker stop coolify coolify-realtime`, then `docker stop coolify-proxy`. Confirm 80 and 443 are free: `ss -ltn | grep -E ':(80|443)\b'` prints nothing.
4. **Move NebulaCtrl's proxy to 80/443.** Either re-run the installer with the default ports,
```sh
curl -fsSL http://127.0.0.1:8100/install.sh | sudo bash -s -- --proxy-http-port 80 --proxy-https-port 443
```
or, as the instance admin, set the ports in **Servers, the host, Proxy, Proxy ports** (review the plan, confirm, then **Deploy proxy**; Reset hands control back to the agent's flags; see [proxy ports](https://nebulactrl.com/docs/operations/proxy-ports)). Traefik is recreated in about a second. The installer refuses while something still holds 80 or 443 and changes nothing in that case. A port set there overrides the installer's flags. See [moving the proxy back to 80/443](https://nebulactrl.com/docs/installing/installer#side-by-side-with-another-proxy).
5. **Switch DNS** where it has to change. A Cloudflare-proxied domain needs no change (the origin is the same host); a name that pointed at the tailnet address stays as it is.
6. **Smoke-test every host name and the external routes**, then start the writers (workers) on NebulaCtrl. Check the dashboard on `https://nebula.example.com` (the installer rewrote its URL without the port).
## 5. Roll back
If something is wrong, you can be back on Coolify in about a minute, because Coolify's data and containers are untouched:
```sh
sudo systemctl stop nebulactrl-agent # stops NebulaCtrl's management
sudo docker stop nebulactrl-traefik # releases 80/443
sudo docker start coolify coolify-realtime coolify-proxy # and your Coolify apps
```
Data written to NebulaCtrl's databases since the cutover is not in Coolify's: if that matters, `pg_dump` the new database and restore it into Coolify's before starting the apps. To retry NebulaCtrl later, re-run the installer with `--proxy-http-port 8080 --proxy-https-port 8443` (it restarts the agent and puts the proxy on the spare ports).
## 6. Soak, then clean up
Run for about **7 days** with Coolify stopped but intact. Then take a final backup of the NebulaCtrl instance (Settings, Instance; the secret key apart from it) and a last Coolify backup, and remove Coolify: its containers, images and volumes (`docker system prune` after listing what it removes) and, if you want it, turn on the firewall and the hardening that you skipped (`--no-firewall` and `--no-hardening` are only install-time options: re-run the installer without them).
## What to expect
- Coolify's custom Traefik files (`/data/coolify/proxy/dynamic/*.yaml`) are not in its API: recreate them as external routes and per-domain access rules.
- Imported HTTP health checks run Coolify's own request **inside the container** (`curl`, else `wget`, against `http://localhost:`), as Coolify did: frameworks with a host allow list (Laravel trusted hosts, Django `ALLOWED_HOSTS`) refuse a probe from outside, whose Host is the container address. Apps without a port deploy as recreate with a short gap.
- Compression, IP allow lists and redirects are options per domain, not per proxy.
- Plan the secrets: variables the importer cannot read stay empty until you fill them in, and `APP_KEY`-style secrets must keep their exact values to keep existing sessions and encrypted data valid.
---
# The control plane
Source: https://nebulactrl.com/docs/installing/control-plane
The control plane is one container (`nebulactrl-server`, with the web UI embedded) and Postgres 17, started by Docker Compose from [`deploy/install/docker-compose.yml`](). **The [installer](https://nebulactrl.com/docs/installing/installer) sets this up; you do not write the files yourself.** This page explains what it made and how to run the same thing by hand.
## Layout
```text
/opt/nebulactrl/
docker-compose.yml written by the installer; edit the repository's copy, not this one
.env every secret and setting, mode 0600
install.conf the installer's remembered options
uninstall.sh
state/ markers (which steps ran)
/var/backups/nebulactrl/control-plane/ local control-plane backups
```
```sh
cd /opt/nebulactrl
docker compose ps
docker compose logs -f nebulactrl # JSON logs; migrations are applied on startup
curl -fsS http://127.0.0.1:8080/readyz # domain mode; http://127.0.0.1:8000 without a domain
```
`.env` holds:
| Variable | Meaning |
|---|---|
| `POSTGRES_PASSWORD` | Database password (hex, because it is embedded in the connection URL) |
| `NEBULACTRL_SECRET_KEY` | Base64 of 32 random bytes. Encrypts every sealed value: variables, git tokens, registry credentials, SSH keys, integration tokens, TOTP seeds, backup passwords (AES-256-GCM). **Exists nowhere else: back it up apart from the database backups.** Rotate it with `nebulactrl-server rekey`, never by replacing it ([secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest)). Alternatives: `NEBULACTRL_SECRET_KEY_FILE` (path to a file holding the key, keeps it out of `docker inspect`) and `NEBULACTRL_SECRET_KEY_PREVIOUS[_FILE]` (earlier keys, during a rotation) |
| `NEBULACTRL_PUBLIC_URL` | The URL users and agents use. An `https://` value turns on the `Secure` session-cookie attribute and is baked into agent install commands |
| `NEBULACTRL_BIND`, `NEBULACTRL_PORT` | Host interface and port the container publishes (`127.0.0.1:8080` with a domain, `0.0.0.0:8000` without) |
| `NEBULACTRL_SETUP_TOKEN` | Required by the setup page until the admin exists |
| `NEBULACTRL_DASHBOARD_UPSTREAM` | `nebulactrl:8080` in domain mode: the dashboard's system route target |
| `NEBULACTRL_DASHBOARD_ALLOW_CIDRS` | Comma separated CIDRs or addresses (IPv4, IPv6, at most 200) that may reach the dashboard's hostname. A malformed value stops the start. Seeds the setting; see [Dashboard access](#dashboard-access) |
| `NEBULACTRL_DASHBOARD_DNS_RESOLVER`, `NEBULACTRL_DASHBOARD_DNS_TOKEN` (or `_FILE`) | DNS-01 provider (`cloudflare`) and its API token for the dashboard's certificate. Both or neither. The token is sealed into the DNS credential store, never logged |
| `NEBULACTRL_LOCAL_ENROLL_TOKEN`, `NEBULACTRL_LOCAL_SERVER_NAME`, `NEBULACTRL_LOCAL_SERVER_IP`, `NEBULACTRL_ACME_EMAIL` | Enrollment of the host's own agent as the first server |
| `NEBULACTRL_BACKUP_HOST_DIR` | Host directory behind the local backup destination |
| `NEBULACTRL_MCP_ENABLED`, `NEBULACTRL_LOG_LEVEL` | Optional: `/mcp` (off by default), `debug`/`info`/`warn`/`error` |
| `PG_*` | Postgres tuning computed from the RAM |
The server also reads `NEBULACTRL_LISTEN`, `NEBULACTRL_AGENT_BINARIES_DIR` and `NEBULACTRL_BACKUP_DIR`; the compose file sets them. `NEBULACTRL_MAX_DUMP_UPLOAD_BYTES` (default 5 GiB) limits the dump files [restored into a database](https://nebulactrl.com/docs/operations/database-backups#restore-from-a-dump-file). It reads `NEBULACTRL_FORCE_UPGRADE=true` (default off) to start after a version change although the backup it takes first failed ([upgrades](https://nebulactrl.com/docs/operations/upgrades#backup-before-a-control-plane-upgrade)). A few advanced timers (`NEBULACTRL_MESH_LINK_GRACE`, `NEBULACTRL_REPLICA_BROKEN_AFTER`) are listed under [advanced settings](https://nebulactrl.com/docs/operations/configuration); leave them unset.
> [!WARNING]
> **Back up `NEBULACTRL_SECRET_KEY` now.** `sudo grep NEBULACTRL_SECRET_KEY /opt/nebulactrl/.env`, into a password manager or vault, apart from the database dumps. If you lose it every stored secret is unrecoverable and every variable must be re-entered; a key that is not in the key ring stops the values sealed with it from decrypting. See [backup and restore](https://nebulactrl.com/docs/operations/backup-restore) and [secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest). Settings → Instance keeps a reminder card until an administrator confirms it.
## Hardening of the containers
- Postgres: internal network only (no route out, no published port), SCRAM-SHA-256, data checksums, capabilities dropped, `no-new-privileges`.
- Control plane: read-only root filesystem, all capabilities dropped, `no-new-privileges`, no Docker socket; its only writable path is the backup directory.
- There is no reverse proxy in the compose file: the host's agent's Traefik serves the dashboard ([single proxy on the control-plane host](https://nebulactrl.com/docs/installing/installer#single-proxy-on-the-control-plane-host)).
## Dashboard access
In domain mode the dashboard is a system route on the control-plane host's proxy. Who may reach it and how its certificate is issued is an instance setting: **Settings → Instance → Dashboard access** (instance admin only), `GET`/`PUT /api/v1/instance/dashboard-access`, or `nebulactrl instance dashboard-access get|set`.
- **Allow list.** CIDRs or addresses, IPv4 and IPv6 (at most 200). It becomes an `ipAllowList` on the route and applies as soon as it is saved: no proxy restart. Anybody else gets `403` from the proxy. Agents on other servers and webhooks use the same hostname, so keep their addresses in the list. *Add my current IP* and the *Tailnet only* preset (`100.64.0.0/10`, `fd7a:115c:a1e0::/48`) fill it in.
- **Lock-out guard.** A list that leaves out the address you are connecting from is refused with `409 dashboard_lockout` unless you confirm (`confirmLockout`; `--confirm-lockout` in the CLI). The address is the one the proxy saw: the last `X-Forwarded-For` entry, trusted only from a private or loopback peer. A caller who reaches the control plane directly (an SSH tunnel to `127.0.0.1:8080`) never meets the list and is never warned; that is also the way back in after a lock-out.
- **Certificate.** Automatic HTTP-01, or DNS-01 with a provider that already has a resolver on the control-plane server (Servers → Proxy). `422 resolver_not_configured` otherwise. A resolver that is staged but not deployed yet sets `proxyDeployRequired`: the dashboard keeps its HTTP-01 certificate until **Deploy proxy** applies the resolver.
- **Where the values come from.** `NEBULACTRL_DASHBOARD_ALLOW_CIDRS` and `NEBULACTRL_DASHBOARD_DNS_RESOLVER`/`_TOKEN` only **seed** the setting. While nobody has edited it the installer's values are written on every start; after the first save the database row wins and the environment is ignored, so edits survive restarts and installer re-runs. When the environment names a DNS-01 provider and token, the control plane, once the host's agent is online, creates the DNS credential `dashboard-` (sealed, scoped to that server), sets the server's resolver, deploys the proxy and only then makes the route use it, retrying with backoff. It never changes an existing credential, so rotate the token in the DNS credential.
- **Audit.** `dashboard.access.update` (an admin's change: networks, provider, whether the lock-out was confirmed) and `dashboard.access.seed` (the installer's values). Tokens are never recorded.
## Running it by hand
If you cannot use the installer (another distribution, a managed Docker host):
1. Install Docker Engine 24+ and the Compose plugin 2.24+, and create the edge network once: `docker network create --label nebulactrl.managed=true nebulactrl-edge`.
2. Build or load the image (`docker build -t nebulactrl:local .`).
3. Copy [`deploy/install/docker-compose.yml`]() next to a `.env` made from [`deploy/install/env.example`]() (mode 0600) and run `docker compose up -d`.
4. Put HTTPS in front of `127.0.0.1:8080` yourself (any reverse proxy; it must pass WebSocket upgrades on `/agent/v1/connect` and must not buffer `text/event-stream` responses on `/api/v1/events` and the log streams), or install an agent on the host as a server (below) and add the system route by setting `NEBULACTRL_DASHBOARD_UPSTREAM=nebulactrl:8080` and the `NEBULACTRL_LOCAL_*` variables.
5. Open `NEBULACTRL_PUBLIC_URL`, create the admin on the setup page, then [add servers](https://nebulactrl.com/docs/installing/servers).
On start the server applies its database migrations (and the job queue's) and waits up to 30 seconds for Postgres.
## First run
While no user exists the **setup page** asks for an email, name, password, team name and (with the installer) the setup token, and creates the admin and the team. Afterwards setup is closed (`POST /api/v1/setup` returns 409). Then:
1. The first server is already there when you used the installer: the control-plane host, **online** within a minute, with **Accept workloads** on.
2. Create a project, open its canvas, add a service (placed on that server, or on one you [add](https://nebulactrl.com/docs/installing/servers)).
3. **Settings → Instance**: add an off-site backup destination and confirm you stored the secret key. **Settings → API tokens** for the CLI, CI or MCP.
The login endpoint is rate limited to 10 attempts per minute per IP. Setup with the token is limited the same way.
## Production notes
- Pin the image version you run; do not run `latest` of anything you cannot roll back ([upgrade](https://nebulactrl.com/docs/installing/upgrade)).
- Keep the control plane's time in sync (NTP).
- Agents keep your apps running through any control-plane restart or upgrade: the control plane never stops remote containers while it is down.
---
# Upgrade and roll back
Source: https://nebulactrl.com/docs/installing/upgrade
## Before you upgrade
1. Read the release notes for schema or protocol changes.
2. Confirm you hold `NEBULACTRL_SECRET_KEY` ([backup and restore](https://nebulactrl.com/docs/operations/backup-restore)). The backup of the database is taken **for you**: the installer backs the control plane up before it changes anything, and the control plane does the same at startup when its version changed ([how it works](https://nebulactrl.com/docs/operations/upgrades#backup-before-a-control-plane-upgrade)). **If that backup fails the upgrade stops**; fix the cause, or re-run with `--force-upgrade` to go on without one.
3. Note the version you run now (`docker compose -f /opt/nebulactrl/docker-compose.yml --env-file /opt/nebulactrl/.env images`).
## Upgrade the control plane (and the host's agent)
Re-run the installer with the new version. Nothing that exists is regenerated:
```sh
curl -fsSL https://nebula.example.com/install.sh | sudo bash -s -- --version 1.4.0
# or, with an image you built or saved yourself:
sudo bash install.sh --image nebulactrl:1.4.0 --image-archive nebulactrl-1.4.0.tar
```
The installer keeps `.env` and the database, loads or pulls the image, recreates only the containers whose definition changed, waits for `/readyz`, and upgrades the host's agent to the binary that ships with the new control plane. Your options are remembered in `/opt/nebulactrl/install.conf`.
- The server applies **forward-only** database migrations on startup, under a Postgres advisory lock. There is no automatic down migration.
- Startup order is safe: Postgres is kept, the server container is replaced, agents reconnect on their own (exponential backoff, up to 60 s) and the control plane **never stops remote containers** while it is down. Your services keep serving through a control-plane restart or upgrade, including those on the control-plane host (the proxy is the agent's, not the control plane's).
- Commands acknowledged before the restart keep their state; the agent redelivers unacknowledged results after reconnecting.
- The upgraded control plane serves new agent binaries immediately, and **agents upgrade themselves** when they reconnect (below).
## Agents upgrade themselves
An agent that is behind the control plane downloads the new binary from its control plane, verifies the SHA-256, replaces itself atomically, restarts when its running work is done and rolls back if the new binary does not start ([details](https://nebulactrl.com/docs/operations/upgrades)). This is automatic by default (Servers page → **Upgrade automatically**); **Upgrade agents** does the same one server at a time on request, and each server's page shows the version skew. Your services keep running.
## Upgrade other servers by hand
You need this once for an agent that predates self-upgrade, for a server that is offline during the upgrade, and when you switched the automatic upgrade off. Re-run the agent installer on the host (no token needed; the identity is kept):
```sh
curl -fsSL https://nebula.example.com/install-agent.sh | sudo sh -s -- --url https://nebula.example.com
```
It downloads the new binary, verifies it against the checksum the control plane publishes, restarts the service, and verifies the service stays up. Managed containers and the Traefik proxy keep running during the agent restart. The wire protocol is versioned; a control plane refuses an unknown **major** protocol version with close code 4400, so upgrade the control plane first and agents second. The server card shows each agent's version.
## Roll back the control plane
Because migrations only go forward, an older server binary may not work against a newer schema. The supported rollback is:
1. `cd /opt/nebulactrl && docker compose down` (do **not** pass `-v`).
2. Restore the pre-upgrade backup (Settings → Instance → History, trigger *Before upgrade*; or `/var/backups/nebulactrl/pre-upgrade-*.dump` from the installer) into a clean database ([restore](https://nebulactrl.com/docs/operations/backup-restore#restore)). Changes made after the dump (deployments, variable edits) are lost.
3. Run the installer with the previous `--version` (or image).
If the new version only changed code (no new migration, see `db/migrations` in the diff), re-running the installer with the previous version is enough.
## Roll back an agent
A failed self-upgrade rolls back by itself. After a good one the previous binary stays at `/usr/local/bin/nebulactrl-agent.prev`: move it over the binary and `systemctl restart nebulactrl-agent`. Or re-run the installer from an older control plane, or place an older binary at `/usr/local/bin/nebulactrl-agent`. The control plane never downgrades an agent, and an agent newer than the control plane is not upgraded. Use an agent whose protocol major version the control plane accepts.
## Roll back an application (not the platform)
That is a deployment rollback inside the UI/API/CLI: it re-deploys an earlier release's image and configuration. It does **not** restore data. See [operations: deployments](https://nebulactrl.com/docs/operations/deployments).
---
# Uninstall
Source: https://nebulactrl.com/docs/installing/uninstall
## The control-plane host
```sh
sudo bash /opt/nebulactrl/uninstall.sh # stops NebulaCtrl, keeps everything that holds data
curl -fsSL https://nebula.example.com/uninstall.sh | sudo bash -s -- --purge # the control plane serves the script too
```
The uninstaller prints what it will do and asks first (`--yes` skips the question; `--purge` and `--remove-workloads` ask you to type `delete`).
| Option | What it adds |
|---|---|
| *(none)* | Stops and removes the control-plane and Postgres containers and the **agent service and binary**. **Kept:** the database volume, `/opt/nebulactrl` (with the secret key), the agent identity, the local backups, and **your deployed apps, their volumes and the proxy, which keep running**. Re-run the installer to bring NebulaCtrl back with all its data |
| `--purge` | Also deletes the database volume, `/opt/nebulactrl` (**including `NEBULACTRL_SECRET_KEY`**) and the agent state. Backups stay |
| `--remove-workloads` | Also removes every container NebulaCtrl manages (your apps, the proxy, helpers), the `nc-vol-*` volumes (**your apps' data**), proxy certificates and `nebulactrl-*` networks |
| `--remove-backups` | Also deletes `/var/backups/nebulactrl` |
| `--revert-host` | Reverts what the installer added: its ufw/firewalld rules (the firewall itself stays on), the fail2ban jail, and `/etc/docker/daemon.json` restored from its backup |
It **never** uninstalls Docker, automatic updates or the hardening fixes, and never touches SSH. Before `--purge`, take a final backup if you may want the data back, and remember a dump is unreadable without the secret key.
Everything the uninstaller does is covered by the installer's test suite: default uninstall, reinstall over the kept data (same secrets, same data, the host re-enrolls with its kept identity), and `--purge --remove-workloads --remove-backups` leaving no container, network, volume or file behind.
## An additional server
```sh
# stops the service, removes the unit, env file and binary; keeps the identity and your containers
curl -fsSL https://nebula.example.com/install-agent.sh | sudo sh -s -- --uninstall
# additionally delete /var/lib/nebulactrl-agent (identity, build workspaces). The host must be re-enrolled to be used again.
curl -fsSL https://nebula.example.com/install-agent.sh | sudo sh -s -- --uninstall --purge
```
(If the control plane is already gone, take the agent installer from any other running control plane (`/install-agent.sh`), or remove the pieces by hand: `systemctl disable --now nebulactrl-agent; rm /etc/systemd/system/nebulactrl-agent.service /etc/nebulactrl-agent.env /usr/local/bin/nebulactrl-agent; systemctl daemon-reload`.)
Uninstalling the agent **does not remove** what it created: the proxy and your services keep running. To delete everything NebulaCtrl created on such a host (this **destroys the data in service volumes**; export what you need first):
```sh
docker ps -aq --filter label=nebulactrl.managed=true | xargs -r docker rm -f
docker images --filter label=nebulactrl.managed=true -q | xargs -r docker rmi -f
docker rm -f nebulactrl-traefik nebulactrl-pages
docker network ls --format '{{.Name}}' | grep '^nebulactrl-' | xargs -r docker network rm
docker volume ls -q | grep -E '^(nc-vol-|nebulactrl-)' | xargs -r docker volume rm
```
Review each list (`docker ps -a`, `docker volume ls`) first; the commands match by label or name prefix, not by what you consider yours. Deleting a service in the UI with *delete volumes* ticked removes that service's volumes only. Revoking the server in the control plane first is optional; it only stops the agent and cancels its pending commands.
Your agents' services keep running after the control plane is gone, but nothing manages them, and agents keep retrying the connection until they are uninstalled.
---
# Deployments
Source: https://nebulactrl.com/docs/operations/deployments
The behavior mirrors what Coolify documents for health checks, rolling updates, stop grace periods and rollbacks, with one deliberate difference: **health checks run from the agent, outside the container**, so images do not need `curl` or `wget`.
## Life of a deployment
```
created → queued → preparing → building → releasing → verifying → succeeded
↘ failed | canceled | rolled_back | rollback_failed
```
Image sources skip `building`. Every transition is recorded with a timestamp (deployment → step timeline). Only **one deployment per service can be active**; a second request gets `409 deployment_in_progress`. Deploy and rollback return `202` with the deployment; they never report success before the agent has finished.
A release is immutable and written *before* dispatch: the resolved source (image + digest, or git URL + commit SHA), a config snapshot and the encrypted, resolved variables.
## Strategies
The plan (`POST /services/{id}/deployments/plan`, shown before you deploy) names the strategy and why:
| Strategy | When | What happens |
|---|---|---|
| **rolling** | the service has **no volumes** and an HTTP port | New container starts and is health checked while the old one keeps serving; traffic switches only after the route is loaded and verified. |
| **recreate** | anything else (volumes, no HTTP port) | The old container is stopped first, then the new one starts. **There is downtime** for the time it takes to start and become healthy. The plan lists the reasons (`has volumes`, `no http port`), as Coolify does. |
### Rolling steps
1. Pull or build. If this fails the old container is untouched (`failed`).
2. Create and start the new container.
3. Health check it. Unhealthy = the deadline passes, the container exits, or it restarts. Then the new container is removed, the outcome is `failed` ("Previous release is still serving" plus the last 50 container log lines).
4. If the service has domains, push the full route table, now pointing at the new container, to Traefik. Traefik applies provider configuration asynchronously, so the agent waits for a per-generation marker router on the probe entrypoint to answer and reads the router statuses. If Traefik disabled any router (for example a bad upstream), the agent pushes the previous table again and the deployment is `failed`.
5. **Verify** through Traefik's internal probe entrypoint (port 8099, never published) using the service's first hostname and health path; any status below 500, 3 tries 2 s apart. If that fails, the previous routes are pushed again and the new container removed: `rolled_back`, or `rollback_failed` if the restore itself fails.
6. Retire: wait 2 s for drain, stop the old container with the **stop grace period**, remove it, prune old images.
### Recreate steps
Pull or build → stop the old container (kept) → start and health check the new one → on failure remove it and restart the old one (`rolled_back`, or `rollback_failed`) → on success load the route and remove the old container.
## Canceling a deployment
A deployment can be canceled at **any step** until the new release is live: `POST /api/v1/deployments/{id}/cancel` (scope `deployments:write`), the **Cancel deployment** button in the Deployments tab and the canvas menu, `nebulactrl deployments cancel `. The deployment ends as `canceled`, with the step it reached in `canceledAtStep` and "Canceled by " in its message. The request and the outcome are audited (`deployment.cancel`, `deployment.canceled` with the step).
- **Before the agent acknowledged the command** (queued, preparing, building with the command still `pending` or `sent`): the command is canceled and the deployment ends as `canceled` at once.
- **Once the agent runs it:** the control plane records the request (`cancelRequestedAt`) and sends a `cancel_deploy` command to the agent. The deployment keeps its status until the agent has aborted and unwound; the UI shows "Canceling…". An offline server receives the request when it reconnects.
What the agent does at each step (the previous release keeps serving unless noted):
| Step reached | Abort |
|---|---|
| **pull** | the pull is stopped; nothing was changed |
| **build** | the build gets `SIGINT` (and `SIGKILL` after 10 s), so BuildKit stops it; the build work directory is removed; no image is produced |
| **pre-deploy** | the command's container is removed (never left for a later redelivery to adopt) |
| **create / start / health / route** | the containers of the new release are removed, including one whose creation was interrupted; if the new route table had already been applied it is withdrawn first |
| **verify** | the previous routes are restored **first**, then the new containers are removed |
| **stop** (recreate only, the old release is stopping) | the stop is not interrupted; right after it, the old release is started again |
| a **recreate** deployment past "stop old" | the previous release is started again, exactly like an automatic rollback |
The result is `canceled` only when everything was restored. If the previous routes or the stopped previous release **cannot** be restored, the result is `rollback_failed` with the manual recovery steps, never `canceled`. A canceled deployment never becomes the service's current deployment and consumes no generation: the staged changes stay staged.
**Too late:** once verification has passed the new release is live and the point of no return is behind it. The agent then reports "Too late to cancel" in the step timeline and the deployment finishes (retire, then `succeeded`). A **post-deploy command** that is still running can be stopped: the deployment stays `succeeded`, with a warning that the command was canceled. A deployment that already finished is `409 cannot_cancel`. A cancel is not an agent shutdown: if the agent stops mid-deploy the command still runs again after the restart, as before.
## Health check semantics
Configured per service (**Settings → Health check**); the agent runs the check against the container's address on its environment network (`nebulactrl-env-`, or the `-int` network when the service blocks outbound internet), not the shared edge network. See [networking](https://nebulactrl.com/docs/operations/networking).
| Type | Passes when |
|---|---|
| `http` (default when a port is set) | `GET ` on the port returns status **200–399**. Default path `/`. |
| `tcp` | the TCP connection opens (default for managed databases) |
| `docker` | the image's own `HEALTHCHECK` reports healthy |
| `none` | the container has simply **run for 10 s without restarting**. The plan shows a warning: nothing proves it works. |
Timing defaults: interval 2 s, timeout 3 s, start period 0 s, **deadline 120 s**, success threshold **2 consecutive passes**. A container that exits or restarts during the deadline fails immediately. As in Coolify, a health check failure during deployment aborts the new release; it does not by itself stop a running one: after a deploy, a container that later becomes unhealthy is shown as unhealthy, not killed. Docker's restart policy (`unless-stopped`) restarts crashed containers.
## Stop grace period
`stopGraceSeconds` (default 30, 1–3600): how long Docker waits after SIGTERM before SIGKILL when a container is stopped for retire, restart or stop. Raise it for workloads that drain slowly.
## Variables and generations
Changing configuration, variables, domains or volumes does **not** redeploy. It increments the service's `generation`; the node shows a **Staged** chip and the environment's staged bar says "N changes to apply". **Review** lists the field-level changes (secret values masked). **Deploy** applies them: one deployment per changed service. Edits use optimistic concurrency: a stale `If-Match` returns `412 generation_conflict` and the UI reloads.
Secret variable values are never returned by any read endpoint, appear in logs only as `••••••`, and travel to the agent inside the command frame at send time, never in stored payloads.
## Rollback
*Roll back to this release* (deployment row, API `POST /deployments/{id}/rollback`, CLI `nebulactrl deployments rollback`) creates a **new deployment** from the old release's source and configuration snapshot. It is deployed with the normal strategy and checks.
> [!NOTE]
> **Application image and configuration only. Databases, volumes and external services are not rolled back.**
If the old release was a git build whose image has been pruned from the host (see below), the API returns `409 image_pruned`; use **Rebuild commit **, which deploys that commit again. Image releases re-pull `repo@digest` when missing (registries may have deleted it).
**Variable values outside the rollback window.** A release keeps the sealed environment it was deployed with only while it is among the newest `imageRetention` succeeded releases (or running, or in flight); an hourly job clears it from older ones ([secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest#release-secret-retention)). Rolling back to such a release deploys its old image and configuration with the service's **current** variables, says so in the timeline and in the rollback response, and fails (changing nothing) if the current variables do not resolve. Within the window, a rollback restores the old values.
Retention: after each successful deploy the agent keeps the newest `imageRetention` built images per service (default **5**) and removes the rest. The server's scheduled [Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup) (daily, and when the disk alert is active) never removes the current or any of these rollback images, and prunes build cache older than 7 days.
## Recovering from `rollback_failed`
`rollback_failed` means the deployment failed *and* the agent could not restore the previous release. The service may be down or serving the wrong release. On the host:
1. `docker ps -a --filter label=nebulactrl.service=`: find which containers exist (names `nc---`, replica 0…N-1).
2. `docker logs ` on the newest one; the deployment's log in the UI has the failure reason.
3. If an old container exists and is stopped (recreate): `docker start `. If routes are wrong: in the UI, **Restart** the service or trigger a deploy of the last good release (rollback target) – the deploy re-applies the full route table.
4. If the host itself is the problem (disk full, Docker unhealthy, proxy error), fix it first; the server view shows the proxy status and disk gauge.
5. Redeploy and confirm `succeeded`. The reconciler also repairs a **missing** container automatically (at most 3 times per hour per service).
## Restart and stop
`restart` stops and starts the existing container (a brief interruption; not a rolling replace) with the configuration it was deployed with; staged changes are applied only by a deploy. `stop` stops it and keeps its configuration; the node shows stopped. Both honor the stop grace period.
## Moving a service to another server
Changing **Placement** is a staged change. Deploying creates a new deployment on the target; the old server's container is not auto-migrated and **volumes are not moved**. Plan data migration yourself.
---
# Builds
Source: https://nebulactrl.com/docs/operations/builds
By default, git services are built on the server that runs them ([ADR 0004](https://nebulactrl.com/docs/decisions/0004-build-on-target-host), [ADR 0006](https://nebulactrl.com/docs/decisions/0006-railpack-default-builder)). A team, project or service can instead build on a dedicated **build server** that pushes to a registry, and the server that runs the service pulls the image by digest: see [Build servers and registries](https://nebulactrl.com/docs/operations/build-servers). The build methods below are the same either way. **Settings → Source → Build method** chooses how.
| Build method | What happens |
|---|---|
| **Automatic** (default) | The repository's `Dockerfile` if there is one, otherwise Railpack. |
| **Dockerfile** | Always the Dockerfile (path configurable); the deployment fails if it is missing. |
| **Railpack** | Railpack, even if the repository has a Dockerfile (which is then ignored). |
| **Starter: Node.js, Python, Go, Ruby, PHP, static site** | A Dockerfile generated from the files in the build context. It is printed at the top of the build log so you can copy it into your repository. |
A Dockerfile always wins in Automatic mode. If you changed *Dockerfile path* away from `Dockerfile` and that file does not exist, the build fails with a clear message instead of falling back to Railpack.
## Preview the build plan
**Settings → Source → Preview build plan** (API: `POST /api/v1/services/{id}/build-plan`, needs `deployments:write`) asks the server that **will build** the service (its target, or the build server it resolves to) to clone the branch head and report:
- the method that applies and why ("No Dockerfile in the repository, so Railpack detects how to build it"),
- for Railpack: the detected providers (`node`, `python`, `staticfile`, …), the toolchain versions it would install, the build steps and the **start command**,
- Railpack's warnings and suggestions (for example "Add a `package-lock.json` for more deterministic installs"),
- whether a Dockerfile was found and, if not, which starter matches the project.
Nothing is built or deployed. That server must be online. The first plan on a host takes longer while Railpack's tooling downloads. The preview uses the **saved** configuration and the current build variables.
## Railpack
Pinned version: **Railpack 0.43.0**, frontend image `ghcr.io/railwayapp/railpack-frontend:v0.43.0@sha256:355ef61f…` (full digest in `pkg/agentprotocol/build.go`).
What Railpack detects: Node (npm, yarn, pnpm, bun), Python, Go, PHP, Ruby, Java, .NET, Deno, Rust, Elixir, static sites (a `Staticfile` or an `index.html`) and more; see the [Railpack docs](https://railpack.com/getting-started). Configure it with a `railpack.json` in the repository or with `RAILPACK_*` build variables, for example `RAILPACK_START_CMD`, `RAILPACK_NODE_VERSION`, `RAILPACK_PACKAGES`. Set build variables under **Variables** with scope *Build*; plain and secret values are both handed to Railpack as BuildKit secrets.
**The port.** Railpack does not know your port. Set the service's *Port*; the platform passes it as `PORT`, which Railpack's Node/Python/static start commands honour (the static site server listens on `$PORT`).
**Start command.** The default comes from Railpack (shown in the preview). To change it, prefer `RAILPACK_START_CMD`. A service-level *Start command* also works: Railpack images start with `bash -c`, so a multi-word command is passed as one quoted script.
**Requirements on the server.** Outbound access to `ghcr.io` (the frontend and Railpack's builder images), `registry-1.docker.io` and the usual package registries. Railpack's build graph needs BuildKit `mergeop`:
- Docker with the **containerd image store** (the default on fresh Docker 29 installs): builds run in the daemon.
- Otherwise the agent creates a `docker buildx` builder named `nebulactrl-buildkit` (a privileged BuildKit container, image pinned by digest) the first time a Railpack build needs it. This needs the `docker-buildx-plugin` package (included with Docker's own packages; Debian's `docker.io` lacks it). Without it Railpack builds fail with that explanation; Dockerfile and starter builds still work.
Check which applies: `docker info --format '{{json .DriverStatus}}'` mentions `io.containerd.snapshotter.v1` with the containerd store.
**Disk.** A first Railpack build pulls ~100–500 MB of builder images and leaves BuildKit cache. The server's [Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup) prunes build cache older than 7 days daily (`docker builder prune --filter until=168h`, plus `docker buildx prune --builder nebulactrl-buildkit` when that builder exists) and keeps the pinned BuildKit and Railpack images. To reclaim everything Railpack brought: `docker buildx rm -f nebulactrl-buildkit` (removes the builder container and its cache), `docker image rm moby/buildkit:v0.34.1 ghcr.io/railwayapp/railpack-frontend:v0.43.0`. On a [build server](https://nebulactrl.com/docs/operations/build-servers) every project has its own builder, `nebulactrl-buildkit-`, and builds always go through it (`buildx build --push`), whatever the daemon's image store.
### Vite, Vite+ and Void apps
Checked against the pinned planner (Railpack 0.43.0), plan only:
| Project | What happens |
|---|---|
| Vite SPA (`vite` in `devDependencies`, `build: "vite build"`) | Detected as a static site, output `dist`, served by Caddy. No settings needed. |
| Vite+ project after `vp migrate` (adds the `vite` alias) | Same as a Vite SPA. |
| Vite+ project with only `vite-plus` (as `vp create` can produce) | **Not detected**: the plan has no start command and the container would not start. Add the build variable `RAILPACK_SPA_OUTPUT_DIR=dist`, or run `vp migrate`. |
| Void app | Only its `target: "node"` mode runs outside Cloudflare. Add a `start` script (`node dist/ssr/index.js`) or set `RAILPACK_START_CMD`; the app listens on `PORT`. Void's `db`, `kv`, `storage`, `auth`, `ai`, sandbox, WebSocket and email modules need Cloudflare and are unavailable; use a [managed database](https://nebulactrl.com/docs/operations/databases) instead. Not verified with a real build. |
## Starter Dockerfiles
Generated from the build context (the *Build context* path):
| Starter | Base image | Install | Start |
|---|---|---|---|
| `node` | `node:24-slim` | `npm ci` / `pnpm` / `yarn` by lockfile (else `npm install`), `build` script if present | `npm start` (or `main`, `server.js`, `index.js`) |
| `python` | `python:3.13-slim` | `requirements.txt`, `Pipfile` or `pyproject.toml` | `python main.py` / `app.py` / `server.py` |
| `go` | `golang:-alpine` → `alpine:3.22` | `go build` of the root package or the single `cmd/` | the binary |
| `ruby` | `ruby:3.4-slim` | `bundle install` | `bin/rails server` or `rackup` |
| `php` | `php:8.4-apache` (+ Composer) | `composer install --no-dev` if `composer.json` | Apache on `$PORT` (document root `public/` when it has an `index.php`) |
| `static` | `nginx:1.29-alpine` | none: serves prebuilt files from `index.html`'s directory (root, `public/`, `dist/` or `build/`) | nginx on `$PORT` |
`.git` is never copied into a starter image; your `.dockerignore` is honoured too. Set the *Start command* when the heuristics do not find one. Starters run no build for `static`: run your site generator in CI and commit the output, or use Railpack.
## Deploying without cache
Stale layers are the usual cause of "works on my machine, fails here". **Redeploy without cache** (inspector ⋯ menu, the Deployments tab and the canvas menu of a git service; API `POST /api/v1/services/{id}/deployments` with `{"noCache": true}`; `nebulactrl services deploy --no-cache`) builds **that one deployment** from scratch:
- Dockerfile, starter and Railpack builds run with `--no-cache --pull`: every layer is rebuilt and the base images are pulled again. This is the same on the target and on a [build server](https://nebulactrl.com/docs/operations/build-servers), where the build also ignores the registry cache (`--cache-from`).
- An image a build server already pushed for the same commit and settings is **not reused** (a normal deploy of an unchanged commit would reuse it). The fresh image becomes the reusable one for later deployments.
- The deployment records it: the timeline shows "Deploying without build cache", the deployment has `noCache: true` and the audit event `deployment.create` carries `noCache`. The next deployment builds normally; there is no setting that makes it permanent.
- `POST /environments/{id}/deploy-staged` and MCP `apply_changes` accept `noCache` too and apply it to the git services of that batch. Both only deploy services with staged changes, and the raw deploy operation is not exposed over MCP, so an AI client cannot rebuild an unchanged service without cache; use the web app, the CLI or REST for that.
- Image services have nothing to build: `noCache` on one is `422 validation_failed`. Rollbacks and reconcile redeploys never build. A server whose agent predates this option answers `422 agent_too_old` instead of quietly building with the cache; update the agent.
Limits: `--no-cache` ignores cached *layers*; BuildKit **cache mounts** (the package-manager caches Railpack mounts, such as the npm or pip download cache) are kept, because BuildKit does not treat them as layers. They speed up downloads but cannot make a build wrong. To drop them as well, run `docker builder prune -af` on the server (the agent also prunes build cache older than a week daily). `--pull` needs the registries of your base images to be reachable, and a `FROM` that names a local-only image fails with it.
## Git submodules
**Settings → Source → Git submodules** (config `submodules`: `auto`, `on` or `off`; `nebulactrl services update --submodules on`):
| Mode | Behaviour |
|---|---|
| **Automatic** (default) | Submodules are checked out recursively when the commit has a `.gitmodules` file. A submodule that cannot be fetched is a **warning** in the build log and the build continues without it, which is what happened before submodules were supported. |
| **Always** | Same checkout; a submodule that cannot be fetched **fails the build**. Use it once you rely on the submodule. |
| **Never** | The submodule folders stay empty. |
Automatic is the default because Coolify turns submodules on, and a repository that has them does not build without them; making it opt-in would keep that failure for everyone who does not find the setting. Because a warning is never worse than the old behaviour, an existing service cannot start failing after an upgrade. The checkout follows the commit's recorded submodule commits, nests up to five levels deep, and is part of the build-reuse key on build servers (changing the mode rebuilds).
**Credentials.** A `.gitmodules` file is untrusted input, so the repository's credentials only go to the repository's own host:
- An HTTPS repository with an access token: the token is used for submodules on the **same host** (and port) over https, and for relative URLs such as `../lib.git`. A submodule written as `git@github.com:org/lib.git` or `ssh://git@github.com/org/lib.git` on that same host is fetched over HTTPS with the token instead, so one token covers a repository and its private submodules on the same provider as long as the token can read them (a GitHub fine-grained token must list the submodule repositories).
- An SSH repository with a [deploy key](https://nebulactrl.com/docs/operations/ssh-deploy-keys): the key and its pinned host key are used for ssh submodules on the **same host and port**. Providers such as GitHub accept a deploy key for one repository only, so a private submodule needs the key added to that repository too, or a user/machine key that can read both.
- Submodules on **another host** are fetched anonymously over `https://`, `http://` or `git://`; a private one fails (a warning in Automatic, a failed build in Always). An `ssh://` submodule on another host is refused, since there is no key or pinned host key for it; change its URL to https on the parent's host or mirror it.
- Passwords in a submodule URL, `file://`, local paths and other schemes are refused.
Not supported: Git LFS (pointer files are checked out as text), shallow clones and per-submodule credentials. A server whose agent predates submodule support ignores the setting in Automatic, and a service set to *Always* is blocked with `agent_too_old` until the agent is updated.
## Upgrading the pinned Railpack
1. Check the release notes of the new version for breaking changes (provider behaviour changes are the usual ones).
2. Resolve the frontend digest without pulling: `docker buildx imagetools inspect ghcr.io/railwayapp/railpack-frontend:vX.Y.Z` and take the top-level `Digest`.
3. Update `RailpackVersion` and `RailpackFrontendImage` together in `pkg/agentprotocol/build.go` (and `BuildKitImage` if you also bump BuildKit; get its digest the same way).
4. Refresh `internal/agent/build/testdata/*-plan.json` / `*-info.json` by running `railpack prepare` from the new frontend image against a Node and a static example, run `go test ./internal/agent/build/`, then `./tests/run.sh -run TestRailpackBuilds -v` (it cleans up the images and cache it created).
5. Ship the new agent. Hosts pick up the new frontend image on their next Railpack build; images already built keep running.
## Troubleshooting
| Message | Cause and fix |
|---|---|
| `railpack could not plan this source` | Railpack recognised no language. Add a Dockerfile, choose a starter, or add `railpack.json`. The preview shows the reason. |
| `requested experimental feature mergeop has been disabled` | Should not surface: the agent retries through the managed builder. If it does, the buildx plugin is missing (`docker buildx version`). |
| `create the BuildKit builder (is the docker buildx plugin installed?)` | Install `docker-buildx-plugin`, or enable the containerd image store, or use a Dockerfile. |
| `dockerfilePath: "…" not found in repository` | Build method Dockerfile (or Automatic with a custom path) and the file is not in the repository at that commit. |
| `starter:static needs an index.html` | Nothing to serve at the root, `public/`, `dist/` or `build/`. |
| Preview: `409 server_offline` | The plan is made on the server that will build; bring it online (or switch the service to build on its target). |
| Preview: `504 build_plan_timeout` | The first plan on a host downloads tooling; try again. |
---
# Git providers and deploy on push
Source: https://nebulactrl.com/docs/operations/git-providers
NebulaCtrl connects to GitHub, GitLab, Gitea, Forgejo and Bitbucket. A connection lets you pick repositories instead of pasting URLs, deploys a service when you push to its branch, shows each deployment as a status on its commit and clones private repositories with short-lived credentials.
Set `NEBULACTRL_PUBLIC_URL` first: it is the address providers deliver webhooks to and redirect OAuth flows back to. Settings → Integrations → Git providers is where connections live (admin scope).
## Connect a provider
| Provider | How | Webhooks |
|---|---|---|
| GitHub (and GitHub Enterprise Server) | One click creates a **GitHub App** (manifest flow) owned by you or an organization. Permissions: contents read, metadata read, pull requests write, commit statuses write. Events: push, pull request. Install it on the repositories you want. | The App's own webhook delivers for every installed repository. |
| GitLab (gitlab.com or self-managed) | A personal/group/project access token with the `api` scope, or an OAuth application (scopes `api read_repository`; redirect URI `/api/v1/git/oauth/callback`). | Registered per repository through the API, with a secret token header. |
| Gitea, Forgejo | Always give the instance URL. An access token, or an OAuth2 application (same redirect URI). | Registered per repository through the API; signed with HMAC-SHA256. Gitea blocks private webhook targets by default: allow the control plane in `[webhook] ALLOWED_HOST_LIST`. |
| Bitbucket (Cloud and Data Center) | A signed push webhook only (optionally a repository access token to clone private repositories). No repository picker, no statuses, no previews. | You create it by hand with the URL and secret NebulaCtrl shows once; the secret can be rotated. |
Credentials, OAuth tokens (refreshed automatically), the GitHub App private key and webhook secrets are encrypted at rest and never returned by the API.
## Deploy on push
Link a git service to a repository in its Source settings (or pick the repository when you create the service). The link and its auto-deploy settings are operational: they apply immediately, never bump the generation and are never rolled back. The clone URL and branch stay in the service config (picking another repository stages that change like any edit).
- **Branch**: a push to `config.gitBranch` deploys. The build is pinned to the pushed commit; the deployment has trigger `git_push` and shows the commit sha, message and author in the Deployments lists.
- **Watch paths** (monorepos): gitignore-style globs relative to the repository root, e.g. `apps/api/**`, `packages/*/src`, `Dockerfile`. `**` crosses directories, a plain directory name watches everything below it, a leading `!` excludes (`!**/*.md`). A push deploys when a changed file matches (and is not excluded). Providers that cap or omit the file list (a GitHub push of over 2,048 commits, a new branch without commits, GitLab pushes with more than 20 commits, every Bitbucket push) cannot be filtered: they deploy.
- **Skip**: `[skip ci]`, `[ci skip]`, `[skip deploy]`, `[nebula skip]` in the commit subject.
- **Busy services**: a push during a running deployment is queued (the newest wins) and deploys when that deployment ends.
- Auto-deploy ships the service's current configuration, including staged changes, exactly like a manual deploy.
## Commit statuses
Every deployment of a linked service reports `nebulactrl/` on its commit: pending while it queues, builds and verifies, then success, or failure (failed, rolled back) / error (canceled), with a link to the deployment. Each state is sent once; failures are retried for several minutes. Reconcile and drain redeployments are not reported. Bitbucket has no status API.
## Webhook security
`POST /hooks/git/{provider}/{connectionId}` is unauthenticated but verified: GitHub, Gitea and Forgejo by HMAC-SHA256 of the body, GitLab by its secret token header, Bitbucket by `X-Hub-Signature: sha256=…`. A delivery that fails verification triggers nothing (HTTP 401). The delivery id is remembered for 24 hours: a repeat is rejected as a replay (HTTP 409). Every delivery is logged with redacted headers (Settings → Integrations → Deliveries; last 200, 14 days): processed, ignored, rejected, replayed or error, and what it started.
## Clone credentials
A linked service clones with credentials minted when the deploy command is sent: a GitHub App installation token restricted to that repository with read access to its contents (valid one hour), or the connection's access token. They travel as send-time secrets only; they are never stored in commands, releases or logs. A service without a link keeps using its own stored git token.
## API, CLI and MCP
`/git/connections*` (connections, repos, branches, deliveries, OAuth/manifest flows) and `/services/{id}/git`. `GET /services/{id}` carries `git` (provider, repo, auto-deploy, watch paths) next to `config.gitUrl` / `config.gitBranch`; MCP `service_get` returns the same. Deployments carry `commit` and trigger `git_push`.
## Removing a connection
Refused while services use it unless forced, which unlinks them. Webhooks NebulaCtrl registered are deleted; a GitHub App stays installed at GitHub. Webhooks of repositories no service uses any more are removed hourly.
---
# Deploy hooks
Source: https://nebulactrl.com/docs/operations/deploy-hooks
A deploy hook is a URL that deploys one service. Call it from a CI pipeline after
a build, from `curl`, or let Docker Hub or GHCR call it after every image push.
It is the same as pressing **Deploy**: the service's current configuration is
deployed, staged changes included, with the same health checks, rolling update
and rollback. The deployment shows **Via deploy hook** in the Deployments list.
```
POST https:///hooks/deploy//
```
The hook is **off** until you turn it on. It needs no login: the secret in the
URL is the credential.
## Turn it on
**Service → Settings → General → Deploy hook** (or the CLI):
1. Turn on **Enable deploy hook**. NebulaCtrl creates a secret and shows the
complete URL **once**. Copy it into your CI system's secret store.
2. Pick the **Set it up** tab for your CI or registry: curl, GitHub Actions,
GitLab CI, Docker Hub or GHCR.
```sh
nebulactrl services deploy-hook # state (never the secret)
nebulactrl services deploy-hook --enable # new secret, prints the URL once
nebulactrl services deploy-hook --rotate # replace the secret, prints the URL once
nebulactrl services deploy-hook --disable # off, secret deleted
```
`NEBULACTRL_PUBLIC_URL` must be set: it is the host of the URL (422
`public_url_required` otherwise).
Who may manage a hook: owners and admins, and developers or API tokens with
`deployments:write` on the project. Anyone with `projects:read` can see whether it
is on, when the secret was created and what the last call did, but never the URL.
### The secret
- 256 bits from the system random source (`ndh_` plus 43 URL-safe characters).
- Only its SHA-256 is stored. The URL is returned by exactly two requests (enable
and rotate) and never again: not in the UI, API, CLI or audit log (a backup holds the hash, which cannot be turned back into the URL).
- **Rotate** replaces the secret at once; the old URL answers 404 from then on.
- **Turn off** deletes the secret. Turning on again creates a new one; an old URL
never comes back.
- A wrong secret, a disabled hook, a deleted service and a malformed id all give
the same `404`, so a guess reveals nothing.
Treat the URL like a password: anyone who has it can deploy that service. Keep it
in a CI secret, not in the repository or a chat. Reverse proxies and CI systems
may log URLs, so rotate it if it might have leaked. For services in a
**production** environment the Settings section says so in a warning.
## What a call does
| Call | Result |
|---|---|
| plain `POST` (curl, CI) | Deploys the current configuration. A git service builds the head of its branch; an image service pulls its configured image again. |
| `?tag=` or `?digest=sha256:…` (image services) | Deploys that image, see below. |
| Docker Hub or GHCR payload | Deploys when the pushed repository and tag are this service's image; otherwise `202` ignored. |
Responses (all `202` unless noted, JSON):
```json
{"status": "deploying", "ignored": false, "deploymentId": "…"}
{"status": "queued", "ignored": false, "reason": "a deployment of this service is running; …"}
{"status": "ignored", "ignored": true, "reason": "pushed tag \"canary\" does not match the service's tag \"stable\""}
```
| Status | Meaning |
|---|---|
| `404` | wrong secret, hook off, service gone |
| `422` | an image reference outside the service's repository (`image_outside_repository`), a tag or digest on a git service, or the service cannot be deployed (for example its server is offline or revoked) |
| `429` | rate limit, with `Retry-After` |
`202 deploying` only says the deployment was created. Follow it with
`GET /api/v1/deployments/` or `nebulactrl deployments status `; a CI job
that must wait can use `nebulactrl services deploy --wait` with an API
token instead.
### A deployment is already running
A service has one active deployment. A hook call that arrives meanwhile is
**queued**, and the newest queued call replaces an older one, exactly like a git
push. When the running deployment ends, the queued call deploys (within about 10
seconds). A queued call whose image no longer belongs to the service (the image
was changed meanwhile) is dropped.
### Rate limit
6 calls per minute per hook with a burst of 3 (`429 rate_limited`, `Retry-After:
10`). Calls with a wrong secret do not use the budget.
### Audit
Every call with a valid secret writes an audit event `deployment.hook` (source,
outcome, image, deployment id, reason; actor "Deploy hook"). Turning the hook on or
off and rotating it write `deploy_hook.enable`, `deploy_hook.disable` and
`deploy_hook.rotate`. No event contains the secret.
## Deploy a specific image: `?tag=` and `?digest=`
For image services:
```sh
curl -fsS -X POST "$HOOK_URL?tag=v1.4.2"
curl -fsS -X POST "$HOOK_URL?digest=sha256:0123…"
```
The tag or digest is applied to the service's own repository: for
`ghcr.io/acme/api:stable`, `?tag=v1.4.2` deploys `ghcr.io/acme/api:v1.4.2`. A
full reference is accepted only when it names the same registry and repository
(`?tag=ghcr.io/acme/api:v1.4.2`); anything else is `422 image_outside_repository`,
so the URL can move a service between versions but never to someone else's image.
`tag` and `digest` together are refused.
**How it fits releases.** The override is **one release, not a configuration
change**: the release records the image reference you passed (in its source), the
service's configured image and its generation are untouched, and nothing becomes
a staged change. Everything else is as for a manual deploy: staged changes are
deployed too, health checks and rollback apply, and **Rollback** to that release
redeploys exactly that image. A later plain deploy (button, hook without `?tag=`)
goes back to the configured image. If you want the service to follow a new tag
permanently, change the image in Settings → Source.
## Registries
Docker Hub and GitHub call the hook with a JSON payload. NebulaCtrl deploys only
when the **pushed repository and tag** match the service's image; every other push
is answered `202` with `status: "ignored"` and the reason, so one webhook per
repository is safe. A service whose image has no tag matches `latest`; a service
pinned by digest never matches a tag push.
### Docker Hub
Repository → **Webhooks** → New webhook, URL = the hook URL.
Payload (from Docker's [webhook documentation](https://docs.docker.com/docker-hub/repos/manage/webhooks/)):
the pushed tag is `push_data.tag`, the repository `repository.repo_name`
(`namespace/name`). Docker Hub sends no signature, so the URL secret is the only
protection.
### GHCR (GitHub Packages)
Repository or organization → Settings → **Webhooks** → Add webhook: Payload URL =
the hook URL, content type `application/json`, **Secret empty**, events: **Packages**.
NebulaCtrl reads GitHub's [`package` event](https://docs.github.com/en/webhooks/webhook-events-and-payloads#package)
(and the older `registry_package` event): the repository is
`ghcr.io//`, the tag is
`package.package_version.container_metadata.tag.name`, and only the `published` and
`updated` actions deploy. A push without a tag (the per-platform manifests of a
multi-architecture image) is ignored; so is GitHub's `ping`. GitHub's optional
`X-Hub-Signature-256` is not verified: the URL is the secret.
Anything else that posts JSON (a body that is neither payload) is a plain call.
## CI examples
GitHub Actions, with the URL in the repository secret `NEBULACTRL_DEPLOY_HOOK`:
```yaml
- name: Deploy to NebulaCtrl
run: curl -fsS -X POST "${{ secrets.NEBULACTRL_DEPLOY_HOOK }}"
```
GitLab CI, with the URL in a masked variable:
```yaml
deploy:
stage: deploy
image: curlimages/curl:8
script:
- curl -fsS -X POST "$NEBULACTRL_DEPLOY_HOOK"
```
`curl -f` makes the job fail on `404`, `422` and `429`.
## API
`GET`, `PUT {"enabled": true|false}` and `POST …/rotate` on
`/api/v1/services/{serviceId}/deploy-hook`; the OpenAPI reference lists them under
**deploy-hooks**. `GET` needs `projects:read`, the others `deployments:write`.
`PUT enabled=true` and `rotate` return the `url` once. The hook endpoint itself
(`/hooks/deploy/…`) is not part of the authenticated API and has no CORS or CSRF
checks: it takes no cookies and no tokens.
---
# SSH deploy keys
Source: https://nebulactrl.com/docs/operations/ssh-deploy-keys
A git service can clone over SSH instead of HTTPS: **Settings → Source → Repository** accepts `ssh://git@host[:port]/org/repo.git` and the scp form `git@host:org/repo.git`. An SSH URL needs a **deploy key** and a **pinned host key**; both are managed per service under Source. They apply immediately and are never staged.
## Deploy key
- **Use SSH: generate a key** creates an Ed25519 key pair on the control plane. The private key is stored encrypted (AES-256-GCM, like every secret), is **write-only** (no endpoint returns it) and travels to the agent as a send-time secret, never in a command payload, release or log. The public key is shown with a copy button, its fingerprint, and, for github.com, gitlab.com, bitbucket.org, codeberg.org and gitea.com, a link to the repository's "add deploy key" page. Read-only access is enough.
- **Upload your own key** stores an unencrypted OpenSSH or PEM private key (Ed25519, ECDSA, or RSA of at least 2048 bits). Passphrase-protected keys are rejected.
- **Generate new key** / **Upload a key** on an existing key is a rotation: the old key stops being used on the next deploy. Add the new public key to the repository first.
- **Remove key** also drops the pinned host keys. A service with an SSH URL and no key fails its deploys with "has no deploy key".
- A pull-request preview of a service copies its key and pins (same repository). Previews from forks do not get them.
## Host key pinning (trust on first use)
The first fetch (a plan, a deploy, **Test connection**) reads the git host's SSH host key and records it per service and host (`host` or `[host]:port`). The control plane and the agent then only accept that key:
- The agent trusts nothing else: it is sent the pinned `known_hosts` line with every deploy and never falls back to `ssh-agent` or `~/.ssh/known_hosts`. A host that is not pinned fails the clone.
- If the host later presents a different key, the deploy fails with "SSH host key for … changed", the refused fingerprint is recorded and shown under **Host key** with **Accept new host key**. Accepting asks for confirmation and pins exactly the fingerprint you reviewed; if the server presents yet another key in between, nothing is pinned (409).
- Accepting is audited (`service.ssh_host.accept`), as are first-use pins (`service.ssh_host.pin`), refused keys (`service.ssh_host.mismatch`) and key changes (`service.ssh_key.create`, `.rotate`, `.delete`).
A mismatch seen only by the agent (for example, interception on the agent's network path only) fails that deploy with the same message in the build log; the control plane's own view of the key is unchanged, so the **Host key** row does not show it.
## Submodules
The deploy key and the pinned host key are also used for `ssh://` and `git@host:` **submodules on the same host and port** as the repository. A submodule on another host or port is not fetched (no key, no pinned host key for it). Hosts that scope a deploy key to one repository (GitHub) need the key added to the submodule repository as well, or a machine-user key that can read both. See [git submodules](https://nebulactrl.com/docs/operations/builds#git-submodules).
## API
`GET /services/{id}/ssh` (`projects:read`), `POST /services/{id}/ssh/key` (generate, or upload with `privateKey`; `rotate: true` to replace), `DELETE /services/{id}/ssh/key`, `POST /services/{id}/ssh/test`, `POST /services/{id}/ssh/host-keys/accept {fingerprint}` (all `projects:write`).
## Limitations
- One key per service; there is no team-level key shared by several services (upload the same key to each).
- One pinned key per host and service. SSH certificates, `@cert-authority` and hashed known_hosts entries are not supported.
- Only `ssh://` and scp-style URLs; passwords inside the URL are rejected. SSH through a proxy or jump host is not supported.
- Webhook-driven auto-deploy and commit statuses need a git connection (HTTPS API); an SSH service can still be deployed by hand, by CLI or API.
---
# Variables and secrets
Source: https://nebulactrl.com/docs/operations/variables
Variables are encrypted at rest (AES-256-GCM). A variable marked secret is write-only: no API, MCP tool or UI returns its value. Changes are staged and reach a container on the next deploy.
## Syntax
A value is plain text with `${{ … }}` expressions:
| Form | Meaning |
|---|---|
| `${{slug.KEY}}` | A variable, export or platform variable of another service in the same environment. |
| `${{KEY}}` | A variable (or platform variable) of the same service. |
| `${{shared.KEY}}` | A shared variable: the environment's value, else the project's. |
| `${{team.KEY}}` | A team variable: one value for every project of the team. |
| `${{vault:/#}}` | A secret in an external secret manager (Vault, OpenBao, Infisical), fetched when a deploy starts and never stored. See [secret managers](https://nebulactrl.com/docs/operations/secret-managers). |
| `$${{` | A literal `${{`. |
| `${{urlencode(x)}}`, `${{base64(x)}}` | Transforms of a reference or a quoted string (resolve time). |
| `${{secret(len, alphabet)}}` | A random string. `len` 1..1024 (default 32). `alphabet`: `alnum` (default), `hex`, `base64`, `base64url`, `digits`, `lower`, or a quoted literal of 2..128 distinct printable ASCII characters without `$ { }`. |
| `${{randomBytes(n, enc)}}` | `n` random bytes (1..1024) as `base64` (default), `hex` or `base64url`. |
| `${{randomInt(min, max)}}`, `${{uuid()}}` | A random integer (inclusive), a UUIDv4. |
Limits: 64 expressions per value, functions nest at most 3 deep, 64 KiB per value. Syntax errors return `422 invalid_template` with the offset. `shared`, `team` and `vault` are reserved: no service can use them as a slug. `vault.KEY` is a syntax error that points at the `vault:/#` form.
## Generators run once
Generators run in the write path only, with `crypto/rand`. The generated text is stored (encrypted) together with the template; a deploy never regenerates it. Writing the same template text again keeps the value and the version, so `nebulactrl vars set TOKEN='${{secret(32)}}'` is safe in CI. `secret`/`randomBytes` force the variable to be secret (an explicit `isSecret:false` is rejected with `secret_required`). Values shorter than 8 characters warn (`weak_secret`); values shorter than 6 characters are not masked in logs.
Rotation is explicit: `POST /services/{id}/variables/{key}/rotate {"dryRun":true}` returns the plan (consumers, deploy order, downtime, warnings, blockers); `dryRun:false` stages the new value and restages consumers. Nothing deploys. Database credentials and `*PASSWORD*` variables of volume-backed services cannot be rotated once deployed (`rotation_unsupported`): the image reads them only when it initialises its data directory. Use a second role or the database panel. `rotateEveryDays` sets a reminder only.
## Layers and restaging
Service variables, environment-shared variables, project-shared variables and team variables. A shared variable reaches a service only through an explicit `KEY=${{shared.KEY}}` row ("Share to services" stages it); a team variable the same way with `KEY=${{team.KEY}}`. Shared values may reference only other shared variables, team variables, external secrets, generators and transforms; team values only other team variables, external secrets, generators and transforms. Cycles are rejected at write time (`reference_cycle`); a reference to something that does not exist yet is a warning, and blocks a deploy (`unresolved_reference`).
When a write changes the resolved environment of other services (a provider's password, a shared value, a deleted environment override), those services are restaged: their generation advances and the staged changes show the row with `origin: inherited` and `via`.
### Team variables
`${{team.KEY}}` is the third shared layer, next to the project and the environment, and applies to every project of the team. Edit them in the UI under Shared variables → Whole team, or through `GET|PUT /team/shared-variables`, `DELETE /team/shared-variables/{key}`, `POST /team/shared-variables/{key}/rotate|share` and `GET …/history`. Same behaviour as the other layers: secrets are write-only, a change restages every service that resolves to the value (in any project), cycles are rejected, versions and audit entries (`shared_variable.set|delete|rotate` with `"layer":"team"`) keep fingerprints only. Only owners and admins may change them; a member or token restricted to some projects has no access, because the values reach every project.
## Platform variables
User keys named `PORT` or starting with `NEBULA_` are reserved. Release-scoped (referenceable): `PORT`, `NEBULA_SERVICE_NAME`, `NEBULA_ENVIRONMENT`, `NEBULA_PRIVATE_HOST`, `NEBULA_PUBLIC_DOMAIN`, `NEBULA_PUBLIC_URL` (always `https://`), `NEBULA_PROJECT_NAME`, `NEBULA_SERVICE_ID`, `NEBULA_ENVIRONMENT_NAME`. Known once the source is pinned (own service only): `NEBULA_GIT_COMMIT_SHA`, `NEBULA_GIT_BRANCH`, `NEBULA_IMAGE`. Per deployment: `NEBULA_DEPLOYMENT_ID`, `NEBULA_RELEASE_ID` (a rollback carries the new deployment's id). Per container (set by the agent): `NEBULA_REPLICA_INDEX`, `NEBULA_REPLICA_COUNT` (the count at the time the container was created).
Exports (for example a database's `DATABASE_URL`) are computed by the resolver, not stored, and never injected into the exporting service. A user variable with the same key wins.
**Ports depend on who asks when the provider is on another server.** `${{api.PORT}}` and every port-bearing export (`DATABASE_URL`, `DATABASE_PORT`, `PGPORT`, `REDIS_URL`, the `DATABASE_WRITE_*` and `DATABASE_READ_*` family of a [replica group](https://nebulactrl.com/docs/operations/database-replicas#writer-and-reader-variables)) resolve **per consumer**: to the provider's mesh port when the provider is exposed over the [mesh](https://nebulactrl.com/docs/operations/mesh#cross-server-service-links) and runs on a different server than the consumer, and to its container port otherwise. The host (`NEBULA_PRIVATE_HOST`, the slug) never changes. The mesh port equals the container port whenever that port is free on the provider's server, so most references keep their value. Editing a variable that references a service on another server never adds a mesh link by itself: the staged bar says "Mesh: adds link A → B (1 device)" and the deploy of the consumer adds it.
## Build-time variables (git services)
Scope `runtime` (default), `build` or `both`. Plain build/both values become `--build-arg`; secret ones (including values that derive from a secret) become BuildKit secrets (`--secret id=KEY,env=NCBS_KEY`, the value in the docker process environment, never argv): use `RUN --mount=type=secret,id=KEY,env=KEY …`. A secret is never offered as a build arg. Build scope needs a git source (`build_scope_requires_git`) and, for secrets, Docker 23+ on the host (`buildkit_required`; there is no silent fallback). Git SHA/branch variables are runtime only, to keep the layer cache.
## Suggestions
Suggestions come from two places: the template env files of the repository and the databases of the environment.
### Keys from `.env.example` (prefill)
The files `.env.example`, `.env.sample`, `.env.template` and `.env.dist` are read from the repository root and the build context (64 KiB and 500 keys at most). `.env`, `.env.local` and `.env.production` are never read. They are read **before the first deploy**:
- through the git provider's API when the service is linked to a GitHub, GitLab, Gitea or Forgejo connection, or the URL is a public repository on github.com or gitlab.com;
- otherwise (Bitbucket, other hosts, SSH, a private repository behind a token, a provider that fails) by a scan-only command on the service's server: a depth-1 fetch of the branch head into memory, nothing checked out, nothing built. A pasted token is sent at send time only and is never stored in the command or the database. An SSH repository can only be read once the service exists and has a deploy key.
Smart defaults per key: a database export of the environment becomes a reference (`DATABASE_URL=${{postgres.DATABASE_URL}}`); a secret-looking key with a placeholder becomes a generator (`${{secret(32)}}`); a safe example value is kept; a value pointing at localhost is cleared; placeholder text such as `changeme` or `` is never proposed as a value; platform keys (`PORT`, `NEBULA_*`) are left out and reported. The comment above a key is its hint.
- **Create dialog:** the Git repository dialog reads the repository as it is filled in and, when keys are found, offers a Variables step. Every key is checked; a key that needs your own value is a field. Creating the service stages the checked keys with it (`source: "suggested"`).
- **Variables tab:** a banner "N keys from .env.example aren't set" opens the same review sheet for bulk add. "Read again" fetches the files now.
- **Kept current:** the files are read again when a git service is created or its repository, branch, build context, server or token changes (a background job), and on every push to the tracked branch. A push that lists its changed files and touches no template file skips the read.
- **Deploy plan:** variables added from the example that still have no value produce the warning "3 variables from .env.example have no value" and list them. It never blocks a deploy; setting a value (any normal write) clears it.
API: `POST /services/{id}/env-template-scan` and `POST /environments/{id}/env-template-preview` (nothing is created or stored by the preview). A repository that cannot be read answers `status: unavailable` with the reason instead of an error.
### Databases
`GET /services/{id}/suggested-variables` also lists databases of the environment that nothing references yet. Accepting a suggestion is a normal staged write.
## History and audit
Every change appends a version (kept: 50 per key, 180 days). Non-secret versions keep their value (restorable as a new version); secrets keep only an HMAC fingerprint derived from the instance secret key (matched against every key of the [key ring](https://nebulactrl.com/docs/security/secrets-at-rest) after a rotation). Audit entries (`variable.set|delete|rotate|restore|share`, `shared_variable.*`) list keys, versions, non-secret values (truncated) and fingerprints, plus the affected services; never a secret value.
## Redaction
Logs are masked for every secret value, every secret substituted into a derived variable, and their URL-escaped and base64 forms (values of 6+ characters), on the server and the agent. Values fetched from a secret manager are masked the same way: the control plane keeps them in memory only (the last three deploys of a service, for at most 30 minutes) and re-fetches them on demand, once a minute at most.
CLI: `nebulactrl vars ls|set|rm|rotate|history`, `nebulactrl vars shared ls|set|rm|rotate|share `.
---
# Networking
Source: https://nebulactrl.com/docs/operations/networking
What can talk to what on an enrolled server, and the three controls you have: the private network every environment gets, opt-in **shared networks** that connect chosen services of different projects, and a per-service **Block outbound internet** switch.
For the threat model behind this (what is isolated and what is not) read the [network section of the threat model](https://nebulactrl.com/docs/security/threat-model).
## The layout on a server
| Network | Created for | Who is on it |
|---|---|---|
| `nebulactrl-edge` | the server | **Traefik**, the error-pages server and `cloudflared` (when a tunnel is on). No app, task or tailnet container is on it. |
| `nebulactrl-env-` | each environment, on the servers it runs services on | the environment's services (alias = the service slug), and Traefik while a route needs it |
| `nebulactrl-env--int` | an environment with a service that blocks outbound internet; Docker `internal`: no gateway | the blocked services, and every other service of the environment (same alias), and Traefik while a route needs it |
| `nebulactrl-shared-` | a team shared network, on one server | the services attached to it (alias `..`) |
| `nebulactrl-mesh` | a mesh-enabled server ([mesh](https://nebulactrl.com/docs/operations/mesh)) | the server's mesh node `nc-mesh-`, which also joins the environment networks of the services it exposes to other servers |
A mesh **link node** (`nc-meshl-:` as usual and a link node carries the connection over the tailnet. The environment boundary still holds: a link belongs to one environment, its access rules are generated for that environment's services only, and containers of another environment on the same server have no alias for it and no route to it. A service that blocks outbound internet still reaches the providers it is linked to, because the link node is on the `-int` network. The providers' ports are the container ports when free on their server, otherwise a mesh port in 15000-15999 (variable references follow).
## Shared networks
Use one when services of **different projects** (or environments) on the **same server** must reach each other: a shared Postgres for several projects, an internal API used by two products.
- A shared network belongs to the **team** and to **one server**. It never spans servers (the mesh does not carry shared networks) and never crosses teams: another team's network does not exist as far as you are concerned (404).
- **Settings → Networks → New network**: a name (unique in the team) and the server. Needs the `servers:write` scope (owners and admins). The page lists every network with its server and the services attached, each with the hostname to copy.
- **Service → Settings → Networking → Shared networks**: tick the networks to join. The list shows the networks on the service's server. The hostname the service gets on each is shown for copying: `..`, for example `api.production.shop`. Needs `deployments:write` on the service's project.
- Attaching and detaching are **staged** like every setting: the service shows *Staged* and **N changes to apply**, the review sheet lists `networks.`, discarding it puts the attachment back as it was, and the next deploy joins or leaves the network. Rolling back restores the networks the old release had (a network that was deleted since is simply left out).
- A service on a shared network is on its environment network too; it keeps reaching its environment by slug and other teams' services only through the shared network.
- A canvas card on a shared network shows a small network chip; hover it for the hostnames.
What the control plane refuses:
| Code | Meaning |
|---|---|
| `network_wrong_server` (422) | the service runs on another server than the network |
| `service_no_server` (422) | the service is not placed on a server yet |
| `egress_blocked` (422) | the service blocks outbound internet (see below): shared networks and outbound blocking are exclusive, because a shared bridge would give the service its internet back |
| `network_in_use` (409) | deleting a network that services are still attached to |
| `network_name_taken` (409) | a network with that name exists in the team |
Moving a service to another server **detaches** its shared networks (they cannot follow it); the change is staged and shows in the review sheet. Deleting a service frees its attachments.
Audit actions: `network.create`, `network.delete`, `network.attach`, `network.detach` (the service and network names, never anything secret).
## Block outbound internet
Service → Settings → **Networking** → *Block outbound internet*. It is part of the service's configuration, so it is **staged and released** like the port or the health check: it takes effect on the next deploy, and a rollback restores the value the old release had.
A blocked service joins only the environment's `-int` network. A Docker `internal` network has no gateway, so the container has **no route to the internet, the LAN or the host**. Every other service of the environment joins `-int` too under the same alias, so it still reaches the blocked service; the proxy reaches it over `-int` as well.
Defaults:
- **Managed databases: on.** A database needs no internet, and backups do not need one either: they stream through the agent. You can switch it off for one database (for example to install an extension on start-up) in its settings.
- **Apps and workers: off.** Most apps call something: package registries, payment or mail APIs, webhooks.
Things to know:
- **Calls out fail.** Anything that resolves or connects to the internet from that container fails: updates, webhooks, third-party APIs, `npm install` at start. Cron tasks and one-off commands run with the same network membership as the service. Builds run on the host and are not affected.
- **Deploy the others too.** A service deployed *before* a sibling started blocking outbound is not yet on the `-int` network; it joins it the next time it is deployed. The deploy plan of the service you changed says so ("Other services in this environment join the internal network the next time they are deployed"). Redeploy the services that need to reach it, or use **Deploy staged** on the environment.
- **The plan warns about outbound-looking variables.** If a service that blocks outbound has variables that look like they call the internet (a public URL, `SMTP_*`, `STRIPE_*`, `AWS_*`, `S3_*`, `SENTRY_*`, …), the plan lists their names. It is a hint from names and values, not a guarantee in either direction.
- **A blocked service cannot join shared networks** (`egress_blocked`). To share a database across projects, allow outbound for it, attach it, and rely on the shared network's membership for access control. A service that may reach the internet is not filtered at all.
- **A tailnet sidecar still works.** The sidecar keeps its own outbound access (it needs it to reach the tailnet's control server) and also joins `-int`, so it reaches the blocked service; only the service container is cut off.
- **A mesh link still works.** The link node joins `-int`, so a blocked consumer reaches the providers it is granted on other servers (and a blocked replica reaches its primary) and nothing else ([mesh](https://nebulactrl.com/docs/operations/mesh#what-runs-on-a-server)).
- Route services (external upstreams and TCP/UDP exposures) have no container, so the setting does not apply to them.
## Address pool
Docker hands out subnets of new networks from its *default address pools* (`172.16.0.0/12` split into /16s and `192.168.0.0/16` split into /20s): about 30 networks per host. Every environment needs one network, or two with an outbound block, so a host with many environments or previews failed with `all predefined address pools have been fully subnetted`.
The agent therefore gives every `nebulactrl-env-*`, `nebulactrl-env-*-int` and `nebulactrl-shared-*` network an explicit subnet out of its own pool, **`10.213.0.0/16` split into /24s: 256 networks** of up to 253 containers each. The edge network `nebulactrl-edge` is created with Docker's defaults and does not use the pool.
| Flag | Environment | Default |
|---|---|---|
| `--network-pool` | `NEBULACTRL_NETWORK_POOL` | `10.213.0.0/16` |
| `--network-prefix-len` | `NEBULACTRL_NETWORK_PREFIX_LEN` | `24` |
How a subnet is picked (one network at a time, under a lock):
- The **first free** subnet of the pool, lowest address first. A subnet freed by a removed network is used again.
- A subnet is free when it overlaps neither the IPAM subnet of **any** Docker network on the host (not only NebulaCtrl's) nor an address range of a network interface of the agent. Inside a container that means the daemon's networks are what counts.
- If Docker still refuses the pick as overlapping (a host route the agent cannot see, for example a VPN), the agent tries the next subnet. After 64 refusals it stops and reports it.
- A network that **already exists keeps its subnet**, whatever it is: upgrading the agent never recreates a network, so networks created before the pool existed stay where they are (and are swept when they empty, then come back from the pool).
- Only IPv4 pools are supported; the networks are IPv4.
When the pool is full the deployment fails with a message such as `network address pool exhausted: no free /24 subnet left in 10.213.0.0/16 for network nebulactrl-env-… (256 subnets …); widen the pool with --network-pool …`. Fix it by, in this order:
1. Removing unused networks: `docker network prune` removes the ones nothing uses; the agent's sweep removes empty `nebulactrl-env-*` and `nebulactrl-shared-*` networks (and leftovers of the old `nc-env-*` / `nc-shared-*` names) by itself after 60 seconds.
2. Using smaller subnets: `--network-prefix-len 26` gives a `/16` pool 1024 networks of 61 containers each.
3. Widening the pool: `--network-pool 10.208.0.0/12` is 4096 networks of /24.
Change the pool if `10.213.0.0/16` collides with something you reach from the host (a LAN, VPN or office network): pick a range none of them uses. Changing it affects only networks created afterwards.
Set them as environment variables in `/etc/nebulactrl-agent.env` (the systemd unit loads it) and restart the agent, for example `NEBULACTRL_NETWORK_POOL=10.208.0.0/12`. An invalid pool or prefix length stops the agent at start.
## Operating notes
- **See what is where:** `docker network ls --filter label=nebulactrl.managed=true` on the server; `docker network inspect nebulactrl-env-` lists the members. The environment id's first eight characters are in the URL of the environment's services (API: `environmentId`).
- **A service cannot reach a sibling after a change:** check that both were deployed since the change (see *Deploy the others too*). A service that only has a shared-network hostname must be on the network *and* the target must be too.
- **Traefik returns 502 for a service that works:** the proxy joins a service's network when its route is applied (every route carries the networks it needs). Redeploy the service; if the proxy itself was recreated, an agent reconnect re-applies every route.
- **A leftover network** after deleting everything: it is removed by the agent's next sweep (it runs every minute and on every reconcile), once nothing is connected to it and it is at least 60 seconds old. Networks with a container still connected are never removed.
- **API:** `GET/POST /networks`, `GET/DELETE /networks/{id}`, `PUT/DELETE /services/{id}/networks/{networkId}`; the setting is `config.blockEgress` of `PATCH /services/{id}`. Omitting `blockEgress` from a config PATCH keeps the current value.
## Limits
- No per-destination egress rules: a service is either allowed out entirely or not at all.
- Networks never span servers and there is no overlay network. Services of one environment on different servers can reach each other through the opt-in [mesh](https://nebulactrl.com/docs/operations/mesh) (TCP only, tagged tailnet devices that count against your plan); across environments and for anything else use a tunnel, a tailnet or a public domain.
- Docker networks are not a sandbox for hostile tenants: everything on a host shares its kernel and Docker daemon. See the [threat model](https://nebulactrl.com/docs/security/threat-model).
---
# Domains and proxy routes
Source: https://nebulactrl.com/docs/operations/proxy-routes
Every server runs its own Traefik (managed by the agent). This page covers what you can publish through it beyond "one hostname → one service": route services (external upstreams and TCP/UDP exposure), per-domain options, and how all of it deploys. Server-level settings (uploaded certificates, the upstream policy, the tailnet) are in [proxy-tailnet-certificates.md](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates).
Everything here is **staged** like any other service change: it shows up as a pending change and takes effect on the next deploy. Writes need the `proxy:write` scope (session users hold it; MCP token presets never get it) in addition to `projects:write`. A server whose agent is too old to render these features answers `422 agent_too_old` instead of silently ignoring them.
## Route services
A route service (`kind: route`) is a canvas node without a container. It has a server, a protocol and one or more upstreams:
| Protocol | What it does |
|---|---|
| `http` | Publishes an upstream NebulaCtrl does not run (a NAS on the LAN, a port on the host, a tailnet device) under one or more hostnames, with the full set of domain options below. |
| `tcp` | Exposes `listenPort` of the server and forwards to the upstream, optionally restricted to client CIDRs and with the PROXY protocol. |
| `udp` | Same for UDP. There is no per-client restriction (see below). |
An upstream is either an external `host:port` or a managed service of the same environment (`serviceId` + `port`) on the **same server**; the latter resolves to that service's current containers, so "expose my Postgres on port 25432" keeps working across deploys of the database.
```
POST /environments/{environmentId}/routes create (Idempotency-Key supported)
GET|PATCH|DELETE /routes/{routeId} PATCH needs If-Match (service generation)
GET /servers/{serverId}/ports TCP/UDP ports held by route services
```
The protocol and the server of a route cannot change; delete and recreate it instead. `DELETE /services/{id}` does the same as `DELETE /routes/{id}`: the listen port is freed and, for a deployed route, the remaining table is pushed to the server in the same transaction, so the proxy stops publishing it. Restart, stop, logs, volumes and scaling do not apply to a route (`422 not_applicable`). Server drains skip route services.
Templates that publish a port (for example a game server) create the same kind of route service.
### Listen ports
TCP/UDP ports are static Traefik entrypoints on the host. A listen port is 1024-65535 and never `80`, `443` or the proxy's internal probe/admin ports (`422 invalid_listen_port`). A port is held per server **and protocol**; a second route asking for the same one gets `409 port_in_use` naming the holder, even before the first is deployed.
**Adding a new TCP/UDP port restarts the proxy on that server** (the agent recreates the Traefik container with the extra published port). Expect about a second of dropped connections, and HTTP is briefly unavailable too. The deploy plan says so (`New TCP port N restarts the proxy on ...`). If the new container fails to start (for example the host port is taken by something else), the agent puts the previous container and routes back and the deployment fails with a message naming the port. Changing only upstreams or CIDRs does not restart anything.
### Security notes for TCP/UDP
- **Docker bypasses ufw/firewalld.** Published ports are opened by Docker's own iptables rules, so a host firewall rule such as `ufw deny 25432` does *not* protect the route. Use `allowCidrs` (and `denyCidrs`) on the route, or a firewall in front of the server (cloud firewall, VPN).
- A TCP route needs either `allowCidrs` or an explicit `openToInternet: true`; the plan warns when it is open to everyone. `denyCidrs` can be added to either (see [IP rules](#ip-rules-allow-and-deny)).
- **UDP has no IP rules.** Traefik has no UDP middleware, so `allowCidrs`, `denyCidrs` and `proxyProtocol` are rejected for UDP (`422 invalid_config`). Put UDP behind a firewall or VPN if it must not be public.
- **Public database ports** are TCP routes with a mandatory allow-list: see [Managed databases](https://nebulactrl.com/docs/operations/databases#public-port).
### Upstream policy
External upstreams are checked against the server's upstream policy twice: by the control plane when you create or edit the route (`422 upstream_denied`) and by the agent when it applies the table (so a stale or hand-edited table cannot bypass it). By default:
- denied: unspecified addresses (`0.0.0.0`), loopback (`127.0.0.0/8`, `localhost`), link-local (`169.254.0.0/16`, which includes cloud metadata endpoints such as `169.254.169.254` and `metadata.google.internal`), the proxy's own containers (`nebulactrl-*`) and the proxy admin port;
- allowed: LAN addresses, CGNAT/tailnet addresses (`100.64.0.0/10`), public addresses, DNS names, and `host.docker.internal` (the Docker host, enabled by default per server).
Admins can allow loopback and link-local per server (Server → Proxy settings). The control plane validates against the policy the agent has *applied*, so a loosened policy takes effect for new routes after "Deploy proxy" on that server.
## Domain options
Every domain of a service or an http route service has a routing rule. Edit it with `PATCH /domains/{domainId}` (`If-Match` = the owning service's generation; the response carries the new ETag). A plain hostname change needs only `projects:write`; everything else needs `proxy:write`.
| Field | Meaning |
|---|---|
| `pathPrefix` | Serve only this path and below, e.g. `/api`. The same hostname can carry several domains with different prefixes; `(hostname, pathPrefix)` is unique (`409 domain_conflict`). |
| `stripPrefix` / `addPrefix` | Remove the prefix before forwarding / prepend a path before forwarding. |
| `kind: redirect` | Answer with a redirect to `redirectTo` (a hostname, not a URL) with `redirectCode` 301/302/307/308 (default 308; Traefik only distinguishes permanent from temporary, so 301/308 render a permanent redirect, which answers 301 to a GET and 308 to other methods, and 302/307 a temporary one: 302 for a GET, 307 otherwise), keeping path and query when `preservePath` (default). A redirect domain takes no options, auth or prefix rewriting. The target must differ from the domain itself and must not close a cycle with the workspace's other whole-host redirects (`422 redirect_loop`). The classic use is `www.example.com` → `example.com`. |
| `wwwRedirect` | `true` on a whole-host `serve` domain (no path prefix) keeps a **www ↔ apex redirect** for it: the counterpart hostname (`www.`, or the host without its `www.`) is routed as a whole-host `kind: redirect` domain with code **301** and `preservePath`, pointing at this domain. So `example.com` with the option on also answers `www.example.com` with a 301 to `https://example.com/…`; on `www.example.com` it redirects the apex to `www`. Both hostnames are routed, so the proxy requests a certificate for each (the redirect copies the domain's `tlsMode`, pinned certificate and DNS-01 provider; an uploaded certificate must cover both names, `422 certificate_mismatch`), and each needs its own DNS record: both rows show their DNS status and the "Point an A record for … to " guidance. The redirect is an ordinary domain row (it shows in the list, in pending changes and in release snapshots, and is rolled back or discarded like any domain). It follows the domain's rename and TLS settings; `false` removes it and deleting the domain deletes it. Domain views carry `wwwPartner` (the counterpart name; empty for wildcards, redirects and path-prefixed domains), `wwwRedirect` (the redirect exists) and, on the redirect row, `wwwRedirectFor`. `422 invalid_www_redirect` for an ineligible domain, `409 domain_conflict` when the counterpart is already taken. Needs `proxy:write`, like every redirect. UI: the **Redirect www.… to this domain** switch in the domain editor, and **Also route … and redirect it here (301)** when adding an apex or `www.` domain. CLI: `nebulactrl domains add --www-redirect`, `nebulactrl domains www on|off`. |
| `tlsMode` | `auto` (ACME when the server has an email and the name is public, otherwise the default or an uploaded certificate), `custom` (never ACME: an uploaded certificate selected by SNI, else the default one), `none` (plain HTTP only, no redirect). |
| `certificateId` | Pin an uploaded certificate (only with `tlsMode: custom`). It must exist on the service's server and cover the hostname, wildcards matching exactly one label (`422 certificate_mismatch`). Traefik still selects by SNI; the pin is validation and documentation. |
| `options` | The typed middleware set, validated as a whole (`422 invalid_options`): `forceHttps` (default true), `compress`, `noindex`, `securityHeaders` (`basic` or `strict`, which adds HSTS), `requestHeaders`/`responseHeaders`, `cors`, `redirects` (regex), `rateLimit`, `access` (IP allow and deny lists with optional trusted-proxy depth for CDNs, see [IP rules](#ip-rules-allow-and-deny)) and `basicAuth`. |
Headers that must not be forged (`Host`, `X-Forwarded-For`, `Set-Cookie`, hop-by-hop headers, ...) are rejected. Path prefixes are `/segment[/segment]` without trailing slash.
### IP rules: allow and deny
A domain (`options.access`) and a TCP route (`allowCidrs`, `denyCidrs`) take two lists of CIDRs or single addresses (IPv4 and IPv6, at most 200 each). They are evaluated **deny first**:
1. a client that matches a `denyCidrs` entry is refused, even when the allow list would admit it;
2. otherwise a client is admitted when it matches an `allowCidrs` entry;
3. an **empty allow list means everyone not denied** (so a deny list alone blocks a bad range and leaves the rest open). On a TCP route that is only accepted together with `openToInternet: true`; on a database public port the allow list is never empty.
Traefik has no deny middleware, so the agent renders both lists as **one `ipAllowList`**: the allow list (or every address when it is empty) with the denied ranges subtracted (the allowed ranges are split around them into CIDRs). The decision is identical to the ordered rules above. A rule set that admits nobody (for example `allow 10.0.0.0/8, deny 10.0.0.0/7`) is rejected (`422 invalid_options` for a domain, `invalid_config` for a route), as is a deny list that splits the allowed space into more than 4096 networks (for example more than 32 single IPv6 hosts carved out of `::/0`, or about 128 single IPv4 hosts out of `0.0.0.0/0`); use broader entries. The web UI shows the ordered effective rules next to the lists. `trustedProxyDepth` (domains) picks the client address behind a CDN, as before; rejected clients get `rejectStatus` (default 403) on a domain, and the connection is closed on a TCP route. The post-deploy verification probe skips access rules, as before.
On a domain locked to Cloudflare (the origin lock, see [Cloudflare](https://nebulactrl.com/docs/operations/cloudflare)) the rules are evaluated against the **real visitor address**: the entrypoints trust `X-Forwarded-For` only from Cloudflare's ranges, and `ipStrategy.depth: 1` picks the last entry, the address Cloudflare saw connect. Entries to its left are whatever the visitor sent and are never read, so `trustedProxyDepth` (access and rate limit) is ignored on such a domain. The lock itself is a separate `ipAllowList` on the direct peer address (Cloudflare's ranges, no `ipStrategy`), runs first and answers a peer outside the ranges with a plain `403 Forbidden`. Traefik's `ipStrategy` can only read the peer address or `X-Forwarded-For`, so `CF-Connecting-IP` is not used; Cloudflare puts the same address at the end of `X-Forwarded-For`. A request from a Cloudflare address without any `X-Forwarded-For` has no visitor address and is refused when a user rule is set.
### Basic auth
```
PUT /domains/{domainId}/auth-users [{username, password?}] replaces the user set
DELETE /domains/{domainId}/auth-users/{username}
```
Passwords are hashed with bcrypt (cost 10) and the hash is stored sealed; responses carry user names only. A user that already exists may omit the password to keep it. Enable the protection with `options.basicAuth: true` *after* adding users (`422 basic_auth_needs_users` otherwise); the last user cannot be removed while it is on. The hashes never appear in route tables, deploy payloads, audit entries or logs: they are attached to the command as `auth:` secrets when it is sent and the agent keeps them only in its own route table. A rule with basic auth enabled and no resolvable users is a hard error on the agent: it never renders an unprotected route.
### Pending changes
Domain edits appear in the pending changes as `domains.` (added/removed) and `domains..` for `stripPrefix`, `addPrefix`, `kind`, `redirectTo`, `redirectCode`, `preservePath`, `tlsMode`, `certificateId`, `options` (compact JSON) and `authUsers` (names). Discarding restores the released value, except `authUsers` (passwords are not kept).
## How a route deploys
A **regular service** deploys as before; its payload now also carries the per-domain rules and the server's applied proxy configuration. Domain edits therefore apply with the service's next deploy (there is no fast path that only reloads routes for a regular service: container names are tied to the current deployment).
A **route service** deploys *route-only*: the deployment has strategy `route_only`, nothing is built, pulled or started. The release snapshots the route config and its domains, and one `sync_routes` command carries the server's full route table (including this route, built from the new snapshot) and the proxy configuration. The agent applies it, reports `route` and `verify` events and a result like any deployment; for an http route it probes the hostname through the proxy and treats status < 500 as routed (set `verifySkip` for upstreams that legitimately answer 5xx). Verification checks routing, not access rules.
- Plan warnings: an external http route without basic auth or an IP allowlist ("this route has no access control" and "verification checks routing, not access rules"), TCP open to the internet, UDP without allowlist, Docker bypassing ufw, and the proxy restart for a new port.
- Plan blocking: `agent_too_old`, an http route without a domain (`route_needs_domain`), a managed upstream whose service is not deployed on the route's server (`upstream_not_deployed`).
- Rolling back a route re-sends the table of the old release. Deleting a deployed route pushes the remaining table.
- When a regular service is deployed, routes that target it are rewritten to its *new* container names in the table the agent applies after the health check, so exposed ports follow the deploy. *Deploy staged* orders routes after the services they point at.
- A reconnecting agent receives the full table (services, external routes, proxy configuration and the auth/certificate secrets) in its reconcile frame and re-applies it.
## Limitations
- Staged options apply on deploy; basic-auth users apply live with the next route push (any deploy, scale or route sync on that server) because they are resolved at send time from the current users.
- A new TCP/UDP port restarts the proxy briefly; there is no hot-add of listeners.
- A route can only target managed services on its own server; cross-server upstreams are external addresses.
- Deleting a managed service does not delete route services that target it: the route drops out of the table (no upstream) until the target is deployed again or the route is removed.
- Basic-auth passwords cannot be recovered or discarded back; set them again.
- The control plane cannot reach upstream addresses itself: reachability is verified from the server by the agent's post-deploy probe (http only).
---
# Import a Compose project
Source: https://nebulactrl.com/docs/operations/compose-import
**Import Docker Compose** has two modes. **Split into services** (this page) reads a `docker-compose.yml` once and stages one ordinary service per Compose service in an environment: the same services you could have created by hand, with normal deploy, rollback, variables, domains and logs; it is a one-time import, not Compose at run time. **Run as stack** keeps the file as it is and runs it with `docker compose` as one service: see [Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks). Nothing deploys until you have reviewed the result and press **Deploy** (or choose "Stage and deploy").
Where: **Create → Import Docker Compose** on the canvas, the CLI `nebulactrl import compose`, or the API (`POST /environments/{id}/compose-imports/plan` and `/compose-imports`). It needs `projects:write` (`deployments:write` for deploy-right-away); reading the file from a git repository also needs a git connection or a public GitHub / GitLab.com repository.
## What the wizard does
1. **Source.** Paste the file, upload it (plus its `.env` and any files it references), or point at a file in a repository: a connected git provider (private repositories; not Bitbucket, which has no file API here) or the https URL of a public repository on github.com or gitlab.com. From a repository the importer also reads `.env`, the `env_file` entries and the bind-mounted files next to the Compose file (at most 40 reads, 1 MiB each).
2. **Review.** The plan shows, per Compose service, what it becomes; every row can be changed. Nothing is written yet. The plan is recomputed from the same input plus your choices, so what you review is what is imported.
3. **Stage.** One transaction creates the services, volumes, files, variables, domains, links, published ports and a canvas group named after the Compose project, and writes one audit event, `compose.import`. Replica counts, shared networks and the repository link follow once it is committed; a failure there becomes a warning, never a rollback of the import.
## How things map
| Compose | NebulaCtrl |
|---|---|
| `image` | image service (the tag is kept; a digest is resolved when it is deployed) |
| `build` (context, dockerfile) | git source built with the repository's Dockerfile. Context and Dockerfile are made relative to the repository root (the Dockerfile path is relative to the context in Compose and to the repository here). The repository is the one you imported from, the `git://`/`https://` context URL (`url#branch:subdir`), or a repository you give in the review. Build `args` and `target` are not applied |
| `command` | start command (shell form is split into words like Compose does) |
| `entrypoint` | the service's **entrypoint** Docker option (replaces the image's entrypoint and its default command, like Compose; `command` is passed to it). An empty entrypoint that would clear the image's is not applied and flagged |
| `environment`, `env_file`, `.env` | variables. Interpolation (`$VAR`, `${VAR:-default}`, `${VAR:?error}`, `$$`) is done by the importer against the `.env` contents; unset variables become empty strings and are listed |
| secrets | a key that looks like a credential (`PASSWORD`, `TOKEN`, `SECRET`, `*_KEY`, `DSN`, ...), a URL with a password or a long random token is flagged as a **secret** (encrypted, never shown again). You confirm or change every flag in the review |
| `ports` | the first HTTP-like container port becomes the service port, with a generated domain suggested (on when Compose published it); Traefik `Host()` labels give explicit hostnames. Other published ports can become **TCP/UDP routes**, off by default, with the Compose host port as the public port. Known database and broker ports are treated as plain TCP (no domain) |
| `PORT` variable | sets the service port; `PORT` and `NEBULA_*` are platform variables and are not imported |
| named `volumes`, anonymous volumes | service volumes (created empty: existing data in your Compose volumes is not copied) |
| bind-mounted **file** (≤ 64 KiB, at most 16 files and 256 KiB) | a config file written into the container before it starts. `*.sh` files get mode 0755. Never put secrets in them: they are stored in plain text |
| bind-mounted **directory** or host path | **rejected** with guidance. Tick "Use a named volume instead" to create a volume at that path; the files in the host directory are not copied |
| Docker socket, `/proc`, `ipc`, `network_mode`, `pid` of another container | never applied; listed per service as danger notes. (`privileged`, `devices`, `pid: host` and the other host-level options are applied or reported, see [Docker options and the target server](#docker-options-and-the-target-server)) |
| `depends_on`, `links` | deploy order (links on the canvas). `condition: service_completed_successfully` makes the dependency a **one-shot** service that must finish first |
| `healthcheck` | an exec health check running the Compose `test` inside the container (interval capped at 10 s, deadline from `start_period` and `retries`); `disable`/`NONE` turn it off. Without one, a service with a port gets a TCP check (not HTTP, so a 404 on `/` cannot block the deploy) |
| `deploy.resources.limits`, `mem_limit`, `cpus` | memory and CPU limits |
| `deploy.replicas`, `scale` | replicas, applied after the import and only for stateless services (no volumes, with a port; at most 10) |
| `stop_grace_period` | stop grace period |
| `cap_add`, `cap_drop`, `security_opt`, `read_only`, `pids_limit`, `hostname`, `dns`, `sysctls`, `user`, `group_add`, `devices`, `gpus` and `deploy.resources.reservations.devices`, `runtime`, `privileged`, `pid: host`, `ulimits`, `shm_size`, `extra_hosts`, `init`, `stop_signal`, `tmpfs` | [Docker options](https://nebulactrl.com/docs/operations/docker-options), judged against the target server: see [below](#docker-options-and-the-target-server). Each value is checked on its own; a refused one is listed with the reason and never takes the others down |
| `networks` | the environment network; every service of the environment reaches the others by name. Several Compose networks are merged (a warning says so). An `external` network attaches to the team's [shared network](https://nebulactrl.com/docs/operations/networking) of the same name when there is one |
| `labels` | ignored, except Traefik `Host()` rules and the load-balancer port |
| `profiles` | services behind a profile are not imported unless you switch them on |
| `extends` (same file), YAML anchors and merge keys | resolved. `extends` from another file and `include:` are not followed |
| `restart`, `container_name`, `logging`, `pull_policy`, `platform`, `tty`, ... | ignored (the platform decides) |
| anything else (`working_dir`, `secrets`, `configs`, `dns_search`, `userns_mode`, `uts`, ...) | listed per service as not applied |
## Docker options and the target server
Compose keys that set container options become the service's typed [Docker options](https://nebulactrl.com/docs/operations/docker-options). Two tiers apply, the same ones the settings page and the stack allow-list use:
| Tier | Compose keys | On a restricted server | On an unrestricted server |
|---|---|---|---|
| every server | `cap_add` (safe list), `cap_drop`, `security_opt: no-new-privileges`, `read_only`, `pids_limit`, `hostname`, `dns` (at most 3 IPs), `sysctls` (curated list only), `user`, `group_add` (numeric IDs), `ulimits`, `shm_size`, `extra_hosts`, `init`, `stop_signal`, `tmpfs`, `entrypoint` | applied | applied |
| unrestricted servers only | `devices`, `gpus` and `deploy.resources.reservations.devices` with `capabilities: [gpu]` (the NVIDIA driver; `count`, extra capabilities such as `compute`), `runtime` (other than `runc`), `privileged`, `pid: host`, `security_opt: seccomp=unconfined` / `apparmor=unconfined`, `cap_add: NET_ADMIN` / `SYS_ADMIN` | **not applied**, listed as "Needs an unrestricted server" in the review | applied, with a warning note per option so you review it |
The target is the server chosen in the review. Without a server (or while the choice changes) the plan assumes a restricted one; switching the server recomputes it. An instance admin turns a server unrestricted under **Servers → the server → Unrestricted**; importing again then applies the options, or you can set them in the service settings afterwards. Nothing is dropped silently: whatever is not applied is listed per service with its reason.
Never applied, on any server, and listed as not applied:
- `devices` given as CDI names (`vendor.com/gpu=all`) or outside `/dev`; a GPU `device_ids` selection (use a count); any device reservation without the `gpu` capability; a GPU driver other than `nvidia`; a second GPU request (the first one is used)
- `security_opt` seccomp or AppArmor profiles, SELinux labels and `systempaths`
- `sysctls` outside the curated list (for example `vm.max_map_count`: it changes the host and cannot be set from a container; declare it as a host requirement in the service settings instead)
- `group_add` names (`video`), `hostname` values that are not one DNS label (`mail.example.com`), `dns` entries that are not IP addresses, `user` values that are not `uid[:gid]`
- `pid: service:...` and `pid: container:...`
- `read_only: true` when the service has config files that are not on a volume (a read-only root filesystem cannot take them): put the file on a volume or turn it off
- `cap_drop` of a capability that is also in `cap_add`
`runtime: runc` is Docker's default and needs nothing. A managed database does not take container options; the review says so.
## References between services
Variables that point at another Compose service by name are rewritten to references so they keep working and show up as edges on the canvas:
- `DB_HOST=postgres` → `DB_HOST=${{postgres.NEBULA_PRIVATE_HOST}}`
- `http://api:3000/v1` → `http://${{api.NEBULA_PRIVATE_HOST}}:3000/v1`
- `redis://cache:6379/1`, `host=db port=5432`, `db:5432`, and a whole-value host name in a host-like variable (`*_HOST`, `*_URL`, `*_ADDR`, `*_SERVER`, ...).
Only unambiguous positions are rewritten. A bare word is never treated as a host in a variable that is not host-like (`POSTGRES_DB=app` stays), an e-mail address is not a host, a longer name (`db.example.com`) is not the service `db`, and a service never references itself. Every rewrite is listed in the review with its before and after (passwords masked); in the variables table you can keep a value as it was ("Keep original host names"). A reference also becomes a deploy-order link, unless it would close a cycle. The services' slugs are what references use; if a slug already exists in the environment the new service gets a numbered one and the references follow it.
A literal `${{` in a Compose value is escaped so it can never become a reference or a generator.
## Managed databases
An **official** database image (postgres, mysql, mariadb, redis, valkey, mongo, clickhouse; plain version tags, optionally `-alpine`, `-bookworm`, ...) with a version the [managed catalog](https://nebulactrl.com/docs/operations/databases) offers can become a managed database. It is **on by default** when eligible; switch it off per service to keep the image as it is. Redis and MongoDB (source-available licences) are off until you switch them on, which acknowledges the licence.
A service is not eligible when it overrides the command, mounts init scripts or other files, has volumes besides the data directory, sets variables the engine does not own, uses an unsupported version or image variant, or (MySQL/MariaDB) connects as `root`. The review says why.
When a service becomes managed:
- user and database name are kept (`POSTGRES_USER`, `POSTGRES_DB`, ...); the **password is generated** and the credential variables of the Compose service are not imported;
- the old password is replaced by `${{db.DATABASE_PASSWORD}}` (`REDIS_PASSWORD` for Valkey and Redis) in the places that use it as a password: the `user:password@` part of a URL, `password=` in a connection string, or the whole value of a password variable;
- a Redis URL without a password (`redis://cache:6379`) gains one: `redis://:${{cache.REDIS_PASSWORD}}@${{cache.NEBULA_PRIVATE_HOST}}:6379`; a variable that only holds the host needs `REDIS_PASSWORD` set by you (the review does not guess);
- the data volume is created empty. Restore a dump or a backup after the first deploy ([database backups](https://nebulactrl.com/docs/operations/database-backups)).
## Secrets and privacy
The plan response never contains a secret value: secret variables come back as `value: ""` with `secret: true`. The pasted source (including `.env`) is processed in memory and not stored; only the resulting variables are, encrypted. The audit event records counts and host names, never values. Config files and non-secret variables are stored in plain text like everywhere else.
## Blockers
The import refuses to run (`422 import_blocked`, listing the reasons) while any of these holds, and the review shows them next to the cause:
- a required `${VAR:?message}` variable has no value;
- a service with a `build` has no repository to build from (give one, or switch it to an image);
- a referenced `env_file` or bind-mounted file was not supplied;
- a hostname is already used by another service, a TCP/UDP port is taken, or the server is revoked or draining.
## CLI
```sh
# review, change nothing
nebulactrl import compose ./docker-compose.yml --project shop --env production --plan
# stage on server box1 (reads .env and referenced files from next to the file)
nebulactrl import compose ./docker-compose.yml --project shop --env production --server box1 \
--domain web=shop.example.com --no-managed cache --publish db=5432/tcp
# from a repository
nebulactrl import compose - --repo acme/shop --connection github-acme --branch main --path deploy/compose.yaml \
--project shop --env production --server box1
```
Per-service decisions are `=` (`--include`, `--exclude`, `--managed`, `--no-managed`, `--git url#branch`, `--image`, `--domain host|auto|off`, `--port`, `--publish port/proto`, `--convert-bind /path`) and `.` for variables (`--secret`, `--not-secret`, `--skip-var`, `--no-rewrite`). `--file '='` supplies a referenced file explicitly. `--deploy` deploys right away. Everything supports `--json`.
`--as-stack` imports the file as one [Compose stack](https://nebulactrl.com/docs/operations/compose-stacks) instead of splitting it (API: `mode: "stack"` on the plan and import requests; split stays the default). The plan lists the components, the suggested domains, the `.env` variables and the allow-list violations (a violation, or `build:`, blocks the import with `422 invalid_compose`; `--server` decides whether an unrestricted server lifts the restricted ones). In this mode `--secret`, `--not-secret` and `--skip-var` take the variable name (``), `--domain` and `--port` take the component name, and the flags that shape split services (`--include`, `--exclude`, `--managed`, `--no-managed`, `--publish`, `--git`, `--image`, `--convert-bind`, `--no-rewrite`) are refused.
```sh
nebulactrl import compose ./docker-compose.yml --project shop --env production --server box1 --as-stack --plan
```
## Limits
- It is an import, not a sync: changing the Compose file later does not update the services, and importing again creates new services (existing slugs get a numeric suffix).
- Volumes, bind-mounted directories and the data of existing containers are not migrated; shared named volumes become independent copies per service (a warning lists them).
- Services run on one server, on the environment network; Compose network separation is not kept.
- Build args and targets, `working_dir`, Compose secrets and configs and the other keys listed above are not applied.
- Only the first HTTP-like port gets a domain; other ports need an explicit TCP/UDP route.
- Reading from git uses the provider's file API (GitHub, GitLab, Gitea, Forgejo) or the public raw host (github.com, gitlab.com); there is no clone, so a file larger than 1 MiB or a repository path outside the Compose file's directory (`..`) cannot be read.
---
# Import from Coolify
Source: https://nebulactrl.com/docs/operations/import-coolify
**Import from Coolify** reads a Coolify v4 instance through its REST API and stages the same projects, environments, applications and databases here. It is a **one-time migration helper**: everything lands as ordinary, *staged* services you review and deploy yourself, and nothing is deployed unless you ask. **Data is never copied** (databases, volumes); the plan prints the follow-up steps, and [Restore from a dump file](https://nebulactrl.com/docs/operations/database-backups#restore-from-a-dump-file) copies a Postgres or MySQL database.
Where: **Create → Import from Coolify** on the canvas, the CLI `nebulactrl import coolify`, or the API (`POST /imports/coolify/plan` and `/imports/coolify/apply`). It needs `projects:write` on an unrestricted token or session (the import creates projects), plus `deployments:write` for `--deploy`. It is **not** available over MCP: a Coolify token and every variable value of another instance must not pass through a model.
## What you need in Coolify
1. **API access enabled** on the instance (Settings → Advanced → API access), reachable from the NebulaCtrl **control plane** (the control plane makes the requests, not your browser).
2. An **API token with the `read` and `read:sensitive` abilities** (Keys & Tokens → API tokens). `read` lists resources; `read:sensitive` is what makes Coolify return **variable values, custom Docker run options, pre/post-deployment commands and database passwords**. Without it the plan is blocked: Coolify answers with those fields hidden, and an import would silently drop them. The token owner must be a team admin or owner. ([Coolify: API permissions](https://coolify.io/docs/api/permissions), [authorization](https://coolify.io/docs/api/authorization).)
Use a separate token and revoke it after the migration. The token is sent in the request body to your NebulaCtrl server, used for that single request, and **never stored, logged, audited or returned**; the plan carries no variable values.
The control plane follows no redirects (so the token cannot be forwarded to another host) and refuses link-local addresses (cloud metadata); private LAN and tailnet addresses are allowed, because that is where Coolify usually lives. Errors never repeat Coolify's response body.
## The flow
1. **Connect / plan.** `nebulactrl import coolify --url https://coolify.example.com --plan` (the token comes from `$COOLIFY_TOKEN`, `--token-file`, or `--token`). The plan lists every resource, its variables **by key** (never values), the references that were rewritten, and everything that is not imported. Nothing is written.
2. **Decide.** Flags (or the wizard's controls) exclude resources, rename projects and services, choose an existing project, override ports and domains, flip secret flags, supply locked values and registry credentials. The plan is recomputed from Coolify on every call, so what you review is what is imported.
3. **Stage.** `nebulactrl import coolify --url … --server ` creates, in one transaction: the projects and environments, services, managed databases, volumes, variables, domains, deploy-order links, and one audit event `coolify.import` (counts only). Git links follow once it is committed; a failure there is a warning, never a rollback.
4. **Review and deploy** from the canvas (staged bar) or `nebulactrl services deploy --wait`. `--deploy` deploys everything right away, one environment at a time.
```sh
export COOLIFY_TOKEN=… # read + read:sensitive
export GHCR_TOKEN=… # a pull credential for ghcr.io (Coolify's own login cannot be exported)
# 1. review
nebulactrl import coolify --url https://coolify.example.com --plan
# 2. stage (nothing runs yet)
nebulactrl import coolify --url https://coolify.example.com --server homelab-1 \
--registry-user ghcr.io=xpertn-bot --registry-token-env ghcr.io=GHCR_TOKEN \
--git-connection github
```
Re-running is safe: a project whose name exists gets a ` (Coolify)` suffix (or use `--rename-project` / `--into`), a service slug that is taken becomes `name-2`, and a hostname already used by a NebulaCtrl service is left out with a warning. Existing services are never changed.
## How things map
### Projects and environments
A Coolify project becomes a NebulaCtrl project and each of its environments an environment (`production` / `prod` is flagged **production**). If a project of that name exists, the new one is named ` (Coolify)`; override with `--rename-project =`, merge with `--into =` (environments with the same name are reused, services are only added), or leave it out with `--skip-project`.
### Applications
| Coolify | NebulaCtrl |
|---|---|
| Docker image (`dockerimage`, or an image name without a repository) | **image service** with the image and tag. A missing tag becomes `latest` with a warning; `sha256:` tags become a digest. Registry credentials are not readable from Coolify: the plan lists every registry host that needs one and the services that use it; give `--registry-user host=user` and `--registry-token-env host=VAR` (stored per service, write-only, sent at deploy time like any registry credential) |
| Dockerfile (`dockerfile`) with a git repository | **git service**, build method *Dockerfile*. Coolify's *Dockerfile location* is relative to its *base directory*; here the path is relative to the repository root, so `base /apps/web` + `/Dockerfile` becomes Dockerfile path `apps/web/Dockerfile`, build context `apps/web` |
| Nixpacks, Railpack, static | **git service**, build method **Railpack** (Nixpacks' successor) with a warning. Coolify's install, build and start commands (and the publish directory of a static site) become the build variables `RAILPACK_INSTALL_CMD`, `RAILPACK_BUILD_CMD`, `RAILPACK_START_CMD` (`RAILPACK_SPA_OUTPUT_DIR`): check the build preview before you deploy. `NIXPACKS_*` variables are not read by Railpack |
| Docker Compose application, or a Dockerfile stored inside Coolify | **not imported**, listed with the reason (import the Compose file with `nebulactrl import compose --as-stack`; commit an inline Dockerfile to a repository) |
| Git repository | the clone URL comes from Coolify (`owner/name` becomes `https://github.com/owner/name`). The branch is kept; a commit Coolify pinned is reported (NebulaCtrl builds the head of the branch). Private repositories: Coolify's GitHub App and deploy keys **cannot be exported**; select a git connection (`--git-connection`) so GitHub/GitLab repositories are linked, or add an SSH deploy key in the service's Source settings |
| `ports_exposes` | the first port is the service **port** (the rest are noted). An application with **no domain, no health check, no published port and no other service pointing at it** becomes a **worker** without a port, with a note; override with `--port =` / `--kind =app` |
| `ports_mappings` (published host ports) | **not imported**: add a TCP/UDP route for the port |
| `fqdn` (every domain) | one **domain** per entry. `http://` entries keep plain HTTP allowed; an explicit `:port` is the target port; a path becomes a path prefix; `noindex_domains` become the *noindex* option; `gzip`/`compress` in the Traefik labels turns on *compression*; auto-generated `sslip.io`/`nip.io`/`traefik.me` hosts become a generated NebulaCtrl hostname. Invalid or already-used hosts are left out with a note. Redirects (www), basic auth, IP allow lists and other Traefik **labels are not imported**: they are listed, and set per domain under Domains → Rules |
| health check | enabled HTTP check → an **exec** check that runs Coolify's own request inside the container: `curl` (else `wget`) against `://:`, with the interval, timeout and start period; deadline = start period + retries × interval. Coolify ran its checks inside the container too, so the image has `curl` or `wget`, and frameworks with a host allow list (Laravel trusted hosts, Django `ALLOWED_HOSTS`, Rails host authorization) accept `localhost` where they refuse a probe from outside addressed to the container IP. A command check → an exec check. Disabled → **none** (the release counts as healthy once it ran 10 s without restarting), or the image's own `HEALTHCHECK` when Coolify found one. Methods other than GET and response-text checks are not supported and are noted; like Coolify's own check, any non-error status passes |
| `limits_memory`, `limits_cpus` | memory (MiB) and CPU limits. Swap, reservation, CPU set and CPU shares are not supported and listed |
| pre-/post-deployment command | the service's **pre-deploy** / **post-deploy** command. Coolify ran them with `sh -c`; a command with shell syntax keeps it by running under `sh -c`, a plain command becomes an argv. The pre-deploy command runs once per deployment in a one-shot container of the new image before the old release is touched |
| persistent storages | **named volumes** (the app-uuid prefix is removed). The data is **not** copied; host bind mounts and file storages are listed as not imported |
| scheduled tasks | listed as not imported (add them under Tasks → Cron) |
| preview-only variables | not imported |
### Custom Docker run options (`custom_docker_run_options`)
The option string is split with a real shell-word parser (quotes, escapes, line continuations; nothing is expanded or executed) and mapped to the typed [Docker options](https://nebulactrl.com/docs/operations/docker-options). A quoted `--entrypoint "docker-php-serversideup-entrypoint bun /var/www/html/bootstrap/ssr/ssr.js"` becomes the exec-form argv `["docker-php-serversideup-entrypoint","bun","/var/www/html/bootstrap/ssr/ssr.js"]` (Coolify splits it the same way); an entrypoint replaces the image's default command, so the start command stays empty.
| Applied | Needs an **unrestricted server** (flagged; applied only when the target server is unrestricted) | Listed as not imported |
|---|---|---|
| `--entrypoint`, `--cap-add`, `--cap-drop`, `--security-opt no-new-privileges`, `--shm-size`, `--ulimit nofile/memlock`, `--add-host`, `--init`, `--tmpfs`, `--hostname`, `--dns`, `--sysctl` (curated list), `--user`, `--group-add`, `--pids-limit`, `--read-only`, `--stop-signal`; `--memory` / `--cpus` when Coolify has no limit of its own | `--privileged`, `--device`, `--gpus`, `--runtime`, `--pid host`, `--security-opt seccomp=unconfined` / `apparmor=unconfined`, `--cap-add NET_ADMIN` / `SYS_ADMIN` | `--network`, `--publish`, `--volume`, `--env`, `--label`, `--restart`, `--ip`, `--cpuset-cpus`, health flags, other ulimits, `-it` and everything the parser does not recognise |
Each option is validated on its own, so one refusal never drops the others. Whatever is not applied is listed per service with the reason; options that need an unrestricted server say so ("An instance admin has to switch the target server to Unrestricted… import again, or set it in the service settings"). Pick the target server first so the plan shows what it accepts.
### Databases
Each database becomes a [managed database](https://nebulactrl.com/docs/operations/databases) with the **same engine** and, where the catalog offers it, the same version and flavor. Coolify's image decides the engine (a `redis` database running `valkey/valkey:8` is Valkey):
| Coolify image | Managed |
|---|---|
| `postgres:` | PostgreSQL `` (Alpine variants become the Debian image, with a note) |
| `postgis/postgis:-` | PostgreSQL with the **PostGIS** flavor (for example `postgis/postgis:18-3.6` stays 18 / PostGIS 3.6; a different PostGIS minor is noted) |
| `mysql:`, `mariadb:` | MySQL / MariaDB: the same version, else the same major, else the next offered (MySQL 8.0 → 8.4 LTS) |
| `valkey/valkey`, `redis`, `eqalpha/keydb`, `dragonflydb/dragonfly` | Valkey (Redis → Redis 8; KeyDB and Dragonfly → Valkey, Redis-compatible) |
| `mongo`, `clickhouse/clickhouse-server` | MongoDB, ClickHouse |
| SQLite, `timescale/…`, `supabase/postgres`, `bitnami/…`, any other image | **not imported**, with the reason; run them as an image service with the same image |
The user and database name are kept; **a new password is generated** and the old one is not imported. A version older than the catalog's oldest is moved to the next offered one (a logical dump restores into a newer version); newer than offered is not imported. Redis and MongoDB have source-available licences and are **off by default**: `--include ` is the acceptance. Public ports, custom config files, init scripts, SSL mode, Docker options and resource limits are listed as not carried over. **The data is not copied**; the plan prints, per database, the follow-up, for example:
```sh
# on the Coolify host
docker exec pg_dump -U expertn -Fc expertn > expertn-postgres.dump
# against NebulaCtrl
nebulactrl databases restore expertn-postgres --file expertn-postgres.dump
```
Key-value databases start empty (cache, queue and session data can usually be rebuilt; drain the queues first). MongoDB and ClickHouse need `mongodump`/`mongorestore` or the engine's own tools.
### Variables
Every variable is imported with its value, except those NebulaCtrl owns or Coolify injects: `PORT` and `NEBULA_*` (platform variables), `COOLIFY_*`, `SERVICE_FQDN_*`, `SERVICE_URL_*`, `SOURCE_COMMIT`, and preview-only variables. Shared-variable references (`{{environment.X}}`) are inlined with the value Coolify resolves, with a note. A literal `${{` in a value is escaped so it stays literal.
- **Secrets.** A variable is a secret when Coolify locked it (*shown once*), when its name looks like a credential (`*_KEY`, `*SECRET*`, `*PASSWORD*`, `*TOKEN*`, `*_DSN`, `API_KEY`, `*_KEK*`, …; public keys and `*_KEY_ID` are exempt), when its value is a URL with a password, or when it became a reference to a database password. The plan says why (`secretReason`) and you can flip any flag (`--secret`, `--not-secret`).
- **Locked variables.** Coolify never returns the value of a locked variable. They are listed with `?` and **not imported** until you supply the value with `--set-env .=` (or `--set`, or in the wizard), or set them on the service afterwards. A supplied value is stored as typed, literally: a `${{…}}` in it stays text. To point a variable at another service (`${{db.DATABASE_URL}}`), set it after the import with `nebulactrl vars set`.
- **Build-time flags.** *Available at build time* maps to the variable scope (`build`, or `both` together with runtime). Build scope exists for **git** services; on image services the variable stays a runtime variable, with a note. Secret build values reach the build as BuildKit secrets (Docker 23+ on the server).
- **References.** A value that points at a Coolify database or application **in the same environment** is rewritten to a NebulaCtrl reference, so it keeps working after the services get new names and passwords:
| In the Coolify value | Becomes |
|---|---|
| the database's internal hostname (its UUID), or an application's UUID / network alias | `${{db.NEBULA_PRIVATE_HOST}}` |
| the database password (whole value, in a URL's user info, or a long password inside text) | `${{db.DATABASE_PASSWORD}}` (`REDIS_PASSWORD` for Redis/Valkey) |
| the exact connection URL Coolify shows for the database | `${{db.DATABASE_URL}}` (`REDIS_URL`, `MONGO_URL`, `CLICKHOUSE_URL`) |
| user, database name or port as the whole value of a fitting key (`*_USER(NAME)`, `*_DB`/`*_DATABASE`/`*_NAME`, `*_PORT`), only for a database the service demonstrably uses | `${{db.DATABASE_USER}}`, `${{db.DATABASE_NAME}}`, `${{db.DATABASE_PORT}}` (`REDIS_PORT` for Redis/Valkey) |
| `http://:13714` | `http://${{ssr.NEBULA_PRIVATE_HOST}}:13714` |
Other text around the match is kept (`redis://default:${{cache.REDIS_PASSWORD}}@${{cache.NEBULA_PRIVATE_HOST}}:6379/1`). A reference to a resource that is **not imported**, or lives in **another environment**, is left as it is and flagged. `--no-rewrite .` keeps one value verbatim. Services that reference each other get a **deploy-order link**, so databases deploy first. The rewritten references are listed in the plan (key and new reference; never the old value).
### Coolify one-click services
A Coolify "service" is a Docker Compose stack from a template. It is **not imported automatically**; the plan lists each one (with its project) and the next step: save its Compose file from Coolify and run `nebulactrl import compose --as-stack`, or install the matching [template](https://nebulactrl.com/docs/operations/templates).
## What stays manual
Registry credentials (unless you pass them), private-repository access, the **data** of databases and volumes, DNS (still pointing at Coolify until you move it), Cloudflare/Traefik middlewares and dynamic route files (per-domain access rules, redirects, the external routes such as a Plex upstream: use a route service), scheduled tasks, port mappings, file mounts, locked variable values, backups and notifications.
## CLI reference
```
nebulactrl import coolify --url [--server ] [flags]
--url, --token-env (default COOLIFY_TOKEN), --token-file, --token connection (token is never printed)
--plan | --dry-run review only; exit code 1 when the plan has blockers
--deploy deploy right away (default: staged)
--server required to import
--git-connection link GitHub/GitLab repositories for builds
--skip-project, --rename-project, --into project choices
--exclude, --include, --rename, --port, --kind, --engine-version per application or database
--domain /=|off change or drop a domain
--secret, --not-secret, --skip-var, --no-rewrite . per variable
--set-env .=, --set .= values of locked variables
--registry-user =, --registry-token-env =
```
Resources are named by their Coolify name or UUID; ambiguous names are rejected with the candidates listed. `--json` prints the plan or the result (no values either way).
## Limits
- Coolify v4 only (API `/api/v1`, read from the `v4.x` OpenAPI); fields that are missing or typed differently in other versions are read leniently and degrade to a note.
- The importer reads, per resource, the application, its variables, storages and scheduled tasks, databases, one-click services and servers: a large instance takes a few seconds per resource.
- Coolify teams: a token sees one team; run the import once per team.
- Variables over 64 KiB and more than 500 variables per service are rejected by the variables API like any other write.
## Troubleshooting
| Message | Meaning |
|---|---|
| `Coolify refused the token` (`coolify_unauthorized`) | wrong or expired token, API access disabled on the instance, or Coolify's **Allowed IPs** (Settings → Advanced) exclude the control plane. On the same host the request arrives from the gateway of Coolify's Docker network (`docker network inspect coolify`, for example `10.0.1.1`): add that address for the import and remove it afterwards |
| `does not look like a Coolify v4 instance` | the URL is not the instance (use the address of the dashboard, not a proxied app) |
| `cannot reach Coolify at …` | the control plane cannot connect (firewall, tailnet, DNS); remember the control plane makes the request |
| `The Coolify token cannot read variable values` | the token lacks `read:sensitive` |
| `import_blocked` | the plan has blockers: run with `--plan` and read the `[blocked]` lines |
| a domain was "left out" | the hostname is already used by a NebulaCtrl service (a previous import) |
---
# Compose stacks
Source: https://nebulactrl.com/docs/operations/compose-stacks
A **stack** runs a Docker Compose file as it is. It is one service on the canvas: you paste or upload a `docker-compose.yml`, and the agent runs `docker compose up` for it on the server you choose. The services in the file are the stack's **components**.
Use a stack when you have a Compose file you do not want to take apart (init containers, `depends_on` conditions, sidecars that share a network namespace, a vendor's reference file). Use [Import a Docker Compose project](https://nebulactrl.com/docs/operations/compose-import) with **Split into services** when you want each part to be its own service with its own domains, variables, replicas, backups and rollbacks.
Where: **Create → Compose stack** on the canvas, **Create → Import Docker Compose → Run as stack**, or the API (`kind: "stack"`, `sourceType: "compose"`, `config.stack.compose`). Decision record: [ADR 0007](https://nebulactrl.com/docs/decisions/0007-compose-stacks).
## What a stack is
- One service: **one deploy, one release, one rollback, one set of staged changes.** Editing the compose file, the variables, the domains or the limits stages a change; **Deploy** applies it to the whole stack.
- Components are not services. They have no variables, domains or settings of their own; the stack's variables are shared by the whole file, and a domain points at one component and one container port.
- The compose project is named `nc-`. Containers are named `nc--` and carry the usual `nebulactrl.*` labels plus `nebulactrl.component`. Named volumes are `nc-_`.
- **Recreate only.** `docker compose up` replaces changed containers in place, so a deploy is not zero-downtime. Components restart on every deploy (the release's deployment ID is part of their labels).
- Requires the **Docker Compose plugin 2.24 or newer** on the server. The server page shows the version it found; without it a stack cannot be deployed there (`compose_unavailable`). The installer installs the plugin; on an existing host: `apt install docker-compose-plugin` (Debian/Ubuntu) or `dnf install docker-compose-plugin` (RHEL family), then restart the agent.
## Variables
The stack's variables (including project and environment shared variables, references and secrets) are used for **Compose interpolation**: write `${DB_PASSWORD}` in the file and set `DB_PASSWORD` in the Variables tab. A variable is only given to a component when the file passes it on (`environment: [DB_PASSWORD]` or `DB_PASSWORD: ${DB_PASSWORD}`), exactly as with `docker compose --env-file`.
- Values are sent to the agent with the deploy command and handed to `docker compose` in a **clean environment**: the agent's own environment (its enrollment token, its Docker settings) never reaches the file. `DOCKER_*` and `COMPOSE_*` keys are dropped, except `COMPOSE_PROFILES`.
- Nothing is written to disk by NebulaCtrl except the compose file itself (with the `${...}` placeholders) and the generated override (no values). Docker stores what a container was created with, as for every container.
- Secret values are masked in deployment logs.
- `env_file:` is not supported (a stack has no files next to the compose file): put the values in the stack's variables.
## What the agent changes: the override
The agent never edits your file. It generates a second file, the **override**, and runs `docker compose -f compose.yaml -f override.yaml`. The override:
| Does | Why |
|---|---|
| adds the managed labels (`nebulactrl.managed`, `.service`, `.deployment`, `.release`, `.environment`, `.component`, `.replica`) | status, logs, metrics, the orphan sweep and routing find the containers |
| sets `container_name: nc--` | stable names; also DNS names for Traefik |
| joins every component to the environment network with the component name as alias (and to the environment's internal network or shared networks like any service) | other services of the environment reach `web` by name; [network isolation](https://nebulactrl.com/docs/operations/networking) is unchanged. The project's own default network stays, so components still reach each other |
| removes `ports:` (host publishing) | nothing is bound on the host; HTTP is routed by Traefik through a domain |
| applies the per-component memory and CPU limits you set in the stack's settings (they replace `mem_limit`, `cpus` and `deploy.resources.limits`) | resource control from the UI |
| sets `restart: unless-stopped` for components that set no `restart` | like every service |
| pins images to the release's digests for a rollback or a reconcile redeploy | a rollback runs the images that were running, even for `:latest` tags |
If **Block outbound internet** is on for the stack, the components join only the environment's internal network (the compose file's own networks are replaced).
## Routing
A stack has no `Port`. Add domains like for any service, but pick the **component** and the **container port**; any port from 1 to 65535 is accepted, and the ports the file declares (`ports:` targets and `expose:`) are offered. Traefik routes to `nc--:` over the environment network, with all the usual domain options (redirects, path prefixes, basic auth, IP rules, TLS).
- `ports:` entries are **never published on the host**, whatever the file says. A published TCP/UDP port (a database, a game server) is therefore not reachable from outside; keep such components internal, or use the [public database port](https://nebulactrl.com/docs/operations/databases) of a managed database. HTTP-like ports become domains.
- After the containers are healthy the agent switches the routes and **verifies through the proxy** (`GET /` through the first domain of the stack must answer below 500). A failed verification restores the previous routes and the previous stack.
- Maintenance mode and traffic analytics are not available for stacks.
## Health, deploy and rollback
- Health is Compose's: the agent runs `docker compose up -d --wait` and waits for every component with a `healthcheck` to be healthy and every other component to be running (`depends_on` conditions work). The wait is bounded by **Wait timeout** (default 300 seconds, at most 3600). A component that exits with code 0 (an init or migration container) counts as healthy ("completed").
- Steps in the deployment view: **pull** (validate the file against the allow-list, normalise it, pull images), **create/start** (`up`), **health**, **route**, **verify**.
- **A failed `up`, health check or verification restores the previous release** (the previous compose file and override are re-applied) and the deployment ends `rolled_back`; if that fails too it is `rollback_failed` and the message says how to recover. The first deployment has nothing to restore: its containers are removed, volumes are kept, and the deployment is `failed`. The last 50 log lines of unhealthy components are attached.
- **Rollback** to an earlier release re-applies that release's compose file with the image digests recorded for it. It never touches data: volumes, databases and external services are not rolled back (the dialog says so).
- **Cancel** works at any step (see [Deployments](https://nebulactrl.com/docs/operations/deployments)).
- **Restart** runs `docker compose restart` for the whole stack, **Stop** stops every component (containers are kept), **Delete** runs `docker compose down --remove-orphans`; **Delete volumes** adds `--volumes`.
## What is checked: the allow-list
Docker is root on the host, so a compose file can ask for the host itself. By default NebulaCtrl refuses a file that does. The agent normalises the file with `docker compose config --format json` (anchors, `extends`, short syntaxes and variables resolved) and checks that model; the control plane checks the file as you wrote it, so mistakes show up as you edit and on save. A violation blocks the deploy (`422 invalid_compose`, with the list).
Refused unless the server is **unrestricted** (below):
- `privileged: true`, and privileged lifecycle hooks
- `network_mode` other than `none` or `service:` (so no `host`, `bridge`, `container:`); `pid`, `ipc`, `uts`, `userns_mode`, `cgroup` that name the host or a container outside the stack
- `devices`, `device_cgroup_rules`, `gpus`, GPU reservations, `runtime` other than `runc`, `cgroup_parent`, `storage_opt`, `isolation`, block-device limits
- `cap_add` beyond the [safe list](https://nebulactrl.com/docs/operations/docker-options#capabilities), including `NET_ADMIN` and `SYS_ADMIN` (the instance-wide "advanced capabilities" switch no longer exists: they need an unrestricted server like everything else here)
- `security_opt` other than `no-new-privileges`: `seccomp=unconfined` and `apparmor=unconfined` need an unrestricted server like the typed [Docker options](https://nebulactrl.com/docs/operations/docker-options); profiles and labels are refused too
- `sysctls` outside the curated list of [namespaced container sysctls](https://nebulactrl.com/docs/operations/docker-options) (the same list plain services may set; host-wide parameters such as `vm.max_map_count` are refused)
- **host bind mounts**, in short or long syntax, relative or absolute: this includes the Docker socket (`/var/run/docker.sock`). Named volumes (declared in the file), tmpfs and anonymous volumes are fine; `volumes_from: container:...` is not
- external volumes, custom volume or network `name:`s, volume `driver_opts` (`device`/`type: none`/`o: bind` mount host paths), non-local volume drivers: they could reach another project's data
- external networks, other network drivers (`host`, `macvlan`, `overlay`), network `driver_opts`: they would bypass the environment isolation
- `file:` secrets and configs, external secrets and configs (use `content:` or `environment:`)
- labels starting with `com.docker.`, `nebulactrl.` or `traefik.`
These work on every server, as they do for plain services: `cap_drop`, `no-new-privileges`, `read_only`, `pids_limit`, `hostname`, `dns`, curated `sysctls`, `user` and `group_add`. Everything marked above as needing an unrestricted server (`privileged`, `devices`, `gpus` and GPU reservations, `runtime` other than `runc`, `pid: host`, unconfined `seccomp` and `apparmor`, `cap_add` of `NET_ADMIN` or `SYS_ADMIN`) is the same set the typed Docker options gate behind the server's Unrestricted switch.
Not supported even on an unrestricted server (the stack model cannot carry them):
- `build:` (the agent runs images; build elsewhere, or use [Split into services](https://nebulactrl.com/docs/operations/compose-import) with a git source)
- `include`, `extends` with a `file:`, `env_file`, top-level `models`, `provider:` services (they need other files or run host-side plugins)
- `scale` or `deploy.replicas` above 1 (container names are fixed)
- service names that are not lower-case letters, digits, `-` or `_`, up to 63 characters (they become container names and DNS aliases)
A component whose name equals the slug of another service in the same environment (or of a component of another stack there) is refused (`stack_alias_conflict`): both would answer to the same name on the environment network.
## Unrestricted (per server)
An **instance admin** can switch the allow-list off for one server: **Servers → the server → Unrestricted**. It is the same switch that lets plain services use [devices, GPUs, a runtime, privileged mode and the other host-level Docker options](https://nebulactrl.com/docs/operations/docker-options#the-unrestricted-switch-per-server), because the two give identical power: anyone who can deploy a one-service stack on an unrestricted server can already ask for `privileged: true` and a host bind mount. The dialog warns that anyone who can deploy on that server can take it over (privileged mode, host networking, devices, the Docker socket, host bind mounts) and asks you to type the server's name. It is recorded in the audit log (`server.unrestricted.enable` / `.disable`, with the number of stacks and of services that use unrestricted-only options). The API is `PUT /servers/{id}/unrestricted` (it was `PUT /servers/{id}/compose` and the field `unrestrictedCompose`; both were renamed without an alias), the CLI `nebulactrl servers unrestricted`.
- The flag is read at **every deploy**, including rollbacks and reconcile redeploys. A rollback to a release that needed it fails on a server that is restricted again; so does the next deploy of such a stack. Switching it off does not stop running stacks.
- Only the allow-list is lifted. Ports are still never published on the host, the unsupported list above still applies, and containers still carry the managed labels and join the environment network (unless the file asks for `network_mode`).
- Keep unrestricted servers for stacks you wrote yourself. See the [threat model](https://nebulactrl.com/docs/security/threat-model).
## Logs, metrics, terminal
- **Logs:** all components interleaved, each line prefixed `component | `; filter by component in the UI (`?component=` in the API). History and drains keep the prefix.
- **Metrics:** the charts show the whole stack (CPU and memory summed over its containers); the last sample lists each component. The component list in the Overview shows the same numbers per component.
- **Terminal and files:** the shell and the file browser open in one component; the component's position in the sorted component list is the `replica` index.
## Import
**Create → Import Docker Compose** has two modes: **Split into services** (the one-time import described in [compose-import.md](https://nebulactrl.com/docs/operations/compose-import)) and **Run as stack**. Stack mode keeps the file as written (nothing is rewritten, `${VAR}` stays), turns the `.env` entries into the stack's variables (credential-looking keys are flagged secret), suggests a domain for components that publish an HTTP-like port (or carry a Traefik `Host()` label), shows the components and any allow-list violations, and stages one stack. Nothing deploys until you deploy. From the CLI: `nebulactrl import compose --as-stack`.
Directory and file bind mounts that the split importer can convert into config files or volumes are **not converted** in stack mode; they are violations: replace them with named volumes or `configs:` with `content:`.
## Known limits
- No rolling updates, replicas, autoscaling, maintenance mode, one-off tasks or cron tasks, volume backups (file-level) or preview environments for stacks.
- `build:` is not supported; images are pulled by tag at deploy time (`pull` happens on every deploy) and a rollback pins the recorded digests.
- Traffic analytics and request origins do not cover stacks.
- Components are not services: no per-component staging, deployments or variables.
- Only HTTP is routed. Published TCP/UDP ports are removed.
- A stack lives on one server; moving it creates a new deployment there and does not move volumes.
- Compose project directories are not backed up beyond the compose file kept in the release; named volumes are your data.
---
# Pull-request previews
Source: https://nebulactrl.com/docs/operations/previews
A preview is a real environment that NebulaCtrl creates for every open pull
request of a repository connected to the project, deploys with the pull
request's head commit, and deletes again when the pull request is closed or
merged. It needs a [git connection](https://nebulactrl.com/docs/operations/git-providers) (GitHub, GitLab, Gitea or
Forgejo) with webhooks, and at least one service of the project's base
environment linked to the repository.
Open it from **Project → Previews**. The page lists the open previews (status,
age, last push, URLs, delete) above the settings. The environment switcher
groups previews under the environment they were cloned from, labelled
`PR #12 · title`; a preview's canvas carries a banner with its pull request,
status and a delete button.
## What a preview is
| | |
|---|---|
| Environment | `environments.kind = 'preview'`, slug `pr-`, name `PR # · `, `parent_environment_id` = the base environment. Standard environment APIs work on it. |
| Services | A copy of the base environment's services. Services connected to the pull request's repository build the head commit on the PR branch; every other service is cloned as it is (image services run the same image, git services of other repositories build their own branch). Route services (external upstreams) are not cloned. |
| Databases | Created **fresh**: the same engine and version, new generated credentials, an empty data volume. Nothing from the base database (data, credentials, extra variables) is copied. References such as `${{postgres.DATABASE_URL}}` resolve to the preview's database. |
| Variables | Copied from the base services, including secret variables, so the preview runs like the base. Variables that contain generators (`${{secret(32)}}`) are generated again. The environment-level shared variables of the base environment are copied; project-level ones apply as before. |
| Volumes | Declared again, **empty**. |
| Domains | Each service that has a public domain in the base environment gets a generated `<service>-pr-<n>.<base domain>` (the server's base domain, else `<ip>.sslip.io`). If the name is taken, the project slug and then a counter are added. Custom domains are never copied (hostnames are unique). |
| Canvas | The base environment's layout, so the preview looks the same. |
The first deployment starts the databases and other providers first; a service
whose variables reference another service waits until that one succeeded. If a
deployment fails, services that depend on it are not started and the preview is
marked `failed` with the reason.
## Lifecycle
| Event | What happens |
|---|---|
| PR opened / reopened | The environment is cloned and deployed. A comment with the status and URLs is posted on the pull request and a commit status `nebulactrl/preview` (pending, then success or failure) is set on the head commit. |
| New push | The services built from the repository redeploy at the new head commit (pinned to the SHA). Other services keep running. A duplicate webhook delivery does nothing. |
| PR closed / merged | The preview is deleted: every service with its container, **volumes** and domains (`remove_service` with `deleteVolumes`), then the environment. The comment is updated to say so. |
| Over the limit | The least recently pushed running preview is **paused**: its services are stopped, volumes stay. The next push to its pull request resumes it (stopped services start again, the pushed commit is deployed). |
| Idle | A preview with no push for the configured idle days is deleted (checked every 10 minutes). Paused previews count as idle too. |
If a push arrives for a pull request without a preview (for example after an
idle deletion) one is created. The sticky comment is found by a marker
(``), so every project that previews a
repository gets one comment per pull request.
## Settings
`GET/PUT /api/v1/projects/{id}/previews/settings` (`projects:read` / `projects:write`):
| Setting | Default | Notes |
|---|---|---|
| `enabled` | off | |
| `baseEnvironmentId` | the production environment | A standard environment of the project. A base environment cannot be deleted while previews of it exist (`409 environment_has_previews`). |
| `forkPrs` | off | See [Fork pull requests](#fork-pull-requests). |
| `maxConcurrent` | 5 | 1–50 previews running at once. Past it the oldest is paused. |
| `idleDays` | 7 | 1–365. |
| `excludedServices` | none | Service slugs of the base environment that previews do not get. |
| `seedCommands` | none | `[{service, command, runOn}]`, below. |
### Seed commands
A preview's database starts empty. A seed command runs a one-off job (the same
mechanism as one-off commands in service settings) in a service of the preview
once **all deployments of the wave succeeded**, for example
`npm run db:migrate && npm run db:seed` in the app that owns the schema.
`command` is a shell command line run with `sh -c` in the service's current
image with its environment and networks. `runOn` is `create` (default: the first
deployment only) or `push` (also after every later deployment). Seed commands
run one after the other; a failing one (non-zero exit or timeout) marks the
preview `failed` and is shown in the comment, the commit status and the list.
Output is in the service's run history.
## Fork pull requests
Off by default: a fork's code is not yours, and a preview runs it. When an admin
turns `forkPrs` on, fork previews **never receive secrets**:
- no secret variable, and no variable whose value references a secret shared
variable or the vault (`${{shared.KEY}}`, `${{vault.…}}`);
- no secret environment-level shared variables;
- no git token, and no repository link: the clone uses the fork's public HTTPS
URL (the services that build the repository get `gitUrl` = the fork and the
PR branch). Private services of other repositories in the project cannot be
cloned for a fork preview;
- no registry credentials for private images.
Credentials of the preview's own databases are generated for the preview. Pull
requests from forks also never get commit statuses or comments through the
service's repository link; the comment and aggregate status of the preview
itself are still posted.
Pull-request previews of same-repository branches are trusted like a push to
that repository: whoever can push a branch can run code with the base
environment's secrets. Restrict who can push, or keep previews off for projects
whose secrets must not reach every contributor.
## Where previews build
Previews follow the [build placement](https://nebulactrl.com/docs/operations/build-servers#where-builds-run) of their services, with one difference: **a preview never builds on a shared build server** unless that server has **Allow previews** on. A preview's code is the pull request's (and for a fork, untrusted), so by default it is built on the preview's own target server, as it was before build servers existed, even when its base services build on a build server. When a build server does allow previews:
- the preview builds there like any deployment, with its own per-project BuildKit builder and cache, and its image is pushed to the registry under the usual repository and a unique `r-<release8>` tag (the repository name has the project and service, not the environment, so a preview's repository can be the base service's repository; deployments always pull the **digest**, never a tag);
- fork previews are still built from untrusted code on that server, with the registry push credential and the other projects' builders on the same host. Keep **Allow previews** off for any build server that matters, or dedicate a build server to previews and give it a registry credential that cannot touch your production repositories.
If a preview's services resolve to a build server but none allows previews, the build runs on the target and the preview's timeline says so ("No build server available; building on <target>"); this never blocks the preview.
## Limits and costs
Each preview runs every service of the base environment (minus exclusions): size
`maxConcurrent` by memory, and exclude heavy services (a search cluster, a
worker) that previews do not need. A paused preview keeps its volumes (disk) but
no memory. Deleting a preview removes its volumes, including the preview
database.
## API
| | |
|---|---|
| `GET /projects/{id}/previews` | the project's previews: status (`creating`, `deploying`, `seeding`, `ready`, `failed`, `paused`, `deleting`), PR, head commit, URLs, `createdAt`, `lastPushAt` |
| `DELETE /previews/{id}` | start deleting a preview (202); needs `projects:write` |
| `GET/PUT /projects/{id}/previews/settings` | above |
Audit actions: `preview.settings.update`, `preview.create`, `preview.deploy`,
`preview.pause`, `preview.delete` (system actor for the automatic ones).
Notifications: `preview.created` and `preview.deleted`.
## Troubleshooting
- **No preview appears.** The base environment must contain a service linked to
the repository the pull request belongs to (Service → Source), previews must
be enabled, and the connection's webhook must deliver (Settings → Git →
Deliveries). Fork pull requests are ignored while `forkPrs` is off (logged as
"fork pull request ignored").
- **`failed` with "could not be deployed".** The message is the deployment's:
a missing server, an unresolved reference to an excluded service, a failed
build. Fix it and push again, or redeploy the service from the preview's
canvas.
- **Preview stuck deleting.** A service of the preview still has a running
deployment; the teardown retries every few seconds until it finishes.
- **Containers of a deleted preview remain.** `remove_service` is delivered like
any command; if the server was offline for longer than 15 minutes the command
expired and the reconciler removes the containers later (volumes may need a
manual `docker volume rm nc-vol-<service id>-…`).
---
# Clone and move
Source: https://nebulactrl.com/docs/operations/clone-and-move
Three operations rearrange what you already run:
| | What it does | Where |
|---|---|---|
| **Clone a service** | A staged copy of one service, in the same or another environment. | Service menu → **Clone**, `nebulactrl services clone`, `POST /services/{id}/clone` |
| **Clone an environment** | A new, non-production environment with a staged copy of every service. The usual "staging from production". | Environment switcher → **Clone environment**, `nebulactrl envs clone`, `POST /environments/{id}/clone` |
| **Move a service** | Re-homes a service in another environment, staged until its next deploy. | Service menu → **Move to environment**, `nebulactrl services move`, `POST /services/{id}/move` |
Nothing runs because of a clone: the new services are staged like any new service
and start when you deploy them. Both clone operations and the move accept
`dryRun` (`--dry-run`): they report what they would do, including conflicts, and change
nothing.
You need write access (`projects:write`) to the source **and** to the target
environment, because a clone carries the source's secrets. A token restricted to
one project cannot clone out of it into another.
## What a clone copies
| | |
|---|---|
| Configuration | Image or repository, build settings, port, health check, start command, resource limits, [Docker options](https://nebulactrl.com/docs/operations/docker-options), pre- and post-deploy commands, files. A Tailnet setting is **not** copied (one tagged device per service). |
| Server | The source's server, or the one you pick (`serverId`). A clone on another server is checked like any new service (online, accepting workloads, Docker options allowed there). |
| Variables | All of them. **Secrets are copied sealed**: each value is opened and sealed again for its new row inside one transaction; no value appears in a response, an audit entry or a log, and the clone's ciphertext is not a copy of the source's. Variables with a generator (`${{secret(32)}}`, `uuid()`, …) are **generated again**, so a staging copy never shares a secret with production. |
| References | See [references](#references-and-slugs). |
| Credentials | The Git token, the registry login and the SSH deploy key (with its pinned host keys) are sealed again for the clone. A git clone is linked to the same repository with **auto-deploy off**; switch it on in the service's Git settings. |
| Volumes | The **definitions** (name, mount path) are copied; the volumes are **new and empty**. Data is not copied (see [volume data](#volume-data)). |
| Databases | A managed database is created **new and empty**: same engine, version and tuning, **new credentials**, an empty data volume. Nothing from the source database (data, credentials, extra variables) is copied. The result carries a note: [restore a dump](https://nebulactrl.com/docs/operations/database-backups#restore-from-a-dump-file) into it. A read replica is not cloned; add a replica to the cloned primary instead. |
| Domains | Never duplicated, because hostnames are unique. See [domains](#domains). |
| Links | A service clone in the **same** environment gets the links the source draws to other services (not the links other services draw to the source). An environment clone copies every link between the cloned services. A clone into another environment copies no link. |
| Environment extras | An environment clone also copies the environment's shared variables (sealed again, generated ones regenerated) and its canvas layout. |
Not copied: backup schedules, cron tasks, deploy hooks, shared-network
attachments (a staging copy must not join production's network), maintenance
mode, replica counts and autoscaling, release and deployment history. Route
services and Compose stacks are not cloned: an environment clone lists them as
skipped; cloning one on its own is refused with `not_applicable`.
### References and slugs
A **service clone** in the same environment gets a new slug (`api` becomes
`api-copy`). References *to the clone itself* in its own variables follow it
(`${{api.PORT}}` becomes `${{api-copy.PORT}}`); references to other services
stay, so the clone talks to the same siblings the source does.
An **environment clone** keeps every slug, so a reference such as
`${{postgres.DATABASE_URL}}` in the cloned `api` resolves to the **cloned**
`postgres`, with the clone's own credentials. The clone is self-contained: no
cloned service points back at production.
When a clone goes to **another environment**, the references of its variables are
checked there. The result lists every reference that does not resolve (a service
that is not in the target environment, a shared variable that does not exist),
so you can add the missing service or change the variable before deploying.
### Domains
A hostname can belong to one domain rule only, so a clone never gets its source's
domains. Choose:
| Mode | Result |
|---|---|
| `drop` (default) | The clone has no domain. Attach new ones in the Networking tab. |
| `suffix` | Each public domain is cloned with a suffix on its first label: `app.example.com` becomes `app-copy.example.com` (same environment), or `app-<environment slug>.example.com` (another environment, or an environment clone). Set `domainSuffix` to choose another. Path rules, port and the other options come along, a custom certificate does not (the clone asks for its own), and basic-auth users are copied (hashes sealed again), so a protected site stays protected. |
Generated domains (`<service>-<env>-<project>.<base>`) are regenerated for the
clone's own name and environment. Redirects, apex domains (they cannot be
suffixed) and Compose component domains are skipped and listed.
A hostname that is already taken is a **conflict**: the whole clone fails with
`409 domain_conflict` (`details.hostnames`) and nothing is created. Choose another
suffix, or drop the domains. A dry run reports the conflicts without failing.
### Volume data
Cloning does not copy volume data. NebulaCtrl's backup and restore machinery is
per service: a volume backup goes to that service's own repository and restores
only into the same service, and database restore creates its sibling from a
backup of the source. Copying data would mean taking a backup, restoring it into
the clone and waiting for both, which is not a cheap side effect of a clone. To
bring data over:
- **Databases:** restore a dump into the cloned database (the database's
**Backups** tab → **Restore from a file**, or `nebulactrl databases restore`;
[restore from a dump file](https://nebulactrl.com/docs/operations/database-backups#restore-from-a-dump-file)), or restore a database backup **as a new
database** ([Database backups](https://nebulactrl.com/docs/operations/database-backups)) instead of cloning.
- **Other volumes:** back the source volume up and restore it into the clone with a
[volume backup](https://nebulactrl.com/docs/operations/backups) once the clone has its own destination, or copy the
files with the web terminal.
## Moving a service
A move changes the environment of a service. It is **staged**: the service keeps
running in its old environment until you deploy it. That deployment joins the new
environment's network and resolves its variables there. Domains, volumes, data,
variables and the release history stay with the service. The service's
**Pending changes** list the move as `environment: Project / old → Project / new`.
Before anything is written the move is checked:
| Check | Result |
|---|---|
| Variables of the service that reference services staying behind (`${{postgres.DATABASE_URL}}`), or project/environment shared variables that do not exist in the target | Listed as `outgoing`. They **break**: the next deploy fails with `unresolved_reference` until you fix them. |
| A variable that references a service the target environment also has (a `postgres` in both) | Listed as `rebound`: it would silently point at the other environment's service. |
| Other services of the old environment that reference the moved one | Listed as `incoming`. They break, and are marked as having staged changes so a deploy of the environment shows it. |
| Canvas links of the service | Removed (listed); draw them again in the new environment. |
| The target already has a service with the same slug | The moved service gets a free one (`api-2`); references to itself are rewritten. |
| A deployment of the service is running | `409 deployment_in_progress`. |
| The service is in a replica group or has a cross-server link | `409 move_blocked`: remove those first. |
| Route services, Compose stacks, the platform project, pull-request previews | Refused (`not_applicable`, `preview_environment`). |
A move that breaks references is refused with `409 move_blocked` (the plan is in
`details.plan`) unless you pass `force` (`--force`; the web app asks you to
confirm). Moving across **projects** of the team is allowed; the result warns that
project-level shared variables, git connections and team settings follow the new
project. Moving a service out of or into a **production** environment needs
approval for guarded clients (AI tokens), like other production changes.
Undo a move by moving the service back.
## In the web app
- **Clone a service:** open the service, then the **⋯ menu → Clone**. Pick the
environment (default: this one), a name, the server and the domain mode; **Check**
runs the dry run and shows conflicts, unresolved references and notes; **Clone**
creates the staged copy and opens it.
- **Move a service:** **⋯ menu → Move to environment**. The dialog shows the plan
(broken references, removed links, a new slug) before you confirm; the service
then shows a *Staged* change you apply with **Deploy**.
- **Clone an environment:** the environment switcher's **Clone environment**
asks for the new name, the domain mode and optionally one server for every
clone, shows the services and notes, and opens the new environment.
## CLI
```bash
# A staging copy of production; domains get -staging
nebulactrl envs clone production --project shop --name Staging --domains suffix
# Clone one service (staged); --dry-run shows what would happen
nebulactrl services clone shop/production/api --env staging --dry-run
nebulactrl services clone shop/production/api --env staging --domains suffix
# Move: check, then do it
nebulactrl services move shop/production/api-copy --env staging --dry-run
nebulactrl services move shop/production/api-copy --env staging --force
```
All three print JSON with `--json`.
## API and MCP
`POST /services/{serviceId}/clone`, `POST /environments/{environmentId}/clone` and
`POST /services/{serviceId}/move` (see the [API reference](https://nebulactrl.com/docs/reference/api)).
Both clones honour `Idempotency-Key`. Over MCP they are generic operations
(`find_operation` / `call_operation`): cloning is a creation like `createService`,
and `moveService` is production-guarded when the source or target environment is
production. Secrets never appear in any of the results.
## Audit
`service.clone`, `environment.clone` and `service.move` are recorded with the
source, the target and counts (services, forced, broken references), never with
values.
## Limits
- No data is copied (see [volume data](#volume-data)); databases start empty.
- Clones stay inside one team. A clone is never production, and never joins
shared networks, the tailnet, or the source's schedules.
- Route services, Compose stacks and read replicas are not cloned; routes,
stacks, replica-group members and services with cross-server links cannot be moved.
- Cloning a service whose source is on a server that is offline works (the clone
is only staged), but deploying the clone needs its own server online.
---
# Replicas and autoscaling
Source: https://nebulactrl.com/docs/operations/scaling
A stateless service can run on 1–10 replicas on its own server. Replicas are
containers of the same release; the server's Traefik load-balances across them
and health-checks every upstream.
**Eligible services:** no volumes, an HTTP port, and a kind other than
`database`. Anything else is fixed at one replica (`422 replicas_require_stateless`).
A deploy of a service that stopped being eligible (a volume was added, the port
removed) runs one replica, resets the count to 1 and turns autoscaling off.
## Manual scaling
`PUT /api/v1/services/{id}/scaling {"replicas": 3}` or
`nebulactrl services scale <service> 3`. It applies immediately and is never a
staged change: a deployed service gets a `scale_service` command that reuses
the running release (no rebuild, no new deployment); an undeployed one stores
the setting for its first deploy. New replicas must pass the health check
before they receive traffic; removed replicas leave the upstream list first and
drain for 2 seconds. Scaling is refused while a deployment is running
(`deployment_in_progress`) or the service is stopped (`service_stopped`).
Deploys keep the current replica count. A rolling deploy starts N new
containers, waits until all are healthy, switches routing to the new set and
retires the old one; if any new replica fails, the old set keeps serving.
## Autoscaling
`nebulactrl services scale <service> --autoscale 1:5:70` (min:max:target%), or
`PUT .../scaling {"autoscaling": {"enabled": true, "min": 1, "max": 5, "targetCpuPercent": 70}}`.
While enabled, the replica count is managed by the autoscaler and a manual
`replicas` value returns `409 autoscaling_enabled`.
A periodic job (every 30 s) judges each deployed service from the per-replica
CPU samples (percent of the replica's CPU limit, or of one core without a limit):
| Rule | Value |
|---|---|
| Scale out +1 | average above the target for 2 minutes |
| Scale in −1 | average below 35 % (never more than half the target) for 10 minutes |
| Cooldown | 3 minutes after scaling out, 10 minutes after scaling in |
| Bounds | min and max, within 1–10 |
Every sample bucket of the window must contain data; a gap restarts the clock.
**Capacity guard.** A scale-out is blocked when the server's free memory is
below one replica's memory (the configured memory limit, else the observed
average per replica) or the server has an active disk alert. The service then
shows `scaling.blockedReason`, e.g. "Scale-out blocked: agent1 has 0.4 GiB
free", until the condition clears. A failed `scale_service` command is shown
the same way ("Scale failed: …").
Each autoscale action is audited as `service.autoscale` (system actor, with the
reason, e.g. "CPU 84% > 70% for 2m"); manual changes as `service.scale`.
## Observability
`GET /services/{id}` reports `scaling.replicas` (desired), `scaling.ready`
(running and healthy) and `observed.replicas[]` per container. The metrics
endpoint sums CPU and memory over the replicas, adds `cpuPerReplica`, and the
newest point carries `replicas: [{index, cpuPercent, memBytes}]`. Runtime logs
merge all replicas, prefixed `[r<N>]` when there is more than one.
Replicas share one server. Spreading replicas across servers is not supported.
---
# Tasks and cron jobs
Source: https://nebulactrl.com/docs/operations/tasks
Four ways to run a command next to a service. All of them run in a **throwaway
container of the service's own image**, on the service's server, with the
service's environment variables, on its environment network (so
`postgres:5432` resolves) and the edge network. They publish no ports, never
restart, use the service's memory and CPU limits, have a timeout, and the
container is removed when the command ends, whatever its outcome.
The command is an **argv array without a shell**. Use `["sh", "-c", "…"]` for
shell syntax. It replaces the image's entrypoint (like `docker exec`, and
unlike `docker run image args`), so an image with `ENTRYPOINT ["/app/server"]`
runs `["/app/server", "migrate"]` as written.
## Cron tasks
Per service, up to 20 tasks: `name`, `schedule`, `command`, `enabled`,
`timeoutSeconds` (default 1800), `concurrency` (`forbid` or `allow`).
- **Schedule**: five fields (`minute hour day-of-month month day-of-week`) or
`@hourly`, `@daily`, `@weekly`, `@monthly`, `@yearly`. Parsed by the in-house
`internal/cronexpr`. The finest resolution is one minute.
- **Time zone**: each task has an IANA `timezone` (`Europe/Berlin`,
`America/New_York`, …; default `UTC`, and every task created before the field
existed stays `UTC`). The schedule is read as wall-clock time in that zone, so
"every day at 03:00" stays at 03:00 across daylight-saving changes (the UTC
instant moves by an hour). A wall time that does not exist (the spring-forward
gap) is skipped for that day; one that happens twice (the autumn hour) runs
once, the first time. An unknown zone is `422 invalid_timezone`. Set it with
`timezone` on `POST /services/{id}/tasks` and `PATCH /tasks/{id}` (changing it
recomputes the next run), `nebulactrl tasks create --timezone` /
`tasks update --timezone`, or the **Time zone** field of the task dialog, which
previews the next runs in the task's zone, in UTC and in your own zone. The
list shows the zone next to the schedule. `POST /task-schedules/preview` takes
`timezone` too. The control plane binary embeds the zone database, so zones
work on minimal images.
- A River periodic job evaluates due tasks every minute. A task that is due runs
**once**, however long the control plane was away: missed ticks are not
replayed.
- `forbid` (default): while the previous run of the task is still active the tick
is recorded as **skipped** ("the previous run is still running"). `allow`
starts another run. A streak of identical skips is recorded once, so it never
pushes real history out.
- A tick is also skipped, with the reason, when the service is stopped, has never
been deployed or its server is offline.
- **History**: the newest 50 finished runs of each task (and 50 one-off runs per
service) are kept with their output; older ones are deleted. Output is
redacted with the service's secret values and capped at 10,000 lines.
- **Run now** starts a run immediately, ignoring the schedule and `concurrency`.
- An acked run whose agent never answers is failed after `timeout + 10 minutes`
("the agent never reported a result").
- **Notifications**: a failed or timed-out **cron** run (and runs started by
features such as preview seeding) emits `task.failed`; runs a person started
from the UI, CLI or MCP do not, you are looking at them.
## One-off commands
`POST /api/v1/services/{id}/run {command, timeoutSeconds?}` (scope
`deployments:write`, audited as `service.run_command` with the command line).
Returns `202` with a run. Follow it with
`GET /task-runs/{runId}/logs?follow=true` (server-sent events; `event: end` carries
`{status, exitCode}`), or `GET /task-runs/{runId}` for the status and exit code.
`409 not_deployed` / `server_offline` when it cannot start.
- UI: the dock's **More** menu and ⌘K ("Run command in …"); the Tasks section of
the service settings also has the button, plus the run history drawer.
- CLI: `nebulactrl services run <service> -- <command…>` streams the output and
**exits with the command's exit code**. `--detach` returns the run id.
- MCP: `run_command` (needs `deployments:write`; waits up to `waitSeconds`,
default 20, returns at most 500 redacted lines, 64 KB) and `get_task_run`.
Docker exec level access is root-equivalent on that service's data, so the scope
is the same one that can deploy. Output is untrusted application text.
## Pre-deploy command
Service settings → Deploy → **Pre-deploy command** (`config.preDeployCommand`,
`preDeployTimeoutSeconds`, default 600, at most 3600).
Order of a deploy on the agent: pull or build the image → **pre-deploy** → stop
the old container (recreate only) → create and start the new one → health check →
route → verify. The pre-deploy container uses the **new image** and the deploy's
exact environment. Until it succeeds nothing of the previous release has been
touched, so for rolling **and** recreate deploys the old release keeps serving
while migrations run.
A non-zero exit (or a timeout) fails the deployment at the step `predeploy`; the
message carries the exit code and the last 20 output lines, and says that the
previous release is still serving. The output is in the deployment logs. The LED
strip keeps its three LEDs (Build, Release, Verify): `predeploy` lights Release
and the card shows a small "pre-deploy" label under it while it runs.
It runs for every deployment that creates a **new release** (manual, staged,
git push, preview). It does **not** run for rollbacks, reconcile redeploys or
server drains: those re-run an existing release and must not touch data.
Beware that migrations are not undone by a rollback.
## Post-deploy command
Service settings → Deploy → **Post-deploy command** (`config.postDeployCommand`,
`postDeployTimeoutSeconds`, default 600, at most 3600). It mirrors the pre-deploy
command, but runs **after** the swap, for work that needs the new release to be
live: cache warm-up, a search reindex, a smoke call.
Order of a deploy on the agent: pull or build → pre-deploy → create and start →
health check → route → verify → **retire the previous release** → **post-deploy**.
The post-deploy container is a one-shot container of the **new image** with the
deploy's exact environment, on the environment network (it reaches the live
service by its name, without an alias of its own). It uses the same memory and
CPU limits as the service and is removed when it ends.
**A failure only warns.** The new release is already serving, so a non-zero exit,
a timeout or a command that cannot start never fails or rolls back the
deployment. Instead:
- the deployment is `succeeded`, and its message ends with
`Warning: post-deploy command failed with exit code N`;
- the timeline shows the step **Post-deploy** with the status `warning` (a
triangle) and the sentence "The new release is live; the deployment was not
rolled back." The command's output is in the deployment logs;
- a notification goes out as `task.failed` with severity **warning** (title
"Post-deploy command failed on <service>"). Channels that subscribe to task
failures at the `warning` level or lower receive it; the deployment itself is
still announced as `deploy.succeeded`.
Like the pre-deploy command it runs for every deployment that creates a **new
release** (manual, staged, git push, preview), not for rollbacks, reconcile
redeploys or drains, and a service that is itself a one-shot has none. The
deployment stays in `verifying` until the command ends (up to its timeout), so a
new deploy of the same service is refused as in progress until then.
## Operational notes
- The agent finds the image from the service's running container on the host,
else from the release (pulling it with the service's registry credentials).
- Containers are named `nc-task-<service>-<run8>` / `nc-predeploy-<service>-<dep8>` / `nc-postdeploy-<service>-<dep8>`
and carry `nebulactrl.task-run`, `nebulactrl.task-service`,
`nebulactrl.task-kind` labels, but **not** `nebulactrl.service`: they are never
replicas. The agent sweeps leftovers (exited ones older than 10 minutes, running
ones far past their deadline) on every reconcile.
- A redelivered `run_task` command (agent restarted mid-run) **adopts** the
existing container instead of running the command a second time. If the agent
shuts down while a task runs, the container is left running and adopted after
restart; the logs written during the gap are not replayed.
- Command payloads persisted in `agent_commands` hold the command line and the
variable **keys** only; values are attached at send time. Do not put secrets
into the command line itself (they would be stored), use variables.
- `GET /services/{id}/task-runs`, `/task-runs/{id}` and the output need
`projects:read`; creating, editing and deleting tasks needs `projects:write`;
running needs `deployments:write`.
---
# Web terminal and file browser
Source: https://nebulactrl.com/docs/operations/terminal
A shell and a file browser inside a running replica container, from the browser
(inspector header ⋯, the canvas dock's **More** menu, ⌘K) and from the CLI.
Nothing is installed in your image: the agent starts the shell with Docker's
exec API, over the connection the agent already holds to the control plane.
```
browser ⇄ WSS ⇄ control plane ⇄ agent WebSocket ⇄ Docker exec / archive API ⇄ container
```
This is a shell **inside a container**. A shell on the server itself is a separate,
opt-in feature with stricter rules: see the [host shell](https://nebulactrl.com/docs/security/host-shell).
## Who can use it
- **Owner, Admin and Developer** of the team. **Viewers never see it** (no
terminal, no files). A per-project restriction on the membership applies:
outside it the service does not exist (404).
- An API token needs the `deployments:write` scope **and** its creator must still
hold one of those roles. Browser sessions and tokens are treated the same.
- **Production** environments need an explicit confirmation: the UI asks with a
hold-to-confirm dialog, and the server refuses the session unless the open
message says `confirmProduction: true`. A script or the CLI must pass it
deliberately (`nebulactrl services shell --production`).
## The terminal
- The shell is `bash` when the image has it, else `sh`. Pick another command in
the dialog (for example `psql -U postgres`); it runs as an argv, with the
container's own user and working directory, and `TERM=xterm-256color`.
- Choose the **replica** when the service runs several. A stopped replica cannot
be opened.
- The terminal resizes with its panel, copies and pastes like any terminal, and
shows the exit code when the shell ends.
- **Limits.** 3 concurrent sessions per person (or token), shared across
services. A session with **no keyboard input for 15 minutes** is closed (a
warning appears one minute before). Closing the browser tab ends the session
and sends SIGHUP to the shell; a process that ignores SIGHUP (`nohup`) keeps
running in the container.
- Output is flow controlled: a runaway `cat /dev/urandom` slows down instead of
filling queues that also carry heartbeats and deployment events.
### What is recorded
| Event | Audit action | Metadata |
|---|---|---|
| a session opens | `terminal.open` | service, replica, container, **the executable only** (never its arguments, which may carry a password), production flag, server |
| it ends | `terminal.close` | duration in seconds, why it ended (`exit`, `client_closed`, `idle`, `agent_disconnected`, `error`), exit code |
**Keystrokes and output are never recorded, logged or stored**, anywhere. The
audit entry for a session is written before the shell starts: if the audit log is
unavailable the session is not opened.
Be aware that a shell in a container sees that container's environment,
including secret variables, and can change the container's filesystem. The
changes disappear on the next deployment, which creates a fresh container. Treat
terminal access like SSH access to that container.
## The file browser
- Lists directories (directories first, up to 2000 entries), previews text files
up to 1 MiB, downloads files of any size (streamed, never buffered whole) and
uploads **one file at a time, up to 50 MB**.
- Paths are absolute paths **inside the container**. The agent reads and writes
through the Docker API, so the host filesystem is out of reach; symlinks are
resolved inside the container's root.
- Listings use the image's `sh` (`stat`/`readlink` when present). Images without
a shell (distroless, `scratch`) fall back to reading the directory's archive,
which is slower on big trees and may report the listing as truncated.
- Uploaded files are created with mode `0644` and owned by root. Use the
terminal to `chown` them when the app runs as another user.
- **Production is read-only.** An Owner or Admin can allow writes per project
(project settings → Terminal & files, or
`PUT /api/v1/projects/{id}/terminal-settings {"allowProductionFileWrites": true}`).
Reads and downloads always work for the roles above.
- Every download or preview is audited as `file.read` (path, size), every upload
as `file.upload` (path, size). File contents are never recorded.
- Volumes: files written into a mounted volume persist, files elsewhere vanish
with the container on the next deployment. The browser shows both without
telling them apart; check the service's volume list.
## API
| Endpoint | Purpose |
|---|---|
| `GET /api/v1/services/{id}/terminal` | availability, replicas, production flag, whether file writes are allowed |
| `GET /api/v1/services/{id}/terminal/ws` | WebSocket (browser session, same origin). First message `{"type":"open","replica":0,"command":[…],"cols":80,"rows":24,"confirmProduction":false}`; binary frames are input/output; `{"type":"resize"}`, `{"type":"close"}`; the server sends `opened`, `idle_warning`, `exit`, `error`, `closed` |
| `GET /api/v1/services/{id}/files?replica=&path=` | list a directory |
| `GET /api/v1/services/{id}/files/content?replica=&path=&preview=` | download or preview a file |
| `POST /api/v1/services/{id}/files/upload?replica=&path=<dir>&name=<file>` | upload the request body (`Content-Length` required) |
| `GET/PUT /api/v1/projects/{id}/terminal-settings` | the production-writes switch (admins) |
A cross-origin page cannot open a terminal with your session cookie: the WebSocket
handshake checks the `Origin` header against the request host and
`NEBULACTRL_PUBLIC_URL`. Behind a reverse proxy, forward `Upgrade` and
`Connection` headers and the original `Host`, and allow long-lived connections
(an idle terminal sends a ping every 30 seconds).
## CLI
```
nebulactrl services shell <service> [--replica 0] [--production] [-- command args…]
```
Needs an interactive terminal on both ends. It opens the same WebSocket with
your token, puts your terminal in raw mode, forwards resizes and restores the
terminal when the session ends. The exit code is the shell's.
## Troubleshooting
| Symptom | Cause |
|---|---|
| "The server is offline" | the agent is not connected; terminals need the live connection and are never queued |
| "Replica N is not running" | the container exited or restarts in a loop; check the logs, then redeploy |
| "could not start a shell: … executable file not found" | the image has no `sh`. Open a command that exists (for example `/app/server`) or debug with the file browser |
| "You already have 3 terminal sessions open" | close one; sessions are per person across all services |
| "production" confirmation fails | the server did not receive `confirmProduction: true` (stale browser tab: reload) |
| the terminal drops after minutes behind a proxy | the proxy's idle timeout is shorter than the 30 s ping interval allows; raise it for `/api/v1/services/*/terminal/ws` |
---
# Docker options
Source: https://nebulactrl.com/docs/operations/docker-options
A service can set a fixed list of **typed** Docker container options. They exist for images that need them (a database that wants a bigger `/dev/shm`, a proxy that binds port 80 as a non-root user, RabbitMQ that stores its data by node name, a media server that needs the GPU) and for hardening (drop all capabilities, read-only root filesystem, a processes limit). There is no free-text `docker run` string: every option is a validated field, so nothing is silently ignored.
Where: the service's **Settings → Docker options**, the CLI (`nebulactrl services docker`), or API `docker` inside the service config (`POST /environments/{id}/services`, `PATCH /services/{id}`). It is release config, so a change is **staged** and applies with the next deploy, appears in the pending changes as `config.docker.<field>` and is frozen into the release (a rollback restores the options of that release).
## Two tiers
The trust rule: **nobody below the instance admin can gain root on a host unless the instance admin made that server _unrestricted_.** Options therefore come in two tiers.
- **Every server.** Options that cannot hand over the host: they only restrict the container, or change something inside its own namespaces.
- **Unrestricted servers only.** Options that give a container the host's devices, GPUs, a different runtime or something equivalent to host root. An instance admin switches a server to unrestricted ([below](#the-unrestricted-switch-per-server)); on any other server these options are refused with `422 requires_unrestricted_server`, and the UI shows the group disabled with the reason.
## Options on every server
| Option | Field | Rules |
|---|---|---|
| Capabilities to add | `capAdd` | Names from the [safe list](#capabilities), with or without the `CAP_` prefix, stored upper-case, no duplicates |
| Capabilities to drop | `capDrop` | `ALL` or Linux capability names; applied before `capAdd`, so `capDrop: [ALL]` with `capAdd: [NET_BIND_SERVICE]` is the hardening pattern. A name in both lists is refused |
| No new privileges | `noNewPrivileges` | sets the `no-new-privileges` security option. Programs that rely on setuid helpers such as `sudo` stop working |
| Read-only root filesystem | `readOnlyRootfs` | writes go only to volumes and `tmpfs`. **Config files cannot be written to a read-only root** (Docker refuses `docker cp` into it, and a tmpfs is mounted over whatever was copied), so every config file of the service must sit under a volume mount; otherwise `422 invalid_config` |
| Processes limit | `pidsLimit` | 0 (no limit) or 1-4194304 processes and threads for the container (cgroup). This is the real per-container limit; see [`nproc`](#ulimits) |
| Hostname | `hostname` | one RFC 1123 label (lowercase letters, digits and dashes, at most 63 characters). **Empty means the service slug**, for every service: the hostname no longer changes with each deploy, which software that stores data by node name (RabbitMQ) needs. **With several replicas every replica gets `<name>-<replica index>`** (`web-0`, `web-1`, ...; an explicit `rabbit` becomes `rabbit-0`, `rabbit-1`, ...), so replicas never share a hostname; a single replica keeps the plain name. The name for an index is the same on every deploy, and the base name is cut if needed to keep the result at most 63 characters. Scaling up adds replicas without restarting the running ones, so a replica that was alone keeps its plain name until the next deploy renames it `<name>-0` |
| DNS servers | `dns` | up to 3 resolver **IP addresses** (IPv4 or IPv6, no hostnames, no duplicates). Docker's embedded resolver still serves container names and forwards the rest to them |
| Sysctls | `sysctls` | up to 16 entries from the [curated list](#sysctls); the value is a number or numbers separated by single spaces |
| Host kernel requirements | `hostSysctls` | not applied: minimum values the **host** needs, for example `{"vm.max_map_count": 262144}`; the plan warns when a server has less ([below](#host-requirements)) |
| User | `user` | `uid[:gid]` as numbers (up to 2147483647) or user and group names |
| Supplementary groups | `groupAdd` | up to 8 numeric group IDs from 1 to 65535, for example the host's `render` group so a non-root process can open a GPU device |
| Resource limits | `ulimits` | `nofile` or `memlock`; `soft` and `hard` are `-1` (unlimited) or 0-2147483647; `soft` must not exceed `hard` unless `hard` is `-1`; each name once |
| Shared memory | `shmSizeMb` | `/dev/shm` size in MiB, 0 (Docker's default 64 MiB) to 65536 |
| Extra hosts | `extraHosts` | up to 20 `name:ip` (IPv4 or IPv6) or `name:host-gateway` entries |
| Labels | `labels` | up to 30; key `[A-Za-z0-9][A-Za-z0-9._/-]*` (127 characters at most), value at most 512 characters, no line breaks. **`com.docker.*`, `nebulactrl.*` and `traefik.*` are reserved** (case-insensitive): the managed labels and the proxy's own can never be set or overwritten |
| Init | `init` | run Docker's init (tini) as PID 1: reaps zombies and forwards signals |
| Stop signal | `stopSignal` | one of `SIGTERM`, `SIGINT`, `SIGQUIT`, `SIGHUP`, `SIGUSR1`, `SIGUSR2`, `SIGWINCH`, `SIGKILL`; the grace period (`stopGraceSeconds`) is unchanged |
| Entrypoint | `entrypoint` | replaces the image's `ENTRYPOINT`; an exec-form argv of up to 32 elements (each at most 4096 bytes, no NUL), the first one the program. The start command (`startCommand`) is passed to it as arguments. Like `docker run --entrypoint`, overriding the entrypoint **drops the image's default command**, so set `startCommand` when the program needs arguments. No shell is involved unless the first element is one. An empty entrypoint (clearing the image's) is not supported |
| tmpfs | `tmpfs` | up to 10 in-memory mounts: absolute, clean `path` (not `/`, not under `/proc`, `/sys`, `/dev`, `/run/docker`, not colliding with a config file mount), `sizeMb` (default 64, at most 65536), optional octal `mode` such as `1777` |
**Hardening is opt-in.** Nothing is switched on by default except the hostname: `no-new-privileges` breaks images that rely on setuid helpers and `capDrop: [ALL]` breaks images that chown their data directory at start, so you choose them per service. The Docker options section offers a one-click **Hardened** preset (`capDrop: [ALL]`, `noNewPrivileges`, `readOnlyRootfs`, `pidsLimit: 512` and a `/tmp` tmpfs) that you then adjust to what the image needs.
A violation is `422 invalid_config` with the reason. The agent validates the same rules again before it creates a container, against the server policy carried in the deploy payload, so an edited or old payload cannot slip past the control plane.
### Capabilities
Safe list (allowed on every server): `AUDIT_WRITE`, `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `FSETID`, `IPC_LOCK`, `KILL`, `MKNOD`, `NET_BIND_SERVICE`, `NET_RAW`, `SETFCAP`, `SETGID`, `SETPCAP`, `SETUID`, `SYS_CHROOT`, `SYS_NICE`, `SYS_PTRACE`. Docker grants most of these by default already; listing one is still useful after `capDrop: [ALL]` and for `IPC_LOCK` (with `memlock`), `NET_BIND_SERVICE`, `SYS_NICE` and `SYS_PTRACE`.
- `SYS_PTRACE` is allowed with a warning (the plan and the UI show it): processes in the container can inspect and tamper with each other.
- **`NET_ADMIN` and `SYS_ADMIN` need an unrestricted server.** They widen the container's power over the network stack and the kernel.
- Anything else is refused: `ALL`, `SYS_MODULE`, `SYS_RAWIO`, `SYS_TIME` and so on. (`capDrop` accepts `ALL` and every Linux capability name.)
### Sysctls
A container can only change **namespaced** kernel parameters. Docker supports the IPC namespace (`kernel.msg*`, `kernel.sem`, `kernel.shm*`, `fs.mqueue.*`) and the network namespace (`net.*`), and "does not support changing sysctls inside of a container that also modify the host system" ([docker run reference](https://docs.docker.com/reference/cli/docker/container/run/#sysctl)). Containers never use the host network or IPC namespace, so each entry below changes only the container's own namespace. `sysctls` accepts exactly:
- IPC: `kernel.msgmax`, `kernel.msgmnb`, `kernel.msgmni`, `kernel.sem`, `kernel.shm_rmid_forced`, `kernel.shmall`, `kernel.shmmax`, `kernel.shmmni`, `fs.mqueue.msg_default`, `fs.mqueue.msg_max`, `fs.mqueue.msgsize_default`, `fs.mqueue.msgsize_max`, `fs.mqueue.queues_max`.
- Network: `net.core.somaxconn` (listen backlog), `net.ipv4.ip_unprivileged_port_start` (bind below port 1024 without a capability), `net.ipv4.ip_forward`, `net.ipv4.conf.all.forwarding`, `net.ipv4.conf.default.forwarding`, `net.ipv4.conf.all.src_valid_mark` and `net.ipv4.conf.all.rp_filter`, `net.ipv4.conf.all.route_localnet` (VPN containers), `net.ipv4.ip_local_port_range`, `net.ipv4.ping_group_range`, `net.ipv4.tcp_fin_timeout`, `net.ipv4.tcp_keepalive_time`, `net.ipv4.tcp_keepalive_intvl`, `net.ipv4.tcp_keepalive_probes`, `net.ipv4.tcp_syncookies`, `net.ipv6.conf.all.disable_ipv6`, `net.ipv6.conf.all.forwarding`, `net.ipv6.conf.default.disable_ipv6`.
Host-wide parameters such as `vm.max_map_count` are refused in `sysctls` with a pointer to `hostSysctls`.
### Ulimits
`nofile` and `memlock` only. The **`nproc` ulimit is gone**: the Linux `RLIMIT_NPROC` is counted per real user ID across the whole host and is not enforced for UID 0, so it gave false assurance. Use `pidsLimit`, a cgroup limit that applies to the container whoever runs in it. Saved configs and release snapshots that carried an `nproc` entry had it removed by migration `0046`.
### Host requirements
A container cannot raise a host sysctl, so an application that needs one (Elasticsearch needs `vm.max_map_count` of at least 262144 for its mmap-based stores) declares it in `hostSysctls`. Nothing is applied. The agent reports the current values of `vm.max_map_count`, `fs.file-max`, `fs.inotify.max_user_watches`, `fs.inotify.max_user_instances`, `kernel.pid_max`, `net.core.rmem_max` and `net.core.wmem_max` with its facts (read again whenever it reconnects), and the **deployment plan** adds a warning for each shortfall with the exact commands:
```
sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count = 262144' | sudo tee -a /etc/sysctl.d/99-nebulactrl.conf
```
The warning does not block the deploy (some applications run without it, slower or with errors). An agent that has not reported the value yet gets a warning that says so.
## Options on unrestricted servers only
| Option | Field | What it does |
|---|---|---|
| Devices | `devices` | up to 16 host device nodes: `hostPath` below `/dev` (for example `/dev/dri/renderD128`, `/dev/kvm`, `/dev/net/tun`), optional `containerPath` (default: the same path), `permissions` a subset of `rwm` (default `rwm`) |
| GPUs | `gpus` | `count` (`-1` for all, or 1-16), `driver` `nvidia`, optional `capabilities` (`compute`, `utility`, `video`, `graphics`, `display`, `compat32`). Needs the NVIDIA Container Toolkit on the host |
| Runtime | `runtime` | an OCI runtime configured in the host's Docker daemon, such as `runsc` (gVisor), `kata` or `sysbox-runc`. A sandboxing runtime raises isolation |
| Privileged | `privileged` | all capabilities, all devices, no seccomp or AppArmor profile: **equivalent to root on the host** |
| Host PID namespace | `pidHost` | the container sees and can signal every process on the host |
| Seccomp unconfined | `seccompUnconfined` | removes the default syscall filter |
| AppArmor unconfined | `apparmorUnconfined` | removes the default AppArmor profile |
| NET_ADMIN, SYS_ADMIN | `capAdd` | see [Capabilities](#capabilities) |
On a restricted server each of them is refused when you save the service, when you move it to that server, when a template that needs it is installed, and again at every deploy, rollback and reconcile redeploy (the plan reports the check as blocking: `requires_unrestricted_server`). Use them only for images you trust: a bug in the NVIDIA toolkit, a driver behind a device node or a kernel exploit without the seccomp filter is a way out of the container.
### The unrestricted switch (per server)
One switch per server controls both these options and [unrestricted Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks#unrestricted-per-server): **Servers → the server → Unrestricted** (API `PUT /servers/{id}/unrestricted`, CLI `nebulactrl servers unrestricted`).
- **Instance admin only** (the earliest-created user, or an admin-scoped token acting for them). A team owner, a developer or a token without the admin scope gets 403.
- **Enabling needs the server's exact name typed**; the dialog warns that anyone who can deploy there can take the server over.
- **Audited** as `server.unrestricted.enable` or `server.unrestricted.disable`, with the server name, the number of stacks and the number of services that use unrestricted-only options.
- It is **read at every deploy, rollback and reconcile**. Disabling it does not stop running containers; the next deploy of a stack or service that needs it fails before anything changes, until the options are removed or the service is moved.
- It was called "Unrestricted Compose" and the instance-wide "advanced capabilities" switch existed next to it. Both are gone: this is the only switch (migration `0046` renamed `servers.unrestricted_compose` to `servers.unrestricted` and dropped the instance setting; the API field `unrestrictedCompose` is `unrestricted`, `PUT /servers/{id}/compose` is `PUT /servers/{id}/unrestricted`, and `PUT /instance/docker-options` no longer exists). A service that already carried `NET_ADMIN` or `SYS_ADMIN` now needs its server to be unrestricted.
## What is never possible
Not configurable on any server, not on any list: `--ip`/`--ip6` (rolling deploys start the new container while the old one runs, so two containers would need the same address; use the private DNS name), host networking on a plain service (routes dial the container on the environment network; a [stack](https://nebulactrl.com/docs/operations/compose-stacks) on an unrestricted server can use it), the restart policy and log options (the platform owns them; see [logs](https://nebulactrl.com/docs/operations/logs)), the host IPC namespace (use `shmSizeMb`), host bind mounts and the Docker socket, `--device-cgroup-rule`, and the host user, UTS or cgroup namespaces. Each container still joins only its environment network (and shared networks you attach); memory and CPU limits stay under NebulaCtrl's control.
## Interactions
- **Replicas and scaling:** every replica gets the same options. The non-secret runtime spec stored in the container's `nebulactrl.spec` label carries them and the server policy they were validated against, so a replica the agent adds on its own (a reconcile or autoscaler scale-up) is created identically. Scaling re-reads the server's unrestricted switch.
- **Managed databases** accept the options too (a larger `shmSizeMb` is the usual one for PostgreSQL). Their image, port and volume stay derived from the engine.
- **Route services** have no container: options do not apply (`422 invalid_config`).
- **Tasks** (one-off, cron, pre- and post-deploy commands) run in their own containers and do not inherit these options.
- **Templates** can set them in a `docker:` block with the same field names ([Templates](https://nebulactrl.com/docs/operations/templates)); a template that uses an unrestricted-only option declares `requiresUnrestricted: true` and installs only on an unrestricted server.
- **Compose import** maps only `cap_add`, `ulimits`, `shm_size`, `extra_hosts`, `init`, `stop_signal` and `tmpfs`; the other Compose keys are listed in [Compose import](https://nebulactrl.com/docs/operations/compose-import).
## Verifying
`docker inspect <container>` on the host shows the result: `Config.Hostname`, `Config.User`, `HostConfig.ReadonlyRootfs`, `HostConfig.CapAdd`, `HostConfig.CapDrop`, `HostConfig.SecurityOpt`, `HostConfig.PidsLimit`, `HostConfig.Sysctls`, `HostConfig.GroupAdd`, `HostConfig.Dns`, `HostConfig.Devices`, `HostConfig.DeviceRequests`, `HostConfig.Runtime`, `HostConfig.Privileged`, `HostConfig.PidMode`, `HostConfig.ShmSize`, `HostConfig.Ulimits`, `HostConfig.Init`, `HostConfig.Tmpfs`, `HostConfig.ExtraHosts`, `Config.StopSignal`, `Config.Entrypoint` and `Config.Labels`. The integration tests `TestPublicDatabasePortAndDockerOptions` and `TestHardeningDockerOptions` assert that on real Docker.
---
# Maintenance mode and error pages
Source: https://nebulactrl.com/docs/operations/maintenance-and-error-pages
Two things share one design: a **maintenance mode** you switch on per service, and **automatic error pages** the proxy shows when there is nothing healthy to forward to. Both are served by the agent's own tiny static server, look like the login screen (the animated ASCII "Eclipse" backdrop over a centred card), and are single self-contained HTML files: no external request, no web font, no tracking.
| Situation | Status | Page |
|---|---|---|
| Service is in maintenance mode | `503` + `Retry-After` | "Down for maintenance" (your title, message and expected-back time) |
| The service answers `502`, `503` or `504`, or has no healthy replica (stopped, crashed, restarting) | the same status | "Service unavailable" / "Service timed out" |
| A host that no service of this server serves | `404` | "Nothing here: No service is configured for this address." |
Every page is sent with `Cache-Control: no-store`, `X-Robots-Tag: noindex, nofollow` and a strict Content-Security-Policy. Error and maintenance pages other than the 404 also carry `Retry-After` (the ETA of a maintenance window as an HTTP date, a fixed delay otherwise: 120 s without an ETA, 300 s once it has passed, 30 s for automatic error pages).
## Maintenance mode
Use it when you want visitors to see a clean page instead of a half-working app: a database migration, a manual upgrade, planned downtime.
- **Turn it on:** the service's Settings → **Maintenance**, the canvas card's **More ▸ Maintenance mode…** (and **End maintenance** once it is on), or the CLI:
```sh
nebulactrl services maintenance acme/production/web on --title "Upgrading the database" \
--message "Nothing is lost. This takes a few minutes." --eta 45m
nebulactrl services maintenance acme/production/web off
```
- **API:** `PUT /api/v1/services/{serviceId}/maintenance` with `{enabled, title?, message?, eta?}` (scope `deployments:write`, audited as `service.maintenance.on` / `service.maintenance.off`). `GET /api/v1/services/{serviceId}/maintenance/preview` returns the rendered page as HTML, optionally for unsaved text (`?draft=true&title=…&message=…&eta=…`); the settings form's **Preview page** opens it. The service resource carries `maintenance: {enabled, title, message, eta, since}`.
- **Text:** title up to 80 characters, message up to 600 (plain text, line breaks kept, always HTML-escaped), `eta` a future time shown in the visitor's local time and sent as `Retry-After`. Empty fields show the default copy. Switching off keeps the text for next time and clears the ETA.
- **While on:** every domain of the service answers `503` with the page. HTTP→HTTPS redirects still happen first, redirect-only domains still redirect, and basic-auth / IP allow rules are not applied to the page (a maintenance page must not need a login). The post-deploy **verification probe keeps reaching the real service**, so a deployment during maintenance still proves the new release works, and ends with the maintenance window intact.
- **Instant, no deploy.** The control plane sends the server its route table (`sync_routes`), and a stopped/offline server picks it up when it reconnects. A service that was never deployed stores the setting and applies it with its first deployment.
- **Operational, never staged.** Like replica counts it does not bump the service's generation, does not create a "changes to apply" entry, is not part of a release and is **never rolled back**. A rollback or a redeploy leaves maintenance exactly as it was.
- **Idempotent.** Repeating a request that changes nothing records no audit event and sends nothing.
- **Not for TCP/UDP routes** (`422 maintenance_unsupported`): there is no page to show on a raw port.
## Automatic error pages
The agent adds an `errors` middleware (status `502-504`) in front of every service router and a lowest-priority catch-all router on ports 80 and 443. Together they replace Traefik's bare `Bad Gateway` / `no available server` / `404 page not found` text:
- A stopped service, a crashed container, a failing upstream health check and a connection error all end in `502` or `503`; a timeout in `504`.
- A request for a host (or a path of a path-scoped domain) that no router matches gets the `404` page. The probe entrypoint is unaffected: an unrouted host is still a `404` there, so post-deploy verification can never pass by accident.
- The status code is the real one, so monitors, CDNs and API clients behave correctly.
**One caveat:** the middleware replaces *any* `502`/`503`/`504` the upstream itself sends, including an application's own JSON error body for a deliberate `503`. If an API must return its own `503` body, answer `429`/`500` instead or keep that status below the proxy.
## How it works
The agent starts a second infrastructure container next to Traefik, **`nebulactrl-pages`**, on the `nebulactrl-edge` network. It publishes nothing; only Traefik reaches it (`nc-pages` → `nebulactrl-pages:8080`).
- **Image:** `nginx:1.30.6-alpine-slim`, pinned by digest (the multi-arch index `sha256:dede8895648b461178c3bc604f7eed4423efe5220b2c4d839f5eb3d34ac3f8f7`), about 5 MB per platform. It is the official image, from the stable branch; it only serves static files and needs no build per service.
- **Hardened:** all capabilities dropped except `CHOWN`/`SETGID`/`SETUID`, 64 MiB memory limit, rotating 1 MB logs, `unless-stopped`.
- **Content** is rendered by the agent from templates embedded in the binary (`deploy/pages`), written into the container before its first start (like Traefik's bootstrap file) and fingerprinted in a label. The nginx configuration maps `/err/<status>` to the error pages and `/m/<serviceId>` to a service's maintenance page, each with the right status and headers.
- **Changing the content recreates the container** (sub-second). The agent orders it so no page ever vanishes while a router still points at it: entering or editing maintenance, the page is rendered *before* the route push; leaving maintenance, the routers move back *first* and the page is dropped after. A recreate causes a short `502` for the other services currently in maintenance, then they recover.
- **It is excluded from the orphan sweep** (like the proxy and tailnet containers) and is repaired by the proxy health loop if someone removes it.
- **Failure is contained.** If the pages server cannot start (no disk, image pull blocked), route tables without maintenance still apply, with Traefik's own error pages and a warning in the agent log; a table that *needs* the pages server (a service in maintenance) is refused with `maintenance pages unavailable` and the previous routes stay live. The next apply retries.
An old agent without pages support ignores the maintenance flag; update the agent first.
## What the pages look like
The same engine as the sign-in screen draws the backdrop on a `<canvas>` (two WebGL2 passes, about 20 fps, paused in hidden tabs, one static frame under `prefers-reduced-motion`). Without JavaScript, without WebGL2 or in forced-colors mode the page falls back to an inline CSS halftone orb, with the same card and text. Fonts are the system UI and monospace stacks. The visited hostname is shown in mono on the card (filled in by a few lines of script; the maintenance page also carries it in the HTML). A small violet mark sits above the card; there is no other branding.
### Rebuilding the assets
Sources live in `deploy/pages` (`page.html.tmpl`, `page.css`, `page.js`, `nginx.conf.tmpl`) and are embedded into both binaries. The backdrop engine is a generated, committed bundle, so `go build` never needs bun:
```sh
go generate ./deploy/pages # or: cd apps/web && bun run scripts/build-pages-engine.ts
```
It writes `deploy/pages/assets/eclipse.js` from the same sources as the login backdrop (`apps/web/src/lib/ascii`). The control plane embeds the same assets to serve the maintenance preview.
## Troubleshooting
- **Visitors still see the app after turning maintenance on.** Check the server is online (the change is a `sync_routes` command; see the service's server page) and `docker logs nebulactrl-traefik`. `docker ps` on the host should show `nebulactrl-pages` running.
- **The generic `404 page not found` text appears.** The agent runs without the pages server (see "Failure is contained" above) or the request reached another reverse proxy first.
- **`docker logs nebulactrl-pages`** shows nginx's error log; access logging is off.
---
# Managed databases
Source: https://nebulactrl.com/docs/operations/databases
A managed database is an ordinary service (`kind=database`) with a typed `database` section in its configuration. NebulaCtrl derives everything engine-specific from that section: image, port, start command, TCP health check, the data volume and its mount path, and the credential variables. You pick an engine and a version; nothing else needs to be configured to get a working database.
Databases are **single-container** (no HA, no automatic failover, no connection pooling; PostgreSQL alone has an [advanced standby replica](https://nebulactrl.com/docs/operations/database-replicas) option with a manual Promote) and always use the `recreate` strategy: a configuration change restarts the database and causes a short outage, which the deployment plan says up front.
## Engines
| Engine | Versions | Image | Licence | Notes |
|---|---|---|---|---|
| PostgreSQL | 18, 17, 16 | `postgres` | PostgreSQL | `postgis` flavor: `postgis/postgis` (GPL-2.0 image). PostgreSQL 18 uses a new data-directory layout, so its volume mounts at `/var/lib/postgresql`. |
| MySQL | 8.4 LTS, 9.7 | `mysql` | GPL-2.0 | |
| MariaDB | 11.8 LTS, 12.3 | `mariadb` | GPL-2.0 | |
| Valkey | 9, 8 | `valkey/valkey` | BSD-3-Clause | **Default key-value engine.** |
| Redis | 8 | `redis` | RSALv2 / SSPLv1 / AGPLv3 | Selectable, with a licence label; creation needs `acceptLicense`. |
| MongoDB | 8.0, 7.0 | `mongo` | SSPL-1.0 | Opt-in, with a licence label; creation needs `acceptLicense`. |
| ClickHouse | 26.8 LTS, 25.8 LTS | `clickhouse/clickhouse-server` | Apache-2.0 | Native-protocol port 9000. Backups are not supported yet. |
Image tags are major-version tags; every release records the resolved digest, so a deployment is reproducible even though the tag floats between patch releases. Redeploying a database picks up the newest patch release of its major version.
The catalog is data (`internal/databases/engines`): a unit test checks that every engine has exactly one default version, a licence, and a licence label when the licence is source-available.
## Creating a database
Canvas → **Create → Database** → pick an engine and version (and flavor, server and database name). Via the API: `POST /environments/{environmentId}/databases`; via the CLI: `nebulactrl databases create`. Creation is **staged** like every other change: it creates the service, the data volume and the generated credentials, and nothing runs until you deploy.
Credentials are ordinary secret variables on the database service, generated once with `crypto/rand` (32 alphanumeric characters) using the engine's own names:
| Engine | Variables |
|---|---|
| PostgreSQL | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` |
| MySQL / MariaDB | `MYSQL_*` / `MARIADB_*`: `ROOT_PASSWORD`, `USER`, `PASSWORD`, `DATABASE` |
| Valkey / Redis | `VALKEY_PASSWORD` / `REDIS_PASSWORD` (the server is started with `--requirepass`) |
| MongoDB | `MONGO_INITDB_ROOT_USERNAME`, `MONGO_INITDB_ROOT_PASSWORD`, `MONGO_INITDB_DATABASE` |
| ClickHouse | `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DB` |
Secret values are never returned by any read. See *Reveal* below.
## Connecting other services
Databases export **derived connection variables**. They are computed from the engine and the credentials when a reference is resolved, are not stored, never injected into the database container and never produce a staged change:
| Engine | Exports |
|---|---|
| PostgreSQL | `DATABASE_URL`, `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USER`, `DATABASE_PASSWORD`, `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE` |
| MySQL / MariaDB | `DATABASE_URL`, `MYSQL_URL`, `DATABASE_HOST`, … `DATABASE_PASSWORD` |
| Valkey | `REDIS_URL`, `VALKEY_URL`, `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD` |
| Redis | `REDIS_URL`, `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD` |
| MongoDB | `MONGO_URL`, `DATABASE_URL`, `DATABASE_HOST`, … |
| ClickHouse | `CLICKHOUSE_URL`, `DATABASE_URL`, `DATABASE_HOST`, … |
In a consumer, set `DATABASE_URL=${{pg.DATABASE_URL}}` (where `pg` is the database's slug). Exports that contain the password are derived secrets: masked everywhere and redacted from logs. Connecting an edge on the canvas proposes exactly this one reference. A service on the **same server** reaches the database by its private hostname. A service on **another server** of the same environment reaches it through a [mesh link](https://nebulactrl.com/docs/operations/mesh#cross-server-service-links): the host is still the slug, and `DATABASE_PORT`, `PGPORT`, `DATABASE_URL` and the other port-bearing exports resolve to the mesh port **for that consumer** (the container port when that port is free on the database's server). The mesh must be enabled on both servers, and the link is added by deploying the consumer, which the plan states with the devices it adds. A PostgreSQL group with replicas also exports `DATABASE_WRITE_*` and `DATABASE_READ_*` ([writer and reader variables](https://nebulactrl.com/docs/operations/database-replicas#writer-and-reader-variables)).
## Reveal a connection string
Secrets stay masked in the UI, API and MCP. To connect from your laptop (`psql`) there is one explicit action: **Copy connection string**, `POST /services/{id}/database/reveal {"key":"DATABASE_URL"}`.
- Requires the `deployments:write` scope (admin included); browser sessions hold every scope. There is no MCP tool for it.
- Audited in the same transaction as `secret.reveal` (service and key, never the value).
- The response is `Cache-Control: no-store` and carries a 60-second `expiresAt`; the UI shows it briefly and does not keep it.
- The reveal response is not an idempotent operation, so the value never lands in stored idempotency bodies.
The internal hostname is only resolvable on the server's private network (and through a mesh link for consumers of the same environment on other servers). To connect from outside, use a tunnel or the TCP routes of the proxy.
## Rotating credentials
The official database images read their password variable **only when they initialise an empty data directory**. Editing `POSTGRES_PASSWORD` on a running database changes the variable and nothing else, so the database and its clients would silently disagree. NebulaCtrl therefore refuses edits of credential variables on a deployed database (`credential_managed`) and offers **Rotate credentials** (`POST /services/{id}/database/rotate-credentials`, `dryRun` for the plan):
- **PostgreSQL, MySQL, MariaDB, MongoDB, ClickHouse (`exec`):** the agent runs a typed command inside the container (`ALTER ROLE` / `ALTER USER` / `changeUserPassword`). The new password reaches the command through its stdin and environment, never argv, never the command payload in the database, and is masked in results. When the agent reports success the variable is updated. **The old password stops working immediately**, so every service using it fails to connect until it is redeployed.
- **Valkey, Redis (`redeploy`):** the server takes its password from the environment on every start. The variable is changed and the database restaged; it restarts on the next deploy.
- **Never deployed:** the new password simply becomes the initial one.
Every service in the environment whose variables reference `${{<slug>.…}}` is **staged** (its generation advances), so the next *Deploy staged changes* redeploys the consumers, databases first. Nothing is deployed automatically. Services that copied the password as plain text are not found: update them by hand. Rotation requires `projects:write` and `deployments:write`, runs one at a time and is blocked while the database is deploying or upgrading.
If the agent is offline the command waits; if it is never delivered the rotation fails with *Not delivered: server offline. No changes were made.*
## Version upgrades
`POST /services/{id}/database/upgrade/plan {"targetVersion":"18"}` returns a plan: method, steps, warnings, blockers and expected downtime. **Take a backup first** ([database backups](https://nebulactrl.com/docs/operations/database-backups)). Downgrades are blocked.
| Engine | Method | What happens |
|---|---|---|
| Valkey, Redis, MySQL, MariaDB, ClickHouse | `redeploy` | The image changes as a staged change; the next deploy upgrades the data files on start. Cannot be reverted without a backup. |
| PostgreSQL | `pg_upgrade` | The database is **stopped**. The agent runs [`tianon/postgres-upgrade`](https://github.com/tianon/docker-postgres-upgrade) (pinned per version pair, e.g. `17-to-18`) with the old volume as the source and a **new volume** as the target, then the service switches to the new volume and image as a staged change. **The old volume is kept untouched** on the server (delete it yourself when you are satisfied). Deploy the staged change to start the new version, then run `vacuumdb --all --analyze-in-stages`. If the conversion fails the old database is started again and nothing changed. |
| PostgreSQL PostGIS, MongoDB | `dump_restore` | Not automatic: restore a backup into a new database at the target version, switch the references, delete the old database. |
A never-deployed database just changes image. `pg_upgrade` copies the data (hard links are not used), so the host needs free disk space for a second copy. The helper needs the checksum setting of the old cluster; the agent retries once with the opposite `initdb` flag when `pg_upgrade` reports a mismatch (PostgreSQL 18 enables data checksums by default, earlier versions do not).
A PostgreSQL primary or replica **cannot be upgraded while the group has [read replicas](https://nebulactrl.com/docs/operations/database-replicas#interactions)** (`replicas_exist`): `pg_upgrade` carries neither the replication block of `pg_hba.conf` nor the replication slots. Remove the replicas, upgrade, and add them again. Scheduled and manual [backups](https://nebulactrl.com/docs/operations/database-backups) run on the primary only.
`tianon/postgres-upgrade` is an owner-approved exception (a personal Docker Hub namespace; its README calls itself a proof of concept). The upgrade path is covered by unit tests and was exercised by hand against a PostgreSQL 17 volume converted to 18.
## Observing a database
The database inspector's **Overview** shows the engine and version, its connection exports (names only) and, on **Refresh**, the version, database size and open connections that the agent reads with a read-only query inside the container. These are reported asynchronously (`POST /services/{id}/database/refresh`, then `GET /services/{id}/database`).
## Public port
A database can be published on a public port of its server, for a client that is not on the server (a laptop, a BI tool, a hosted job). It is off by default and always restricted by an IP allow-list.
| | |
|---|---|
| Where | the database's **Networking** settings: *Expose on a public port*; API `PUT /services/{id}/database/public-port` (`GET` reads it, `DELETE` closes it) |
| What it creates | one TCP route service (see [proxy routes](https://nebulactrl.com/docs/operations/proxy-routes)) on the database's server, marked as the database's public port. It shows on the canvas like other route services and is **staged**: deploy it to apply, like any change |
| Allow-list | **mandatory** (`422 allowlist_required`): CIDRs or single addresses. `0.0.0.0/0` or `::/0` is accepted only with `confirmOpen` set to the database slug (the UI asks you to type it, `422 open_confirmation_required`) |
| Deny list | optional, evaluated first, so it wins even over a confirmed `0.0.0.0/0` ([IP rules](https://nebulactrl.com/docs/operations/proxy-routes#ip-rules-allow-and-deny)) |
| Port | 1024-65535, never `80`, `443` or a proxy port; the UI suggests the engine's own port (5432 …) and moves to another one when it is taken (`409 port_in_use`) |
| Scopes | `projects:write` and `proxy:write` to change it, `projects:read` to read it |
**A new public port restarts the proxy on that server.** The proxy publishes each TCP port as a static entrypoint, so the agent recreates the Traefik container when the set of listening ports changes: about a second of dropped connections, and HTTP is briefly unavailable. The UI and the deployment plan warn about it, and `newEntrypoint` in the API says whether it applies. Changing only the allow or deny list of a port that is already published restarts nothing.
**TLS.** The proxy forwards raw TCP and never terminates TLS on this port, so a database that terminates TLS itself keeps doing so end to end. The managed databases do **not** enable TLS on their own, so by default the connection, password included, is not encrypted. Keep the allow-list to networks you trust, or tunnel the connection through a VPN or a [per-service tailnet](https://nebulactrl.com/docs/operations/service-tailnet).
**Docker bypasses host firewalls** (ufw, firewalld): the allow-list is the control, not a host firewall rule ([proxy routes](https://nebulactrl.com/docs/operations/proxy-routes#security-notes-for-tcpudp)). The proxy sees the client address Docker hands it, so a server behind another NAT or proxy sees that device's address, not the real client's.
**The external connection string** uses the server's **public IP** (set it in the server's settings; `422 no_public_address` otherwise) and the public port. Like the private exports it is only revealed on request: `POST /services/{id}/database/public-port/connection` (`deployments:write`), uncached, shown for 60 seconds and recorded in the audit log as `secret.reveal` with `public: true` (never the value).
## Advanced
- **[Standby replicas](https://nebulactrl.com/docs/operations/database-replicas)** (PostgreSQL only, advanced). A copy of the database that streams to another server and can answer read-only queries, so you can promote it by hand if the first server is lost. It is not a backup and not automatic failover, so keep your scheduled backups. On the same server it only helps with read load. In the database panel it sits in a collapsed **Advanced** section.
## Limits
- One container per database; no HA, no automatic failover and no connection pooling. PostgreSQL can have up to 5 asynchronous [standby replicas](https://nebulactrl.com/docs/operations/database-replicas) with a manual Promote; the other engines have none.
- A public port is a TCP route with an IP allow-list, not a managed TLS endpoint; there is no per-engine TLS setup (see [Public port](#public-port)). Closing it (`DELETE`, or removing the route) closes the listener on the server; the proxy is recreated only when its set of ports changes.
- Settings (`postgresql.conf`, `my.cnf`), init scripts, TLS and extension management are not exposed. Choose the PostGIS flavor for geospatial data.
- ClickHouse and MongoDB are lightly exercised; the integration tests cover PostgreSQL and Valkey.
---
# Postgres standby replicas and manual Promote
Source: https://nebulactrl.com/docs/operations/database-replicas
> **Advanced feature, PostgreSQL only.** Most databases do not need it. Start with [scheduled backups](https://nebulactrl.com/docs/operations/database-backups); add a standby replica only if one of the cases below is yours.
A **standby replica** is a second Postgres service in the same environment that follows the [managed database](https://nebulactrl.com/docs/operations/databases) with streaming replication and serves read-only queries. In the database panel it sits in a collapsed **Advanced** section, which opens by itself when the database already has replicas, is one, or is fenced. If the primary is lost, you can **Promote** a replica by hand and the platform repoints your apps.
## When to use it
- **A standby on another server.** A copy of the database streams to a second server over the [mesh](https://nebulactrl.com/docs/operations/mesh). If the first server is lost, you promote the copy by hand and the apps follow after you deploy the staged rewrite. This is the case the feature is made for.
- **Read offload.** Reports, exports or read-heavy queries run against `DATABASE_READ_URL` and leave the primary's CPU for writes. A replica on another server also spares the primary's disk and network.
## When not to use it
- **As a backup.** A replica is **not a backup**: a `DROP TABLE` or a bad `UPDATE` replicates within seconds. Keep [scheduled backups](https://nebulactrl.com/docs/operations/database-backups); there is no point-in-time recovery.
- **On the same server, for safety.** A replica on the primary's own server shares its disk and host, so it does not survive losing that server. It only helps with read load. The add dialog says so and still lets you do it.
- **As automatic failover.** NebulaCtrl does not watch the primary and does not switch anything by itself: there is no automatic failover, no leader election and no witness. Promote is a decision you make and confirm.
There is no synchronous replication and no cascading replicas. See [limits](#limits).
Scope: **PostgreSQL only** (16, 17 and 18, plain and PostGIS flavor). MySQL, MariaDB, Valkey, Redis, MongoDB and ClickHouse have no replicas, and their panels do not show this section.
## How it works
- A replica is an ordinary `kind=database` service of the primary's environment, with its own data volume and its own deployments, logs and cancel. Its engine, version, flavor, database name and user match the primary.
- Replication is **asynchronous** and physical (`pg_basebackup`, then a WAL stream with a replication slot).
- A primary has **at most 5 replicas** (`replicas_limit`). A replica cannot have replicas of its own (`replica_not_primary`).
- The primary needs **no restart and no configuration change**: Postgres' defaults already allow replication. Adding a replica creates a dedicated replication role `nebula_r_<id>` on the primary, adds a managed block to `pg_hba.conf` for it, reloads the configuration and caps the WAL the primary keeps for a lost replica (see [Disk use on the primary](#disk-use-on-the-primary)). The role, the block and the slot are removed again when the replica is removed.
- The replica keeps the primary's setting of outbound traffic: **Block outbound internet stays on**. Its only route to the primary is the environment network (same server) or the mesh link.
## Add a replica
Primary's database panel → **Advanced → Standby replica → Add standby replica**, or:
```
POST /services/{primaryId}/database/replicas
{"name": "pg-replica", "serverId": "…", "acceptMeshChanges": false}
```
`"dryRun": true` returns the plan (`200`) without changing anything. Needs `projects:write` and `deployments:write`; the CLI is `nebulactrl databases replicas add`. The real request answers `202` with a **saga**, not a staged change:
1. **prepare**: the role, the `pg_hba.conf` block and the slot cap on the primary;
2. **link**: nothing on the same server; across servers, the exposure, link and grant on the [mesh](https://nebulactrl.com/docs/operations/mesh) and a connectivity check;
3. **clone**: one agent command runs `pg_basebackup` into the replica's new volume (large databases take a while: the default time limit is 6 hours; **cancel** deletes the half-written volume);
4. **deploy**: the replica service is deployed through the normal deploy path;
5. **streaming**.
The saga is re-entrant: a restart of the control plane or a disconnected agent resumes it. Replica states are `pending`, `preparing`, `cloning`, `deploying`, `streaming`, `lagging`, `broken`, `needs_rebuild`, `promoting`, `removing` and `failed`.
The replica copies the primary's memory and CPU limits, Docker options and stop grace period; edit them afterwards like any service. It is placed on the server you choose and **cannot be moved to another server afterwards** (`replica_pinned`: a move does not move volumes anyway); use [Rebuild](#rebuild) instead.
The replica's start command refuses to start with an **empty data directory** (it exits with code 3 and says "rebuild the replica"), because the stock entrypoint would otherwise initialise a fresh, empty primary that looks healthy.
### Same server and cross-server
| | Same server | Across servers |
|---|---|---|
| Path | the replica joins the environment network and connects to the primary's slug on port 5432 | the replica connects to a **link node** alias on the replica's server, which forwards over the mesh to the primary's server node and then to the primary |
| Needs | nothing extra | [mesh](https://nebulactrl.com/docs/operations/mesh) **enabled on both servers** (`mesh_server_required` blocks otherwise, with a one-click plan to enable it), `/dev/net/tun` on the replica's server (`mesh_no_tun`), an agent that supports it (`mesh_agent_too_old`), the mesh not paused, the coordinator reachable (`mesh_coordinator_unreachable` for a new replica) |
| Devices | none | adds the link node (and the server nodes if enabling the mesh), the plan says "adds N devices" and asks for `acceptMeshChanges` (`mesh_confirmation_required` without it) |
| Encryption | local | WireGuard between the link node and the server node; the last hop on each host is local |
| Removing the last replica | releases the role, slot and block | also releases the grant, then the link and the exposure |
The replication link and an app's [service link](https://nebulactrl.com/docs/operations/mesh#cross-server-service-links) are separate nodes with separate rules: a replica's link cannot reach an app port.
### Disk use on the primary
A replica that stops reading makes the primary keep WAL for its slot. NebulaCtrl sets `max_slot_wal_keep_size` to the smaller of **4 GiB** and **10 % of the server's disk**, so a dead replica cannot fill the primary's disk: past the cap the slot is invalidated (`wal_status = lost`), the replica becomes `needs_rebuild` and the primary carries on. The replica row shows the WAL retained by its slot.
## Writer and reader variables
Every member of a replica group (the primary and each replica) exports the same set of **derived variables**, in addition to the [usual ones](https://nebulactrl.com/docs/operations/databases#connecting-other-services). They are computed when a reference is resolved, are not stored and never produce a staged change on their own:
| Export | Value |
|---|---|
| `DATABASE_WRITE_URL`, `DATABASE_WRITE_HOST`, `DATABASE_WRITE_PORT` | the group's **current primary** |
| `DATABASE_READ_URL`, `DATABASE_READ_HOST`, `DATABASE_READ_PORT`, `DATABASE_READ_HOSTS` | the replicas that are `streaming` or `lagging` and **reachable by the consumer** |
Use them as `${{pg.DATABASE_WRITE_URL}}` and `${{pg.DATABASE_READ_URL}}`, where `pg` is the slug of any member. The URL and password exports are masked and redacted like `DATABASE_URL`.
Which replicas a consumer gets:
1. the replicas **on the consumer's server**, if any;
2. otherwise the replicas on other servers, reached through the mesh (the port is the mesh port, resolved per consumer);
3. otherwise **the writer**: with no usable replica, `DATABASE_READ_*` equals `DATABASE_WRITE_*`.
One replica gives a plain URL; several give a libpq multi-host URL, `postgres://user:pass@h1:5432,h2:5432/db` (a driver without multi-host support can use `DATABASE_READ_HOST`). `DATABASE_URL`, `PGHOST` and the others keep describing the service they are read from: on a replica that is the read-only server.
When a replica joins, leaves or changes **health class** (`streaming` or `lagging` against `broken`, `needs_rebuild` and the like; **not** its lag numbers), the services that reference any member are **staged**, never deployed. There is no reader proxy or pooler: the application picks the URL.
## Lag and health
The agent polls every 30 seconds, on the local socket, without any command traffic and without secrets. Lag comes from the **primary's** `pg_stat_replication` (bytes behind, and the replay lag in milliseconds); if that is unavailable it falls back to the replica's own received-versus-replayed difference and marks it stale. Time-based lag measured on the replica alone is not used, because it reads as unknown while the database is idle.
| State | Meaning |
|---|---|
| `streaming` | connected and replaying |
| `lagging` | more than 64 MiB or 60 seconds behind for 5 minutes |
| `broken` | the replica has not been connected to the primary for more than 2 minutes (the receiver stopped, the link is down or the primary is gone) |
| `needs_rebuild` | the slot was lost (`wal_status = lost`), or the replica follows a primary that was promoted over it |
Cards show the lag, the primary shows a `n replicas` chip and the replica `Replica of <slug>`. An observation older than 90 seconds is shown as "unknown". Notifications (see [notifications](https://nebulactrl.com/docs/operations/notifications)) fire on a **transition**: `database.replica.lagging` and `database.replica.broken`, de-duplicated.
Two Postgres behaviours to know: `hot_standby_feedback` is **off**, so long queries on a replica never bloat the primary, and `max_standby_streaming_delay` is 30 seconds, so **a long query on a replica can be cancelled** when it conflicts with replay. Retry it or use a dedicated longer-running copy.
`GET /services/{id}/database/replication` returns the status; `GET …/database` includes `replication` (`role`: `standalone`, `primary`, `replica` or `fenced`, the primary, the replicas with state and lag, and whether a rebuild is possible).
## Promote
**Promote** turns a replica into the new primary. It is **manual**. The dialog says so: "Manual. NebulaCtrl does not detect failures or switch automatically." Use it for planned maintenance or after you have decided the primary is gone.
`POST /services/{replicaId}/database/promote`, with `"dryRun": true` first for the plan, then the real request with `{"confirm": "<replica slug>", "acceptDataLoss": false}`. It needs `projects:write` and `deployments:write`, and a production environment needs the production approval like a deploy ([production guard](https://nebulactrl.com/docs/reference/mcp#production-approval)). Only one promotion per replica runs at a time (`promote_in_progress`). There is no MCP tool to promote.
### The plan
The dry run reports:
- the **mode**: `switchover` when the primary's server is online and its container is running, otherwise `forced`;
- the last observed lag and how old that observation is;
- the **consumers** whose variables reference the old primary, listed **per server**, with the links (and devices) the new primary needs for each consumer server;
- a warning `mesh_server_required` for a consumer on a server without the mesh: it cannot reach the new primary until the mesh is enabled there;
- the other replicas that will need a rebuild, and the backup schedules that will move;
- blockers: the replica is not `streaming`, `lagging` or `broken` (`replica_not_streaming`), its agent is offline (`replica_agent_offline`), or another promotion is running (`promote_in_progress`).
### Switchover (the old primary is reachable)
No committed data is lost.
1. **Freeze**: the old primary is set read-only and other sessions are ended.
2. **Stop and fence**: it is stopped cleanly, a fence is recorded and its restart policy is set to `no`.
3. **Final position**: the platform reads the last checkpoint from the **stopped** data directory and waits (5 minutes) until the replica has replayed it. If that times out, the old primary is restarted, unfrozen and unfenced, the promotion fails and **nothing was lost**.
4. **Promote** the replica (`pg_promote`) and clear its replica settings.
5. **Repoint** (below).
6. The old primary stays **fenced and stopped, with its data kept**.
### Forced (the old primary is unreachable or revoked)
The fence is recorded **first**, so the old primary cannot run as a primary again, then the replica is promoted. Nothing is read from the lost server. It needs `"acceptDataLoss": true` (`data_loss_not_accepted`, 422). **Everything the replica had not yet received is lost**: with asynchronous replication that is up to the replication lag, shown in the plan ("up to the replication lag, which is `<n> MiB` as of `<t>`, or unknown, will be lost").
### Fencing
A fenced service is carried in every reconcile: the agent stops its container, sets its restart policy to `no` and refuses to start it on a deploy. There is **no "unfence"**, because running two writers is how data diverges. A fenced old primary can be [rebuilt as a replica](#rebuild) of the new primary or deleted. **Honest limit:** on a forced Promote, if the old primary's host comes back and Docker restarts the container (`unless-stopped`) before the agent reconnects and fences it, writes accepted in that window are lost; there is no merge, and rebuild is the only way back.
### Repoint: consumers on any server
In one transaction the platform:
- turns the replica into a standalone database, keeps its credentials (references to the old primary become literal copies of the same values) and clears its replica settings;
- marks the **other replicas `needs_rebuild`**, now following the new primary;
- **moves the backup schedules** to the new primary (the backup history stays with the old service);
- **stages a rewrite of the consumers**: in every consumer on **any server**, every `${{<old>.KEY}}` reference to the old primary becomes `${{<new>.KEY}}`. Nothing is deployed: the rewrites show in the staged bar, can be reviewed and discarded, and apply with **Deploy staged**;
- derives the grants the new primary needs for those consumers, adding the links the plan listed; the old primary's grants are dropped after the grace period.
A consumer that copied a connection string as plain text is not found; update it by hand. A consumer on a server **without the mesh** cannot reach a primary that moved to another server until the mesh is enabled there. A forced Promote while the old primary's server is dead depends on nothing from that server. It is audited as `database.promote` and `database.fence`.
## Rebuild
**Rebuild** (`POST /services/{id}/database/rebuild`, on a replica or a fenced primary; the CLI is `nebulactrl databases rebuild`) clones the database again from its primary into a **new data volume**; the old volume stays until you delete it. Use it for `needs_rebuild`, `broken` replicas whose slot is gone, a replica that followed a primary that was promoted over it, and the **old primary after a Promote**: its body may name the primary to follow (`followServiceId`), by default the replica that was promoted over it. There is no `pg_rewind`: rebuild is simpler and safe, at the cost of copying the whole database. Audited as `database.replica.rebuild`.
## Interactions
**Backups.** [Database backups](https://nebulactrl.com/docs/operations/database-backups) run on the **current primary only**. A schedule or a manual backup on a replica is refused (`backup_replica_unsupported`): a `pg_dump` on a standby can be cancelled by replay conflicts. A Promote moves the schedules with the role.
**Restore.** Restoring into a **new** database creates an ordinary standalone database, without replica settings. Restoring **in place** is refused on a replica (`replica_read_only`) and allowed on a primary with a warning: the restore replicates to every replica as ordinary writes.
**Upgrades.** A major version upgrade of a primary or a replica is **blocked while the group has replicas** (`replicas_exist`): `pg_upgrade` carries neither the `pg_hba.conf` block nor the replication slots. Remove the replicas, upgrade, add the replicas again. The minor version can drift when a floating major tag is redeployed: the plan warns, and you should redeploy the replicas first.
**Deleting** a primary that has replicas is refused (`replicas_exist`), and a replica cannot be deleted as a plain service (`replica_remove_required`): use **Remove replica** so its slot, role and `pg_hba.conf` block on the primary are dropped. **Credentials**: rotation on a replica is blocked (`replica_read_only`); rotate on the primary, the new password replicates and the replica's references stay consistent. A [public port](https://nebulactrl.com/docs/operations/databases#public-port) on a replica is allowed (it is a read-only server) and the plan says so. **Rollback** of a replica restores its configuration; the empty-data guard keeps a restored replica setting from doing harm on a promoted service.
**Removing a replica** stops it, drops its slot, role and block on the primary, deletes its service and then waits for the agent to remove its container and **data volume** before the replica is gone: the replica stays in `removing` until the agent reports the removal done, and a removal the agent reports as incomplete (a volume still in use, say) is sent again up to three times in all. A replica whose deploy is still running is stopped once that deploy is over, so it cannot stay connected to the primary; a drop the primary refuses (a slot still in use) is likewise sent again up to three times in all, because a slot left behind keeps the primary's WAL. An agent that is offline, or that does not answer within 10 minutes, does not hold the replica forever: the removal is given up with a warning in the control plane log, and a queued removal still runs when the agent returns. A drop that keeps failing or a primary that is offline is given up the same way: the warning names the slot, which you can drop on the primary with `SELECT pg_drop_replication_slot('<slot>')`.
**Servers.** A revoked server's replicas become `failed`; replicas whose primary was on it become `broken` and Promote is offered. [Draining](https://nebulactrl.com/docs/operations/draining-servers) a server moves services but not volumes, so a replica cannot be moved (`replica_pinned`).
## API, CLI and audit
| | |
|---|---|
| Add | `POST /services/{id}/database/replicas` |
| Remove | `DELETE /services/{id}/database/replica` (on the replica) |
| Promote | `POST /services/{id}/database/promote` |
| Rebuild | `POST /services/{id}/database/rebuild` |
| Status | `GET /services/{id}/database/replication` |
| CLI | `nebulactrl databases replicas add\|ls\|rm`, `nebulactrl databases promote <replica> [--yes] [--accept-data-loss]`, `nebulactrl databases rebuild <service> [--follow]` |
| Audit | `database.replica.add`, `database.replica.remove`, `database.replica.rebuild`, `database.promote`, `database.fence` |
Errors: `replica_wrong_engine`, `replicas_limit`, `replica_not_primary`, `replicas_exist`, `replica_remove_required`, `replica_pinned`, `replica_read_only`, `replica_not_rebuildable`, `replica_not_streaming`, `replica_agent_offline`, `database_fenced`, `backup_replica_unsupported`, `mesh_server_required`, `mesh_confirmation_required`, `mesh_link_not_ready`, `primary_not_running`, `promote_in_progress`, `promote_confirmation_required`, `data_loss_not_accepted`, `agent_too_old`. The replication password is generated, sealed, delivered at send time only, and never appears in a payload, the audit log, the API or the logs. MCP has a read-only view (`get_database` shows replication); there is no tool to add, promote or rebuild.
## Limits
- **Manual only.** NebulaCtrl never promotes by itself, never detects that a primary is gone and has no witness or fencing device: it fences by stopping the container through the agent, which needs the agent to be reachable (or the host to stay down).
- **Asynchronous.** A forced Promote loses what the replica had not received. There is no synchronous option, no delayed replica, no cascading replica, no logical replication and **no point-in-time recovery**. Keep [backups](https://nebulactrl.com/docs/operations/database-backups).
- **Postgres only.** No other engine, no reader proxy or pooler, and no replica outside the primary's environment.
- At most **5 replicas** per primary; a replica is pinned to its server.
- Replicas across servers depend on the [mesh](https://nebulactrl.com/docs/operations/mesh): see its [failure modes](https://nebulactrl.com/docs/operations/mesh#failure-modes) and what was **not verified** there (Tailscale SaaS, NAT and relay fallback across real networks, hosts with a firewall, large databases).
- Split-brain after a real power loss, large-database clone times and clones over a real WAN were not exercised.
---
# Database backups
Source: https://nebulactrl.com/docs/operations/database-backups
NebulaCtrl backs up **managed databases** (the database panel's engines) on a schedule or on demand, keeps them in a destination you choose, prunes old ones, and restores them. This page is about your databases. Backing up the control plane itself (Postgres and `NEBULACTRL_SECRET_KEY`) is [backup-restore.md](https://nebulactrl.com/docs/operations/backup-restore).
## How a backup runs
The dump never touches the control plane's disk or memory.
1. The control plane records a backup run and queues a `db_backup` command for the agent of the database's server. The command names the destination and the object key; it carries **no credentials**. The destination's keys and the database password are attached to the frame at send time and live only in agent memory for the run.
2. The agent runs the engine's own dump tool **inside the database container** (an argv vector, no shell; credentials go through the exec environment, not the command line, except MongoDB whose tools only take them in the URI).
3. The dump is streamed: `dump stdout → (gzip) → SHA-256 → destination upload`. There is **no temp file**, so a 50 GB dump does not need 50 GB of spare disk. S3 uploads are multipart with 16 MiB parts, at most two in flight.
4. The exit status is checked **before** the stream may end. A dump that fails halfway makes the upload fail: the multipart upload is aborted (or the local temp file removed), no object and no manifest are left behind, and the run shows `failed` with the dump tool's own error.
5. After the upload the agent writes `<object>.manifest.json` (engine, version, database, format, size, SHA-256, project/service names) and verifies the stored size. A restore into a fresh control plane can understand backups from the bucket alone.
Runs have a real state: `queued` → `running` (the agent acknowledged) → `succeeded` | `failed`. A run is `skipped` when it was not attempted (database stopped, not deployed, or the server offline) and `expired` when the command was never delivered. Nothing is ever reported as a success that did not happen.
| Engine | Tool (in the container) | Stored as | Restore |
|---|---|---|---|
| PostgreSQL | `pg_dump --format=custom --no-owner --no-acl` | `.dump` | `pg_restore --single-transaction --exit-on-error` |
| MySQL | `mysqldump --single-transaction --quick --routines --events --triggers` (as `root`) | `.sql.gz` | `mysql` |
| MariaDB | `mariadb-dump …` (as `root`) | `.sql.gz` | `mariadb` |
| MongoDB | `mongodump --archive --gzip` | `.archive.gz` | `mongorestore --archive --gzip` |
| Valkey / Redis | `valkey-cli --rdb -` / `redis-cli --rdb -` | `.rdb` | **not automated** (see below) |
| ClickHouse | not supported yet | | |
Key-value dumps are RDB snapshots. They are real backups, but loading an RDB means replacing the data file of a stopped server, which NebulaCtrl does not do yet: download the object and place it as `dump.rdb` in the data volume yourself.
## Destinations
Settings → **Backups** (admin scope). Two kinds:
- **S3-compatible** (AWS S3, Cloudflare R2, Backblaze B2, Wasabi, Hetzner Object Storage, SeaweedFS, Garage, …). Built on `aws-sdk-go-v2`. For a custom endpoint choose *path-style* if the store needs it and keep **compat mode** on (it is the default for custom endpoints): since 2025 the AWS SDK adds CRC32 checksums to every request, which several S3 implementations reject. Compat mode limits them to operations that require them. Use a **bucket-scoped key** with `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket` and `s3:AbortMultipartUpload`.
- **Local directory on the server** — flagged **not off-site**. Files are written under the agent's backup directory (`--backup-dir`, default `<state-dir>/backups`, `/var/lib/nebulactrl-agent/backups`), in the sub-directory you name. A destination can not point outside that directory; to use another disk mount it there or set `--backup-dir`. A backup on the database's own host does not survive the loss of that host, so use it as a first copy, not the only one. A local backup lives on one server and can only be restored onto a database on that same server.
Keys are sealed with `NEBULACTRL_SECRET_KEY` and are write-only in the API and UI (`accessKeySet: true`, never the value). Restoring the control plane without its secret key loses them.
**Test** writes, reads back and deletes a probe object from the agent of a server you pick (the agent is the one that talks to the destination). The result and any error stay on the destination.
A destination that schedules or stored backups use can not be deleted, and its endpoint, bucket, prefix and directory can not change while backups live in it.
## Schedules and retention
A database panel → **Backups** → *Add schedule*: a destination, a cron expression, a timezone and retention.
- Cron is the standard 5 fields (`minute hour day-of-month month day-of-week`) or `@hourly`, `@daily`, `@weekly`, `@monthly`, `@yearly`. Ranges, lists, steps (`*/15`, `10-50/10`) and month/weekday names work. When both day fields are restricted a day matches if either does (classic cron). The parser is in-house (`internal/cronexpr`), with its next-run calculation tested across daylight-saving changes: a wall time that does not exist (spring forward) is skipped that day, a repeated one (fall back) runs once.
- At most one backup every **5 minutes**.
- A tick runs every minute. Missed runs are not replayed; after downtime a schedule runs once and moves on.
- **Missed backups are reported.** A schedule whose backup did not succeed within its window (the next due time after the last success plus a grace of a quarter of the interval, 15 minutes to 6 hours) raises `backup.missed` once, until the next success. It covers the case where nothing ran at all (control plane or server down, destination unreachable). See [Service alerts](https://nebulactrl.com/docs/operations/service-alerts#missed-backups).
- **Retention**: keep the newest *N* successful backups of the schedule (default 7) and/or delete those older than *D* days. `0` disables a limit. The newest backup is never pruned, and **locked** backups are never pruned and do not count towards *N*. Backups made on demand are never pruned automatically. Pruning deletes the object through the agent of the server that made it (an offline server retries after the next successful backup).
## Restore
Pick a backup → **Restore**.
- **To a new database (default).** NebulaCtrl creates a sibling database service next to the original (same engine, version and settings, its own volume, **fresh credentials**), deploys it empty, waits until it runs, then streams the dump into it. The original is untouched. Point your apps at the new database when you are satisfied, then delete the old one. For restores onto the same server this needs disk for both copies.
- **In place (advanced, destructive).** Type the database's slug to confirm. NebulaCtrl first takes a safety backup (marked *pre-restore*) to the same destination; only if that succeeds it restores over the database with `--clean` semantics. If the safety backup fails nothing is changed. NebulaCtrl can not stop your applications' writes: stop dependants first if consistency matters.
The agent verifies the object's SHA-256 against the recorded value while restoring; a mismatch fails the restore and reports it. Restore to a new database only happens after the new database's deployment succeeded; a failed deployment fails the restore and leaves the sibling for you to inspect or delete.
A backup is not proven until it restores. Restore the latest one into a new database now and then and look at the data.
## Restore from a dump file
Moving a database in from somewhere else (another server, Coolify, a laptop): **Database → Backups → Restore from a file**, or
```sh
nebulactrl databases restore shop/production/postgres --file shop.pgdump
nebulactrl databases restore shop/production/postgres --file shop.sql.gz --in-place --confirm postgres --destination s3
```
| Engine | Accepted files |
|---|---|
| PostgreSQL | `pg_dump -Fc` (custom), `-Ft` (tar) or plain SQL, each optionally `.gz` |
| MySQL, MariaDB | plain SQL from `mysqldump` / `mariadb-dump`, optionally `.gz` |
| MongoDB | `mongodump --archive`, optionally `.gz` |
The format is detected from the file's first bytes; the wrong engine, a `pg_dumpall` cluster dump, zstd/xz/zip, UTF-16 text and Redis/Valkey RDB files are refused with the reason, before anything is changed. Make a Postgres dump with `pg_dump -Fc -U <user> <db> > shop.pgdump`.
**How the file travels.** The upload goes through the control plane to the server's agent as a stream and is piped into the engine's own client (`pg_restore`, `psql`, `mysql`, `mongorestore`) inside the database container. Nothing is stored on the control plane or written to the agent's disk, and at most one 256 KiB window is in memory, so a 5 GB file works. The default limit is **5 GiB** (`NEBULACTRL_MAX_DUMP_UPLOAD_BYTES`); a larger file is refused at the start. Throughput is bounded by the window over the round trip to the agent. If you front the control plane with a proxy that caps request bodies (a Cloudflare-proxied hostname does at 100–500 MB), upload through a direct or DNS-only hostname.
**Targets.** As for backups, **a new database is the default**: a sibling is created, deployed empty, and the file is restored into it; the original is untouched. **In place** needs the database's slug and a backup destination for the safety backup (default: the one of its schedule); only after that backup succeeded can the file be sent. The restore waits in `awaiting_upload` until the target runs, then the client sends the file (the CLI and web app do this for you; the raw endpoint is below).
**What PostgreSQL does.** The file is restored into a *scratch database* created from `template0`, and swapped in only when it restored completely: the live database is renamed away, the scratch database takes its name, the old one is dropped (connections are ended at that moment). A broken or abandoned file therefore leaves the data exactly as it was, and PostGIS dumps work because their extensions and schemas are created by the dump itself in an empty database. `pg_restore` runs with `--no-owner --no-acl --role=<managed user>`: every object ends up owned by the managed user and grants are dropped. For plain SQL the same is done on the stream: ownership, `GRANT`/`REVOKE`, role, tablespace, `CREATE/DROP/ALTER DATABASE` and `\connect` lines are skipped (the log counts them). In-place needs room for both copies while it runs.
**MySQL, MariaDB, MongoDB** are restored directly into the target database; they have no atomic swap. `CREATE/DROP DATABASE` and `USE` lines and `DEFINER=` clauses of a SQL dump are removed so the data lands in the managed database; `mongorestore` restores the databases the archive contains (`--drop` for in-place). A failed in-place restore can leave part of the dump there: restore the safety backup, or prefer a new database.
Progress and errors (the client's own messages, with credentials masked) are kept as the restore's log, shown in the dialog and in `GET /dump-restores/{id}`. The control plane records the SHA-256 of what it relayed. A restore whose transfer is lost (control plane restart, agent offline) is failed with "nothing was switched over"; start it again.
API: `POST /services/{id}/dump-restores` (202; `mode`, `name`, `serverId`, `confirmSlug`, `destinationId`, `fileName`, `sizeBytes`), `GET /dump-restores/{id}`, `GET /services/{id}/dump-restores`, `POST /dump-restores/{id}/cancel`, and the raw upload `PUT /api/v1/dump-restores/{id}/upload?fileName=` (body = the file, `Content-Length` required; 202 once the agent has the whole file; 409 `dump_restore_not_ready` until the status is `awaiting_upload`). Statuses: `provisioning`, `pre_backup`, `awaiting_upload`, `uploading`, `restoring`, `succeeded`, `failed`. Needs `deployments:write` (and `projects:write` for a new database). Audited as `backup.dump_restore.create|upload|succeeded|failed|cancel`. The create operation is not available over MCP (it needs a binary upload); reads are.
Limits: PostgreSQL 16+ images, agents of this release or newer (an older agent never answers and the upload is refused after 60 s), no `pg_dumpall` cluster dumps, no URL or object-storage import, no resumable uploads.
## API, audit, permissions
- Destinations: `GET/POST /backup-destinations`, `PATCH/DELETE /backup-destinations/{id}`, `POST …/test`. Admin scope to write.
- Per database: `GET/POST /services/{id}/backup-schedules`, `GET/POST /services/{id}/backups` (202), `GET /services/{id}/backup-restores`.
- `PATCH/DELETE /backup-schedules/{id}`, `GET/DELETE /backups/{id}`, `PUT /backups/{id}/lock`, `POST /backups/{id}/restore` (202), `GET /backup-restores/{id}`.
- Creating, restoring and deleting need `deployments:write` on the project; reading needs `projects:read`. Creates honour `Idempotency-Key`.
- Every change is audited (`backup.create`, `backup.restore`, `backup.delete`, `backup.lock`, `backup.schedule.*`, `backup.destination.*`); credential changes record only that they happened.
- Finished backups and restores are published to a notifier hook; channels (email, Slack, …) attach to it in Phase 6b.
## Limits
- Logical dumps only: no point-in-time recovery, no volume snapshots.
- No client-side encryption: protect the bucket and its key. A planned opt-in is `age` encryption; a lost key makes backups unreadable, so it will be explicit.
- Backups of one database are not coordinated with another's.
- Deleting a project or database removes its backup history rows but leaves the stored objects: delete them from the destination if you do not need them.
---
# Volume backups
Source: https://nebulactrl.com/docs/operations/backups
NebulaCtrl backs up the **named volumes of an ordinary service** (anything that is not a managed database): uploads, SQLite files, a Git server's repositories, an app's `data` directory. Managed databases keep their own logical dumps ([database backups](https://nebulactrl.com/docs/operations/database-backups)); the control plane's own backup is [backup-restore.md](https://nebulactrl.com/docs/operations/backup-restore).
A volume backup is **a file-level backup**. Out of the box it is *crash-consistent*: the files are exactly as they were at one instant, as if the power had been cut. Apps that journal their writes (SQLite, PostgreSQL, Valkey with AOF, most key-value stores) start from such a copy without trouble. For *application-consistent* data add a **before-backup command** (below).
## How a backup runs
The point-in-time copy is taken fast; the slow part, moving the data, happens afterwards while the service runs normally.
1. **Before-backup command** (optional). Run inside the service's container with a timeout, for example `sqlite3 /data/app.db ".backup '/data/.nc-backup/app.db'"` or `redis-cli BGSAVE`. It runs **before** the freeze, because Docker refuses to exec into a paused container. If it fails, the backup fails with "Nothing was backed up." and the service is never frozen (unless you chose *continue anyway*).
2. **Freeze.** The agent takes the service's per-service lock (so no deployment can recreate the container in the middle), then runs `docker pause` on the container. This uses the kernel's cgroup freezer: every process stops, **TCP connections stay open and nothing is reset**; clients see a latency spike, not an outage.
3. **Local copy.** A helper container copies the volumes into a temporary *staging volume* with `cp -a --reflink=auto`. On filesystems with copy-on-write (btrfs, XFS with reflink) this takes well under a second regardless of size; elsewhere it is a plain copy, bounded by the freeze budget.
4. **Thaw.** `docker unpause`. The freeze lasted only as long as step 3.
5. **After-backup command** (optional) runs now (for `live` mode: after the upload).
6. **Upload.** The helper runs **restic** on the staging copy and uploads into the service's repository in the background. restic is incremental and deduplicating: unchanged data costs nothing, and the data is encrypted on the server before it leaves it. The staging copy is deleted afterwards.
The service is only ever paused (or stopped) for step 3, **never for the upload**, and never longer than the freeze budget.
### Consistency modes
| Mode | What happens | Use it for |
|---|---|---|
| **Brief freeze** (default) | pause → copy → unpause. Requests stall for about a second on copy-on-write hosts. The default budget is **5 s** (1–60). | most services |
| **Stop during backup** | stop → copy → start. Connections are reset and the service is unavailable until it is back. | apps that cannot be paused safely: leases or heartbeats shorter than the freeze (Consul/etcd-style, Kafka, RabbitMQ), apps with multi-file invariants and no crash recovery |
| **Keep running, no freeze** | no pause, no copy: restic reads the live volumes. The backup can hold files from different moments. | static uploads, or together with a before-backup command that produces a consistent file |
**The freeze budget is a hard limit.** If the copy does not finish within it, the agent thaws the service, discards the copy and fails the backup with *"Snapshot took longer than 5 s; use Stop mode or a pre-backup hook"*. There is no silent fallback to another mode, and the service is never left paused. A timer unfreezes independently of the copy, retries the unpause, and reports it loudly if it cannot.
A service that is **not running** is backed up without any freeze (nothing is writing), whatever mode is set. If a deployment (or another action on the service) is in progress for two minutes, the backup is `skipped` and the service is not touched.
Hosts without copy-on-write need free disk for the staging copy (about as much as the volumes use). The agent checks this **before** freezing and fails early with the numbers instead of filling the disk.
## Set it up
Service → **Backups** tab (shown for services that have at least one volume):
- **Destination**: the same destinations as database backups (Settings → Backups): S3-compatible storage or a directory on the server. A directory is **not off-site**; it does not survive the loss of that server.
- **Consistency**, **freeze budget**, before/after commands (program and arguments; no shell unless you write `sh -c '…'`; optional user and timeout, default 5 min, and *if it fails: stop the backup / continue anyway*), and **excludes** (restic patterns such as `*.tmp`).
- **Retention**: keep the last *N* daily, weekly and monthly backups (defaults **7 / 4 / 6**). `0` removes that limit; all zeros keeps everything. **Locked backups are never removed.**
- **Schedule** (optional): a cron expression and timezone, at most one run every 5 minutes. A schedule that finds the server offline records a `skipped` run with the reason.
- **Missed backups are reported.** Like database schedules, an enabled volume schedule that produced no successful backup within its window raises `backup.missed` once per streak. See [Service alerts](https://nebulactrl.com/docs/operations/service-alerts#missed-backups).
*Back up now* uses the saved settings. All volumes of a service go into **one snapshot** (`/vol/<name>`), so they are consistent with each other.
### Where the data goes
- One restic repository **per service and destination**, at `<destination prefix>/volumes/<serviceId>`. Backups of different services never share a repository.
- The repository is encrypted (AES-256 with Poly1305) with a password NebulaCtrl generates on the first backup. It is stored sealed with `NEBULACTRL_SECRET_KEY` and reaches the agent only as a send-time secret in memory.
- **Download the recovery key** (Backups tab, admin scope, audited). Without the key, or without the secret key that seals it, the backups can not be read. Restoring the control plane without its secret key loses them, exactly as it loses destination credentials.
Restoring by hand without NebulaCtrl (restic ≥ 0.14):
```sh
export RESTIC_PASSWORD='<recovery key password>'
export RESTIC_REPOSITORY='<repository from the recovery key>' # e.g. s3:https://s3.example.com/bucket/volumes/<serviceId>
# S3 also needs AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
restic snapshots
restic restore <snapshot>:/vol/data --target /path/to/new/volume
```
## Restore
Choose a backup → **Restore**, pick the volume.
- **Restore into a new volume (default).** The data goes into a **new** Docker volume (`nc-vol-<id>-<name>-r<yymmddhhmmss>`). Nothing is overwritten. The volume row is then pointed at the new volume as a **staged change** ("`/data · restored from Jan 12, 03:30`", counted in the staged bar). **Nothing changes until you press *Deploy*.** The old volume is kept, unmounted, and listed under the service's Volumes as a retired volume you can delete when you are happy. *Discard* in the staged bar undoes the swap (the unused restored volume then also shows as retired).
- **Restore in place (guarded).** For when two copies do not fit on the disk. Type the service's slug. A **safety backup** of the current state is taken first (`pre_restore`); if it fails, nothing is changed. Then the agent stops the service, empties the volume, restores into it and starts the service again.
A rollback of a deployment **never** changes which volume a service uses.
### Browse files and download
A backup's **Browse files** shows the snapshot's directories. Download a single file, a folder as `.zip`, or a whole volume as `.zip`. Downloads are audited, stream through the control plane (nothing is stored there) and are limited to 1 GiB per file. Listings show up to 5,000 entries per directory.
## What runs on the server
Everything that touches volume bytes runs in a **short-lived helper container** from the official `restic/restic` image, **pinned by digest** (`agentprotocol.ResticImage`, restic 0.19.1). The agent may itself run in a container and cannot read `/var/lib/docker`.
- The image is pulled on first use only (~30 MB). Docker cleanup never removes it.
- Helpers carry the label `nebulactrl.helper=backup` and **never** `nebulactrl.managed`, so the orphan sweep and Docker cleanup leave them alone. They run with no capabilities beyond file ownership and read access, a read-only root filesystem, `no-new-privileges`, 512 MiB / 1 CPU, no published ports and (local destinations) no network at all.
- The repository password and S3 keys are passed **per exec in the environment**, never in the helper's create config (which `docker inspect` would show) and never in a command line. restic's cache lives in the volume `nebulactrl-backup-cache`.
- A **local destination** lives under the agent's backup directory. On the bare host the helper bind-mounts it. When the agent itself runs in a container (dev, compose), the helper mounts the volume or bind of the agent container that holds the backup directory; the directory must therefore be on a mounted volume.
- Leftovers are swept: every two minutes and on every reconnect the agent removes helpers and staging volumes no running backup owns, thaws a container a dead backup left paused and starts one a dead `stop`-mode backup left stopped. A paused container no command holds is always thawed.
- The agent reports a frozen service as running and healthy; the reconciler never restarts it.
## Limitations
- **Crash-consistent is not application-consistent.** An app with no journal or with short leases can restore into a state it cannot start from. Use a before-backup command or stop mode, and test a restore now and then.
- A freeze can cost a stalled request or a failed health probe if it outlasts the proxy's health check (about 3 s). Keep the budget small, or use stop mode.
- Hosts without copy-on-write copy the data while frozen; large volumes (over a few GB) will exceed a 5 s budget there. Use stop mode, a before-backup command that writes a small consistent file plus excludes, or `live` mode for static data.
- Applying a restored volume is a recreate deploy (services with volumes never get a rolling deploy).
- Retention runs after each successful backup and deletes with `restic forget --prune`; a server that is offline retries after the next backup. Rows of forgotten backups stay as history.
- Not offered yet: restoring selected files into the live volume, restoring as a new service, test restores, bandwidth limits.
## API
Operations are tagged `volume-backups` in the OpenAPI document: `GET/PUT/DELETE /services/{id}/volume-backup`, `POST/GET /services/{id}/volume-backups`, `GET/DELETE /volume-backups/{id}`, `PUT /volume-backups/{id}/lock`, `POST /volume-backups/{id}/restore`, `GET /volume-backups/{id}/files` and `…/files/content`, `GET /services/{id}/volume-restores`, `GET /services/{id}/retired-volumes`, `DELETE /retired-volumes/{id}`, `POST /services/{id}/volume-backup/recovery-key`.
---
# Control-plane backup and restore
Source: https://nebulactrl.com/docs/operations/backup-restore
NebulaCtrl's state is **one Postgres database plus one secret key**. They must be backed up together, and they must be stored in different places:
| Item | Contains | Without it |
|---|---|---|
| Postgres database `nebulactrl` | projects, environments, services, config, releases, deployments, audit log, API token hashes, agent identities (public keys), encrypted secrets, canvas layouts, job queue | everything is gone |
| `NEBULACTRL_SECRET_KEY` (and `NEBULACTRL_SECRET_KEY_PREVIOUS`, if you rotated) | the AES-256-GCM key ring for every sealed value: variables, git and registry credentials, SSH keys, integration tokens, TOTP seeds, backup passwords | the database restores but **every secret sealed with a missing key is unreadable** and must be re-entered |
The key never lives in the database, and **a backup never contains it** (a dump next to the key would defeat the encryption). Store it in a password manager or secrets vault, apart from the backups but retrievable on the day of a disaster. The key is a 32-byte value; `nebulactrl-server keygen` makes a new one. **Rotate with the procedure in [secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest#rotating-the-key)** (new key as `NEBULACTRL_SECRET_KEY`, the old one as `NEBULACTRL_SECRET_KEY_PREVIOUS`, then `nebulactrl-server rekey`), never by just replacing the key: values sealed with a key that is not in the ring cannot be read. **Keep every old key for as long as backups taken before the rotation might be restored.**
The settings page shows a **fingerprint** of the key (12 hex characters; it identifies the key without revealing it) and writes the same fingerprint into every backup. After a restore, the command prints it again, so you can check that the key you are about to start with is the one the backed-up instance used.
## What is *not* in the backup
- **Your services' data.** Docker volumes (`nc-vol-…`) and anything databases or apps keep on the agent hosts are not copied by the control-plane backup. *A Docker volume is not a backup.* Managed databases have their own [scheduled backups](https://nebulactrl.com/docs/operations/database-backups); for everything else use the application's own dump tools or snapshots of the host.
- **Proxy certificates** (volume `nebulactrl-traefik-acme` on each agent host, holding `acme.json`). They are re-issued automatically; back the volume up only if you want to avoid rate limits during a rebuild.
- **Built images.** They are rebuilt from git (the release stores the commit) or re-pulled.
- **Browser sessions.** Session rows are left out on purpose: after a restore everyone signs in again.
- **Agent identities** live on each host in `/var/lib/nebulactrl-agent/agent.db` *and* as public keys in the database. After a restore from a backup, agents reconnect with the same identity as long as it is in the backup.
## Scheduled control-plane backup
**Settings → Instance → Control-plane backup** (instance administrator only: the first account). It runs `pg_dump` against the control plane's own database and streams the result to one **instance backup destination**, which belongs to no team. It is independent of the per-team destinations used by [database backups](https://nebulactrl.com/docs/operations/database-backups), though it uses the same kinds of storage.
| | Default | Notes |
|---|---|---|
| Schedule | daily at **03:30 UTC** | a 24-hour time of day in the UI; the API takes any 5-field cron (UTC), at most one run every 5 minutes |
| Retention | newest **14** successful backups | 1 to 365; the newest backup is never removed |
| State | **on** after the [installer](https://nebulactrl.com/docs/installing/installer): a daily backup to the local directory below. Otherwise **off** until a destination is set up and you switch it on | |
What a run does:
1. `pg_dump --format=custom --no-owner --no-acl --exclude-table-data=sessions` streams into the destination. There is no temporary file; a dump that fails (or an upload that fails) leaves no object behind.
2. The upload is verified (size at the destination equals the bytes dumped).
3. A manifest `…dump.manifest.json` is written last, with the SHA-256, the PostgreSQL version, the key fingerprint and the schema version. A dump without a manifest is not a restore point.
4. Retention deletes the oldest backups of this destination beyond the count (dump and manifest). Backups made for an earlier destination stay where they are: changing the destination never deletes anything.
Objects are stored as `<prefix>/nebulactrl-instance/<yyyy>/<mm>/nebulactrl-<yyyymmddThhmmssZ>-<id8>.dump`.
Details worth knowing:
- **One run at a time.** A scheduled slot that comes due while a run is still going is skipped, not queued. A run has five hours.
- **Missed slots.** If the control plane is down at 03:30, one backup runs as soon as it is back up; the next slot is computed from then.
- **Failures are loud.** A failed run is recorded with the reason (credentials and the database password are masked), written to the audit log (`instance.backup.failed`) and sent as a **Backup or restore failed** notification to the instance administrator's team (the oldest team). Turn on **Backup or restore succeeded** to also hear about good runs. Set up an email or chat channel under [notifications](https://nebulactrl.com/docs/operations/notifications), otherwise nobody is told.
- **Test the destination** with the **Test** button: the control plane writes, reads back and deletes a small probe object.
- **Back up now** starts a run immediately, whether or not the schedule is on.
- **Before an upgrade** the control plane runs a backup by itself (trigger *Before upgrade*), whether or not the schedule is on, and refuses to upgrade when it failed: [upgrades](https://nebulactrl.com/docs/operations/upgrades#backup-before-a-control-plane-upgrade).
- Every change and every run is audited (`instance.backup.update`, `.run`, `.test`, `.succeeded`, `.failed`, `.key_acknowledged`). Credentials are recorded as "changed", never as values.
### Destinations
| Kind | Where | Off-site? |
|---|---|---|
| **S3-compatible bucket** | AWS S3, Cloudflare R2, Backblaze B2, Wasabi, MinIO, SeaweedFS, Garage … | yes |
| **Directory on the control-plane host** | a sub-directory of `NEBULACTRL_BACKUP_DIR` | **no**, labelled "not off-site" |
A directory is only a first copy: if the host is lost, so are the backups. Use a bucket, or sync the directory elsewhere. In the installer's Compose setup `NEBULACTRL_BACKUP_DIR` is `/var/lib/nebulactrl/backups` inside the container, bind-mounted from `/var/backups/nebulactrl` on the host (`--backup-dir` or `NEBULACTRL_BACKUP_HOST_DIR` changes it). Outside Docker set the variable yourself; without it the local option is unavailable.
S3 permissions for a bucket-scoped key: `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket`, `s3:AbortMultipartUpload`. Credentials are stored encrypted and never shown again. Turn on bucket encryption and keep the key's scope to this bucket: the dump contains password hashes, API token hashes and the *encrypted* secrets, so treat the bucket as sensitive.
### Requirements
- **`pg_dump` on the control-plane host.** The container image ships the PostgreSQL 17 client tools. A host install needs `postgresql-client` of at least the database server's major version. Without them the page says so and backups do not start.
- A reachable database through `NEBULACTRL_DATABASE_URL` (password and TLS settings are passed to `pg_dump` through its environment, never on the command line).
### The reminder card
Until an administrator confirms that the instance secret key is stored separately, a **Store your instance secret key separately** card shows on the settings pages. **I stored the key** dismisses it for everyone and is audited. It does not check anything: it is a promise, so keep it.
## Restore
Restoring replaces the control plane's knowledge with the backup's. Anything that happened after the backup (deployments, edits) is forgotten, and **running containers on agent hosts are not touched**: the reconciler reconciles them against the restored desired state when agents reconnect, and removes managed containers that belong to no known deployment after a 10 minute grace period.
A restore is a **command-line operation into an empty database**, run with the control plane stopped:
```
nebulactrl-server restore --from <source> [--at <timestamp>] [--database-url <url>]
```
| Flag | Meaning |
|---|---|
| `--from` | where the backups are: `s3://bucket/prefix` or a directory (the destination's directory, or any parent of it) |
| `--at` | restore the newest backup created **at or before** a time: RFC 3339 (`2026-10-27T03:30:00Z`), `"2026-10-27 03:30"` (UTC), a date (the end of that UTC day) or `latest` (the default) |
| `--backup-id` | restore exactly this backup (id or its first 8 characters); overrides `--at` |
| `--list` | list the backups at the source (id, time, size, PostgreSQL version, key fingerprint) and exit |
| `--database-url` | the **empty** target database; default `$NEBULACTRL_DATABASE_URL` |
| `--tmp-dir` | where the dump is downloaded and verified before restoring; needs room for one dump (default: the system temp directory) |
| `--endpoint`, `--region`, `--path-style` | S3-compatible stores (`s3://` sources only) |
For an S3 source the credentials come from the environment, never from the command line: `NEBULACTRL_RESTORE_ACCESS_KEY` and `NEBULACTRL_RESTORE_SECRET_KEY` (or the standard `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`). They are not read from the control plane's database, because that database is exactly what you are restoring. The command does not need `NEBULACTRL_SECRET_KEY`.
What the command does: it finds the backup, **downloads it and verifies its SHA-256 and size** against the manifest, refuses a target that already holds any table, and then runs `pg_restore --single-transaction --exit-on-error`. A damaged backup is rejected before a byte reaches the database, and a restore that fails half-way is rolled back, leaving the database empty so you can try another backup. On success it prints the key fingerprint the original instance used.
### Step by step with Docker Compose
```sh
cd /opt/nebulactrl # where the installer put docker-compose.yml and .env
# 1. stop the control plane, keep Postgres running
docker compose stop nebulactrl
# 2. recreate an empty database
docker compose exec -T postgres psql -U nebulactrl -d postgres \
-c 'DROP DATABASE nebulactrl WITH (FORCE)' -c 'CREATE DATABASE nebulactrl OWNER nebulactrl'
# 3a. restore from the local backup volume ...
docker compose run --rm --no-deps nebulactrl restore \
--from /var/lib/nebulactrl/backups --tmp-dir /var/lib/nebulactrl/backups --at latest
# 3b. ... or from a bucket
docker compose run --rm --no-deps \
-e NEBULACTRL_RESTORE_ACCESS_KEY=... -e NEBULACTRL_RESTORE_SECRET_KEY=... \
nebulactrl restore --from s3://my-bucket/nebulactrl --endpoint https://s3.example.com --path-style \
--tmp-dir /var/lib/nebulactrl/backups --at 2026-10-27T03:30:00Z
# 4. start the control plane with the ORIGINAL NEBULACTRL_SECRET_KEY in .env
docker compose up -d nebulactrl
```
(`--tmp-dir` points at the volume because the container's `/tmp` is a RAM-backed tmpfs.) `docker compose run … restore --list --from …` shows what is available first.
Without Compose, run the same `nebulactrl-server restore` against your database, with `pg_restore` of at least the dump's PostgreSQL major version on the `PATH`.
Verify: `curl -fsS "$NEBULACTRL_PUBLIC_URL/readyz"`, sign in (everyone has to: sessions are not backed up), open a service and confirm a secret variable shows as *set*; deploy a harmless change to prove secrets decrypt. If a deploy fails to resolve variables with a decryption error, the key does not match the backup: compare it with the fingerprint the restore printed.
Restore with the **same or a newer** `nebulactrl-server` than the one that wrote the backup. A newer one migrates the schema forward on start; an older one warns that the backup has a newer schema.
To restore onto a **new machine**: install the same version (see [upgrade](https://nebulactrl.com/docs/installing/upgrade)), put the original `NEBULACTRL_SECRET_KEY` in `.env`, restore as above, and keep (or re-point DNS for) `NEBULACTRL_PUBLIC_URL`, because agents dial that URL. If the URL must change, re-run the installer on each host with the new `--url`.
### When the restore says no
| Message | Cause and fix |
|---|---|
| `the target database is not empty` | restore only into a new, empty database (`CREATE DATABASE nebulactrl`), never over a live one |
| `SHA-256 … it is damaged` / `incomplete or damaged` | the object at the source is not the one that was written; pick another with `--at` or `--backup-id` |
| `no control-plane backups were found at the source` | wrong `--from` or prefix; run `--list`; a dump without a manifest is not offered |
| `no backup at or before …; the oldest one is from …` | `--at` is older than every backup; retention removed it |
| `pg_restore was not found` | install the PostgreSQL client tools (the image has them) |
| `pg_restore: error: … unsupported version` | the client tools are older than the dump's PostgreSQL; install a newer `postgresql-client` |
| `S3 credentials are missing` | set `NEBULACTRL_RESTORE_ACCESS_KEY` and `NEBULACTRL_RESTORE_SECRET_KEY` |
## Manual backup (alternative)
If you cannot use the scheduled backup, a plain dump works and restores with the same tools:
```sh
cd /opt/nebulactrl
mkdir -p backups
docker compose exec -T postgres pg_dump -U nebulactrl -d nebulactrl --format=custom \
> "backups/nebulactrl-$(date -u +%Y%m%dT%H%M%SZ).dump"
```
Restore it into an empty database (steps 1 and 2 above), then load it with `docker compose exec -T postgres pg_restore -U nebulactrl -d nebulactrl --no-owner < backups/<file>.dump` and start the control plane (step 4). (`nebulactrl-server restore` only reads backups written by the scheduled backup, because it relies on the manifest.) Also record: the `.env` file (it holds the key and the database password), the image tag or commit you run, and your DNS records.
Restoring **a managed database** from a dump file you made yourself (`pg_dump`, `mysqldump`, `mongodump`) is a different operation: see [restore from a dump file](https://nebulactrl.com/docs/operations/database-backups#restore-from-a-dump-file).
## Test it
A backup you have never restored is a hope. Quarterly: restore the latest backup into a scratch Compose project (`docker compose -p nebula-restore-test …` with a different `NEBULACTRL_PORT`), start it **without agents enrolled to it** (it has the same agent identities, so do not point real agents at it), and check that you can sign in and see your projects.
---
# One-click templates
Source: https://nebulactrl.com/docs/operations/templates
NebulaCtrl ships a hand-reviewed catalog of self-hostable apps (`templates/catalog/*.yaml`, over 300 of them, embedded in the server binary). **Add to canvas** turns a template into ordinary, staged services in an environment: nothing is deployed until you press Deploy (or choose "Add and deploy"). A template is **not** a Docker Compose project (to turn one into services, see [Compose import](https://nebulactrl.com/docs/operations/compose-import)); every template service becomes an independent service on the canvas, on one server, with normal deploy, rollback, variables, domains and logs.
## What an install creates
For each template service, in one transaction:
- a service (image pinned as `repo:tag@sha256:…`), or a managed database through the databases package (`database:` block);
- volumes, config files, variables (secrets generated once and stored encrypted), at most one domain, and published TCP/UDP ports (proxy routes);
- manual links for `dependsOn` (deploy order, canvas edges), plus the edges derived from `${{key.VAR}}` references;
- a canvas group named after the template, and a `template_instances` row (the stack, deletable as a unit);
- one audit event `template.install`.
Deploying a stack uses *deploy staged* for exactly those services: providers deploy first (references and `dependsOn`); a one-shot provider must finish before its dependants start.
## Format (schema 1)
Strict YAML: an unknown field is an error. One file per template, `templates/catalog/<id>.yaml`, file name equals `id`.
```yaml
schema: 1
id: umami # [a-z0-9-], equals the file name
revision: 1 # bump when an install of the new text would differ (image, config, env, files, volumes...)
name: Umami
summary: One line (<= 200 chars).
description: Optional longer text.
category: web # homelab | media | storage | games | devtools | ai | data | web | analytics | automation | productivity | security
tags: [analytics, privacy]
icon: umami # simple-icons slug (optional)
links: {website: https://umami.is, docs: https://umami.is/docs, source: https://github.com/umami-software/umami}
upstream:
license: MIT # SPDX id or name of the UPSTREAM APP (required)
licenseUrl: https://… # required for source-available and proprietary
class: permissive # permissive | copyleft | source-available | proprietary
origin: {type: original} # or coolify|dokploy with path, commit, license (attribution; required)
requires: {memoryMb: 512, diskGb: 2, arch: [amd64, arm64]} # arch defaults to both; list only what the images support
heavy: false # true: several GB / lots of RAM / slow first start (see "Heavy templates")
requiresUnrestricted: false # true: a service uses devices, GPUs or another unrestricted-only option; installs only on an unrestricted server
inputs: # install form; used as ${input:KEY}
- {key: ADMIN_EMAIL, label: Admin email, format: email, required: true}
- {key: ADMIN_PASSWORD, label: Admin password, secret: true, default: "${{secret(20)}}"}
services: # 1..8, ordered
- key: app # [a-z0-9-]; `${{app.VAR}}` references use it
name: Umami
kind: app # app (default) | worker | database
image: ghcr.io/umami-software/umami:3.4.1 # explicit, non-floating tag
digest: sha256:… # multi-arch index digest; set by tools/template-pin
port: 3000 # http port (0 = none)
command: [..] # optional start command (argv)
healthcheck: {type: http, path: /api/heartbeat, startPeriodSeconds: 15, deadlineSeconds: 180}
env:
DATABASE_URL: ${{db.DATABASE_URL}}
APP_SECRET: ${{secret(64)}}
TITLE: ${input:TITLE}
secrets: [OTHER_KEY] # env keys that are secret although they have no generator
volumes: [{name: data, path: /app/data}]
files: [{path: /etc/app/config.toml, mode: "0444", content: "…"}]
ports: [{name: game, port: 25565, protocol: tcp, publish: true}]
domains: [{generate: true}] # or {host: "${input:DOMAIN}"}; max one
dependsOn: [db]
oneshot: false # run-to-completion job (migrations): exit 0 = success, never restarted
resources: {memoryMb: 512, cpus: 1}
docker: # container options, typed, same names as a service's Docker options (see "Docker options")
entrypoint: [/bin/sh, -c]
shmSizeMb: 512
ulimits: [{name: nofile, soft: 65536, hard: 65536}]
stopGraceSeconds: 60
- key: db
name: Umami DB
kind: database
database: {engine: postgres, version: "17", databaseName: umami} # managed database; version must be one the engine catalog offers
group: Umami # canvas group title (default: name)
notes: ["Default login admin / umami: change it."]
```
### Categories and licence classes
The gallery has twelve categories (ids in the `category` field; the order below is the display order): `homelab` (dashboards, networking), `media`, `storage` (files, sync, backups), `games`, `devtools`, `ai`, `data` (databases, search, brokers, backend as a service), `web` (CMS, wikis, shops, forms), `analytics` (web analytics, uptime, observability), `automation`, `productivity` (notes, tasks, documents, chat, mail, ERP) and `security` (identity, SSO, password managers). A template has exactly one.
`upstream.class` says what the upstream licence allows, and the gallery filters on it:
| Class | Meaning | Examples of licences |
|---|---|---|
| `permissive` | Open source without copyleft | MIT, Apache-2.0, BSD, ISC |
| `copyleft` | Open source, derivative works must stay open | GPL, AGPL, LGPL, MPL, EUPL |
| `source-available` | The source is public but the licence is not OSI-approved, or open-core code is restricted (needs `licenseUrl`; the UI warns) | BUSL, SSPL, Elastic, FSL, Sustainable Use |
| `proprietary` | Closed source that is free to run (needs `licenseUrl`; the UI warns) | Plex |
Open source in the gallery's licence filter means `permissive` plus `copyleft`. The class describes the app the image runs, not the image's base layers and not the repository's docs.
### Inputs and formats
An input is `text` by default, a password field when `secret: true`, and a select when it has `options`. `format:` makes it a typed field: `email`, `url` (http or https), `hostname` (DNS name or IP, no scheme or path) or `number` (whole number). The form renders the matching browser field, and the install and the plan refuse a non-empty value that does not fit with 422 `invalid_input`. Rules the validator enforces: the format is one of those five, a default must fit it, secret and select inputs stay plain text, and a **required** input without a default whose key ends in `_EMAIL`, `_URL` or `_URI` must declare the matching format.
### Managed database versions
`database.version` must be a version the engine catalog (`internal/databases/engines`) offers: for example postgres 16-18, mysql 8.4 and 9.7, mariadb 11.8 and 12.3, valkey 8 and 9, redis 8, mongodb 7 and 8. Anything else fails an install with `invalid_database`, so the validator (and with it every tier-1 test) rejects it. A template that needs another version or a database user other than `postgres` uses an image service with `kind: database` instead.
### Variables
Env values use the variable grammar:
| Syntax | Meaning |
|---|---|
| `${input:KEY}` | install-time input (expanded when installing; also allowed in `files[].content`, `command`, `domains[].host`) |
| `${{key.VAR}}` | variable of another template service (rewritten to its real slug at install) |
| `${{VAR}}` | variable of the same service (including platform variables such as `${{NEBULA_PUBLIC_URL}}`, `${{NEBULA_PRIVATE_HOST}}`, `${{PORT}}`) |
| `${{secret(32)}}`, `${{randomInt(1000,9999)}}`, `${{uuid()}}`, `${{randomBytes(32,"hex")}}` | generated once at install |
| `$${{…}}` | a literal `${{` |
Managed databases export `DATABASE_URL`, `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USER`, `DATABASE_PASSWORD` (SQL engines) or `REDIS_URL`, `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD` (valkey/redis; valkey also `VALKEY_URL`). Reference them as `${{db.DATABASE_URL}}`. Engines: `postgres`, `mysql`, `mariadb`, `valkey`, `redis`, `mongodb`, `clickhouse` (versions are majors as strings, e.g. postgres `"17"`, valkey `"9"`). Use a managed database whenever the app can use one of those engines; use an image service with `kind: database` for anything else (for example a vector-enabled Postgres build).
Environment keys named like credentials (`*PASSWORD*`, `*SECRET*`, `*TOKEN*`, `*API_KEY*`, `*SALT*`…) must not have a literal value: use a generator, an input or a reference.
### Docker options
A service may carry a `docker:` block. It is the same typed set a service's own Docker options take (`docs/operations/docker-options.md`), with the same JSON names (`entrypoint`, `shmSizeMb`, `ulimits`, `capAdd`, `capDrop`, `sysctls`, `hostSysctls`, `user`, `groupAdd`…), strictly decoded (an unknown key is an error) and validated by the very same validator at catalog load, so a template can never carry an option a service could not. An install writes the block into the staged service's configuration; it is an ordinary, editable setting afterwards.
- Free text values (an `entrypoint` script, `user`, `groupAdd`) may use `${input:KEY}`; a **secret** input is refused there, because the block is stored in clear text like `command` and `files`.
- An `entrypoint` replaces the image's ENTRYPOINT and drops the image's default command; `command` becomes its arguments. The way to say compose's `entrypoint: []` is to put the command's program into `entrypoint` and keep the rest in `command`.
- A managed database (`database:`) takes no docker block.
- `readOnlyRootfs` needs every `files` entry on a volume.
- **Unrestricted-only options** (`devices`, `gpus`, `runtime`, `privileged`, `pidHost`, `seccompUnconfined`, `apparmorUnconfined`, and the capabilities `NET_ADMIN` and `SYS_ADMIN`) hand a container the host's hardware or root. A template using one must say `requiresUnrestricted: true` at the top level, and the validator checks the flag is set exactly when one is used. The gallery marks such a template, the plan shows a failed `unrestricted` check on any other server, and the install is refused with 422 `requires_unrestricted_server` (an instance admin makes a server unrestricted in its settings). The catalog ships hardware variants as separate templates for that reason: `jellyfin-hwaccel` (the render device) and `ollama-with-open-webui-gpu` (NVIDIA GPUs).
- `hostSysctls` (for example `vm.max_map_count: 262144` for Elasticsearch) is a host minimum that nothing applies: the deploy plan warns, with the command to run, when the server's value is lower.
### Health checks
Every non-one-shot service needs one (`none` is not allowed in the catalog).
- `http` / `tcp`: probed by the agent from outside the container (no curl in the image needed).
- `exec`: `command: [CMD, pg_isready, -U, app]` or `[CMD-SHELL, "…"]`, executed inside the container by Docker; the agent waits for it to report healthy.
- `docker`: use the image's own `HEALTHCHECK`.
### Files
`files` are written into the container before it first starts (tar copy). Content is stored in plain text in the service config; never put secrets in files. Paths are absolute without `..`; at most 16 files, 64 KiB each. A service with files is not scaled beyond one replica.
### Ports and game servers
`ports` declares non-HTTP container ports. With `publish: true`, the install creates a TCP or UDP proxy route on the same host port (open to the internet; the proxy restarts briefly the first time a new port is added on a server). Install fails with `port_in_use` when another route on that server already owns the port; the plan reports it first. Ports below 1024 cannot be published.
### One-shot services
`oneshot: true` services run once per deploy (database migrations, seed jobs). They have no port, domain or health check; exit code 0 is success, anything else fails the deployment. Services that depend on them (via `dependsOn`) wait for it to finish.
## Curation rules
A template ships only when every item holds:
1. The image is published by the project itself (or is a Docker Official Image) on Docker Hub, GHCR or Quay. No community forks of dead projects.
2. Pinned `tag` **and** `digest` of the multi-arch index, verified with `go run ./tools/template-pin` (a registry manifest check; it never pulls). Floating tags (`latest`, `stable`, `main`…) are rejected by the validator.
3. `upstream.license` is present and classified. **Source-available** apps (n8n, Directus, Outline, …) are shipped with `class: source-available` and a `licenseUrl`; the UI shows a badge and the licence link.
4. A real health check and sensible ports, volumes and resource hints.
5. Static validation passes (`go test ./internal/templates/`), and a sample of templates is deploy-tested in `tests/integration` (images are pulled, tested and removed again). The digests of the whole catalog are checked against the registries with manifest requests only (`NEBULACTRL_TEMPLATE_REGISTRY_CHECK=1`); no image is pulled to pin or verify one.
Translated templates (Coolify is Apache-2.0, Dokploy MIT) carry an `origin` block (`type`, `path` into the upstream repository, `commit`, `license`) and are covered by `templates/NOTICE`, which carries both licences and states what NebulaCtrl changed. They are reviewed by hand; the translator never runs at runtime.
Reasons a draft is **rejected** and never ships (the list of rejected ids with their reasons is `templates/drafts/REJECTED.txt`): the image is not published by the project (community repackaging, wrapper images) unless the project's own docs recommend it; the upstream is archived or idle for 12 months; only floating tags exist; it needs host bind mounts, the Docker socket, privileged mode, devices, host networking, wildcard DNS or a second public domain; it needs an out-of-band bootstrap that cannot be expressed as inputs and files; it needs more than 8 services; or there is no health check that proves the app started. Self-hosted edge stacks (Pangolin with Gerbil and its own Traefik) are rejected for a structural reason, described in the next section.
### Edge stacks (why there is no Pangolin template)
A template turns into ordinary services on the environment network behind the agent's Traefik, which owns 80/443. A stack that **is** its own edge cannot be expressed that way. [Pangolin](https://github.com/fosrl/pangolin) is the example (the community edition is dual-licensed AGPL-3.0 and the Fossorial Commercial License, as is Gerbil; the `ee` image runs under the commercial licence with a key), and the blockers are concrete:
- Pangolin generates Traefik configuration through an HTTP provider (`/api/v1/traefik-config`) and authenticates every resource with the Badger plugin, which is declared in Traefik's *static* configuration. The agent's Traefik is configured only through its REST provider and cannot load either, so Pangolin's dashboard could sit behind our proxy but the resources it exists to publish could not.
- Pangolin's own Traefik must therefore bind 80/443. Templates cannot publish ports below 1024, and a service never publishes host ports at all: only TCP/UDP routes (1024-65535) through the agent's Traefik exist.
- In the upstream compose file Traefik runs with `network_mode: service:gerbil`, because it reaches the sites' WireGuard addresses on `wg0` inside Gerbil's network namespace. Service containers only join the environment network, so that topology cannot be created from a template (a [Compose stack](https://nebulactrl.com/docs/operations/compose-stacks) may use `network_mode: service:<other service>`, but stacks never publish host ports and templates cannot create stacks).
- Gerbil needs `NET_ADMIN` (unrestricted servers only; `SYS_MODULE`, in the upstream compose file, is never allowed) and UDP 51820 and 21820 on the host. A UDP route through the agent's proxy makes the upstream see the proxy's own socket instead of the peer's address, and Gerbil's relay and NAT hole punching work from the peer's source address.
Product changes that would make it possible, in order of size: a service option to share another service's network namespace (unrestricted servers only); a per-server **external proxy** mode (the agent does not start Traefik, or moves it off 80/443, and a service may publish host ports directly, unrestricted servers only); and template support for a Compose stack. Until then, run Pangolin on its own host with its own installer and connect sites with its connector (`fosrl/pangolin-cli up site`, which Pangolin's docs now recommend for Docker, built on the Newt networking stack) from NebulaCtrl-managed or any other hosts. A connector template is not shipped either: it needs a site ID and secret created in a running Pangolin (the same out-of-band bootstrap that keeps `tailscale-exitnode` out), and the connector image is changing under it.
### Heavy templates
`heavy: true` marks a template whose install downloads several GB (large images, models, game files), asks for 4 GB of RAM or more (the catalog test enforces this) or takes many minutes to start: azuracast, gitlab-ce, the AI stacks, the Steam game servers. The gallery card shows a "Heavy" badge and the plan warns. **Heavy templates are never part of the integration sample and need a manual check**: deploy them yourself on a server with the room, or name them in `NEBULACTRL_TEST_TEMPLATES`.
Three static rules guard every template in `go test ./internal/templates/` (tier 1, no network): the catalog has at least 300 valid templates and uses every category; every one has a licence class (with `licenseUrl` for the restricted classes), an origin that points into the upstream catalog at a commit, a pinned `tag@digest` (the all-zero placeholder is rejected) and no leftover draft marker; and every template survives the snapshot encoding the update flow depends on (below).
## Versions and updates
A template has two version numbers. The **revision** (`revision:`) counts the template's own changes: bump it whenever installing the new text would produce something different (a new image tag, another health check, a changed variable, file or volume); metadata-only edits (summary, tags, category, icon, links) do not need one. The **app version** is not written by hand: it is the tag of the template's main image (the first image service that is not a one-shot job or a database), for example `3.4.1`, shown on the card.
An install records both (`template_instances.template_revision`, `template_version`) and a **snapshot** of the template as installed (`template_snapshot`, the template text with its `${input:KEY}` placeholders and generators, never an input value, so no secret). The catalog is embedded in the server binary, so after an upgrade the older revision exists nowhere else; the snapshot is what an update diffs against.
**Update available.** When the catalog's template differs from a stack's snapshot, the stack's group header on the canvas and the service's Settings tab show an update marker. It opens a diff (also `GET /template-instances/{id}/update` and `nebulactrl templates update <instance>`), with every change in one of four states:
| State | Meaning |
|---|---|
| `apply` | Will be staged: the service still has what the old revision installed |
| `kept` | Left alone because you changed the field on the service (image, a health check setting, a file you edited, a variable you changed or removed); a secret that exists is never overwritten |
| `manual` | Never done by an update: new or removed services, a database engine or version (use the database's Upgrade action), published ports and domains, removed variables, a changed volume mount path or a removed volume, and values built from an input whose value is stored only as a secret |
| `applied` | The service already has the new value (a retry after a partial update) |
**Stage update** (`POST /template-instances/{id}/update`, `--apply` in the CLI) writes the `apply` changes as ordinary staged edits through the normal service paths (service update with its validation, variables with generated secrets, volumes), so the staged bar shows them and **Deploy** applies them in the usual way; nothing deploys by itself, and rollback works as always. Inputs the new revision added are asked for in the dialog (required ones must be filled; generated ones are generated). The request carries the revision you reviewed: a newer one in the catalog fails with 409 `template_changed`. A failure part-way is safe to retry, because anything already applied shows as `applied`. The instance then records the new revision, version and snapshot (kept and manual items do not come back as updates), and an audit event `template.update` lists what was staged.
Installs made before updates existed have no snapshot: they show "newer revision available (reinstall)" and cannot be updated in place.
## What the tests deploy
Static validation covers the whole catalog on every run. `TestTemplatesDeploySample` (full tier) installs templates through the API, deploys them, probes the generated domain and removes everything again, and a run is bounded: a **rotation of 3 templates per day** (non-heavy, deterministic, every one is reached in a few months) plus **at most 8 of the templates changed against origin/main**, picked deterministically and rotating by day so a big change set is covered over several runs. Heavy templates are left out. The test logs which templates it deployed and which it skipped. Required inputs are filled with realistic values of their format (`ops@nebulactrl.dev`, `https://example.nebulactrl.test`, `host.nebulactrl.test`, `587`, a select's first option, a generated-style password). `NEBULACTRL_TEST_TEMPLATES` overrides the choice: a comma separated list of ids, or `all`.
## Tools
```
go run ./tools/template-pin templates/catalog # verify digests (no pulls)
go run ./tools/template-pin -update templates/catalog/x.yaml # resolve tag → digest, rewrite the file
go test ./internal/templates/ # schema + pinning + licence for the whole catalog
NEBULACTRL_TEMPLATE_REGISTRY_CHECK=1 go test -run Registry ./internal/templates/
./tests/run.sh full -run '^TestTemplatesDeploySample$' # deploy a bounded sample (below)
NEBULACTRL_TEST_TEMPLATES=umami,ghost ./tests/run.sh full -run '^TestTemplatesDeploySample$' # exactly these (heavy ones too)
NEBULACTRL_TEST_TEMPLATES=all ./tests/run.sh full -run '^TestTemplatesDeploySample$' # exhaustive sweep, hours
go run ./tools/template-translate … # one-time draft generator, see tools/template-translate/README
go run ./tools/template-translate -enrich <drafts> -enrich-out <dir> # tags, digests, licences, rejections (no pulls)
go run ./tools/template-translate -check templates/catalog/x.yaml # static validation of one file
go run ./tools/template-translate -resolve redis:7.4 # digest and platforms of an image (HEAD request)
```
## In the web app
- **Create → Templates** (or ⌘K → "New from template…", or the "Browse templates" button on an empty canvas) opens the gallery: search (every word must match the name, summary, tags, licence or database engine; names rank first), twelve category chips and three licence chips (open source, source-available, proprietary), each with a count that follows the search and the other filter, and a card per template (services, memory, database engines, licence, a TCP/UDP marker for game servers). A link `…/p/<project>/<env>?template=<id>` opens the gallery on that template.
- A card opens the plan step. The left side describes the template and lists the services it creates; the right side is the install form (secret inputs are password fields and say when they are generated if left empty), the server picker with its CPU, memory and disk headroom, and a live preflight from the plan endpoint that re-runs when the server or an input changes. **Add to canvas** stages the stack; **Add and deploy** also deploys it. Both stay disabled until a fresh preflight has no failed check. **Source-available** and **proprietary** templates carry a warning badge with the licence name and link.
- After the install the canvas takes the group and the card positions the control plane saved, and fits the new group into view.
- A group that is a template stack has **Update** while the catalog has a newer revision (it opens the diff above), **Deploy stack** (deploys what is staged in it; in production it opens the review sheet) and **Delete stack** (type the template id, optionally delete the volumes, hold to confirm) in its header. A service's Settings tab says which template it came from.
- Service Settings can edit what templates configure: an exec health check (`CMD` or `CMD-SHELL` first), **Files** (at most 16 of 64 KiB, stored in plain text: never put secrets there) and the **one-shot job** switch.
## CLI and MCP
```
nebulactrl templates ls [--category games] [--license source-available] [--search minecraft]
nebulactrl templates show n8n
nebulactrl templates plan umami --project acme --env production --server box1 [--input KEY=VALUE]
nebulactrl templates install umami --project acme --env production --server box1 [--deploy]
nebulactrl templates installed --project acme --env production # shows version, revision and UPDATE
nebulactrl templates update <instance-id> [--apply] [--input KEY=VALUE] # diff, or stage it
nebulactrl templates rm <instance-id> --confirm umami [--delete-volumes]
```
MCP (when `NEBULACTRL_MCP_ENABLED=true`): `list_templates` and `plan_template` (scope `projects:read`), `install_template` (`projects:write`; `deploy: true` also needs `deployments:write`, published ports need `proxy:write`). They call the same service methods as REST and are audited as `mcp`.
## Install semantics worth knowing
- **Everything is staged.** Installing never deploys unless `deploy: true`. The staged bar then shows N staged services; Deploy applies them in dependency order.
- **Secrets are generated once**, at install time, and stored encrypted. Inputs with generator defaults (for example an admin password) are generated once and reused wherever the template references them. The plan lists secret names only.
- **The stack is not special afterwards.** Editing, redeploying, rolling back or deleting one service uses the normal service paths. `Delete stack` removes every service of the install, its published ports and its canvas group.
- **Different servers are not supported**: all services of a template land on one server (private networking is per host).
- **Image digests** are pinned in the installed service (`repo:tag@sha256:…`), so a template revision never changes what an existing install runs; only staging an [update](#versions-and-updates) does, and only as a staged change you deploy.
- **Docker Hub rate limits** apply to the pulls of a stack (anonymous pulls are limited per IP); give the service registry credentials if a pull fails with `toomanyrequests`.
## API
`GET /api/v1/catalog/templates?q=&category=&license=`, `GET /api/v1/catalog/templates/{id}`, `POST /api/v1/environments/{id}/templates/{templateId}/plan`, `POST /api/v1/environments/{id}/templates/{templateId}/instances`, `GET /api/v1/environments/{id}/template-instances`, `GET /api/v1/template-instances/{id}/update` (diff), `POST /api/v1/template-instances/{id}/update` (`{revision, inputs}`, stages it; needs `projects:write`), `DELETE /api/v1/template-instances/{id}?confirm=<templateId>&deleteVolumes=<bool>`. Installing needs `projects:write` (plus `proxy:write` for templates that publish ports); `deploy: true` additionally needs `deployments:write`.
---
# Add a server
Source: https://nebulactrl.com/docs/installing/servers
## The first server
The [installer](https://nebulactrl.com/docs/installing/installer) enrolls the control-plane host itself: it installs the agent there and creates the server (flagged **control plane**), which is online by the time you finish setup. Its proxy serves the dashboard and your apps. Its **Accept workloads** setting decides whether services can be scheduled on it (on by default; see [the installer](https://nebulactrl.com/docs/installing/installer#accept-workloads)). The rest of this page is for *additional* servers.
## Additional servers
Each other Docker host that runs services needs the NebulaCtrl agent. The agent is a single static binary; the control plane serves it at `/agent/v1/download/{amd64|arm64}` and serves the installer at `/install-agent.sh`. Check the [requirements](https://nebulactrl.com/docs/installing/requirements) first: Debian 12/13, Ubuntu 22.04, 24.04 or 26.04, or RHEL/Rocky/AlmaLinux 9, Docker 24+ with the Compose plugin 2.24+, ports 80/443 free. The agent installer needs Docker to be installed already; it does not install it.
## 1. Create an enrollment token
In the UI: **Servers → Add server**. Enter a name and, optionally:
| Field | Used for |
|---|---|
| Public IP | DNS checks (A records are compared against it) and generated `*.sslip.io` domains |
| Base domain | Generated domains become `<service>-<env>-<project>.<base-domain>` |
| ACME email | Contact for Let's Encrypt certificates issued by this host's Traefik |
The dialog shows the install command. The token (`nct_enroll_…`) is **single use, expires after one hour and can be revoked**; only its hash is stored.
## 2. Run the installer on the target host
```sh
curl -fsSL https://nebula.example.com/install-agent.sh | sudo sh -s -- \
--url https://nebula.example.com --token nct_enroll_XXXXXXXX
```
The script:
1. checks that it runs as root, on Linux with systemd, with `curl`/`wget`, and that Docker is installed, reachable and **24 or newer**; it warns if port 80/443 is already in use;
2. downloads the right binary for `uname -m` to `/usr/local/bin/nebulactrl-agent`;
3. writes `/etc/systemd/system/nebulactrl-agent.service` (reference copy: [`deploy/agent/nebulactrl-agent.service`]()) and, on first install, `/etc/nebulactrl-agent.env` (mode 0600) containing the one-time token;
4. enables and starts the service and fails if it does not stay running.
Options: `--proxy-http-port N` and `--proxy-https-port N` move Traefik's public listeners; `--proxy-admin-port N` moves its admin port (default 9000, bound to 127.0.0.1 only); `--network-pool CIDR` and `--network-prefix-len N` set the address range environment networks are allocated from (default `10.213.0.0/16` in /24s; see [Networking](https://nebulactrl.com/docs/operations/networking#address-pool)); `--insecure-dev` allows a plain-`http://` control plane (development only). Re-running the installer upgrades the binary and keeps the identity (no token needed).
On first start the agent generates an Ed25519 keypair, redeems the token, stores its identity in `/var/lib/nebulactrl-agent/agent.db` (mode 0600), starts its **Traefik proxy container** (`nebulactrl-traefik`, network `nebulactrl-edge`) and connects. The server turns **online** within seconds, with its Docker and agent versions, CPU/memory/disk and proxy status.
## 3. Verify
- UI: the server card shows *online*, a recent "last seen", and **Proxy: running**. A **Proxy error** (typically port 80 in use) is shown with its reason; fix it and restart the agent (`systemctl restart nebulactrl-agent`).
- Host: `systemctl status nebulactrl-agent`, `journalctl -u nebulactrl-agent -f`, `docker ps --filter name=nebulactrl-traefik`.
- Firewall: allow inbound **80/tcp, 443/tcp, 443/udp** from the internet to this host for public services. **No inbound port is needed for the control plane**; the agent only needs outbound HTTPS to it.
## What lives on the host
| Item | Where |
|---|---|
| Agent binary, unit, token file | `/usr/local/bin/nebulactrl-agent`, `/etc/systemd/system/nebulactrl-agent.service`, `/etc/nebulactrl-agent.env` |
| Identity and in-flight command state | `/var/lib/nebulactrl-agent/agent.db` |
| Git build workspaces | `/var/lib/nebulactrl-agent/builds/<deploymentId>` (removed after each build) |
| Proxy | container `nebulactrl-traefik`, network `nebulactrl-edge`, volume `nebulactrl-traefik-acme` (ACME certificates in `acme.json`: keep it) |
| Your services | containers named `nc-<service>-<deploymentId8>-<replica>`, labelled `nebulactrl.managed=true`; volumes `nc-vol-<serviceId8>-<name>`; networks `nebulactrl-env-<environmentId8>`, `nebulactrl-env-<environmentId8>-int` and `nebulactrl-shared-<id8>` |
## Revoking and re-enrolling
See [operations: servers](https://nebulactrl.com/docs/operations/servers). Short version: **Servers → server → Revoke** closes the agent's connection immediately and the agent exits with "agent revoked; re-enroll". To re-enroll that host, run `install-agent.sh --uninstall --purge`, create a new enrollment token and run the installer again.
## Several agents
Add one server per Docker host. Each service is placed on one explicitly chosen server. Hosts do **not** share a private network: services on different hosts cannot reach each other by service name ([limitations](https://nebulactrl.com/docs/operations/limitations)).
---
# Servers and agents
Source: https://nebulactrl.com/docs/operations/servers
## Server states
| State | Meaning |
|---|---|
| `pending` | An enrollment token exists; no agent has redeemed it yet |
| `online` | A heartbeat arrived within the last 30 s |
| `offline` | No heartbeat for 30 s (agent down, host down, or network cut). Marked by a sweep that runs every 10 s. |
| `revoked` | The identity was revoked; the agent cannot reconnect |
Every enrolled agent holds its own Ed25519 identity. A connection is authenticated by signing a fresh server nonce; nonces older than 30 s are rejected.
## What happens when a server goes offline
- **Services keep running.** The agent and Traefik are independent of the control plane connection; the proxy continues to serve.
- **New commands** (deploys, restarts, …) are stored first, then sent. If the agent never received a command, it expires after **15 minutes**: the deployment becomes `failed` with "Not delivered: server offline. No changes were made."
- **Commands the agent already acknowledged never time out into a terminal state**, because the true outcome is unknown while it is offline. The deployment stays active and the canvas shows "Waiting for <server>" with the offline time. When the agent reconnects it replays the result and the deployment settles. If you know the host is gone, revoke the server to cancel what is pending.
- Reads (logs, current status) show stale data marked as stale.
## On reconnect: reconcile
When an agent connects, the control plane sends it the desired set of active deployments. The agent re-applies its route table, reports what is actually running (`observed`), and removes managed containers that belong to no desired deployment and no in-flight command **and are older than 10 minutes** (manually started containers without the `nebulactrl.managed=true` label are never touched).
If an active deployment's container is **missing** (for example someone ran `docker rm`), the control plane redeploys the same release (`trigger=reconcile`), **at most 3 times per service per hour**. A fourth miss records the audit event `reconcile.limit_reached` and the node shows an error; fix the cause (out-of-memory kills, disk full, a manual removal) and deploy again.
## Build servers
A server can be a **build server**: **Settings → Role** chooses *Deployments* (the default), *Builds* (no workloads, builds only) or *Both*, with a cap on concurrent builds and switches for emulation and pull-request previews. A build server builds images, pushes them to a registry and never needs the images afterwards; the servers that run services pull them by digest. Everything about roles, where builds run, registries, rollback and troubleshooting is in [Build servers and registries](https://nebulactrl.com/docs/operations/build-servers).
What changes for the rest of this page:
- **Offline build server.** If it is offline when a deployment is dispatched, the build falls back to the target and the timeline says so; a build command it never received expires after 15 minutes and the deployment then builds on the target. If it goes offline **after** it acknowledged a build, the deployment waits (the canvas shows "Waiting for <name>") and never times out, exactly like any acknowledged command above; it settles when the agent reconnects and replays the result.
- **Revoking a build server** cancels its pending commands. If a deployment is waiting on a build there, cancel it and deploy again (placement then picks another build server or builds on the target). **Already built releases are unaffected**: their images live in the registry and roll back without it.
- **A server whose role is *Builds*** does not accept workloads, so it is not offered for services, drains or templates ([draining servers](https://nebulactrl.com/docs/operations/draining-servers)).
- **Agents** must be current to build (`build.push` capability); older agents are never chosen.
## Revoke
**Servers → server → Revoke** (hold to confirm) or `POST /api/v1/servers/{serverId}/revoke` (scope `servers:write`; CLI: `nebulactrl servers revoke <id>`).
- The agent's socket is closed with status 4401 and the agent exits with "agent revoked; re-enroll".
- Pending commands for that server are cancelled.
- The containers on the host **keep running** until you remove them (see [uninstall](https://nebulactrl.com/docs/installing/uninstall)).
- Revocation is recorded in the audit log.
Revoke when a host is retired, lost, or suspected compromised. A compromised agent host is a compromised root account ([threat model](https://nebulactrl.com/docs/security/threat-model)); revoking only cuts its access to the control plane, so also rotate every secret that was deployed to it.
## Agent versions
Each server card shows when its agent is behind the control plane. Agents upgrade themselves (automatically, or with **Upgrade agents**), verify the binary and roll back if the new one does not start: [Upgrades](https://nebulactrl.com/docs/operations/upgrades).
## Re-enroll a host
1. On the host: `curl -fsSL <url>/install-agent.sh | sudo sh -s -- --uninstall --purge` (deletes the old identity).
2. In the UI create a **new** server (**Add server**) or token; tokens are single use.
3. Run the new install command on the host.
The new server gets a new ID. Services are placed on a specific server, so move them by changing **Placement** in each service's settings and deploying; this creates a new deployment on the target, and **volumes are not migrated**. To empty a whole server in one operation, see [Draining servers](https://nebulactrl.com/docs/operations/draining-servers).
If an agent was merely moved to new hardware, copy `/var/lib/nebulactrl-agent/agent.db` (mode 0600) to the same path before starting it and it keeps its identity.
## Lost the agent database
If `agent.db` is lost on a still-valid server, the agent has no identity and cannot connect. Revoke the server in the UI, then re-enroll as above.
## Moving the control plane
Agents dial `NEBULACTRL_PUBLIC_URL`. To change it, update `.env`, then re-run the installer on each host with the new `--url` (it rewrites the unit and restarts the agent; the identity is kept).
---
# Build servers and registries
Source: https://nebulactrl.com/docs/operations/build-servers
By default a git service is built on the server that runs it ([builds](https://nebulactrl.com/docs/operations/builds), [ADR 0004](https://nebulactrl.com/docs/decisions/0004-build-on-target-host)): nothing to set up, no registry. A **build server** is an optional second path: one enrolled server builds the image, **pushes it to an OCI registry**, and the server that runs the service **pulls it by digest**. Use it when builds should not compete with production for CPU and disk, when untrusted Dockerfiles should not run next to production data, or when a release must be deployable on any server without a rebuild. The design and the alternatives are in [ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers).
Nothing changes until something resolves to **Build server** (below). Services that use an image, Compose stacks and databases never build, so they are unaffected.
## How a deployment works
1. The control plane resolves the commit, picks a build server and a registry, and writes the release.
2. The build server clones the pinned commit, builds one image for the **target's architecture** with BuildKit, pushes it as `<registry>/<namespace>/<project>-<service>:r-<release8>`, and returns the manifest digest.
3. The control plane records the image (`registryRef` on the release, a row in the registry's image list) and sends the normal deploy command to the target, which pulls `repo@sha256:…` and runs the usual steps (health check, routing, retire the old container).
The deployment's timeline shows both servers (**Built on X, deployed on Y**) and a **push** step. The build log is the build server's output; the pull and deploy lines come from the target. Build, push and pull all count as the **building** status.
What goes where:
| Data | Build server | Target |
|---|---|---|
| Git token or SSH deploy key, build variables | yes | no (it never clones) |
| Registry **push** credential | yes | no |
| Registry **pull** credential | no | yes |
| Runtime variables and secrets | **never** | yes |
Secrets are attached to the command when it is sent and are never stored in the command row ([threat model](https://nebulactrl.com/docs/security/threat-model)).
## Server roles
Open the server's page, **Settings → Role**:
| Role | Runs workloads | Builds | Flags |
|---|---|---|---|
| **Deployments** (default) | yes | no | `accept_workloads` on, `build_enabled` off |
| **Builds** | no | yes | `accept_workloads` off, `build_enabled` on |
| **Both** | yes | yes | both on |
`accept_workloads` keeps its meaning (placement of services, drains, templates, platform services); build eligibility never reads it. A **Builds** server therefore cannot be chosen for a service, a drain or a template, and a **Both** server builds on the same Docker daemon that runs its workloads.
More settings on a build server:
| Setting | Default | Meaning |
|---|---|---|
| **Max concurrent builds** | 2 | 1 to 16. The cap is per build server; a build over the cap **queues on the control plane** and the deployment stays *preparing* with "Waiting for a build slot on <server> (N builds running)". |
| **Allow emulation** | off | Allow builds for another CPU architecture ([below](#cross-architecture-builds-and-emulation)). |
| **Allow previews** | off | Allow [pull-request previews](https://nebulactrl.com/docs/operations/previews) to build here ([below](#per-project-builders-and-preview-isolation)). |
Changing the role is audited (`server.build_role`). Use a dedicated host for **Builds**: the BuildKit builder is privileged, so a compromised build host exposes the git credentials, build variables and push credentials of every project that builds there. Give it the agent and Docker only, and revoke it like any server if it is suspect ([servers](https://nebulactrl.com/docs/operations/servers#revoke)).
A server needs a current agent to be a build server (the `build.push` capability). An older agent is never chosen; the deployment falls back to the target.
## Where builds run
Placement is resolved **service → project → team**. The first level that is not *inherit* wins; the team default is **Target**.
| Level | Where to set it | Choices |
|---|---|---|
| Team | **Settings → Builds** (`GET/PUT /team/build-settings`) | Target or Build server, plus the default registry |
| Project | **Project → Settings → Build location** | Inherit, Target or Build server; an optional pinned build server; an optional registry |
| Service | **Service → Settings → Build** | Inherit, Target or Build server; an optional pinned build server; an optional registry |
The pinned server and the registry follow the same order (service, then project). The CLI sets the service override with `--build-location` and `--build-server` on `nebulactrl services create` and `services update`.
The deploy **plan** shows the result before you deploy: the build server, the registry, the platform and every warning (fallback, emulation, a cold cache).
**Choosing the server.** A pin always wins. Otherwise NebulaCtrl hashes the service over the **eligible** servers, so the same service keeps landing on the same build server and its BuildKit cache stays warm. A server is eligible when it is `build_enabled`, online and not revoked, has the `build.push` capability, matches the target's architecture (or may emulate it), is allowed to build previews if this is a preview, and the registry exists and is healthy. A server at its concurrency cap is still eligible: the build queues.
**Reusing a build.** If a live image already exists for the same commit, build configuration, build variables (compared by a keyed hash, never stored), builder version and platform, the build is skipped: the deployment's timeline says "Reusing the image already built for commit <sha7> (no build needed)" and it deploys that digest. This is also why a redeploy that only changes a runtime variable is fast.
**Rebuild commit** (offered when a rollback target is gone) uses the **current** effective placement, not the one the old release was built with.
## Registries
A registry is a team object (**Settings → Registries**). A build server needs one; there are two kinds.
### Bring your own registry
Any OCI/Docker v2 registry with **basic authentication**. Fields:
| Field | Meaning |
|---|---|
| **Host** | `ghcr.io`, `registry.example.com:5000`. No scheme. |
| **Namespace** | The path under the host, possibly empty. The repository is `<host>/<namespace>/<project>-<service>`. |
| **Push credential** (required) | Username and password or token that may create repositories and push. Write-only: never returned, sealed with the instance key. |
| **Pull credential** (optional) | A read-only credential for the **target** servers. Without one, the push credential is used for pulls too. Use it. |
**Test registry** (per server, `POST /registries/{id}/check?serverId=`) asks that server to reach the registry and try a push and a pull with the stored credentials, and shows the registry's own error. Run it for the build server **and** for each target before the first deploy. The same check runs automatically at the start of every build, before minutes are spent building.
Recipes (credentials and names are the provider's; check its current console):
**GitHub Container Registry (GHCR)**
- Host `ghcr.io`, namespace your user or organisation **in lowercase**.
- GHCR accepts **classic personal access tokens only**. Push: a token with `write:packages`. Pull: a second token with `read:packages`. The username is your GitHub login.
- A classic token is account-wide, not limited to one repository; create a dedicated bot account for it if that matters. New packages are private by default and linked to the account, not to a repository, so set the package access for the organisation if other members need it.
**Docker Hub**
- Host `docker.io`, namespace your Docker Hub user or organisation.
- Create an **access token** (push: Read & Write; pull: Read-only) and use your Docker Hub username with it.
- NebulaCtrl creates one repository per service (`<project>-<service>`); check your plan's private-repository limit and the pull rate limits.
**GitLab**
- Host `registry.gitlab.com` (or your instance's registry host), namespace `<group>/<project>`: the images live under that GitLab project.
- Push: a **deploy token** with `write_registry` (or a personal or project access token with `write_registry`). Pull: a deploy token with `read_registry`. The username is the deploy token's username.
**Harbor**
- Host your Harbor hostname, namespace the **Harbor project name**. The Harbor project must exist first: Harbor does not create projects on push.
- Create a **robot account** for the project: push and pull for the push credential, pull only for the pull credential. Use the robot's full name (it contains a `$`) as the username.
Other self-run registries (`registry:3`, Gitea, Forgejo, a cloud provider's registry with a static credential) work the same way. A registry on plain HTTP is refused unless it is on a loopback address, because Docker daemons only accept plain HTTP for loopback (the integration test uses one).
**Not supported:** AWS ECR, Google Artifact Registry and Azure ACR hand out tokens that expire within hours; a stored password does not work with them and NebulaCtrl does not refresh tokens ([limits](#limits)).
### Set up for me (managed registry)
The registry is a [platform service](https://nebulactrl.com/docs/operations/platform-services) of kind `registry`: CNCF Distribution `registry:3.1.2`, pinned by digest and run unmodified, on a server you choose, with a data volume and a public hostname. Setting it up creates the registry object for you.
- **Needs** a public hostname, port 443 on that server's proxy and the server's ACME email: Docker daemons require TLS for any registry that is not on loopback, so the certificate has to be real (a `.localhost` name is the development exception). Put it on the **build server** (the push stays local and only pulls cross the network) or on the control-plane server.
- **Credentials are short-lived and scoped.** The control plane is the registry's token server (`/registry/token`, Docker token authentication). Every build and every pull gets an expiring credential limited to **one repository and the actions needed** (15 minutes by default, longer for a build), which the registry exchanges for a token valid at most 15 minutes. No long-lived registry password exists on any host, and a revoked server cannot get a new one.
- **Large uploads.** The proxy's default 60 second request-read timeout can cut a big layer upload over a slow link, so the registry's domain has **large uploads** on, which raises it to 30 minutes on that server (the proxy container is recreated once).
- **Retention is managed** (below).
- **Back it up?** The images can be rebuilt from their commits, so it is optional; if you want a copy, back up its volume like any service ([volume backups](https://nebulactrl.com/docs/operations/backups)).
## Cross-architecture builds and emulation
A deployment has exactly one target server and so one CPU architecture. v1 builds **one single-platform image** for that architecture (for example `linux/arm64`). There are no multi-arch manifests.
- A build server of the **same architecture** is always eligible.
- A build server of **another architecture** is eligible only if **Allow emulation** is on for it **and** it reports that platform in its facts (`facts.emulation`, detected from the kernel's `binfmt_misc` QEMU handlers). Otherwise it is skipped, and with no eligible server the deployment [falls back to the target](#fallback-to-the-target).
- Emulation is **much slower** than native builds, especially for compilation. Railpack builds one platform per build and has no cross-compile shortcut, so every install and build step runs emulated.
- To enable it on an amd64 build host: `docker run --privileged --rm tonistiigi/binfmt --install all`, then restart the agent so it reports the platform.
- A faster alternative: build natively on a server of the target's architecture, or cross-compile in your Dockerfile with `FROM --platform=$BUILDPLATFORM …` and `TARGETPLATFORM`/`TARGETARCH`, so the slow part runs natively even on the amd64 builder.
- Placement only considers build servers whose architecture matches the target (or that may emulate it), so an image for the wrong architecture is not built in the first place. A release records the platform it was built for (`platform` in its source); do not move a release's service to a server of the other architecture without rebuilding the commit, or the container fails with `exec format error`.
## Per-project builders and preview isolation
A build server runs **one BuildKit builder per project**, named `nebulactrl-buildkit-<project8>`, each with its own cache. Cache mounts and layer caches are therefore not shared between projects, so one project's build cannot read or poison another's cache. It costs disk: the same base layers are cached once per project. Garbage-collection limits are set on each builder, and an agent-level daily prune stays as a backstop ([Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup#build-servers)). Remove a builder and its cache with `docker buildx rm -f nebulactrl-buildkit-<project8>`; the next build recreates it.
A per-project builder isolates **caches**, not the kernel: the builder is privileged and a container escape on the build host reaches every project that builds there. That is inside the [trusted-team boundary](https://nebulactrl.com/docs/security/threat-model); do not put mutually untrusting teams' projects on one build server.
**Pull-request previews never build on a shared build server.** They build on their target, as before, unless a build server has **Allow previews** on. Fork pull-request code is untrusted ([previews](https://nebulactrl.com/docs/operations/previews#fork-pull-requests)); keep **Allow previews** off on any build server that holds credentials you care about, or dedicate one build server to previews.
## Fallback to the target
A deployment never blocks because there is no build server. When nothing usable exists, the build runs on the **target**, exactly as without this feature, the timeline shows **"No build server available; building on <target> (<reason>)"** (a *placement* event), and the deploy plan carries the same warning. This happens when:
- no server is `build_enabled`, online, and capable (`build.push`);
- the architecture does not match and emulation is off or not reported;
- the build server was **offline when the deployment was dispatched** (nothing had been sent yet);
- no registry is configured, or a managed registry is not healthy;
- the deployment is a preview and no build server allows previews.
Fallback is decided **before** the build command is sent. A build server that goes offline **after** it acknowledged the command is a different case: the true outcome is unknown, so the deployment waits ("Waiting for build server <name>", with the time it went offline) and never times out into a terminal state ([servers](https://nebulactrl.com/docs/operations/servers#what-happens-when-a-server-goes-offline)). When it reconnects it replays the result. Cancel the deployment, or deploy again after switching the service to **Target**, if the host is gone.
## Rollback and retention
**A rollback of a registry release needs no build server.** It is an ordinary deploy of `repo@digest` on the **current** target: the build server can be offline, removed or revoked, and the target does not need the image locally. It does need the **registry up** and the digest **still present** in it. The target also keeps its newest `imageRetention` images locally (default 5), so a rollback inside that window works without the registry too.
If the digest is gone, the rollback does not start the new release and the running one keeps serving: for a **managed** registry NebulaCtrl knows beforehand from its image list and refuses with 409 `image_pruned`; for a **BYO** registry it finds out when the target's pull fails (the deployment fails at the pull step, "Previous release is still serving", with the registry's own message). Either way, **Rebuild commit <sha7>** deploys the commit again.
**What is kept.** For every service the **live set** is the current release, the releases of deployments in flight, and the newest `imageRetention` succeeded releases (the ones **Roll back to this release** can pick). Each image NebulaCtrl pushes is recorded with its own repository and digest in the registry's image list (**Settings → Registries → Images**, `GET /registries/{id}/images`), whose state is `live`, `expired` or `deleted`.
- **Managed registry:** an **hourly sweep** deletes the digests of images already marked `expired` (those outside the live set) through the registry API, one explicit digest at a time; it never touches a `live` image or a digest it did not record, so a push whose result was not yet stored is safe. Blobs nothing references are then reclaimed by `registry garbage-collect` (never with `--delete-untagged`) in a **weekly window** that starts between 02:00 and 05:00 UTC: builds for this registry queue, running pushes get up to 2 hours to finish, the registry is set read-only (a short redeploy), the collection runs, and the registry returns to read-write. Pulls work during the window; pushes get `405`.
- **BYO registry:** NebulaCtrl **never deletes** anything and does not mark anything expired: its images stay `live` in the list for good, and the registry keeps growing until you set a lifecycle rule.
**What to keep in a BYO registry**
- **Do not enable "delete untagged" rules.** A digest-pinned rollback target can look untagged to a registry that only counts tags.
- Keep **at least `imageRetention` recent tags per repository, plus a margin**: every release has a unique, never-moved tag `r-<release8>`, so rules that keep "the newest N tags" keep the newest releases. Ten is a comfortable number for the default of 5.
- Never delete the digest of the **current** release: a reconcile redeploy, a drain or a move to another server pulls it again.
- Delete by age only if the age exceeds how long you want rollbacks to work.
Lifecycle rules that fit:
| Registry | Rule |
|---|---|
| GitLab | **Cleanup policy**: keep the most recent **10** tags per image name, remove tags older than 90 days, name regex `r-.*`, and leave **untagged** removal off. |
| Harbor | **Tag retention**: retain the most recent **10** pushed artifacts per repository, for the project. Do not add an untagged-artifact purge. |
| GHCR | Keep the newest 10 versions of each package, with a scheduled workflow that deletes older **tagged** versions only (for example the `actions/delete-package-versions` action with `min-versions-to-keep: 10`), and **never** `delete-only-untagged-versions`. |
| Docker Hub | No tag-count rule for tagged images; prune by hand if needed. |
**Local images.** A build server pushes with `--push` and nothing is loaded into its Docker image store, so there is no image keep-list on a build-only server; only BuildKit cache, the pinned BuildKit and Railpack images and clone workdirs remain ([Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup#build-servers)). A target keeps the images of registry releases in its keep list ([Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup#what-it-never-removes)).
## Troubleshooting
| Situation | What happens | What to do |
|---|---|---|
| Build server **offline at dispatch** | Nothing is sent; the deployment [falls back to the target](#fallback-to-the-target) and says so. If the server drops off right after dispatch and never received the build command, it expires after 15 minutes and the deployment then builds on the target ("Build server <name> did not take the build; building on <target>"). | Bring it back, or set the service to **Target**. |
| Build server **offline after it acknowledged** | The deployment waits; it never times out ("Waiting for build server <name>"). When it reconnects it replays the result (a repeated build is cheap thanks to the cache). | Wait, or cancel and deploy again; to move on, switch to **Target**. |
| **Registry unreachable** from the build server | Fails in the pre-build check, before the build, with `cannot reach <host>: …` and the underlying error. During a push BuildKit retries; the final failure is `failed` at the **push** step. The target is untouched. | Run **Test registry** for that server; check DNS, the firewall, the certificate and the registry's own status. |
| **Push authorization failed** (`401`, `403`, `denied`, `unauthorized`) | The build fails with the registry's own message; no retry. | Replace the push credential; GHCR needs `write:packages`; Harbor needs the project to exist; a deploy token needs `write_registry`. Then **Test registry**. |
| **Pull from the target fails** (auth, unreachable) | The pull happens before the old release is touched: the deployment is `failed` with "Previous release is still serving". | Fix the pull credential or the network, then **Redeploy**: the digest exists, so nothing is rebuilt. |
| **Digest missing** from the registry | Managed: rollback refused with 409 `image_pruned`. BYO: the pull fails and the running release keeps serving. | **Rebuild commit <sha7>**; review your lifecycle rule. |
| **Wrong architecture** | Placement skips build servers of another architecture unless emulation is on and reported; with none left the build [falls back to the target](#fallback-to-the-target). | Use a build server of the target's architecture, or enable emulation, or build on the target. |
| **Managed registry read-only** (garbage-collection window) | Builds for the registry queue until the window ends; pushes get `405` meanwhile. | Retry in a few minutes. |
| **Build server disk full** | A preflight refuses to start with too little free space, runs an emergency cache prune and retries once; otherwise `failed` with BuildKit's error. | Free space or raise the build server's disk; see [Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup). |
| **Build timeout** | Fails after 30 minutes, as for target builds. | Fix the slow step; for emulated builds consider a native builder. |
| **Too many builds** | Each build server runs at most **Max concurrent builds**; the rest queue, per server, not globally ("Waiting for a build slot on <server> (N builds running)"). | Raise the cap if the host can take it, or add a build server. |
| **Control plane restarted mid-build** | Commands are durable. The build continues on the agent; its result is delivered when the agent reconnects. | Nothing. |
| Same tag pushed again, target runs old code | Cannot happen: targets deploy the **digest**, never the tag. | |
| `blob unknown to registry` / `denied` on push to a third-party registry | The registry's own message is shown verbatim in the build log with the server's name. | Run **Test registry**; check quotas and the repository's visibility settings with the provider. |
The build log of a registry deployment lists both servers' steps. Use **Deployments → the deployment → Logs** and read the `build` and `push` lines for the build server and the `pull` and `deploy` lines for the target.
## Limits
- **One platform per build, no multi-arch manifests.** A release runs on servers of one architecture; moving it across architectures means a rebuild. The extension path is native builds on two build servers joined with `docker buildx imagetools create`.
- **No ECR, Google Artifact Registry or Azure ACR token refresh.** Only registries with a static basic credential work. A long-lived access key through a registry that supports static credentials works; tokens that expire in hours do not.
- **No registry-free transport.** There is no `docker save | load` relay through the control plane; a build server needs a registry that both it and the targets can reach. Without a registry, build on the target.
- **BYO registries are never cleaned up** by NebulaCtrl, and a lifecycle rule can delete a rollback target.
- **GHCR credentials are account-wide** (classic tokens only); other registries scope as finely as their credentials allow.
- **The build server is a shared privileged host** for every project of the team; per-project builders isolate caches only ([threat model](https://nebulactrl.com/docs/security/threat-model)).
- **The managed registry needs a public hostname with a real certificate**, and is one per team like every platform service ([platform services](https://nebulactrl.com/docs/operations/platform-services#limits)).
- Provenance and SBOM attestations are off, so the registry holds a single plain manifest per release.
- Stacks, image services, databases and platform services do not use build servers.
- Plain-HTTP registries are accepted on loopback addresses only.
- Registry cache export (`--cache-to type=registry`) is not offered; the persistent per-project builder is what keeps builds fast.
## API and audit
| | |
|---|---|
| `GET/PUT /team/build-settings` | team default location and registry |
| `GET /registries`, `GET /registries/{id}`, `GET /registries/{id}/images` | registries and their pushed images (`projects:read`); credentials are never returned |
| `POST /registries`, `PATCH /registries/{id}`, `DELETE /registries/{id}` | external registries (`admin`): `name`, `host`, `namespace`, `pushUsername`, `pushPassword`, optional `pullUsername`, `pullPassword`. Deleting is refused with `409 registry_in_use` while the team's build settings, a project or a service references it, and `409 managed_registry` for a managed one (remove that under platform services). |
| `POST /registries/{id}/check?serverId=` | test a registry from a server (`admin`); waits up to 25 s for the agent and answers `{ok, push, pull, error}`; `409 server_offline` if the agent is not connected |
| `PATCH /servers/{id}` | `build_enabled`, `build_max_concurrent`, `build_allow_emulation`, `build_allow_previews`, `accept_workloads` |
Audit actions: `registry.create`, `registry.update`, `registry.delete`, `registry.check`, `server.build_role`, and `deployment.build_server` (which build server built a release). The deployment resource carries the build server's id next to the target's.
---
# Draining servers and the fleet
Source: https://nebulactrl.com/docs/operations/draining-servers
## When to drain
Drain a server to empty it before maintenance (kernel update, reboot, disk work) or before retiring a VPS. A drain moves **every** service placed on the server to other servers, one at a time, and closes the server to new placements.
Draining is not required to revoke a server, but revoking leaves the containers running and the services still placed on it ([Servers and agents](https://nebulactrl.com/docs/operations/servers#revoke)). Drain first, then revoke.
**Volumes are not copied.** A service with volumes, and any service of kind `database`, starts with empty volumes on its target. Move the data yourself before you drain (see [Stateful services](#stateful-services)).
In the UI: open the server's detail page and use the drain sheet (the final button is hold-to-confirm: "Hold to drain <server>"). The same flow is available over the API.
## Plan (dry run)
`POST /api/v1/servers/{serverId}/drain/plan` changes nothing. It returns one move per service on the server, plus a capacity verdict. The body is optional: `{"targetServerId": "<id>"}` sends every service to that server; omitting it picks a target per service.
Move order: stateless services first, services with volumes or of kind `database` last. A failure stops the drain, so data-bearing services are touched last.
### Automatic target
Without `targetServerId`, each service goes to the eligible server with the most free memory. A server is eligible when it is online, not the source, not revoked, and its drain state is neither `draining` nor `drained`.
Free memory (headroom) of a candidate:
```
total memory − max(memory used, committed limits) − memory already assigned by this plan
```
- *Committed limits* is the sum of the memory limits (`memoryMb`) of the services already placed on the candidate.
- *Assigned* is what earlier moves of the same plan already booked on it, so one big server does not receive everything unless it still has the most room.
- A service's demand is its memory limit; with no limit, its latest usage sample (at most 2 minutes old); with neither, nothing is booked.
- Ties are broken by server name, then ID.
- A server that has not reported its memory ranks below every server that has. If it is the only candidate, the plan still picks it and warns that capacity is unknown.
With an explicit target the server must exist, differ from the source, not be revoked, and not be draining or drained (`422 server_draining`).
### What the plan reports
Per move: `targetServerId`/`targetServerName`, `stateful`, `warnings[]`, `blocking[]`. Overall: `capacity.targetFits` and `capacity.reasons[]`.
Blocking problems (the drain cannot start):
| Code | Cause |
|---|---|
| `no_target` | No online, schedulable server can take the service |
| `target_offline` | The chosen target is offline |
Capacity: `targetFits` is `false` when a target would exceed its memory after the assigned moves; `reasons` say by how much. If a target has not reported its memory the check is skipped for it and a reason says so. The start endpoint does not re-check `targetFits`; read it before you start.
Warnings the plan can emit:
- the target was chosen automatically (and how much memory remains free, or that capacity is unknown);
- volumes are not copied / a database starts empty on the target;
- the service has never been deployed: only its placement changes, nothing is started;
- the service has staged changes: they are deployed together with the move;
- the service has no memory limit (and possibly no usage data): the capacity check approximates or cannot account for it;
- a deployment of the service is in progress: the drain waits for it to finish;
- the service has domains and the target has a different public IP: point the A records to the target after the move;
- the source server is not online: the old container cannot be stopped until its agent reconnects.
## Stateful services
A service is *stateful* when it has at least one volume or its kind is `database`. On the target it starts with empty volumes.
1. Back up or copy the data to the target host yourself.
2. Start the drain with `"confirmStateful": true`. Without it the API answers `422 confirmation_required` naming the stateful services. In the UI this is the checkbox "I have moved the data, or I accept losing it."
Nothing in the drain code copies or deletes volume data.
## Starting the drain
`POST /api/v1/servers/{serverId}/drain` with:
```json
{
"moves": [{"serviceId": "…", "targetServerId": "…"}],
"confirmStateful": true
}
```
Rules enforced on start:
- Every service on the server appears in `moves` exactly once (`422 incomplete_moves` lists the missing ones; `422 invalid_moves` for duplicates or services not on this server).
- Every target passes the checks above and is **online** (`422 target_offline` otherwise). Plan blockers (`no_target`) are returned as errors.
- `409 drain_in_progress` if the server is already draining.
- `Idempotency-Key` is honored.
The call returns `202`. In one transaction the server's drain state becomes `draining`, a drain run is stored, a background job is enqueued, and the audit event `server.drain.start` is written (including the plan and `confirmStateful`).
## How the moves run
Moves run **sequentially**, in plan order. Per service:
1. If a deployment of the service is active, wait (3 s polls) until it ends.
2. Change the service's placement to the target. This is a normal service update, so it increments the service generation and is audited under the initiator.
3. If the service was never deployed, it is marked done ("Not deployed yet: only the placement changed.").
4. Otherwise create a deployment with trigger **`drain`** for the release content that is running now (a git service is pinned to its current commit SHA; a drain never ships new code). Staged config and variable changes are deployed with it.
5. Wait until that deployment ends. `succeeded` marks the move done. Per the endpoint description, the old container is removed once the new one is live.
A service that was deleted during the drain is skipped as done. A drain job interrupted by a control-plane restart resumes from the stored progress and adopts an already running `drain` deployment instead of starting a second one.
The job acts as the initiator with `projects:read`, `projects:write` and `deployments:write`, so deployments and audit entries are attributed to them.
When every move is done, the run is `completed`, the server's drain state becomes `drained`, and `server.drain.complete` is audited.
Follow progress with `GET /api/v1/servers/{serverId}/drain` (the running drain, else the newest finished one, else `{"drain": null}`). Each move has the status `pending`, `moving`, `deploying`, `done` or `failed`, and `deploymentStatus` live from its deployment.
## Failure semantics
The first move that fails **stops the drain**:
- Failure means the placement update was rejected, or the `drain` deployment ended in anything but `succeeded` (`failed`, `rolled_back`, `rollback_failed`, `canceled`, or the deployment disappeared).
- That service's placement is **reverted** to the source server, so the placement matches what is actually running. If its placement or generation was changed meanwhile, the revert is skipped and the move's message says "the service's placement was changed meanwhile and was not reverted".
- Services not yet started are **untouched** and keep running on the source. Services already moved stay on their targets; nothing is moved back.
- The run is `failed`; the server's drain state is `failed`, with `drainMessage` = `Moving <service> to <server> failed: <reason>`. `server.drain.fail` is audited.
In state `failed` the server accepts new placements again (the scheduling guard applies only to `draining` and `drained`). Fix the cause, then either **Retry (re-plan)** in the UI, which plans the services that are left, or undrain.
## Cancel
`POST /api/v1/servers/{serverId}/drain/cancel` asks the drain to stop **after the move in flight**. A move that is moving or deploying is finished first; no further move starts. Nothing is rolled back.
| Moves done at that point | Result |
|---|---|
| 0 | Run `canceled`; server drain state returns to `none` |
| N ≥ 1 of M | Run `canceled`; server drain state `failed` with message **"Canceled after N of M moves"** |
`409 no_drain_in_progress` when no drain is running. `server.drain.cancel` is audited.
## Undrain
`POST /api/v1/servers/{serverId}/undrain` sets a `drained` or `failed` server back to `none` so it can receive services again. Services that were moved **stay where they are**.
- While a drain is running: `409 drain_in_progress` (cancel first).
- On a server whose state is already `none`: no change, the server is returned.
## Scheduling guard
A server in drain state `draining` or `drained` refuses new placements: creating a service on it, or changing a service's placement to it, returns `422 server_draining` ("<name> is being drained"). Editing a service that stays on the server is not affected. The same guard rejects such a server as an explicit drain target. The UI greys these servers out in the service placement select and in the drain target list.
Drain states: `none`, `draining`, `drained`, `failed`. `drainState` and `drainMessage` appear on the fleet overview; the drain run itself carries `createdAt`/`finishedAt`.
## Required scopes
The drain touches every project placed on the server, so **tokens restricted to specific projects are refused** (`403`) on every drain endpoint. Session users hold every scope.
| Endpoint | Scopes |
|---|---|
| `GET /fleet` | `servers:read` |
| `POST /servers/{serverId}/drain/plan` | `servers:read`, `projects:read` |
| `GET /servers/{serverId}/drain` | `servers:read`, `projects:read` |
| `POST /servers/{serverId}/drain` | `servers:write`, `deployments:write`, `projects:write` |
| `POST /servers/{serverId}/drain/cancel` | `servers:write`, `deployments:write` |
| `POST /servers/{serverId}/undrain` | `servers:write` |
## API endpoints
All under `/api/v1`, tag `fleet`:
| Method and path | Purpose |
|---|---|
| `GET /fleet` | Capacity and placement of every server |
| `POST /servers/{serverId}/drain/plan` | Dry run |
| `POST /servers/{serverId}/drain` | Start (`202`) |
| `GET /servers/{serverId}/drain` | Progress |
| `POST /servers/{serverId}/drain/cancel` | Stop after the move in flight |
| `POST /servers/{serverId}/undrain` | Make schedulable again |
Deployments created by a drain have trigger `drain`.
## Fleet overview
`GET /api/v1/fleet` (UI: the Fleet view on the Servers page) lists every server of the workspace with:
- status, `drainState`, `drainMessage`;
- `cpus`, `memTotalBytes`, `diskTotalBytes`, and the latest `cpuPercent`, `memUsedBytes`, `diskUsedBytes` from heartbeats;
- `committedMemBytes`: the sum of the memory limits of the services placed on the server;
- `unlimitedServices`: how many placed services have no memory limit (they are not in the committed sum);
- `overcommitted`: `true` when `committedMemBytes` is greater than `memTotalBytes` (never when the total is unknown; equal is not overcommit);
- `services[]`: name, project, environment, kind, `memLimitBytes` (0 = unlimited), latest usage, `stateful`, and a `status` (`undeployed`, `deploying`, `running`, `unhealthy`, `stopped`, `missing`, `failed`, `unknown`).
`totals` sums servers, services, memory total, used and committed.
Server numbers always cover all services. The `services[]` rows are limited to the projects the caller may see.
Overcommit is a warning, not a limit: the code only flags it. The drain planner uses `max(used, committed)` in its headroom formula, so it steers moves away from overcommitted servers.
---
# Certificates, upstream policy and tailnet egress
Source: https://nebulactrl.com/docs/operations/proxy-tailnet-certificates
Server-level proxy settings. Every server runs its own Traefik, so everything on this page belongs to **one server** and is **staged** until you press **Deploy proxy** (`POST /servers/{id}/proxy/deploy`). Nothing reaches the agent before that, except that the *applied* configuration keeps riding along with every route push.
All of it needs an agent that advertises the `proxy.v2` capability. For an older agent, staging any of it answers `422 agent_too_old`; update the agent first (the proxy view shows `agentTooOld`).
Scopes: certificates, policy and tailnet changes need `proxy:write`; reads need `servers:read`. The loopback/link-local toggles and the Tailscale integration need `admin`. MCP token presets never receive `proxy:write`.
## Staged, plan, deploy
| Step | Endpoint | Effect |
|---|---|---|
| Stage | `POST /servers/{id}/certificates`, `DELETE /certificates/{id}`, `PATCH /servers/{id}/proxy/settings` | Bumps `proxy.generation`. The proxy view shows `staged: true` while `generation > appliedGeneration`. |
| Plan | `POST /servers/{id}/proxy/plan` | `{restartRequired, reasons[], warnings[], diff[]}`. Contacts nothing. |
| Deploy | `POST /servers/{id}/proxy/deploy` (`Idempotency-Key`, 202) | Sends a `proxy_configure` command built from the **desired** state (staged uploads included, staged removals excluded). One at a time per server (409 `proxy_deploy_in_progress`). |
| Result | agent → control plane | On success: certificates become applied (removed ones are deleted for good), policy and tailnet are copied to their *applied* columns, `appliedGeneration` catches up, the tailnet device id and status are stored. On failure nothing changes; `lastDeploy` shows the agent's message. |
The command payload contains **no** private key, bcrypt hash or Tailscale auth key. They are attached to the outgoing frame at send time and never persisted (`agent_commands.payload`, audit rows, API responses and logs are all free of them).
The commands wait for a server that is offline and expire after 15 minutes, like every command.
## Custom certificates
Use them for wildcard or internal certificates that Let's Encrypt (HTTP-01) cannot issue, for certificates from your own CA, and for hosts that are not reachable from the internet.
```
curl -X POST https://nebula.example.com/api/v1/servers/$SERVER/certificates \
-H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -H 'Content-Type: application/json' \
-d "$(jq -n --rawfile c fullchain.pem --rawfile k privkey.pem '{name:"wildcard-apps", certPem:$c, keyPem:$k}')"
```
What is checked (each failure is a 422 with its own `code`):
| Check | Code |
|---|---|
| each PEM at most 64 KB | `pem_too_large` |
| `certPem` holds only `CERTIFICATE` blocks (the **first** is the leaf, the rest is the chain) | `invalid_certificate` |
| the key is an unencrypted PKCS#8, PKCS#1 (RSA) or SEC1 (EC) key; Ed25519 works | `invalid_key`, `encrypted_key` (remove the passphrase: `openssl pkey -in key.pem -out key-plain.pem`) |
| key and leaf match | `key_mismatch` |
| not expired, not yet valid (5 minutes of clock skew tolerated) | `certificate_expired`, `certificate_not_yet_valid` |
| at least one DNS or IP SAN (browsers ignore the common name) | `certificate_no_san` |
| same certificate twice / same name twice on a server | 409 `certificate_exists`, `certificate_name_taken` |
NebulaCtrl records the SANs as `domains`, the issuer, validity and the SHA-256 fingerprint of the leaf. The key is sealed with the workspace secret key (`NEBULACTRL_SECRET_KEY`) and is **never returned** by any endpoint. The audit event `certificate.upload` carries the name and the domains only.
How Traefik uses it: the certificate goes to the proxy inline (no files, no restart) and is selected by **SNI** against its SANs, wildcards included (`*.apps.example.com` matches exactly one label). A host without a matching certificate gets Traefik's default certificate. Pin a certificate to a domain with `tlsMode: custom` and `certificateId` (the domain must be covered by the certificate, otherwise 422 `certificate_mismatch`); `tlsMode: auto` keeps using Let's Encrypt where the server has an ACME email.
Deleting: a certificate that was never deployed disappears at once; an installed one is **staged for removal** (`change: remove`) until the next deploy. A certificate a domain still pins cannot be deleted (409 `certificate_in_use`).
Status and expiry: `status` is `pending` (never applied), `active` (loaded by the proxy) or `error` (the proxy could not load it, or it expired). The daily `CertExpiryJob` writes `expires in N days` into the status message at 30 and 7 days, records the audit event `certificate.expiring` once per threshold, and marks expired certificates `error`. **There is no renewal for custom certificates**: upload the new one, repoint the domains, deploy, delete the old one.
## Upstream policy (SSRF guard)
External route services and TCP/UDP routes dial operator-supplied addresses, so the same policy runs on the control plane (create/update → 422 `upstream_denied`) and on the agent (before applying a table).
| Target | Default |
|---|---|
| unspecified addresses, `nebulactrl-*` containers, the proxy admin port | always denied |
| loopback (`127.0.0.0/8`, `::1`, `localhost`) | denied; **admin** can allow per server (`allowLoopback`) |
| link-local incl. cloud metadata `169.254.0.0/16`, `metadata.google.internal` | denied; **admin** can allow per server (`allowLinkLocal`) |
| `host.docker.internal` (the Docker host) | allowed; `allowHostGateway: false` removes the mapping from the proxy container |
| LAN, tailnet CGNAT `100.64.0.0/10`, public addresses, DNS names | allowed |
`PATCH /servers/{id}/proxy/settings {"upstreamPolicy": {...}}` stages the change. The *applied* policy (what the agent enforces and what route pushes carry) only changes with a successful **Deploy proxy**. Changing `allowHostGateway` recreates the proxy container (the plan says so).
## Tailnet egress
Goal: route to services on your tailnet (a NAS, another VPS) from this server's proxy. This is **egress from the proxy**, not publishing NebulaCtrl on the tailnet.
| Mode | What runs | When |
|---|---|---|
| `off` | nothing | default |
| `managed` | a `tailscale/tailscale` container `nebulactrl-tailscale` joined to the proxy container's network namespace in **kernel mode**; identity in the volume `nebulactrl-tailscale-state` | the host has no Tailscale; needs the Tailscale integration |
| `host` | nothing: the host already runs Tailscale and the proxy reaches 100.x addresses through it | `facts.hostTailscale.detected` (an interface address in `100.64.0.0/10`); 422 `host_tailscale_not_detected` otherwise |
Routes then target the device's tailnet IP (`100.x.y.z:port`) from the device picker (`GET /integrations/tailscale/devices`). External routes to `100.64.0.0/10` are allowed by the default policy.
### Managed mode details
- **Kernel mode needs `/dev/net/tun` and `NET_ADMIN`/`NET_RAW` on the Tailscale container only.** It is not privileged, has no Docker socket and no host network. The Traefik container gets no extra capability.
- Sharing the proxy's network namespace means the Tailscale container is **recreated whenever the proxy container is** (new TCP/UDP port, host gateway change, MagicDNS toggle). Its identity persists in the volume, so no new key is needed.
- Hostname: default `nebula-<server name slug>`. The real name is read back from the node (`status.fqdn`); a name clash makes Tailscale append `-1`, so never construct it yourself.
- **Provisioning** happens when the mode becomes `managed`, the hostname changes, or you ask for it (`reprovision`). The control plane mints a **single-use, pre-authorized, non-ephemeral, tagged** key (1 hour, description `nebulactrl server <name>`) at send time and attaches it to the command frame; the agent recreates the node with it. Steady-state deploys carry no key (`TS_AUTH_ONCE`).
- **Disabling deletes the device** from your tailnet: the agent removes the container and the state volume, then a River job (`tailnet_delete_device`, retried with backoff, a no-op when the integration is gone or the device already is) deletes it through the API. Re-enabling creates a new device with a new tailnet IP. Re-provisioning and hostname changes replace the device the same way. The plan states it.
- **Device removed in the admin console, or the key was rejected**: the node reports `needs_attention` ("Re-provision"). `POST /servers/{id}/proxy/tailnet/reprovision` mints a fresh key, wipes the state and recreates the node.
- **MagicDNS** is off by default. When on, `100.100.100.100` is added to the proxy container's DNS so routes may name devices (`nas.tailnet.ts.net`). It restarts the proxy container. Prefer IPs; MagicDNS names also depend on your tailnet's DNS settings, and a MagicDNS resolver in the proxy container affects every upstream hostname lookup of that container.
- Heartbeats carry the observed state (`off | provisioning | online | offline | needs_attention`, FQDN, tailnet IPs, device id) into `servers.tailnet_status`; the server list shows it for badges.
### Setting up the Tailscale integration
One per workspace: *Settings → Integrations → Tailscale* (`PUT /integrations/tailscale`, admin). Credentials are an **OAuth client**, not an API access token. The same integration powers the [per-service tailnet](https://nebulactrl.com/docs/operations/service-tailnet) (one sidecar and device per service), the [server mesh](https://nebulactrl.com/docs/operations/mesh) (server and link nodes) and can instead point at a **Headscale** server; per-server egress (this page) is Tailscale-OAuth only and is unavailable with Headscale (`ErrWrongProvider`).
1. In the Tailscale admin console create **tags** and let the OAuth client's tag own them (policy file):
```jsonc
{
"tagOwners": {
"tag:nebula": ["autogroup:admin"], // held by the OAuth client; owns the service tags
"tag:nebula-svc": ["tag:nebula"],
"tag:nebula-mesh": ["tag:nebula"] // only if you use the server mesh
},
"acls": [
// Let your people reach what NebulaCtrl publishes, and nothing from the nodes back:
// add NO rule with src "tag:nebula-svc" unless the proxy must dial tailnet devices.
// The proxy node DOES need to reach the devices you route to, e.g.:
{"action": "accept", "src": ["tag:nebula-svc"], "dst": ["tag:homelab:*"]}
]
}
```
New tailnets default to allow-all: tighten the ACLs, otherwise every tailnet member can reach the proxy node and the proxy node can reach everything.
2. *Settings → OAuth clients → Generate*: scopes **`auth_keys`** (write) and **`devices:core`** (write), restricted to the owner tag (`tag:nebula`). Optional `feature_settings:read` lets the test report whether HTTPS certificates are enabled, and optional `policy_file:read` lets the [mesh](https://nebulactrl.com/docs/operations/mesh#the-acl-mode) run its read-only drift check. Do **not** grant `all`, `dns` or `users`. **`policy_file` (write) is requested only if you opt in to the mesh's managed ACL push on Tailscale; a token with it can rewrite your entire tailnet policy, so leave it off unless you want that** (the default mesh mode is advisory: NebulaCtrl shows the rules and you paste them).
3. In NebulaCtrl save the client id, the secret, the tailnet (`-` = the client's own tailnet) and the tags (default `tag:nebula-svc`; allowed tags must include the defaults, and `tag:nebula-mesh` if you use the [server mesh](https://nebulactrl.com/docs/operations/mesh#providers)). The secret is sealed with `NEBULACTRL_SECRET_KEY`, write-only (`secretSet: true` in responses; omit `clientSecret` on later PUTs to keep it) and absent from logs and audit entries (`integration.tailscale.update` lists changed field names only).
4. **Test connection** (`POST /integrations/tailscale/test`): exchanges a token, lists devices, mints a 60-second single-use key with the default tags and revokes it. No residue. If Tailscale rejects 60 seconds, it retries with 1 day.
The integration cannot be removed while a server uses (or is staged to use) a managed tailnet (409 `integration_in_use`).
### Trust notes
- Whoever holds `proxy:write` and the integration can put a node with your **tag** on the tailnet; the tag's ACL is the boundary. Keep the tag's reach minimal and review the ACL.
- The OAuth secret can mint keys for every tag its owner tag owns. Rotate it (new secret, `PUT`) if it leaks, and revoke the client in the admin console.
- Auth keys live one hour, are single use, and never touch disk on the control plane. A command delivered twice mints twice; an unused key expires on its own.
- The [mesh](https://nebulactrl.com/docs/operations/mesh) adds another tag, `tag:nebula-mesh`, for every server and link node of the team. Its generated rules are by default **advisory**: you paste them. Only if you opt in does NebulaCtrl write the policy (`policy_file` scope), and then the control plane holds a credential that can rewrite the whole policy; the mode is audited as `mesh.acl.managed.enable`, is refused if another team shares the tailnet, and replaces only rules whose every source and destination is a mesh-node IP.
- Nothing contacts Tailscale until Test, Deploy proxy (key mint), a device removal job, or the device picker.
## Limitations
- **Not verified against a real tailnet.** The Tailscale API client is tested against a fake server, and the container shape follows Tailscale's documented kernel-mode setup (spike on Headscale/containers by the design phase). Exercise it on a staging tailnet first.
- No renewal for custom certificates; DNS-01 and automatic wildcard issuance are not part of this release.
- The upstream policy and the tailnet MagicDNS/host-gateway toggles take effect per server, not per route.
- A removed tailnet device is deleted by a background job; if the Tailscale API is unreachable the job retries (10 attempts) and then gives up, leaving the device for you to delete in the admin console.
- Tailscale-side name clashes (`name-1`) are surfaced, not prevented.
---
# Proxy ports
Source: https://nebulactrl.com/docs/operations/proxy-ports
Every server's Traefik publishes **HTTP** and **HTTPS** (tcp, plus udp for HTTP/3) on host ports. The agent starts with `--proxy-http-port` and `--proxy-https-port` (80 and 443 by default; the installer writes them into the agent's systemd unit). The **Proxy ports** card on **Servers, the server, Proxy** moves them later, without touching the host: for example to run NebulaCtrl beside another proxy first, and to take over 80 and 443 at cutover.
It follows the same staged pattern as the other server-level proxy settings (see [Certificates, upstream policy and tailnet egress](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates)): you stage a change, the plan says what it does, and **Deploy proxy** applies it.
## Where the ports come from
The proxy view (`GET /servers/{id}/proxy`) has a `ports` object:
| Field | Meaning |
|---|---|
| `appliedHttp`, `appliedHttps` | The host ports the proxy listens on now, as the agent reports them in its heartbeat |
| `source` | `flag`: the agent's own service flags. `server`: this server setting. Empty until the agent reports |
| `http`, `https` | The ports a Deploy proxy would apply: the staged setting, else the agent's own |
| `setHttp`, `setHttps` | The staged setting (`null` = the agent's own port) |
| `agentHttp`, `agentHttps` | The ports of the agent's flags: what **Reset to agent defaults** returns to |
| `staged` | The setting differs from the applied one and awaits Deploy proxy |
| `busyOther` | Of 80 and 443, the ports another process on the host holds right now (reported by the agent) |
| `supported` | The agent can change its ports from the server (capability `proxy.ports`) |
A server setting **overrides the agent's flags**, and the agent keeps it across restarts. Re-running the agent installer with other `--proxy-http-port` / `--proxy-https-port` flags then changes nothing: use **Reset to agent defaults** first, or change the setting here. Without a setting the flags rule, as before.
## Change the ports
Only the **instance administrator** (the first account) can change them, because a wrong port can cut the dashboard off. Everyone with `servers:read` sees them.
```bash
# Preview: the plan of the deploy as if the ports were staged. Stores nothing.
curl -X POST $URL/api/v1/servers/$ID/proxy/plan -H "Authorization: Bearer $TOKEN" \
-d '{"ports": {"http": 80, "https": 443}}'
# Stage them (needs proxy:write and the instance administrator).
curl -X PATCH $URL/api/v1/servers/$ID/proxy/settings -H "Authorization: Bearer $TOKEN" \
-d '{"ports": {"http": 80, "https": 443}}'
# Apply (recreates the proxy container).
curl -X POST $URL/api/v1/servers/$ID/proxy/deploy -H "Authorization: Bearer $TOKEN" -H "Idempotency-Key: $(uuidgen)" -d '{}'
```
`0` in `ports.http` or `ports.https` resets that port to the agent's own. A token can stage ports only with the `admin` scope and only when its creator is the instance administrator. The change is audited as `server.proxy.update` with the keys `httpPort` and `httpsPort` and their new values (`null` for a reset); the deploy is audited as `server.proxy.deploy` with `httpPort` and `httpsPort` (`0` = the agent's own).
Refused with `422`:
| Code | When |
|---|---|
| `invalid_proxy_ports` | A port outside 1 to 65535, HTTP and HTTPS equal, or a port equal to the proxy's admin port (9000) |
| `proxy_port_clash` | A TCP route service reserves one of them (a UDP route clashes only with the HTTPS port) |
| `agent_too_old` | The agent lacks `proxy.ports` |
Also `403 instance_admin_required` for everyone else.
### What the plan says
A changed port adds `proxy.httpPort` / `proxy.httpsPort` to the plan's diff and sets `restartRequired`: *Changing a published port recreates the proxy container (about 1 second of dropped connections).* Warnings:
- **A port another process holds**, from the agent's `busyOther`: the agent refuses to move onto it (see below).
- **The dashboard**, when the server runs the NebulaCtrl dashboard: `NEBULACTRL_PUBLIC_URL` and the agent's own URL are not changed, but the dashboard is then served on the new HTTPS port.
- **Let's Encrypt HTTP-01**, once per affected domain (see below).
### What the agent does
Before it touches the running container, the agent checks that every web port the proxy does not publish yet is free: no process listens on it and no other container publishes it (Coolify's proxy, for instance). If one is taken the deploy **fails with a result that names the port** (`port 443 (HTTPS, tcp) is already in use on this server by the container coolify-proxy; the proxy keeps running on :8080 and :8443 ...`) and the old container keeps serving. Otherwise it recreates Traefik; if the new container still cannot start (a race), the previous container is created again, exactly as for a failed TCP/UDP port. The applied ports, the `source` and the redirects' public HTTPS port follow the change.
## Let's Encrypt HTTP-01 needs port 80
Let's Encrypt validates HTTP-01 on **port 80** only. While the proxy listens elsewhere, a domain with TLS `auto`, no DNS-01 resolver, no covering wildcard or uploaded certificate and no Cloudflare Tunnel cannot get its certificate. The domain's certificate status (`cert.warning` of the domain view), the service deploy plan and the proxy plan say:
> This server's proxy listens on :8080, not :80: Let's Encrypt HTTP-01 cannot validate app.example.com. Use a DNS-01 resolver (Servers, Proxy, DNS-01) for this domain.
Use [DNS-01 or wildcard certificates](https://nebulactrl.com/docs/operations/dns01-wildcard-certificates) to issue every certificate before the move to 80 and 443. The warning disappears when the proxy listens on 80.
## Moving beside another proxy, then to 80 and 443
1. Install the agent with `--proxy-http-port 8080 --proxy-https-port 8443`, or set 8080 and 8443 here.
2. Issue the certificates through DNS-01.
3. At cutover, stop the other proxy, then set the ports to 80 and 443 (or **Reset to agent defaults** if the flags say 80 and 443). The plan lists the restart; Deploy proxy applies it. If the other proxy still holds a port the deploy fails and the proxy keeps running on its alternate ports.
## Limitations
- The admin entrypoint (default 127.0.0.1:9000) is not part of this setting: it stays an agent flag (`--proxy-admin-port`).
- The control plane's own port and `NEBULACTRL_PUBLIC_URL` are not changed; update them yourself when the dashboard's server moves its HTTPS port.
---
# DNS-01 and wildcard certificates
Source: https://nebulactrl.com/docs/operations/dns01-wildcard-certificates
Let's Encrypt validates a wildcard (`*.example.com`) only through the **DNS-01**
challenge: Traefik creates a TXT record at `_acme-challenge.<domain>` through
your DNS provider's API. NebulaCtrl supports **every DNS provider of lego**, the ACME
client inside Traefik: **19 with a form and credential help** (Cloudflare, Amazon Route 53,
Hetzner, DigitalOcean, Gandi, OVHcloud, Porkbun, Namecheap, Google Cloud DNS, Azure DNS,
deSEC, Linode, Vultr, Scaleway, Bunny, DNSimple, IONOS, GoDaddy and Netcup) and **any other
lego provider** through a generic form (provider code plus name/value variables).
## Concepts
| Thing | What it is |
|---|---|
| Credential | The provider's API variables, sealed with the instance secret key. Scope **team** (usable on every server) or **server**. Created by administrators; secret values are write-only. |
| Resolver | One per provider and server (Traefik `certificatesresolvers.nc-dns-<provider>`). It is static proxy configuration with the credential as the proxy container's environment. |
| Wildcard certificate | `*.<domain>` (and the base domain with "Include the base domain") requested through a resolver. Optional: **also cover generated domains** and **serve as the default certificate** (`defaultGeneratedCert`, so unknown hosts get a valid certificate instead of Traefik's self-signed one). |
| Domain resolver | A single domain can name a resolver instead (useful for hosts that are not reachable on port 80). |
A resolver needs the server's **ACME email**. The environment is process-wide, so
there is **one credential per provider per server**; for two accounts of one
provider delegate `_acme-challenge` with a CNAME (Traefik documents this).
## Variables (verified against go-acme/lego v5.5.2, vendored by Traefik v3.7.16)
Where the names come from. Traefik v3.7.16's `go.mod` requires `github.com/go-acme/lego/v5 v5.5.2`
and its ACME provider calls `dns.NewDNSChallengeProviderByName` (`pkg/provider/acme/provider.go`).
Lego reads each provider's credentials from the process environment, with the names in
`providers/dns/<code>/<code>.toml` at tag `v5.5.2`; the lego documentation renders that file at
`https://go-acme.github.io/lego/dns/<code>/`. The catalog in
`pkg/agentprotocol/dns_lego_catalog.go` is generated from those files by `tools/dnscatalog`, and
`TestCatalogMatchesPinnedTraefik` fails when `proxy.TraefikImage` moves without regenerating it:
```sh
go run ./tools/dnscatalog -traefik v3.7.16 -o pkg/agentprotocol/dns_lego_catalog.go
```
Providers with a form (`R` = required; `OneOf` = set every variable of one method):
| Provider (code) | Variables | Lego page |
|---|---|---|
| Cloudflare (`cloudflare`) | `CF_DNS_API_TOKEN` R (Zone:Zone:Read + Zone:DNS:Edit), `CF_ZONE_API_TOKEN` | [cloudflare](https://go-acme.github.io/lego/dns/cloudflare/) |
| Amazon Route 53 (`route53`) | `AWS_ACCESS_KEY_ID` R, `AWS_SECRET_ACCESS_KEY` R, `AWS_REGION` R, `AWS_HOSTED_ZONE_ID` | [route53](https://go-acme.github.io/lego/dns/route53/) |
| Hetzner (`hetzner`) | `HETZNER_API_TOKEN` R | [hetzner](https://go-acme.github.io/lego/dns/hetzner/) |
| DigitalOcean (`digitalocean`) | `DO_AUTH_TOKEN` R | [digitalocean](https://go-acme.github.io/lego/dns/digitalocean/) |
| Gandi (`gandiv5`) | `GANDIV5_PERSONAL_ACCESS_TOKEN` R | [gandiv5](https://go-acme.github.io/lego/dns/gandiv5/) |
| OVHcloud (`ovh`) | `OVH_ENDPOINT`; OneOf: `OVH_APPLICATION_KEY` + `OVH_APPLICATION_SECRET` + `OVH_CONSUMER_KEY`, or `OVH_CLIENT_ID` + `OVH_CLIENT_SECRET`, or `OVH_ACCESS_TOKEN` | [ovh](https://go-acme.github.io/lego/dns/ovh/) |
| Porkbun (`porkbun`) | `PORKBUN_API_KEY` R, `PORKBUN_SECRET_API_KEY` R | [porkbun](https://go-acme.github.io/lego/dns/porkbun/) |
| Namecheap (`namecheap`) | `NAMECHEAP_API_USER` R, `NAMECHEAP_API_KEY` R | [namecheap](https://go-acme.github.io/lego/dns/namecheap/) |
| Google Cloud DNS (`gcloud`) | `GCE_SERVICE_ACCOUNT` R (the key JSON, stored on one line), `GCE_PROJECT`, `GCE_ZONE_ID` | [gcloud](https://go-acme.github.io/lego/dns/gcloud/) |
| Azure DNS (`azuredns`) | `AZURE_CLIENT_ID` R, `AZURE_CLIENT_SECRET` R, `AZURE_TENANT_ID` R, `AZURE_SUBSCRIPTION_ID`, `AZURE_RESOURCE_GROUP` | [azuredns](https://go-acme.github.io/lego/dns/azuredns/) |
| deSEC (`desec`) | `DESEC_TOKEN` R | [desec](https://go-acme.github.io/lego/dns/desec/) |
| Linode (`linode`) | `LINODE_TOKEN` R | [linode](https://go-acme.github.io/lego/dns/linode/) |
| Vultr (`vultr`) | `VULTR_API_KEY` R | [vultr](https://go-acme.github.io/lego/dns/vultr/) |
| Scaleway (`scaleway`) | `SCW_SECRET_KEY` R, `SCW_ACCESS_KEY`, `SCW_PROJECT_ID` | [scaleway](https://go-acme.github.io/lego/dns/scaleway/) |
| Bunny (`bunny`) | `BUNNY_API_KEY` R | [bunny](https://go-acme.github.io/lego/dns/bunny/) |
| DNSimple (`dnsimple`) | `DNSIMPLE_OAUTH_TOKEN` R (an account token; a user token is rejected) | [dnsimple](https://go-acme.github.io/lego/dns/dnsimple/) |
| IONOS (`ionos`) | `IONOS_API_KEY` R (`<prefix>.<secret>`) | [ionos](https://go-acme.github.io/lego/dns/ionos/) |
| GoDaddy (`godaddy`) | `GODADDY_API_KEY` R, `GODADDY_API_SECRET` R | [godaddy](https://go-acme.github.io/lego/dns/godaddy/) |
| Netcup (`netcup`) | `NETCUP_CUSTOMER_NUMBER` R, `NETCUP_API_KEY` R, `NETCUP_API_PASSWORD` R | [netcup](https://go-acme.github.io/lego/dns/netcup/) |
Each form has a **credential help** panel: a button to the provider's token page (prefilled where the
provider allows it: Cloudflare and OVHcloud), the permissions to select and the provider's own docs.
The pages were opened on 2026-10-27; they are listed in `apps/web/src/lib/dns-provider-guides.ts`.
Provider caveats worth knowing:
- **GoDaddy:** lego's provider authenticates with a *classic developer key*. GoDaddy's developer
docs call it "scheduled for retirement" and the v3 Domains APIs need a Personal Access Token
instead. Expect this provider to stop working when GoDaddy turns classic keys off; delegate
`_acme-challenge` with a CNAME to another provider or move the zone.
- **Namecheap:** API access is opt-in and the calling IP must be on the account's allowed list. The
proxy calls from the server's public IPv4 address.
- **Gandi:** a Personal Access Token expires after at most one year; the credential needs rotating.
- **OVHcloud:** use one authentication method only; lego refuses a mix.
- **Google Cloud DNS:** paste the whole service account key file; it is compacted to one line.
### Any other lego provider
**Add credential → Any other lego provider.** Type the provider's lego code (the form suggests the
other 200 or so from the catalog), then add name/value variables. The form links the lego page of the
code and lists every variable lego reads for it. The control plane checks each name against the
catalog **for that code** and refuses anything else with 422 `invalid_credential`, so a credential can
never put `TRAEFIK_*`, proxy or loader variables (`HTTPS_PROXY`, `LD_PRELOAD`) in the proxy
container's environment. Left out of the catalog on purpose: the `exec` and `manual` providers
(they run a program or wait on stdin) and variables that read a file (`*_FILE`, `*_PATH`: the proxy
container has nothing to read). Values of variables whose name does not say they are an identifier or
a setting (`_ID`, `_USER`, `_REGION`, `_ENDPOINT`, `_URL`, `_TTL`, `_TIMEOUT`, …) are write-only.
A generic credential needs at least one variable; which ones depends on the authentication method,
so read the provider's lego page.
`GET /api/v1/dns/providers` lists the providers with a form; `GET /api/v1/dns/lego-providers` lists
the rest with their variables. Agents older than this feature (no `dns01-lego` capability) are
refused for every provider but the original four (422 `agent_too_old`).
The proxy container has **one environment**: two resolvers cannot both set the same variable name
(for example Route 53 and Lightsail both read `AWS_ACCESS_KEY_ID`). Staging the second resolver
fails with 422 `dns_variable_clash`; use a different provider for one of them.
Only these names can ever reach the proxy container; the agent drops everything else.
## Set it up
1. **Servers, Proxy, DNS-01 certificates**: add a credential (administrator), then pick it
for the provider's resolver, then add a wildcard certificate.
2. **Deploy proxy.** The plan lists the changes and says the proxy container is
recreated (about a second of dropped connections). A rotated credential shows as
"credential rotated" and recreates it too. Nothing reaches the server before this.
3. Add a wildcard domain to a service (`*.example.com`, or pick the resolver for it);
it is served by `HostRegexp` and covered by the certificate. Point a wildcard `A`
record at the server.
The certificate's **status and expiry** come from the proxy itself: the agent opens a TLS
connection with the wildcard's SNI and reads the certificate Traefik serves. "Waiting for
certificate" means the proxy still serves its default certificate; an error shows
Traefik's ACME log line for the resolver. Renewal is Traefik's job (30 days before expiry).
Use **Let's Encrypt staging** on a resolver while testing; staging certificates are not
trusted by browsers.
## How secrets travel
- The database holds the values only sealed. `agent_commands.payload` carries variable
**names**; the values are attached to the outgoing frame at send time (`dns01:<credential>:<NAME>`)
and written to the container environment by the agent. The agent keeps them in its
`0600` state file like uploaded certificate keys, so the proxy can be recreated offline.
- `docker inspect` on the host shows the environment: Docker access is root-equivalent
anyway (see the threat model). Use API tokens limited to the zone.
## Limits
- Resolver and credential changes are an intentional proxy restart; they are never hot.
- Observed state uses the agent's TLS handshake; a CA outage looks like "waiting".
- `defaultGeneratedCert` is rendered through the REST provider; verify it against Let's
Encrypt **staging** before relying on it (the anchor router that requests the wildcard is
the primary mechanism).
---
# Cloudflare DNS
Source: https://nebulactrl.com/docs/operations/cloudflare
NebulaCtrl can keep the DNS records of your service domains in Cloudflare in step with where the services run. One **team-level connection** (an API token) covers every zone the token can see. This page covers the connection, the records NebulaCtrl manages, the per-domain proxy toggle, the DNS status and the [origin lock](#origin-lock) that keeps proxied domains reachable only through Cloudflare. Cloudflare Tunnel is a separate part of the integration (see the *Tunnel* section at the end).
Nothing here is required: without a connection, domains work exactly as before and you point DNS yourself.
## Connect Cloudflare
**Settings → Integrations → Cloudflare** (administrators only).
1. In Cloudflare, **My Profile → API Tokens → Create Token → Custom token** (or an account-owned token under **Manage Account → Account API tokens**) with:
| Permission | Needed for |
|---|---|
| Zone → Zone → **Read** | listing zones, the zone picker (required) |
| Zone → DNS → **Edit** | creating, updating and deleting records (required) |
| Zone → Zone Settings → **Read** | the SSL-mode warning for proxied records, and the SSL-mode check of the origin lock (optional) |
| Zone → Zone Settings → **Edit** | turning on Authenticated Origin Pulls for the [origin lock](#origin-lock) (optional; without it the lock is IP-only) |
| Account → Cloudflare Tunnel → **Edit** | tunnels only (optional) |
Limit **Zone Resources** to the zones NebulaCtrl should manage.
2. Paste the token. NebulaCtrl verifies it with Cloudflare (it must be able to list zones) before it stores it. The token is encrypted with `NEBULACTRL_SECRET_KEY`, is write-only and never appears in API responses, logs, audit entries or notifications.
3. Pick the defaults for **new** domains: *manage DNS automatically* (default on) and *proxied* (default off).
If you already created a Cloudflare DNS-01 credential (Servers → Proxy → DNS-01 certificates), choose it under **Or reuse a DNS-01 credential** instead of typing the token again. It must have the scopes above. The token is copied; the two features then rotate independently.
The zone list is cached and refreshed on **Test connection**, when the token changes and every six hours. Domains are matched to the zone with the longest matching name (`app.eu.example.com` goes to `eu.example.com` if you have that zone, else `example.com`). **Disconnect** stops all management; tick *Also delete the DNS records NebulaCtrl created* to remove them from Cloudflare first.
## What NebulaCtrl manages
A domain that falls into a connected zone gets records created for its hostname:
| Server | Record |
|---|---|
| public IPv4 set | `A` to the server's public IP |
| public IPv6 set | `AAAA` to the server's public IP |
| behind a Cloudflare Tunnel | proxied `CNAME` to `<tunnel-id>.cfargotunnel.com` |
Records follow the domain through its life:
* **domain added** — the record is created right away (before the deployment), so DNS can propagate. Domains that appear through a template, a pull-request preview or a route service are picked up by their first DNS check, seconds after creation;
* **domain removed, service deleted** — the record is deleted; if two domains share a hostname (different paths) the record stays until both are gone;
* **hostname renamed** — the new name gets a record, the old one is removed;
* **the server's public IP changes** — every record of the services placed on that server is updated;
* **a service moves to another server** (including a [drain](https://nebulactrl.com/docs/operations/draining-servers)) — the record is retargeted when the deployment on the new server succeeds, not before, so traffic keeps reaching the old server until the new one is live;
* **a tunnel is created or removed on the server** — address records and the tunnel CNAME replace each other.
Syncs run as background jobs with retries. A sweep every ten minutes repairs drift (for example a record deleted in the Cloudflare dashboard). Every change is audited as `dns.record.create`, `dns.record.update` or `dns.record.delete` (hostname, zone, type, target, never the token). If a sync cannot succeed (Cloudflare errors, missing permission, rejected token) the domain shows the error and NebulaCtrl sends a **DNS sync failed** [notification](https://nebulactrl.com/docs/operations/notifications) (once per failure, at most every six hours).
### Ownership: only records NebulaCtrl created are touched
Every record NebulaCtrl creates carries the comment `nebulactrl:<domainId>` and, where your Cloudflare plan supports tags, the tag `nebulactrl:managed`. NebulaCtrl modifies or deletes a record only if **it carries that marker and NebulaCtrl recorded creating it**. In particular:
* A record you created is **never modified or deleted**, not even when it points to the wrong place. It is reported (`present` or `wrong_target`) and left alone.
* If you edit the comment of a managed record, it stops being managed: NebulaCtrl will not touch it again (and forgets it when the domain is removed).
* Records of other types (TXT, MX, …) are never read or changed.
To hand a record over, use **Adopt existing record** (below). Adopting takes over exactly one record and is the only way an unmarked record is changed. It is refused (`dns_conflict`) when an unmarked record of a conflicting type exists (a `CNAME` cannot sit next to an `A` record): remove that one in Cloudflare yourself.
## Per-domain settings and status
In the domain dialog of a service, a **Cloudflare DNS** section appears for hostnames in a connected zone:
* **Manage DNS automatically** — turn off to manage this domain's DNS yourself. The status is still shown.
* **Proxy** — *DNS only* (grey cloud) or *Proxied* (orange cloud). Tunnel domains are always proxied.
* **Status** (live from Cloudflare):
| Status | Meaning |
|---|---|
| `managed` | a record created by NebulaCtrl points to the server or tunnel |
| `present` | a record you created already points to the right place; untouched |
| `wrong_target` | a record exists but points elsewhere (NebulaCtrl's own drifted record, or yours) |
| `missing` | no record exists yet |
| `not_in_zone` | the hostname is in none of the connected zones; NebulaCtrl does not manage it |
* **Fix DNS** — creates or corrects the record in one click. For a record that is not NebulaCtrl's the button reads **Adopt and fix** and asks first. Fixing also switches automatic management on for the domain.
The same data is available as `GET /api/v1/domains/{id}/dns`, `PUT /api/v1/domains/{id}/dns` (`proxied`, `manage`) and `POST /api/v1/domains/{id}/dns/fix` (`adopt`); `nebulactrl domains ls <service>` shows the status column.
### Proxied records and the SSL/TLS mode
NebulaCtrl serves HTTPS and redirects plain HTTP to it. With a proxied record, Cloudflare connects to your server using the zone's **SSL/TLS mode**:
* **Flexible** or **Off** — Cloudflare talks plain HTTP to your server, which redirects to HTTPS: visitors get a **redirect loop**. NebulaCtrl shows a warning on every proxied domain of such a zone (when the token may read zone settings). Set the zone to **Full (strict)** in Cloudflare → SSL/TLS, or switch the domain to *DNS only*.
* **Full** and **Full (strict)** work. *Strict* needs a certificate Cloudflare trusts at the origin; NebulaCtrl's Let's Encrypt certificates are fine.
Certificate issuance keeps working behind the proxy: HTTP-01 challenges pass through Cloudflare, and [DNS-01](https://nebulactrl.com/docs/operations/dns01-wildcard-certificates) does not involve your server at all. The *DNS check* of a domain reports "proxied through Cloudflare" instead of comparing addresses, because a proxied name resolves to Cloudflare, not to your server.
A proxied record also needs a publicly routable address: Cloudflare cannot reach a private (LAN) IP, which is flagged as a warning.
## Origin lock
A proxied (orange-cloud) domain is meant to be reached **through** Cloudflare: its WAF, rate limits and bot rules only protect the site if visitors cannot skip them by connecting to the server's IP. The origin lock enforces that. It is **on by default for every proxied domain** and needs nothing but the Cloudflare connection above.
### When a domain is locked
| Domain | Locked? |
|---|---|
| Proxied, DNS managed by NebulaCtrl, lock switch on (default) | **yes**, once the DNS record is actually proxied |
| Proxied, but you switched the lock off | no. The domain shows a warning: *Anyone who finds this server's IP can bypass Cloudflare.* |
| DNS only (grey cloud) | no. It must stay reachable directly (a server can host proxied and DNS-only domains side by side) |
| Served through a [Cloudflare Tunnel](#cloudflare-tunnel-per-server) | no. It has no public origin to protect; cloudflared connects from inside |
| DNS **not** managed by NebulaCtrl (*Manage DNS automatically* off) | no. NebulaCtrl cannot know the record is proxied, so it never locks on a guess. Turn management on to lock it |
| Wildcard hostnames, redirect rules | no (the proxy cannot attach a per-host TLS option to a wildcard, and a redirect exposes nothing) |
The lock is decided **per hostname**: when one hostname has several rules (different paths) they all lock, or none does, and your opt-out wins.
The lock follows the DNS record, not just the toggle. NebulaCtrl locks a domain only after its record is proxied (so visitors are never locked out while DNS still points straight at the server) and unlocks it **before** the record is switched back to DNS only.
### The three layers
The domain's DNS panel (Service → Domains → the domain) shows **Origin locked to Cloudflare** with each layer and its state, and the opt-out switch.
1. **IP lock.** The router of the domain answers only connections whose **TCP peer** is in Cloudflare's published IPv4 and IPv6 ranges; every other peer gets `403 Forbidden` with Traefik's plain body (nothing about the server). It is applied per route in Traefik, not with `ufw` or other host firewall rules: Docker-published ports bypass `ufw`, and DNS-only domains on the same server must stay reachable. On port 80 a locked domain only redirects to HTTPS, behind the same lock; Let's Encrypt HTTP-01 validation is answered by Traefik's own ACME router and is never behind the lock.
2. **Authenticated Origin Pulls (AOP).** The TLS option of the domain requires a client certificate (`RequireAndVerifyClientCert`) signed by Cloudflare's origin-pull CA, so a connection that did not come from Cloudflare cannot even finish the handshake. NebulaCtrl turns AOP on for the zone and only then makes the proxy require the certificate. States:
* **On**: AOP is on in the zone and the proxy requires the certificate.
* **Unavailable**: AOP cannot be used; the domain stays **locked by IP only** and the panel says why:
* *the token cannot change zone settings*: add **Zone → Zone Settings → Edit** to the token (the panel links to the token page). The next sync, within ten minutes, turns AOP on;
* *the zone's SSL/TLS mode is Flexible or Off*: Cloudflare does not connect to the origin over HTTPS, so there is no client certificate to present. Set the mode to **Full (strict)** (the same change also fixes the redirect loop above). The panel shows the live mode, so a fix shows at once and the next sync enables AOP.
* **Off**: not applied (the domain is not locked, or the zone has not been checked yet).
*How it is turned on.* Authenticated Origin Pulls with Cloudflare's own certificate is the zone setting `tls_client_auth` ("global" AOP): NebulaCtrl reads `GET /zones/{zone_id}/settings/tls_client_auth` and, when it is off, sets `PATCH /zones/{zone_id}/settings/tls_client_auth` with `{"value":"on"}`: the [Edit zone setting](https://developers.cloudflare.com/api/resources/zones/subresources/settings/methods/edit/) endpoint, as described in Cloudflare's [global AOP guide](https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/set-up/global/). The API reference lists **Zone Settings Write** as the permission for that endpoint (the dashboard names it *Zone → Zone Settings → Edit*). It is distinct from *zone-level* AOP with your own certificate (`origin_tls_client_auth/settings`) and from per-hostname AOP; both are out of scope. The setting is zone-wide: it makes Cloudflare present its certificate to every origin of the zone, and origins that do not ask for it ignore it. Turning it on is audited as `cloudflare.zone.aop.enable`; **NebulaCtrl never turns it off.** The state is stored per zone and re-checked at every DNS sync of a locked domain (trusted for at most five minutes; the ten-minute sweep runs one), so switching AOP off in the dashboard is repaired within about ten minutes.
*The pinned CA.* The proxy trusts exactly one CA for this, Cloudflare's origin-pull CA, embedded in the control plane binary:
| | |
|---|---|
| Source | `https://developers.cloudflare.com/ssl/static/authenticated_origin_pull_ca.pem` (downloaded 2026-10-27) |
| Subject | `CN=origin-pull.cloudflare.net, OU=Origin Pull, O=CloudFlare, Inc.` |
| SHA-256 fingerprint (DER) | `9A:1A:C2:B4:BE:15:F9:F2:7E:EE:20:A7:34:CB:A4:E9:89:8F:61:00:1B:3B:D7:C8:4B:69:B5:6A:3E:25:A2:B9` |
| Valid until | 2029-11-01 |
To refresh it (a release task), download the file again, replace `internal/integrations/cloudflare/data/authenticated_origin_pull_ca.pem`, read the new fingerprint with `openssl x509 -in <file> -noout -fingerprint -sha256` and update `OriginPullCAFingerprint` in `originlock.go`. A unit test fails when the file and the constant differ, and when the certificate is within 180 days of expiry.
3. **Real client IP.** The entrypoints trust `X-Forwarded-For` **only from Cloudflare's ranges**; from any other peer Traefik discards it before anything reads it. On a locked domain the visitor address is the **last** entry Cloudflare appended (Cloudflare adds the address that connected to it, which is the `CF-Connecting-IP` value; everything to its left is supplied by the visitor and is never read). Your IP allow and deny rules, the rate limit, the access log, [traffic analytics](https://nebulactrl.com/docs/operations/traffic-analytics) and the request-origin map all use that address, while the Cloudflare IP lock checks the TCP peer: two separate checks that look at two different addresses. A `trustedProxyDepth` set on a locked domain is ignored (a larger depth would read a forgeable entry).
Traefik's IP strategy can read only the TCP peer or `X-Forwarded-For`, so `CF-Connecting-IP` itself is not consulted; the value is the same by construction. The control plane stores no visitor addresses: analytics keep per-minute counts per country and city computed on the agent from the access log. The control plane's **own** sign-in rate limit and audit actor addresses use the connecting address (behind a proxy that is the proxy's), whether or not the dashboard is proxied through Cloudflare: that is unchanged by the origin lock.
### The IP ranges
The ranges are Cloudflare's, kept current without a release:
* A job fetches `https://api.cloudflare.com/client/v4/ips` (no authentication) **every three hours**. If the API fails or its answer is rejected, it falls back to `https://www.cloudflare.com/ips-v4` and `ips-v6`.
* An answer is accepted only when it is plausible: valid CIDRs of the right family, no range wider than `/8` (IPv4) or `/16` (IPv6), at least 10 IPv4 and 5 IPv6 ranges, and **not shrinking either family by more than half** compared with the list in use. Anything else is refused and the **last good list stays in use**.
* A list built into the binary (fetched 2026-10-27) is used on first start and on installs without internet access.
* The list is stored with its hash, source and `fetched_at`; the last error and the first failure of the current streak are recorded. **Settings → Integrations → Cloudflare** shows the source (Cloudflare API, published text files, built-in copy), the number of ranges, when it was fetched and last changed, any fetch problem, and **Refresh now** (administrators; `POST /api/v1/cloudflare/ips/refresh`). `GET /api/v1/cloudflare/ips` returns the same.
* **When the content changes** the new ranges are pushed to every server that serves a locked domain as a route update, in the same transaction as the change (no redeploy), the change is audited as `cloudflare.ips.updated` (what was added and removed) and teams with a Cloudflare connection get a **Cloudflare IP ranges changed** [notification](https://nebulactrl.com/docs/operations/notifications). Because Traefik reads the trusted forwarded-header ranges from its static configuration, a server whose locked set of ranges changed has its proxy container recreated (about a second of dropped connections, like adding a TCP port); a server without locked domains is not touched.
* When fetching has failed for **24 hours**, teams get a **Cloudflare IP ranges stale** notification (once per outage). Locked domains keep using the last good list; if Cloudflare adds ranges in the meantime, visitors served from them are blocked until a fetch works.
### Turning the lock off for a domain
The **Lock the origin to Cloudflare** switch in the domain's DNS panel (or `PUT /api/v1/domains/{id}/dns` with `{"originLock": false}`) opts one domain out. It asks for confirmation, is applied to the proxy at once, and is audited (`domain.dns.update` with `originLock: false`). While a proxied domain is opted out the panel keeps showing: *Anyone who finds this server's IP can bypass Cloudflare.*
### What the lock does not do
* **It proves "from Cloudflare", not "from your Cloudflare account".** Cloudflare's ranges and its origin-pull certificate are shared by every Cloudflare customer: someone who learns your server's IP can point a zone of their own at it and pass both layers. Hide the IP (the Cloudflare Tunnel needs no open port at all), or add a secret header check at the application, if that matters.
* Non-HTTP routes (TCP/UDP) are not covered; restrict them with their own allowed CIDRs.
* DNS-only domains on the same server stay reachable directly by design.
* The lock does not change certificate issuance: HTTP-01 passes through Cloudflare only if the zone lets the challenge reach port 80 (an *Always Use HTTPS* rule or an SSL mode that contacts the origin on :443 can prevent it); use a [DNS-01 resolver](https://nebulactrl.com/docs/operations/dns01-wildcard-certificates) for such domains.
## Failure modes
| Symptom | Cause and fix |
|---|---|
| Connect fails with *Cloudflare rejected the token* | wrong token, expired, or it lacks Zone:Read. Create a new token with the permissions above |
| Status *not in zone* | the token cannot see the zone (add it to the token's Zone Resources, then **Test connection**) |
| Domain error mentioning a permission | the token has Zone:Read but not DNS:Edit on that zone |
| Connection card shows an error | Cloudflare returned 401 for the stored token; paste a new token. Records are not changed while the token is rejected |
| `wrong_target` that **Fix** will not change | the record is not NebulaCtrl's: use *Adopt and fix*, or edit it in Cloudflare |
| Redirect loop on a proxied domain | zone SSL mode is Flexible/Off, see above |
| A record is still in the zone after deleting a domain | the cleanup runs within seconds, or at the next sweep; it never deletes a record whose marker was edited away |
| Origin lock shows *Authenticated Origin Pulls unavailable* | the token lacks **Zone → Zone Settings → Edit**, or the zone's SSL mode is Flexible/Off; the domain is locked by IP only until fixed |
| A proxied domain answers `403 Forbidden` from a browser that goes through Cloudflare | the visitor was served from a Cloudflare range the stored list lacks: check **Cloudflare IP ranges** (age, last error) and press **Refresh now** |
| `525` (SSL handshake failed) from Cloudflare on a locked domain | AOP is off in Cloudflare while the proxy requires the certificate; the next sync (within ten minutes) turns it on again, or switch the domain's lock off while you investigate |
| *Pending* next to the lock | the DNS record is not proxied yet (or DNS is not managed); the lock follows once it is |
## Tokens and rotation
Replace the token in Settings → Integrations at any time (paste the new one; the old one stops being used immediately). Revoke the old one in Cloudflare afterwards. NebulaCtrl never needs a Global API Key.
## Cloudflare Tunnel per server
A tunnel lets Cloudflare reach a server without any inbound port: **ports 80 and 443 may stay closed**. Enable it on **Servers → (server) → Proxy → Cloudflare Tunnel**. It acts at once and is not staged for *Deploy proxy*.
**Needs:** the Cloudflare connection with an account selected, a token that also has **Account → Cloudflare Tunnel → Edit** (a token without it fails with `cloudflare_tunnel_forbidden`), an agent that advertises the `tunnel.cloudflared` capability, and the `servers:write` + `proxy:write` scopes (administrators).
**What happens when you enable it**
1. NebulaCtrl creates a remotely-managed tunnel in your account (`nebulactrl-<server>-<id>`) and pushes its ingress rules: every hostname of the server's services goes to `http://nebulactrl-traefik:80` with that hostname as the Host header, then a final `http_status:404` catch-all. The rules follow the server's domains (checked every minute and after the connector connects).
2. The agent runs `cloudflare/cloudflared` (pinned by digest) as `nebulactrl-cloudflared` on the `nebulactrl-edge` network, restart `unless-stopped`, no published ports, all capabilities dropped. Its token is fetched from the Cloudflare API when the command is sent and written to a file in the container; it is never stored in the database, in a command payload or in an environment variable.
3. When the connector reports its first connection the tunnel is **active**: domains of this server in connected zones (DNS managed by NebulaCtrl) get a proxied CNAME `<tunnelId>.cfargotunnel.com` instead of A/AAAA records. Until then DNS is unchanged.
4. For those domains the proxy skips HTTP-01 (Cloudflare terminates the public HTTPS; the origin keeps Traefik's default certificate, or the domain's DNS-01 resolver or a covering wildcard) and serves port 80 directly instead of redirecting to HTTPS, which would loop.
**Status** shows in the server's proxy tab and on the server card (connected, connecting, offline, needs attention) with the number of connections, taken from the connector's `/ready` endpoint and reported with every heartbeat. A missing or stopped connector on an online server is re-created automatically (at most every five minutes); *needs attention* (for example a rejected token) waits for **Re-sync**.
**Disabling** points the domains back at the server's public IP, removes the connector and deletes the tunnel at Cloudflare about a minute later. Open ports 80 and 443 before the DNS change propagates. If Cloudflare refuses the delete (token lost the permission), the state stays *Removing* with the reason; fix the token and retry, or **Force remove** to drop the record (delete the tunnel in the Cloudflare dashboard yourself). Revoking a server tears its tunnel down the same way. A Cloudflare connection cannot be removed while a server still has a tunnel.
**Notes:** the connector's metrics port (20241, `/ready` and counters) is reachable from containers on `nebulactrl-edge`, not from outside. Domains outside connected zones, or with DNS management turned off, keep their own DNS and certificates even when the tunnel is active. Hosts that cannot route to container IPs (Docker Desktop) report the connection count as unknown and use the container's health instead.
---
# Per-service tailnet
Source: https://nebulactrl.com/docs/operations/service-tailnet
Put a single service on your tailnet: a private HTTPS (or HTTP) address and raw TCP ports reachable only from your tailnet, with no public port. Typical uses: an admin UI, a database you reach from your laptop, a game server for friends on the tailnet.
This is **ingress to one service**. Routing the proxy to devices on your tailnet is a different feature (see [tailnet egress](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates#tailnet-egress)), and so is letting services of one environment on different servers reach each other (the [server mesh](https://nebulactrl.com/docs/operations/mesh)). The mesh uses the same integration and the same pinned image, but its own devices (server nodes and link nodes, tag `tag:nebula-mesh`), and does not touch a service's sidecar: both can be on at once, and each is a tagged device that counts against your plan.
## How it works
- One sidecar container per service (`nc-ts-<serviceId8>`) runs the pinned `tailscale/tailscale` image in **userspace networking mode**. It joins only the service's environment network and proxies to `http://<service-slug>:<port>`, the alias every replica shares, so one device fronts all replicas and survives rolling deploys and scaling.
- No `NET_ADMIN`, no `/dev/net/tun`, `cap_drop: ALL`, `no-new-privileges`, no published ports, no Docker socket, no routes advertised: a tailnet peer reaches only the ports the serve config names. The agent reads the node's state with `docker exec tailscale status --json` (a fixed argv, no shell).
- Settings are `config.tailnet` on the service. They are **staged** like every setting (the Staged chip, `pending-changes`, per-field discard) and applied by the next deployment. Nothing contacts Tailscale until you deploy.
- The device is created with a **single-use, pre-authorized, tagged, non-ephemeral auth key minted at the moment the command is sent**. The key travels in the outgoing frame only: it is not in `agent_commands.payload`, the audit log, the API, deployment logs or agent state, and the agent masks it in everything it reports. One-off keys are consumed at the first login; a redelivered command mints another and the unused one expires within an hour.
- The device identity lives on the volume `nc-ts-<serviceId8>-state` (and survives restarts, recreations and agent upgrades). The serve config lives on `nc-ts-<serviceId8>-config`, is written by the agent and re-applied live by Tailscale's file watcher, so changing exposures never restarts the sidecar.
- The real MagicDNS name is **read back from the node** (`status.fqdn`), never constructed: a name clash makes Tailscale append `-1`.
## Setting it up
1. Connect a provider in *Settings → Integrations → Tailscale*: Tailscale (OAuth client) or Headscale. See [Tailscale](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates#setting-up-the-tailscale-integration) for the OAuth client, tag ownership and ACLs, and [Headscale](#headscale) below. The integration's **allowed tags** bound what services may use.
2. Service → Settings → **Tailnet**: switch on *Reachable over your tailnet*, then pick what to expose:
- **Web**: HTTPS on 443 to the service port (Tailscale; needs *HTTPS Certificates* enabled in the tailnet DNS settings) or plain HTTP on 80 (Headscale).
- **TCP ports**: `listen → target` forwards (per connection, encrypted by WireGuard). Chips fill common database ports. *TLS* terminates TLS with the node certificate (Tailscale only).
- Optional hostname (default `<service>-<environment>`), extra tags (from the allow-list), **Tailnet only** and Funnel.
3. Save, then **Deploy**. The plan says what will happen ("creates device `web-production` tagged `tag:nebula-svc`").
The canvas card shows a chip next to the host chip with the node name and a status dot, with the text state (`Online`, `Provisioning…`, `Offline`, `Needs attention`) in its tooltip and accessible label, a "Public" mark when Funnel is on and a lock icon (tooltip "Tailnet only") when public domains are paused.
## Tailnet only
*Tailnet only* (`hidePublic`) removes the service's public routers: its domains are kept (and shown paused) but never routed, on any route-table push, until you turn it off and deploy. It follows the **applied** configuration, so a staged toggle changes nothing until the deploy that applies it. Services that other route services target stay reachable through those routes.
## Funnel
Funnel publishes the web exposure on the **public internet** through Tailscale's relays. It is off by default, lives under *Advanced* and needs a typed confirmation (the hostname). It bypasses NebulaCtrl's proxy: no access logs, no traffic analytics, bandwidth limits apply and the hostname is public. Your tailnet policy must grant the `funnel` node attribute to the tags. Audit: `tailnet.funnel_enabled`. Funnel is not available with Headscale (422 `tailnet_funnel_unsupported`).
## What the plan and API refuse
| Code | Meaning |
|---|---|
| `tailnet_not_configured` (422) | no Tailscale/Headscale integration in this team |
| `tailnet_tag_not_allowed` (422) | an extra tag is not in the integration's allowed tags (stops a project member from assigning a more privileged tag) |
| `tailnet_https_disabled` (422) | web on Tailscale while a connection test found HTTPS certificates disabled; enable them or expose TCP ports only |
| `tailnet_funnel_unsupported` / `tailnet_tls_unsupported` (422) | Headscale has no Funnel or TLS termination |
| `agent_too_old` (422) | the server's agent does not advertise `tailnet.service` |
| `invalid_config` (422) | hostname not an RFC 1123 label, bad tag syntax, no exposure chosen, web without an HTTP port, duplicate or colliding ports, one-shot or route services |
## Identity, re-provisioning and removal
- A node is **re-created** (old device deleted first, so the new one keeps its name; new tailnet IP) when its identity changes (hostname, tags, provider or login server), when the service moves to another server, or when it needs attention. Exposure changes never re-create it.
- **Needs attention** appears when the device was deleted in the admin console or its key was rejected: the sidecar exits or sits in `NeedsLogin`, the agent reports it and the control plane flags it. *Re-provision* (`POST /services/{id}/tailnet/reprovision`, `deployments:write`) mints a fresh key, wipes the state volume and recreates the node; the next deployment also does it. The sidecar's last log lines are included in the message.
- **Disabling** (or deleting the service) removes the sidecar and both volumes, then a background job (`tailnet_delete_device`, retried with backoff, a no-op when the integration or the device is gone) deletes the device through the API. Re-enabling creates a new device with a new IP. The plan states it. Orphaned devices from a failed job are left for you to delete in the admin console.
- **Rollback never touches tailnet state.** A rollback, a reconcile redeploy and a drain run an existing release: they keep the node (and "Tailnet only") as last applied, or re-create it on the new server for a drain. The staged-changes list afterwards may show `config.tailnet.*` differences from the old release; deploying them is a no-op unless you changed the tailnet since.
- A failure of the tailnet step **never fails or rolls back the app**: the deployment keeps the app's outcome and carries a `Warning: tailnet …` message; the node shows `Needs attention` with the reason.
- If the Tailscale API is unreachable at send time the deployment is *not delivered* ("could not resolve the command's secrets … No changes were made."), as for any send-time secret. Only deployments that create or re-create the device need the API.
## Headscale
Pick *Headscale* in the integration card:
- **Login server**: the Headscale URL (also the REST base).
- **Credential**:
- *API key*: NebulaCtrl creates a **single-use pre-auth key per provisioning** (`POST /api/v1/preauthkey`, tags from the key, never `--advertise-tags`), expires unused keys, finds nodes by tailnet IP and deletes them (`DELETE /api/v1/node/{id}`) on disable and re-provision. Optional *user* owns the keys; empty creates tags-only keys (nodes belong to `tagged-devices`).
- *Pre-auth key*: one **reusable tagged key** handed to every sidecar. Nodes cannot be deleted through the API, and the key stays visible to anyone with Docker access on the hosts. Prefer the API key.
- Headscale has **no Serve HTTPS and no Funnel** (feature matrix, issue juanfont/headscale#1921): the web exposure is plain HTTP on port 80 and TCP ports forward over WireGuard. The agent writes the HTTP serve config only after the node has its real DNS name. The tags must be allowed by Headscale's policy (`tagOwners`); the connection test mints and expires a key to check that. Clients need Headscale ≥ 0.26 API shape (`/api/v1`, string ids) and a Tailscale client ≥ the version Headscale requires; check your server's `/swagger` if a call is rejected.
## Tailscale policy
Do not give the sidecars more reach than they need: they never initiate, so add no ACL with `src: tag:nebula-svc`. Example (see the OAuth page for `tagOwners`):
```jsonc
"acls": [
{"action": "accept", "src": ["autogroup:member"], "dst": ["tag:nebula-svc:443", "tag:nebula-svc:5432"]}
]
```
A new tailnet is allow-all: tighten it. Hostnames become public in Certificate Transparency logs when HTTPS is on; do not put secrets in hostnames. One tagged device per service counts against your plan's device quota.
## Limits and known behaviour
- **Not verified against a real tailnet.** The API clients are tested against fake Tailscale and Headscale servers and the container shape follows the documented userspace setup (and a spike on Headscale). Serve HTTPS, Funnel, OAuth key minting and device deletion on Tailscale SaaS need one run on a staging tailnet before you rely on them.
- The web path load-balances **per connection, not per request** across replicas (connection pooling in Tailscale Serve); TCP forwards balance per connection too. Idle pooled connections to a retired replica are replaced.
- Serve adds `Tailscale-User-*` identity headers only for user nodes and the backend is reachable on the environment network: **do not use them as authentication.**
- Memory: the sidecar is capped at 256 MiB (not measured per node).
- Sidecar auth key lives in the container environment until the node is recreated; one-off keys are inert after first login.
- Egress from the app to tailnet hosts is not provided by this feature (the app container does not share the sidecar's network namespace). Reaching a service on **another server of the same environment** is what the [mesh](https://nebulactrl.com/docs/operations/mesh) does; reaching arbitrary tailnet hosts is still not available.
---
# Server mesh and cross-server links
Source: https://nebulactrl.com/docs/operations/mesh
The **mesh** joins the servers of a team to the team's tailnet (your Tailscale or the managed Headscale) so that a service on one server can reach a service on another server **inside the same environment**. Two features use it:
- **Cross-server service links**: an app on server A reaches a database, an API or a [read replica](https://nebulactrl.com/docs/operations/database-replicas) on server B by its service name, for example `postgres:5432`.
- **Read replicas across servers**: the replication stream of a [Postgres read replica](https://nebulactrl.com/docs/operations/database-replicas#same-server-and-cross-server) travels over the mesh.
It is a handful of small, ordinary containers that NebulaCtrl starts on your servers, plus an access policy it generates. This page explains what they are, what they cost, what can go wrong and how to test it.
The mesh is **off** until you enable it per server, and it needs a Tailscale or Headscale integration first.
## What runs on a server
| Container | One per | What it does | Privileges |
|---|---|---|---|
| **Server node** `nc-mesh-<server>` | mesh-enabled server | The server's identity on the tailnet. It accepts connections from peers and forwards them (`tailscale serve`, TCP only) to the services you expose. | None to speak of: userspace networking, `cap_drop: ALL`, `no-new-privileges`, no `/dev/net/tun`, no host network, no Docker socket. |
| **Link node** `nc-meshl-<link>` | environment, local server, remote server and purpose, **only while a link needs it** | The way out: containers of the environment dial `<service>:<port>` and the link node carries the connection to the remote server node. | `NET_ADMIN`, `NET_RAW` and `/dev/net/tun` **in its own network namespace only**, the same shape as the proxy's [managed tailnet container](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates#managed-mode-details). Not privileged, no Docker socket, no host network. |
Both use the same pinned `tailscale/tailscale` image as the other tailnet features. **Neither runs in the host's network namespace**, so NebulaCtrl changes no firewall rule, installs nothing on the host and does not interact with ufw or firewalld. Tunnel traffic is outbound UDP and HTTPS from containers. If the host already runs Tailscale, it keeps working next to the mesh (NebulaCtrl never adopts or manages it).
A link node joins the environment's network (and its `-int` network when the environment has one, see [networking](https://nebulactrl.com/docs/operations/networking#the-layout-on-a-server)) under the **aliases of the services it reaches**. That is why a consumer that **blocks outbound internet** still reaches its providers: the link is on the internal network, and nothing else is.
A link node needs `/dev/net/tun` on its host (the agent reports it). A server without it can still join the mesh and *be* a provider; it cannot be the consuming side of a link: the plan says `mesh_no_tun`. On most distributions `modprobe tun` is enough.
## Providers
The mesh uses the integration you already have, in *Settings → Integrations → Tailscale* (see [Setting up the Tailscale integration](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates#setting-up-the-tailscale-integration) and [Headscale](https://nebulactrl.com/docs/operations/service-tailnet#headscale)):
| Provider | Joining | Access rules (ACL) |
|---|---|---|
| **Managed Headscale** ("Set up for me", [platform services](https://nebulactrl.com/docs/operations/platform-services#tailnet-control-server)) | single-use tagged pre-auth keys through its API key | **Managed**: NebulaCtrl owns the whole policy |
| **Your own Headscale** with an API key | same | **Managed** when its policy mode is `database`; **advisory** when it is `file` (the server refuses writes) |
| Headscale with a **pre-auth key** only | joining only; nodes cannot be deleted through the API, so removal leaves devices behind | **Advisory** |
| **Tailscale** (OAuth client) | single-use tagged auth keys | **Advisory** by default; **managed** only if you opt in |
Every mesh node carries **one tag** for the whole team, `tag:nebula-mesh` by default. It must be in the integration's allowed tags, and on Tailscale it needs a `tagOwners` line (see [the policy snippet](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates#setting-up-the-tailscale-integration)). Auth keys are **single-use, pre-authorized, non-ephemeral, tagged and valid for 1 hour**; they are minted when the command is sent, travel in the outgoing frame only, and are never in command payloads, the audit log, the API or logs. A node keeps its identity in a volume, so steady state needs no key.
## Turning it on
1. Connect the integration.
2. Server → **Mesh** card → **Enable**, or `PUT /servers/{id}/mesh {"enabled": true, "acceptMeshChanges": true}` (`servers:write`; the change is accepted with `202`). Add `?dryRun=true` to see the plan: the devices it adds, whether `/dev/net/tun` is present, blockers and warnings. Without `acceptMeshChanges` an enabling that adds devices answers `mesh_confirmation_required`. It is audited as `mesh.enable` (`mesh.disable` for the opposite).
3. The server node appears under *Settings → Mesh* with its tailnet IP, name, peers (direct or relayed, latency), its exposures and its state: `provisioning`, `online`, `offline` or `needs_attention` (`unknown` when the last observation is older than 60 seconds). A link node can also be `degraded` (no direct path) or `removing`.
Link nodes are never created by hand. They appear when a deployment, a canvas edge, a replica or a Promote needs one ([below](#who-creates-links)).
## Access rules (ACL)
NebulaCtrl **generates** the rules; you do not write them. They come from three tables only: the server nodes, the links and the *grants* (which link may reach which exposed service). The result is always an explicit allow list, so the default is deny:
```jsonc
// one rule per link; src is the link node's tailnet IP, dst the remote server node and the granted ports
{"action": "accept", "src": ["100.64.0.12"], "dst": ["100.64.0.7:15432"]}
```
- A link belongs to **one environment**, so its IP is the source of rules for that environment's services only. A grant can only join a link of environment E to a service of environment E; that is enforced when the grant is created, so one environment can never open another environment's port. This is how access is limited **per team** (a team is a tailnet) **and per environment** without a tag per environment.
- Replication links and service links are **different nodes** with different source IPs, so a consumer's link can never reach a replication port and the other way round.
- Rules are IP and port based, sorted, and deterministic. IPv6 is left unmatched, which means denied.
- Rules are re-generated when nodes, IPs, links or grants change (debounced, a few seconds). A re-provisioned node gets a new IP and its rules follow.
**Settings → Mesh → Preview rules** (`POST /mesh/acl/preview`) shows the rules and, for Tailscale, the snippet. The page also lists each rule with why it exists: `link → server:port because replica <slug> of <primary>`.
### The ACL mode
`PUT /mesh/settings {"aclMode": "managed" | "advisory" | "off"}` needs the `admin` scope.
| Mode | What NebulaCtrl does |
|---|---|
| **managed** (Headscale) | Writes the **whole** policy document (`PUT /api/v1/policy`, after `POST /api/v1/policy/check`): the tag owners, the rule that lets members reach service tags, and the generated mesh rules. **Anything you edit by hand in that policy is overwritten.** Needs the API key credential and Headscale's `policy.mode: database`. |
| **advisory** (default on Tailscale) | Shows the generated `tagOwners` and `acls` snippet to paste into your tailnet policy, and writes nothing. With the optional `policy_file:read` scope the **ACL check** (`POST /mesh/acl/check`, `servers:read`) compares the stored policy with the generated rules: it lists the generated rules that are missing and the outdated NebulaCtrl rules still present, and it cannot prove that a broader rule of yours does not already allow the traffic. **The connectivity check ([Test](#test)) is the ground truth.** |
| **managed** (Tailscale, opt-in) | Patches **only** the mesh rules in your policy and keeps your comments (HuJSON), through the Tailscale policy API. A rule is treated as ours only when every source and destination is a mesh-node IP; only those are replaced, and `tagOwners` for the mesh tag is added. It validates before it writes and refuses to write over a policy that changed in between (`If-Match`). Enabling is audited as `mesh.acl.managed.enable`. |
| **off** | No rules are shown or applied; your tailnet's default policy decides. |
> **Read this before you opt in on Tailscale.** The managed push needs the OAuth client to carry the **`policy_file`** scope (write access to the policy). **A token with that scope can rewrite your entire tailnet policy**, not just the mesh rules, and NebulaCtrl's control plane holds that credential: a compromised control plane could open every port on the tailnet. NebulaCtrl asks for it only if you switch the mode on (the default scopes stay `auth_keys` and `devices:core`, with optional `policy_file:read` for the drift check), and it refuses the mode when another team uses the same tailnet (`mesh_acl_shared_tailnet`) and when the scope is missing (`mesh_policy_scope_missing`). If you do not want that trust relationship, stay advisory and paste the snippet.
A **replica or link deploy waits** until its link node is online and the policy state is `applied`, `advisory` or `off`; in advisory mode the wait ends with a successful connectivity check, never with an assumption. The state is shown on the Mesh page: `unknown`, `applied`, `pending`, `drift`, `advisory` or `error`.
With an advisory policy and a tailnet whose default is allow-all, a link node can reach **any** exposure port of the remote server node. The Mesh page warns about it. Tighten the tailnet policy (a new tailnet starts allow-all) or use a managed mode.
## Devices and plan limits
Every mesh node is a **tagged device**:
```
devices added = server nodes (one per enabled server) + link nodes (one per environment × server pair × purpose in use)
```
and they come on top of the devices of [per-service tailnets](https://nebulactrl.com/docs/operations/service-tailnet) and the proxy's managed node. Every plan or dialog that adds a link states "adds N devices", and the page shows the number the mesh uses.
- **Tailscale**: Personal includes **50 tagged resources** and charges **$1 per month for each one beyond that**; it allows up to 6 users and 3 ACL groups. These are the published numbers at the time of writing: check [tailscale.com/pricing](https://tailscale.com/pricing) for your plan, because NebulaCtrl cannot read your quota and does not stop at it.
- **Headscale**: one instance with SQLite and no per-device limit. It is the single place where *changes* happen (see [failure modes](#failure-modes)).
A link exists only while something needs it; removing the last reference releases the grant, then the link, then the exposure, and deletes the device through the API (not possible with a pre-auth key credential: the UI says so).
## Cross-server service links
A **consumer** `C` on server A reaches a **provider** `P` on server B, both in the **same environment**. `P` is any service with a TCP port: an app, a worker, a database or a replica. UDP, Compose stacks and route services cannot be exposed (`mesh_not_exposable`), and **cross-environment links are not built**: shared networks stay on one server.
How it fits together:
1. **Exposure.** On B's server node a forward `meshPort → tcp://<service>:<port>` is created. When `P`'s running container carries the mesh alias it uses that, otherwise its container name; the agent decides by inspecting, so exposing a running service **needs no restart**.
2. **Link.** On A, a link node for (environment, A → B). Its aliases are the **slugs of the providers it reaches**.
3. **Grant.** The link may reach this exposure, because of a reference, a canvas edge or a replica.
**What the consumer dials is `<provider slug>:<port>`**, exactly like on one server. The port is the container port **when that port is free** on server B's node; otherwise the first free port in 15000-15999. It is allocated once and stays. In the common case `http://api:8080` works unchanged; the port differs only when two services on server B use the same one.
Variable references follow without you doing anything. `${{api.NEBULA_PRIVATE_HOST}}` is the slug, which is the alias. `${{api.PORT}}`, the database exports (`DATABASE_URL`, `DATABASE_PORT`, `PGPORT`, `REDIS_URL`, the `DATABASE_WRITE_*` and `DATABASE_READ_*` exports) and each group member's port become **consumer-aware**: they resolve to the mesh port when the provider runs on another server than the consumer, and to the container port otherwise. See [variables](https://nebulactrl.com/docs/operations/variables#platform-variables).
### Who creates links
Links cost devices, so they are **never created by editing a variable**. They are added only by an explicit action:
- **Deploying or deploy-staged on the consumer.** The plan lists the links it adds ("adds N devices") and any blocker; deploying is the confirmation.
- **Creating a canvas edge** between services on different servers. The dialog states the devices and asks for confirmation (`acceptMeshChanges`); without it the API answers `mesh_confirmation_required`. The edge replaces the old warning that different servers share no private network.
- **Adding a replica or promoting** one ([database replicas](https://nebulactrl.com/docs/operations/database-replicas)).
- Deploying a provider that **moved server**.
Editing a variable that references a service on another server only stages: the staged bar shows "Mesh: adds link A → B (1 device)". A background **repair** pass every 5 minutes only **removes** what nothing refers to any more (after 10 minutes of grace; an edge's grant goes at once) and restores rows lost by a crash for pairs you already confirmed. It never adds a device.
### What a deploy checks
For a consumer that needs a link the plan blocks, or warns, with:
| Code | Meaning |
|---|---|
| `mesh_server_required` | mesh is not enabled on a server involved (with "Enable mesh on `<server>`") |
| `mesh_no_tun` | the consuming server has no `/dev/net/tun` |
| `mesh_agent_too_old` | the server's agent does not manage mesh nodes: update it |
| `mesh_paused` | the mesh is [paused](#the-paused-migration-window) |
| `mesh_not_exposable` | the provider is UDP, a stack, a route service or has no port |
| `mesh_route_pending` | warning: a reference to a provider on another server has no link yet |
After the link is added the deployment **waits up to 5 minutes** for every needed link to be online and the policy applied. On timeout it fails with "Not deployed: mesh link not ready. No changes were made." and nothing of the consumer changed.
### Security of a link
A consumer reaches exactly the provider ports it is granted, through its environment's link only. Other environments on the same server have no alias for it and no route to the link, and the policy source is the link's own IP. **Other services of the same environment on the consuming server can use the link too**: that is the same trust boundary as everything else in an environment. Traffic is encrypted by WireGuard between the link node and the server node; **it is not encrypted between the server node and the provider container on the same host** (that hop is local), and it adds no authentication of its own: the provider's own credentials still apply.
## DERP: public relays are the fallback
Peers connect directly when they can. When two servers have no direct UDP path, WireGuard traffic falls back to **DERP relays**. The **managed Headscale** is configured with **Tailscale's public DERP map** and has no relay of its own, so the fallback is Tailscale's public relays. Traffic stays end-to-end encrypted: a relay sees ciphertext, the endpoint addresses and the volume, not the content. This is disclosed on the Mesh page and in the "Set up for me" dialog. It also means the managed Headscale server needs outbound internet access to fetch the map (it is the one platform service that does not block outbound). An embedded DERP server is **not built**. Allowing inbound UDP 41641 on the servers only improves the chance of a direct path; it is optional and NebulaCtrl never opens it for you.
If you use your own Tailscale or Headscale, its DERP configuration applies.
## The paused migration window
Managed Headscale used to run with a policy *file*. The mesh needs the `database` policy mode so NebulaCtrl can write rules, so **an existing platform Headscale is migrated once**: redeployed with the new configuration. Fresh installs start with the new mode and have no window.
During the migration the mesh is **paused**:
- the Mesh page shows a banner with the reason (for example "Headscale is being migrated");
- no key is minted, no node is synced or removed, no ACL is written, and replica and Promote operations wait (they resume by themselves);
- a consumer deploy that needs a **new** link fails with "Not deployed: mesh paused. No changes were made.";
- **established tunnels, existing nodes and [service sidecars](https://nebulactrl.com/docs/operations/service-tailnet) keep working.**
The job redeploys Headscale through the platform service's own provisioning, waits for it to be healthy, writes the base policy plus the mesh rules, reads them back and un-pauses. Between the restart and the first write Headscale has **no policy, which means allow-all**; that lasts seconds, and nothing can join because the mesh is paused. If a step fails, the mesh **stays paused** with the reason (the banner starts with "Headscale is being migrated" and says what failed). **Retry** the Headscale service on the [platform services](https://nebulactrl.com/docs/operations/platform-services#tailnet-control-server) page to resume; the job also looks again every 10 minutes and is safe to run more than once. While Headscale itself is down, see the next section.
## Failure modes
| What happens | Effect | What you see |
|---|---|---|
| **The coordinator is down** (the Headscale host, or Tailscale's control plane) | Established WireGuard sessions keep carrying traffic, so a running replication stream continues (what a node does if it restarts while the coordinator is down was not verified: assume it cannot rejoin). No new joins, no ACL changes, no re-provisioning. | Banner "Mesh coordinator unreachable: existing connections keep working, changes wait". Adding a replica is refused with `mesh_coordinator_unreachable`. The page warns when Headscale and a replication member run on the **same server** (a shared failure). |
| **A server cannot reach DERP and has no direct path** | The link is down. | Link state `degraded` or `offline`; Test returns `no_path`. |
| **A server is offline** | Node and link states go stale. | Observations older than 60 seconds show "unknown", not "online". |
| **A server is revoked** | Its node and link devices are deleted through the API, exposures, grants and links are dropped and the rules are regenerated. Replicas on it become `failed`; replicas whose **primary** was on it become `broken` and Promote is offered. | Server list and Mesh page. |
| **A node is deleted in the tailnet admin console** | The node reports `needs_attention`. | **Re-provision** mints a new key, wipes the node state and gives it a **new tailnet IP**; the rules follow. |
| **The integration is removed or changed** | Blocked while the mesh is enabled: 409 `integration_in_use`. | |
| **The mesh is paused** | See [above](#the-paused-migration-window). | Banner with the reason. |
| **A link was never granted (or the rule is missing)** | Connections time out. | The link's connectivity check returns `timeout`; a replica using it goes `broken`. |
An auth key that never produced an online node within an hour is revoked by a sweeper. Node keys have no expiry once joined (tagged devices are exempt on Tailscale and Headscale), so rotation means **re-provisioning** a node, rotating the OAuth secret or the Headscale API key (a platform Headscale's key lasts 365 days; **Reconnect** mints a new one).
NebulaCtrl never depends on MagicDNS for the mesh: `TS_ACCEPT_DNS=false` on every node so Docker's own DNS keeps resolving aliases. Names on the tailnet are cosmetic.
## Test
Connectivity is what proves a mesh works, in every ACL mode.
- **Server → Mesh → Test**, or `POST /servers/{id}/mesh/check` (`servers:write`, optionally `{"peerServerId": "…"}`; the default is every other server with a mesh node): the agent reads the server node's view of the tailnet (`tailscale status`, plus a ping) and reports, per peer server node, `connected`, `refused`, `timeout` or `no_path` with the latency and a message.
- **Link checks** run for you: before a replica or a deploy that needs a link goes on, and in advisory mode until the check connects, the agent dials `<link node>:<mesh port>` through the environment network for each grant. `connected` means the whole path (link node, policy, server node, provider) works for that pair; `timeout` usually means a missing rule or no path (an ACL that does not allow the port looks the same as a dead remote node); `refused` means the path works and nothing listens.
- The **ACL check** is a different, read-only comparison of policy against rules (see [the ACL mode](#the-acl-mode)).
The check proves **intent, not exclusivity**: it cannot show that nothing else is allowed. For that, read the rules (Preview) and your tailnet's own policy.
## Limits
- **Same environment only**: no cross-environment links.
- **TCP only**: no UDP, no Compose stacks, no TLS termination and no DNS names for remote services; a consumer dials `<slug>:<port>`.
- **Not verified here**: Tailscale SaaS (OAuth, policy API, tag ownership, device quota), NAT traversal and DERP fallback across real networks, WAN latency and MTU, hosts with ufw or firewalld enabled, and `/dev/net/tun` inside docker-in-docker. The design (server node, link node, policy, replication through it) was checked in a spike with Headscale 0.29.6 and the pinned `tailscale/tailscale` image on one machine, and the control plane and agent code is covered by unit tests against fake Tailscale, Headscale and Docker; run it on a staging tailnet before you rely on it.
- No exit nodes, subnet routes, Tailscale SSH or Taildrop, and no management of a Tailscale that already runs on the host.
- Headscale is a single instance: it is where changes happen, not a path for established traffic.
Related: [database replicas](https://nebulactrl.com/docs/operations/database-replicas), [per-service tailnet](https://nebulactrl.com/docs/operations/service-tailnet), [platform services](https://nebulactrl.com/docs/operations/platform-services), [networking](https://nebulactrl.com/docs/operations/networking), the [threat model](https://nebulactrl.com/docs/security/threat-model).
---
# Host hardening
Source: https://nebulactrl.com/docs/security/host-hardening
NebulaCtrl audits each server with [Lynis](https://cisofy.com/lynis/) (CISOfy's open-source host auditing tool), shows the result on the server's **Security** tab, and offers a short, curated list of **safe fixes** it can apply for you and undo again.
It is deliberately conservative. A fix is in the list only if it is reversible, changes nothing a service or a login depends on in normal use, and cannot cut you off from the host. Everything else Lynis suggests is shown with an explanation and the commands to do it yourself.
**NebulaCtrl never changes SSH settings, firewall rules or user accounts.** There is no fix for them in the catalog, a test pins it, and the Security tab says so next to those findings. The installer's own ufw rules (SSH, 80, 443) are the only firewall change NebulaCtrl ever makes.
## What runs where
| Where | What |
|---|---|
| Installer (`install.sh`) | The last step runs `nebulactrl-agent hardening run` on the control-plane host: installs Lynis, audits, applies the default fixes, audits again and prints the index before and after. `--no-hardening` skips it. |
| Agent | Runs everything on the host. An agent installed on the host runs the scripts directly as root. An agent that runs in a container (the dev stack) starts a short-lived privileged helper container from its own image whose command enters PID 1's namespaces with `nsenter`, the same mechanism as the [host shell](https://nebulactrl.com/docs/security/host-shell). The helper is removed by the daemon when the script ends. |
| Control plane | Schedules the weekly audit, starts applies and undos, keeps the parsed report and the state of every fix, audits who did what. |
| Dashboard | Server → **Security**: the index, the fixes, what is left, the history. |
The agent advertises the capability `host.hardening`; a server whose agent is older gets 422 `agent_too_old` and the tab says to update it.
Operating system package updates and reboots are a separate feature with their own checks and actions: see [Server patching](https://nebulactrl.com/docs/operations/server-patching). A hardening fix that only takes effect after a reboot sets the same *reboot required* note.
## The audit
`lynis audit system --quick --no-colors --cronjob`, run from `/` (Lynis treats a working directory that has a `db/` or `include/` folder as its own installation). The agent reads `/var/log/lynis-report.dat` ([report files](https://cisofy.com/documentation/lynis/get-started/#report-files): `key=value` lines, repeated keys end in `[]`) and keeps these keys: `lynis_version`, `hostname`, `os_name`, `os_version`, `report_datetime_end`, `hardening_index`, `lynis_tests_done`, `vulnerable_packages_found`, `warning[]`, `suggestion[]` (`ID|text|details|solution|`) and `details[]` (`ID|section|desc:…;field:…;prefval:…;value:…;|`, which is where the per-key sysctl values and sshd options of KRNL-6000 and SSH-7408 live). A report without `hardening_index` means the audit did not finish and is rejected. The audit only reads the host.
Lynis is installed on request, never silently:
* The installer installs it on the control-plane host.
* The Security tab shows **Install Lynis and run audit** on a server that does not have it. Clicking it is the consent.
* The weekly audit never installs anything; on a server without Lynis it records a *skipped* run.
The installer prefers CISOfy's own repository, because the distributions ship older Lynis releases: it fetches the signing key from `packages.cisofy.com`, checks that it is an armored PGP key, installs it to `/usr/share/keyrings/cisofy-lynis.asc`, adds `deb [signed-by=…] https://packages.cisofy.com/community/lynis/deb/ stable main` and refreshes only that source (RHEL family: `/etc/yum.repos.d/cisofy-lynis.repo` with `gpgcheck=1`). If the repository is not reachable it falls back to the distribution's `lynis` package on Debian and Ubuntu. To remove the repository: delete `/etc/apt/sources.list.d/cisofy-lynis.list` and the keyring file. Sources: [Lynis get started](https://cisofy.com/documentation/lynis/get-started/), [CISOfy package repository](https://packages.cisofy.com/community/).
The weekly audit is a control-plane job that sends a `host_hardening` command once a week to every online server (switch it off per server on the tab). Why the control plane and not a timer on the host: the result, the history and the audit log live in one place, and a server that was offline simply gets its audit when it comes back.
## The safe fixes
Lynis test IDs link to CISOfy's page for the control: `https://cisofy.com/lynis/controls/<ID>/`. "Default" fixes are what **Apply safe fixes** and the installer apply; the others are offered one by one.
| Fix | Default | Lynis test | Check (already satisfied when …) | Fix | Rollback | Downside |
|---|---|---|---|---|---|---|
| `sysctl-kernel` | yes | [KRNL-6000](https://cisofy.com/lynis/controls/KRNL-6000/) | each key has the value, or is stricter, or does not exist | `kernel.dmesg_restrict=1`, `kernel.kptr_restrict=2`, `kernel.sysrq=0`, `kernel.ctrl-alt-del=0`, `kernel.yama.ptrace_scope≥1`, `dev.tty.ldisc_autoload=0`, `net.core.bpf_jit_harden=2`, `kernel.core_uses_pid=1`; written live to `/proc/sys` and to `/etc/sysctl.d/90-nebulactrl-sysctl-kernel.conf` | previous values written back, drop-in deleted | `kptr_restrict=2` hides kernel addresses from root-run profilers; `ptrace_scope=1` stops non-root users attaching a debugger to processes that are not their children |
| `sysctl-fs` | yes | KRNL-6000, [KRNL-5820](https://cisofy.com/lynis/controls/KRNL-5820/) | as above | `fs.suid_dumpable=0`, `fs.protected_hardlinks=1`, `fs.protected_symlinks=1` | as above | none known |
| `sysctl-net` | yes | KRNL-6000 | as above | no ICMP redirects accepted or sent (v4, v6), no source routing, `log_martians=1`, `tcp_syncookies=1`, ignore broadcast pings and bogus ICMP errors | as above | `log_martians` is noisy on an internet-facing host (the kernel rate-limits it) |
| `sysctl-fs-protected` | no | KRNL-6000 | as above | `fs.protected_fifos=2`, `fs.protected_regular=2` | as above | a root job that opens another user's lock file in `/tmp` fails; audit your jobs first |
| `core-dumps` | yes | KRNL-5820 | both drop-ins exist | `/etc/systemd/coredump.conf.d/nebulactrl-hardening.conf` (`Storage=none`, `ProcessSizeMax=0`) and `/etc/security/limits.d/nebulactrl-hardening.conf` (`* hard core 0`) | both files deleted | no core files for post-mortem debugging |
| `umask` | yes | [AUTH-9328](https://cisofy.com/lynis/controls/AUTH-9328/) | `UMASK` in `/etc/login.defs` is 027, 0027, 077 or 0077 | `UMASK 027` | the original `UMASK` line restored verbatim | files you create by hand (also with `sudo`, because `pam_umask` applies) are 640/750; a service running as another user cannot read a config you wrote with `sudo tee` until you `chmod` it |
| `password-ageing` | yes | [AUTH-9286](https://cisofy.com/lynis/controls/AUTH-9286/), AUTH-9230 | `PASS_MAX_DAYS` set and not 99999, `PASS_MIN_DAYS`≥1, `SHA_CRYPT_MIN_ROUNDS` and `MAX_ROUNDS`≥5000 | sets the missing ones to 365 / 1 / 5000 / 10000 in `/etc/login.defs`; a stricter value of yours is kept | the original lines restored verbatim | new human accounts must change their password yearly and not twice in a day. **Defaults for new accounts only**: existing accounts, root and system accounts are never changed (no `chage`) |
| `cron-permissions` | yes | [FILE-7524](https://cisofy.com/lynis/controls/FILE-7524/) | `/etc/crontab` ≤600 and `/etc/cron.{d,hourly,daily,weekly,monthly}` ≤700 (missing ones are ignored) | `chmod 600` / `chmod 700`, tighten only, previous modes recorded | previous modes restored | other local users cannot read the system crontabs. `cron.allow`/`deny` and `at.allow`/`deny` are left alone (on Debian and Ubuntu `crontab(1)` reads them through its setgid group) |
| `protocols` | yes | [NETW-3200](https://cisofy.com/lynis/controls/NETW-3200/) | each module is blocked or loaded right now | `install <m> /bin/false` and `blacklist <m>` for `dccp sctp rds tipc` in `/etc/modprobe.d/nebulactrl-protocols.conf` | file deleted | software that really uses SCTP, RDS or TIPC stops working. A module loaded right now is left alone |
| `firewire` | yes | [STRG-1846](https://cisofy.com/lynis/controls/STRG-1846/) | as above | `firewire-core firewire-ohci ohci1394` | file deleted | none unless a FireWire device is attached |
| `filesystems` | yes | FILE-6430 | as above | `cramfs freevxfs jffs2 hfs hfsplus`. **`squashfs` (snap) and `udf` (cloud config drives) are deliberately not blocked.** | file deleted | a host that mounts one of those filesystems can no longer |
| `legal-banner` | yes | [BANN-7126](https://cisofy.com/lynis/controls/BANN-7126/), [BANN-7130](https://cisofy.com/lynis/controls/BANN-7130/) | the file has at least five of Lynis's legal key words | puts a two-line banner above the existing text of `/etc/issue` and `/etc/issue.net` | the original restored, unless you edited the file since (then it is left as it is) | cosmetic; shown on the console login. Over SSH only if sshd's `Banner` points at it, which NebulaCtrl never sets |
| `usb-storage` | no | [USB-1000](https://cisofy.com/lynis/controls/USB-1000/) | blocked or loaded | `usb-storage` blocked | file deleted | USB disks, backup drives included, stop working |
| `tmp-mount` | no | [FILE-6310](https://cisofy.com/lynis/controls/FILE-6310/) | `/tmp` is mounted `nosuid,nodev`, or the drop-in exists and `tmp.mount` is enabled | a systemd drop-in `Options=mode=1777,strictatime,nosuid,nodev` for `tmp.mount` and `systemctl enable tmp.mount`; **takes effect at the next boot** | drop-in deleted, unit disabled | `/tmp` becomes RAM-backed. Only offered when systemd is init, the host is not a container, has ≥2 GB RAM and `/tmp` is not already a mount (Debian 13 already is). `noexec` is deliberately not set: package scripts run from `/tmp` |
| `auditd` | no | [ACCT-9628](https://cisofy.com/lynis/controls/ACCT-9628/), ACCT-9630 | `auditd` is active and the rule file exists | installs `auditd`/`audit` if missing, rules that watch `/etc/passwd`, `group`, `shadow`, `gshadow`, `sudoers` and `sudoers.d` | rule file deleted; `auditd` stopped only if NebulaCtrl started it; the package stays | runs a daemon; logs grow (8 MB files, 5 kept). Unavailable in containers |
### How a fix is applied
Each fix is POSIX shell (`internal/lynis/catalog.go`) with four parts: a *precondition* (is it possible here), a *check*, the *apply* and the *rollback*. The same text is shown on the Security tab as the commands to do it by hand.
1. If the check already passes, the outcome is `unchanged` and nothing is touched (and NebulaCtrl will not claim it as its own).
2. If the precondition fails (a container, no systemd, too little memory, no `/etc/login.defs`), the outcome is `skipped` with the reason.
3. Apply runs with `set -e`. If it fails halfway, its rollback runs and the outcome is `failed` with the message.
4. After apply the check runs again. If it does not pass, the fix is rolled back and the outcome is `failed` "did not take effect". A fix is therefore in place or not at all.
5. State is kept in `/var/lib/nebulactrl/hardening/` (previous sysctl values, original `login.defs` lines, previous file modes, backups, a marker per applied fix). **Rollback only undoes fixes it applied**: for a fix that was already compliant it says so and does nothing. A later edit of yours (another line in `login.defs`) survives a rollback.
Sysctl values are only ever tightened: a value that is already stricter than the target (for example `ptrace_scope=3`) is kept, and a key the kernel does not have, or will not let the agent write (some containers), is left alone and not reported as a failure.
### Deliberately not in the catalog
Lynis's KRNL-6000 profile asks for several values NebulaCtrl will not set. The Security tab lists each with the reason:
| Lynis wants | Why not |
|---|---|
| `net.ipv4.conf.all.forwarding=0` (also `ip_forward`) | Docker routes container traffic through it; turning it off cuts every container off the network |
| `net.ipv4.conf.all.rp_filter=1` | strict reverse-path filtering drops legitimate asymmetric traffic and breaks Tailscale, WireGuard policy routing and multi-homed hosts. Loose mode (2) is the safe choice if you want it |
| `kernel.modules_disabled=1` | one-way until reboot, and Docker loads kernel modules |
| `kernel.unprivileged_bpf_disabled=1` | cannot be rolled back without a reboot; the common value 2 already blocks unprivileged BPF |
| SSH-7408 (sshd options), SSH key or root-login settings, `chmod 600 /etc/ssh/sshd_config` | SSH is how you reach the host; a wrong value locks you out |
| Firewall rules (FIRE-*) | specific to what runs on the host |
| Installing PAM password-strength modules, AIDE, rkhunter, compilers' permissions, repartitioning for `/home` and `/var` | each needs a decision or can break a login or a build; the tab explains and gives the commands |
## Who may do what
* **Reading** the Security tab needs the `servers:read` scope.
* **Audit now, Apply safe fixes, Undo and the weekly switch** need `servers:write` **and** the Owner or Admin role (403 otherwise). They are recorded in the audit log: `server.hardening.audit`, `server.hardening.apply`, `server.hardening.rollback` and `server.hardening.update` with who, the server and the fix IDs; when a run finishes, `server.hardening.run` records the index before and after and the outcome of every fix. The weekly run is audited when it finishes (actor: system).
* One run at a time per server (409 `hardening_running`). An offline server gives 409 `server_offline`.
## API
All under `/api/v1/servers/{serverId}/hardening` (see `openapi/openapi.json`):
| Method and path | Purpose |
|---|---|
| `GET /hardening` | The Security view: index (now and at the first audit), Lynis version, every fix with its state, the remaining findings with explanation and commands, recent runs |
| `PUT /hardening` | `{weekly: bool}` |
| `POST /hardening/audit` | `{installLynis?: bool}` → 202 with the run |
| `POST /hardening/apply` | `{fixIds?: []}` (empty: the default set) → 202 with the run |
| `POST /hardening/rollback` | `{fixIds: [...]}` → 202 with the run |
Fix states: `applied` (NebulaCtrl applied it), `compliant` (already in place), `pending`, `unavailable` (with the reason) and `unknown` (no audit yet). After an apply or an undo the agent audits again, so the index on the page is the one after the change.
## Command line on the host
`nebulactrl-agent hardening` works directly on the machine as root, with no control plane:
```sh
sudo nebulactrl-agent hardening list # the fixes and which are default
sudo nebulactrl-agent hardening audit [--install] # Lynis audit, read-only
sudo nebulactrl-agent hardening run # install Lynis, audit, apply defaults, audit again
sudo nebulactrl-agent hardening apply --fix umask,core-dumps
sudo nebulactrl-agent hardening rollback --fix umask # undo
```
Undo what the installer applied before you uninstall NebulaCtrl, because `uninstall.sh` removes the agent binary: `hardening rollback --fix sysctl-kernel,sysctl-fs,sysctl-net,core-dumps,umask,password-ageing,cron-permissions,protocols,firewire,filesystems,legal-banner`.
## Verified on a real host
On a disposable Debian 13 (trixie, arm64) OrbStack machine with `openssh-server`, Lynis 3.1.7 installed by the agent from CISOfy's repository:
| | Hardening index |
|---|---|
| Before | 62 |
| After `hardening run` (11 fixes applied) | **71** |
| After rolling all of them back | 62, and every sysctl value, `login.defs`, `/etc/issue`, `/etc/issue.net` and the cron directory mode were byte-for-byte as before |
SSH was checked before and after: `/etc/ssh/sshd_config` has the same SHA-256, `sshd -T` is identical, and key-based logins as root and as an unprivileged user still worked after the fixes. The findings that disappeared from the report: AUTH-9230, AUTH-9286, AUTH-9328, NETW-3200 (all four protocols), STRG-1846, BANN-7126 and BANN-7130, and KRNL-6000 went from nine differing keys to the five that are deliberately excluded. What stays on that machine (and is explained in the tab): the five excluded sysctl values, SSH-7408, USB-1000 (optional fix), the separate-partition suggestions, the packages and tools Lynis recommends, and FILE-7524 for `/etc/ssh/sshd_config` (mode 644).
Limits of that check: OrbStack machines are containers on a shared VM kernel, so `tmp-mount` and `auditd` correctly reported *unavailable* ("this host is a container") and were not exercised, and the global sysctls were applied to the shared VM kernel only for the minutes of the test (then rolled back and diffed). The helper-container path was exercised against a real Docker daemon with a read-only audit: the helper entered the VM's namespaces with `nsenter`, reported the fix states and left no container behind.
## Data
Three tables: `server_hardening_settings` (weekly audit on/off; no row means on), `server_hardening_state` (the newest report, the state of every fix, the first index, the reboot flag) and `server_hardening_runs` (30 runs per server).
## Sources
* Lynis: [Get started](https://cisofy.com/documentation/lynis/get-started/) (installation, `--cronjob`, report files), [CISOfy package repository](https://packages.cisofy.com/community/), [control pages](https://cisofy.com/lynis/controls/) for each test ID above.
* Kernel settings: [kernel.txt](https://www.kernel.org/doc/Documentation/sysctl/kernel.txt), [fs.txt](https://www.kernel.org/doc/Documentation/sysctl/fs.txt) (the pages Lynis's own profile links to).
* What each Lynis test accepts (for example the five legal key words of BANN-7126, the exact `install <module> /bin/false` lines of NETW-3200, STRG-1846 and USB-1000, the `UMASK` values of AUTH-9328) was read from the test sources in `/usr/share/lynis/include/` of the installed Lynis 3.1.7.
---
# Docker cleanup
Source: https://nebulactrl.com/docs/operations/docker-cleanup
Images, build cache and stopped containers pile up on a Docker host. NebulaCtrl cleans each server on a schedule, so a disk does not fill up with last month's builds, and never at the cost of a rollback.
Each server has a **Docker cleanup** card under *Settings* on its page. The schedule is **on by default**.
## When it runs
| Trigger | When |
|---|---|
| Scheduled | By the server's **schedule**, by default daily at **04:00 UTC**, for every online server with cleanup on. A server that enrolled after a slot, or whose schedule was just changed, waits for the next one: a new schedule never catches up on earlier slots. A server that was offline at a slot is cleaned as soon as it is back online. |
| Disk trigger | When the root disk reaches the server's *disk trigger* (by default its *disk alert threshold*, 80 %) and no cleanup started on it in the last 6 hours. |
| By hand | *Run cleanup now*, `POST /api/v1/servers/{id}/cleanup/runs`. Works with the schedule off. |
Only one cleanup runs on a server at a time. An offline server is never sent one: the command would be stale by the time it connected.
## What it removes
| Step | Removed | Notes |
|---|---|---|
| Stopped containers | Containers that are **not** managed by NebulaCtrl and have been stopped for at least an hour | **Only when the switch is on. Off by default.** Never removes a container's volumes. Helper containers NebulaCtrl starts itself (`nebulactrl.helper` label, `nebulactrl-hostshell-*`) are never touched. |
| Dangling images | Images with no tag that no container uses | No age limit. |
| Unused images | Tagged images that no container uses, older than the **unused image age** (default 72 hours) | Age is the later of the image's build date and the time it was last pulled or tagged, so an old image you pulled an hour ago is not old. |
| Build cache | Cache older than the **build cache age** (default 7 days) | The daemon's and, when it exists, the managed BuildKit builder's. |
| Networks | Empty NebulaCtrl environment and shared networks | The same networks the agent already sweeps every minute. |
## What it never removes
- **Volumes.** A scheduled, emergency or manual cleanup never removes a volume, and never removes a container together with its volumes. Removing unused volumes is a separate action (below).
- **Rollback images.** For every service on the server the control plane sends the agent the list of images to keep: the **current release**, the releases of any **deployment in flight**, and the **newest `imageRetention` succeeded releases** (the ones *Roll back to this release* can pick; default 5). The agent never deletes an image in that list, whatever its age or whether a container uses it. The list matches git builds by local image ID and image releases by reference and digest (a release built on a [build server](https://nebulactrl.com/docs/operations/build-servers) by its registry `repo@digest`, plus the release it was built from), so a tag that moved on does not orphan an older release.
- **Pinned infrastructure images.** Traefik, the pages server, cloudflared, Tailscale, BuildKit and the Railpack frontend, and any helper image registered with `agentprotocol.PinImage`. The agent knows these itself and keeps them even when nothing uses them right now. On a build server this includes the per-project BuildKit builders' image.
- **Images in use** by any container, running or stopped.
A cleanup command without the keep list is refused by the agent. If the list could not be computed the command is not sent.
## Build servers
A [build server](https://nebulactrl.com/docs/operations/build-servers) pushes what it builds to a registry (`docker buildx build --push`) and nothing is loaded into its Docker image store, so there are no release images on it and no keep list to compute: a **Builds** server's cleanup has nothing to protect. What it accumulates is:
- **BuildKit cache**, one builder per project (`nebulactrl-buildkit-<project8>`). Each builder is created with its own garbage-collection limits (a reserved size and a maximum size), so BuildKit evicts the oldest records by itself when a project's cache grows. The daily age-based prune above stays as a backstop and prunes every builder.
- The pinned BuildKit and Railpack frontend images, and any Railpack builder images pulled for a build.
- Clone workdirs, which are removed when the build ends.
Two guards protect running builds:
- **No prune while building.** The agent holds a lock for the length of every build and never prunes build cache underneath a running build. (Pruning mid-build kills it at "importing to docker", a known failure of other platforms.)
- **Disk preflight.** A build refuses to start with too little free space, runs an emergency cache prune and retries once; otherwise the build fails with BuildKit's error. The disk alert threshold and the disk trigger above apply to a build server like any server.
On a **Both** server the usual rules apply for the workloads, and the cache of the build side is pruned as above.
The images of the **managed registry** are not cleaned up here: the control plane deletes expired digests through the registry's API and collects blobs in a weekly window ([build servers: rollback and retention](https://nebulactrl.com/docs/operations/build-servers#rollback-and-retention)).
## Results
Every run is recorded (the last 30 per server) with the trigger, the space it freed, the number of items removed and what each step did, including how many images it kept for releases. Open a run on the card to see the steps and the first removed names.
- Freed space is the difference in Docker's image storage before and after (shared layers count once) plus the build cache the prune reported. Containers and networks free little and are counted as items only.
- A step that fails (the daemon refuses an image, buildx is gone) is reported on its row; the other steps still run, and the run is marked *Failed* while still showing what it freed.
- An image that became used between listing and removal is not a failure.
- Each finished run is written to the audit log as a system action (`server.cleanup`); starting one by hand, changing the settings and the volume action are audited with the person (`server.cleanup.start`, `server.cleanup.update`, `server.cleanup.volumes`).
- A run that **freed more than 1 GB, or failed**, sends a `server.cleanup` notification (see [Notifications](https://nebulactrl.com/docs/operations/notifications)): info for a large reclaim, error for a failure. Smaller successful runs are silent.
## Removing unused volumes
*Remove unused volumes…* (`POST /api/v1/servers/{id}/cleanup/volumes`) is the only way a cleanup deletes a volume. It asks you to type the server's name and deletes, with their data, the volumes that **no container (running or stopped) mounts** and that **NebulaCtrl does not manage**: anonymous volumes and volumes created outside NebulaCtrl. Volumes of your services (`nc-vol-*`, anything labeled as managed, anything the control plane lists as a service volume, including the volumes of deleted services) are kept. It runs nothing else and the schedule never starts it. It cannot be undone.
## Settings and API
| Setting | Field | Default | |
|---|---|---|---|
| Scheduled cleanup | `enabled` | on | Turns the scheduled and the disk-trigger runs on or off. *Run cleanup now* always works. |
| Schedule | `cron` | `0 4 * * *` | A 5-field cron expression (or `@daily`, `@weekly`, ...) evaluated in **UTC**. Runs must be at least 15 minutes apart (`422 validation_failed` otherwise). The page offers *Daily at 04:00*, *Every 6 hours* and *Sundays at 03:00* as shortcuts. |
| Unused image age | `unusedImageMinAgeHours` | 72 | 1 to 8760 hours. Rollback images and pinned infrastructure images are kept whatever the age. |
| Build cache age | `buildCacheMinAgeDays` | 7 | 1 to 365 days. |
| Disk trigger | `diskTriggerPercent` | follows the disk alert threshold | 50 to 99 % of the root disk. With a value set, the heartbeat's disk usage decides (not the alert flag). At most one emergency run every 6 hours. |
| Stopped containers | `removeStoppedUnmanaged` | off | Also remove stopped containers NebulaCtrl does not manage. |
`GET` and `PUT /api/v1/servers/{id}/cleanup` read and change them (the `servers:write` scope for `PUT` and the runs). `PUT` replaces every setting: `0`, an empty string or an omitted field restores that default. The change is audited (`server.cleanup.update`, with the values). The server's own *disk alert threshold* is a separate setting.
## Limits
- The agent's own daily build-cache prune is gone: the cleanup does that step now. With the schedule off, build cache is only pruned by a manual run.
- The keep list is computed when the command is queued. A deployment that starts afterwards is protected by its age (an unused image is only removed after its age limit, 72 hours by default; a dangling one is removed at once, so avoid running a cleanup by hand while you redeploy the same tag).
- A server whose agent is older than this feature (`docker.cleanup` capability) shows *This agent is too old* and is skipped.
---
# Server patching
Source: https://nebulactrl.com/docs/operations/server-patching
A Docker host is also an operating system that needs its security updates. NebulaCtrl shows, for every server, how many package updates are waiting (and how many of them are security updates), whether the host asks for a reboot, and gives the **instance admin** two actions: **Apply updates** and **Reboot**.
Nothing is installed and nothing reboots without a person asking. The installer's one-time setup of unattended security updates (`deploy/install/install.sh`) is separate and keeps working; this page is about seeing the state and acting on it from the dashboard.
## What you see
| Where | What |
|---|---|
| Servers list, every card | `3 updates (1 security)` with a status icon, or `Up to date`; a **Reboot required** badge when the host asks for one. A server that has not been checked yet says nothing. |
| Server page, header | The same two facts beside the status. |
| Server page → **Updates** tab | The counts, the time of the last check, the package manager, the pending packages (security first, risky ones with a warning), **Check for updates**, **Apply updates**, **Reboot**, and the history of runs with the upgrade's log. The tab's badge reads `Reboot` or `N security` when something needs you. |
| Notifications | `server.reboot_required` (see below). |
The numbers are the ones the host's package manager reports after refreshing its package lists. They are only as fresh as the last check: the tab shows when it ran and marks a check older than 12 hours as out of date.
## How it is checked
The agent runs a read-only script on the host and reports the result. Everything runs on the host the way [host hardening](https://nebulactrl.com/docs/security/host-hardening) does: an agent installed on the host runs the script directly as root; an agent that runs in a container (the dev stack) starts a short-lived privileged helper container from its own image whose command enters PID 1's namespaces with `nsenter` (the host shell's mechanism). The agent advertises the capability `host.patching`; a server whose agent is older gets `422 agent_too_old`.
| Package manager | Pending updates | Security updates | Reboot required |
|---|---|---|---|
| **apt** (Debian, Ubuntu) | `apt-get update`, then the `Inst` lines of `apt-get -s upgrade` (what `apt-get upgrade` would do; packages that would need new dependencies are *kept back* and not counted) | An update whose origin is a `-security` suite (`jammy-security`, `stable-security`, ESM `…-infra-security`) | `/run/reboot-required` exists; the packages that asked are in `/run/reboot-required.pkgs` |
| **dnf** (RHEL family) | `dnf check-update --refresh` | The packages in `dnf check-update --security` | `needs-restarting -r` says so (exit 1); without it, a newer installed `kernel-core` than the running kernel |
A host with neither manager is reported as *not managed here*. The check also reads the kernel's boot id (`/proc/sys/kernel/random/boot_id`), which is how a reboot is proven (below). If the package lists cannot be refreshed (no network), the numbers are the last ones the host downloaded and the tab says so.
**When it runs.** A control-plane sweep looks every 30 minutes and checks each **online** server that has had no run (of any kind, with any outcome) in the last **6 hours**, so a server is checked at most every 6 hours and a server whose checks fail is retried at the same pace, not on every sweep. **Check for updates** starts one by hand (it needs the Owner or Admin role and the `servers:write` scope; it changes no package). Only one run per server is in flight at a time (`409 patching_running`); an offline server gets `409 server_offline`.
## Apply updates
**Apply updates** upgrades the packages on the host as root: `apt-get upgrade` (never `dist-upgrade`, so nothing is removed and no new package is pulled in) or `dnf upgrade`, with `Dpkg::Options::=--force-confold` so configuration files you changed are kept. **Security only** upgrades just the packages that have a security update (`apt-get install --only-upgrade <those packages>`, `dnf upgrade --security`). It never reboots by itself.
Who and how:
* **Instance admin only** (the first user, or an `admin`-scoped token acting for them), with the `servers:write` scope. Owners who are not the instance admin get `403`. Upgrading packages and rebooting are root operations on the host; this is the same rule as the *Unrestricted* switch.
* **A check first.** The plan comes from the last check; before one has completed the request returns `409 no_check_yet`. Nothing to upgrade is `422 no_updates`.
* **A plan.** `GET /api/v1/servers/{id}/patches/plan?securityOnly=` returns what would be upgraded, the packages that deserve a warning and why (a new kernel, glibc, systemd and NVIDIA drivers need a reboot; Docker/containerd/runc restart the Docker daemon, which stops containers unless Docker's live-restore is on; openssh-server, the bootloader, a database server installed on the host, Tailscale/cloudflared), whether a reboot will probably be needed, how many services run on the server, and the production services among them. The dashboard shows it in the confirmation.
* **A confirmation.** The request must carry `confirm`, the server's name (`422 confirmation_mismatch`). The dashboard asks you to type it.
* **A production guard.** When the server runs services of a **production environment**, or is the host of the control plane itself, the request must also set `acknowledgeProduction` (`422 production_ack_required` otherwise). A token or MCP client that must obtain approvals for production changes (see [production approval](https://nebulactrl.com/docs/reference/mcp#production-approval)) additionally needs the human approval for each production environment on the server, and the call is refused when approvals are unavailable. The audit row names the production services affected.
* **Audited.** `server.patches.apply` records who, which server, whether security only, how many updates and the production services; `server.patches.run` records the outcome (system action).
The upgrade runs **detached on the host** (`systemd-run`, falling back to `nohup setsid`) and writes its output and exit code under `/var/lib/nebulactrl-patching/` (files older than 7 days are removed on the next run). The agent polls for the result. This matters because upgrading Docker restarts the daemon, which restarts the helper container and, for an agent that runs in a container, the agent itself: the upgrade finishes anyway, and the agent attaches to it again when it is back (a start request for a run id that already started attaches instead of starting another). Host contact lost for up to 15 minutes is tolerated; a run whose process vanished without an exit code (the host rebooted mid-upgrade) is reported as failed, and running it again resumes where apt or dnf stopped.
When it ends, the agent checks again, so the new counts and the reboot flag are stored with the run. The run keeps the **last 48 KiB of the package manager's output** as its log (shown in the history; `GET /api/v1/servers/{id}/patches/runs/{runId}`). The log is delivered when the upgrade ends, not streamed while it runs. A non-zero exit status marks the run `failed` with the log and the new state.
## Reboot
**Reboot** restarts the whole server. Every service on it goes down until the host boots and Docker starts them again (containers with the `unless-stopped` policy that NebulaCtrl creates come back by themselves). It has the same gates as Apply updates: instance admin, the server's name typed as `confirm`, `acknowledgeProduction` when production or the control plane runs there, the production approval for guarded principals, and an audit row (`server.patches.reboot`, then `server.patches.run`). The dashboard's confirmation warns that services go down and lists the production services.
The agent schedules the reboot a few seconds ahead (`systemd-run --on-active=5s systemctl reboot`, else `shutdown -r +1`) so it can still answer, and reports the boot id it will replace. The run is then **`rebooting`**: NebulaCtrl waits for the agent to reconnect. Once the server is online again the control plane asks it for a check and compares the boot id; **a different boot id finishes the run as `succeeded`**. If the agent reconnects before the host went down, the boot id is the same and the wait continues. If the server does not come back within **20 minutes**, the run fails with that message (check the console; services stay stopped until the host boots). A server revoked meanwhile fails the run too.
While the host is down the usual `server.offline` and `server.online` notifications fire; they are not suppressed for a planned reboot. On the control-plane host, the dashboard is unavailable too until it is back.
## Notification: `server.reboot_required`
Sent once each time a server **starts needing a reboot**: the first check that sees the flag, or an upgrade that sets it. It is not repeated while the flag stays set, and a new streak after a reboot is announced again. Severity warning, group Servers, on by default for a new channel. Server events belong to no project, so only workspace-wide channels receive them (see [Notifications](https://nebulactrl.com/docs/operations/notifications)). There is no notification for pending updates themselves.
## API
All under `/api/v1/servers/{serverId}`; the reads need `servers:read`.
| Operation | Endpoint | Notes |
|---|---|---|
| `getServerPatches` | `GET /patches` | Counts, packages, reboot-required, last check, the run in flight, production services on the server, recent runs. The server list and `GET /servers/{id}` carry a `patches` summary. |
| `planServerPatches` | `GET /patches/plan?securityOnly=` | What an upgrade would do; changes nothing. |
| `getServerPatchRun` | `GET /patches/runs/{runId}` | One run with its log. |
| `checkServerPatches` | `POST /patches/check` | `202`; Owner or Admin. |
| `applyServerPatches` | `POST /patches/apply` | `202`; instance admin; body `{confirm, securityOnly?, acknowledgeProduction?}`. |
| `rebootServer` | `POST /patches/reboot` | `202`; instance admin; body `{confirm, acknowledgeProduction?}`. |
Errors: `409 server_offline`, `409 patching_running`, `409 no_check_yet`, `422 agent_too_old`, `422 server_revoked`, `422 patching_unsupported`, `422 no_updates`, `422 confirmation_mismatch`, `422 production_ack_required`, `409 approval_required`. The three writes accept an `Idempotency-Key`.
**MCP.** The reads and *check* are available as operations (`call_operation`). `applyServerPatches` and `rebootServer` are **not exposed over MCP**: they change the host as root (`host-root`, see [MCP](https://nebulactrl.com/docs/reference/mcp)).
## Limits
* **apt and dnf only.** zypper, pacman, apk and the like report *not managed here*.
* **`apt-get upgrade` semantics.** Packages held back because they need new dependencies or removals are neither counted nor upgraded; do those by hand. Third-party repositories are included as the host has them configured.
* **No automatic upgrades or reboots.** Nothing is installed or rebooted without a person asking. (Use the installer's unattended security updates for that; this page then tells you when a reboot is pending.)
* **A reboot is not drained.** The server's services go down; move them first with [draining](https://nebulactrl.com/docs/operations/draining-servers) if that matters.
* **No live log.** The upgrade's output is stored when it ends (the last 48 KiB).
* **Checks need an online agent** that can reach the host (a helper container or the host itself) and, for the refresh, the distribution's mirrors.
---
# Upgrades: agents and the control plane
Source: https://nebulactrl.com/docs/operations/upgrades
Two things happen when NebulaCtrl upgrades:
- **Every agent catches up on its own.** An agent that is behind the control plane downloads the new binary from the control plane it is connected to, checks it, replaces itself and restarts. If the new binary does not start, it puts the old one back.
- **The control plane backs up its database first.** Before it migrates its schema, it takes a control-plane backup and refuses to go on when that backup failed.
Upgrading the control plane itself is still an installer re-run ([upgrade guide](https://nebulactrl.com/docs/installing/upgrade)). It is not done from the dashboard, because that would need the Docker socket on the control plane.
## Agent upgrades
### What happens
The control plane runs version `V` and serves the matching agent binary at `/agent/v1/download/<amd64|arm64>`. For an agent on an older version it sends one `agent_upgrade` command that carries `V` and the **SHA-256** of that binary. The command travels over the agent's authenticated connection and holds no secret.
The agent then:
1. downloads the binary from the control plane URL it was started with (over TLS; plain HTTP only where the agent already accepts it) into a temporary file next to its own binary, hashing while it reads;
2. compares the hash with the one in the command; a mismatch deletes the file and fails the upgrade, nothing changed;
3. runs `<new> --version` and requires `V` (catches a wrong architecture or a corrupt file);
4. keeps the running binary as `<binary>.prev` (a hard link, or a copy) and writes `upgrade.json` in its state directory;
5. replaces its binary with one atomic `rename`: the path holds a whole binary at every moment;
6. reports success, waits until its **other running commands are done** (a build, a backup, a deployment; at most 45 minutes), then exits so that systemd (`Restart=always`) or the container's restart policy starts the new binary.
Your services and the proxy keep running; only the agent restarts. The control plane marks the upgrade **succeeded** when the agent reconnects with the target version.
### If the new binary does not work
The new binary starts "on probation". It is kept once it has stayed connected to the control plane for 20 seconds. It is rolled back (`<binary>.prev` is renamed over it and the agent exits to restart) when:
- it started 3 times without staying connected (crash loop), or
- it was not connected within 5 minutes of its start.
The agent remembers the rolled-back version and does not install it again by itself for 24 hours; the control plane does not retry it automatically either. The server's page shows the reason (**Last upgrade** → *The agent rolled the upgrade back: …*). **Retry anyway** on the server page (or `force` in the API) installs the version again.
After a successful upgrade `<binary>.prev` stays as one generation for a manual rollback: `mv /usr/local/bin/nebulactrl-agent.prev /usr/local/bin/nebulactrl-agent && systemctl restart nebulactrl-agent`.
### Automatic and staged
- **Automatic upgrade** (Servers page, on by default; instance administrator only). When an agent that is **provably behind** connects (both versions are release versions and the agent's is lower), it is upgraded right away. Each server upgrades independently when it connects, which is also how a control-plane upgrade spreads: agents reconnect over a few minutes. A development build (`dev`, `1.4.0-3-gabc1234`) or an agent that is newer than the control plane is never upgraded automatically.
- **Upgrade agents** (button on the Servers page; Owners and Admins). Upgrades the servers that can be upgraded now **one at a time**, in name order: the next server starts when the previous agent has reconnected with the new version, and the **first failure stops the rollout**, leaving the remaining servers untouched. Needs the `servers:write` scope. A single server has its own **Upgrade agent** button.
Both paths are audited: `server.agent.upgrade` (a person), `server.agent.upgrade.auto`, `server.agent.upgraded`, `server.agent.upgrade.failed`, `server.agent.upgrade.rollout`, `server.agent.upgrade.halted` and `instance.agent_auto_upgrade.update`.
### Version skew
Every server card shows how its agent compares with the control plane: *Behind 1.5.0*, *Newer than the control plane*, *Version not comparable* (a development build), or nothing when they match. The same data is `GET /api/v1/instance/upgrade`.
| Skew | Meaning | Upgrade |
|---|---|---|
| `current` | same version | none |
| `behind` | the agent is older | automatic, or on request |
| `ahead` | the agent is newer | never; upgrade the control plane first |
| `unknown` | a version is not a release version | on request only |
### When an agent cannot upgrade itself
The server page says why. The common reasons:
| Reason | What to do |
|---|---|
| The agent predates self-upgrade | Re-run the agent installer on the host once ([upgrade guide](https://nebulactrl.com/docs/installing/upgrade#upgrade-other-servers-by-hand)); later upgrades are automatic. |
| The directory of the agent binary is not writable (a read-only container image) | Run the agent with a writable binary directory, or upgrade by re-creating the container from the new image. |
| The server is offline | Upgrades only run on a connected agent. |
| This control plane ships no binary for the server's architecture | Only `amd64` and `arm64` Linux are built. Development control planes have no binaries directory. |
An agent that runs in a container upgrades the binary inside the container's writable layer; the container restart policy restarts it. A container that is recreated from its image starts the image's version again and is upgraded again at the next connect, so for containers also bump the image tag.
### Files
| File | Holds |
|---|---|
| `<binary>.prev` | the previous binary |
| `<state-dir>/upgrade.json` | the last upgrade: states `swapped`, `committed`, `rolled_back`, `abandoned`, the start counter, the version not to retry |
The agent's last upgrade also appears in the host facts (`facts.agentUpgrade`) so the control plane learns about a rollback it was not connected for.
## Backup before a control-plane upgrade
The control plane takes a **control-plane backup** (the [instance backup](https://nebulactrl.com/docs/operations/backup-restore): the same destination, dump format and manifest, trigger *Before upgrade*) before it migrates its database in two situations:
- **The installer is re-run.** Before it changes anything, the installer runs `nebulactrl-server pre-upgrade-backup` inside the running control plane. If that command does not exist (a control plane older than this feature), it dumps the database with `pg_dump` to `/var/backups/nebulactrl/pre-upgrade-<time>.dump` (mode 0600) instead.
- **The server starts with another version than the one that last ran on this database.** The check runs at startup, **before** the migrations: it compares its version with the one stored in the database. A development build (a version that is not a release such as `1.5.0`), a new database and an unchanged version skip it. If the installer already backed up for exactly this version in the last 24 hours, the server does not back up a second time.
What counts as failure:
| Outcome | What happens |
|---|---|
| Backed up | The upgrade goes on. The run shows in Settings → Instance → History as *Before upgrade*. |
| **The backup failed** (a destination is configured, but pg_dump or the upload failed) | **The upgrade stops.** The installer exits before it changes anything; the server exits at startup without migrating. The old control plane keeps running. |
| No destination is configured (no S3 bucket, no `NEBULACTRL_BACKUP_DIR`) | The upgrade goes on with a warning, and the dashboard says *No backup before the upgrade*. The installer always configures a local destination. |
| Forced | You asked to go on without it: `sudo bash install.sh --force-upgrade`, or `NEBULACTRL_FORCE_UPGRADE=true` for a server start. The outcome is recorded as *forced*. |
You can take it yourself at any time: `cd /opt/nebulactrl && docker compose exec nebulactrl nebulactrl-server pre-upgrade-backup` (exit status 0 backed up or nothing configured, 3 the backup failed, `--force` turns a failure into a warning).
The outcome of the last pre-upgrade backup is shown in Settings → Instance → Control-plane backup, and returned by `GET /api/v1/instance/upgrade` (`preUpgradeBackup`). Restoring it is the same as any control-plane backup: [restore](https://nebulactrl.com/docs/operations/backup-restore#restore).
The backup does not contain `NEBULACTRL_SECRET_KEY`. Keep it apart from the dumps.
A large database makes the startup wait for the dump (limit 30 minutes). If the container's health check gives up earlier, take the backup first with the installer, which has no such limit: its result is reused.
## API
| Operation | Endpoint | Needs |
|---|---|---|
| `getInstanceUpgrade` | `GET /api/v1/instance/upgrade` | `servers:read` |
| `updateInstanceUpgrade` | `PATCH /api/v1/instance/upgrade` `{agentAutoUpgrade}` | `servers:write`, instance admin |
| `upgradeServerAgent` | `POST /api/v1/servers/{serverId}/agent-upgrade` `{force?}` | `servers:write`, Owner or Admin |
| `upgradeServerAgents` | `POST /api/v1/servers/agent-upgrades` `{serverIds?, force?}` | `servers:write`, Owner or Admin |
The upgrade endpoints answer `202`. Errors: `409 server_offline`, `agent_current`, `agent_newer`, `upgrade_in_progress`, `nothing_to_upgrade`; `422 agent_predates_upgrade`, `no_agent_binary`. They are not available over MCP: replacing the binary of a root process is for people.
## Settings
| Setting | Where | Default |
|---|---|---|
| Upgrade agents automatically | Servers page | on |
| `NEBULACTRL_FORCE_UPGRADE=true` | control-plane environment | off: a failed pre-upgrade backup stops the start |
| `--force-upgrade` | installer | off |
---
# Host shell
Source: https://nebulactrl.com/docs/security/host-shell
An interactive shell **on the server itself**, from the browser (server page →
**Host shell**). It is not the [web terminal](https://nebulactrl.com/docs/operations/terminal), which
runs inside one service's container: the host shell sits outside every
container, as the agent's user (root on a normal install). From it you can read
every volume, every container's environment and the Docker daemon's socket.
Treat it like SSH as root.
It is **off by default and opt-in per server**.
```
browser ⇄ WSS ⇄ control plane ⇄ agent WebSocket ⇄ a pty on the host (or a helper container)
```
## What has to be true before a shell opens
| Check | Refusal |
|---|---|
| An Owner or Admin **switched it on** for this server (**Server → Host shell → Allow a shell on this server**; audited as `server.shell.enable`) | `403 host_shell_disabled` |
| The person is an **Owner or Admin** of the team, with a signed-in browser session. Developers and Viewers never get it; an **API token or MCP principal never gets it**, whatever its scopes | `403 forbidden` / `403 session_required` |
| Their account has **two-factor authentication** (Settings → Security) | `403 two_factor_required` |
| The server is online | `server_offline` |
| **Production**: a server that runs any service of a production environment needs the guarded confirm in the UI (hold to confirm) and `confirmProduction: true` in the open message | `confirmation_required` |
| At most **2** open host shells per person | `too_many_sessions` |
The first three are answered over plain HTTP before the WebSocket upgrade, so a
script sees the status and the problem code. The handshake also enforces the
`Origin`, like the service terminal: another website cannot open a shell with
your session cookie.
Switching the setting **off** ends the sessions that are open on that server and
refuses new ones.
## What is recorded
| Event | Audit action | Metadata |
|---|---|---|
| the setting changes | `server.shell.enable`, `server.shell.disable` | who (the audit actor), server |
| a session opens | `server.shell.open` | session ID, server, production flag, how it runs (`host` or the helper container's name) |
| it ends | `server.shell.close` | duration in seconds, why it ended (`exit`, `client_closed`, `idle`, `agent_disconnected`, `host_shell_disabled`, `error`), exit code |
**Keystrokes and output are never recorded, logged or stored**: not in the audit
log, not on the control plane, not on the agent. The open entry is written before
the shell is handed over: if the audit log is unavailable, the shell is killed and
not opened. A session with **no input for 15 minutes** is closed (a warning
appears one minute before).
## How the agent runs it
- **An agent installed on the host** (the supported production setup,
`install-agent.sh` with systemd) spawns the shell itself on a pseudo
terminal: `/bin/bash -l` when the host has bash, else `/bin/sh -l`, with
`TERM=xterm-256color`, in the agent user's home directory. The shell gets a
minimal environment (`TERM`, `HOME`, `USER`, `PATH`, `SHELL`, locale). **The
agent's own variables, the enrolment token among them, are not passed on.**
Closing the session hangs up the shell's process group (SIGHUP, then SIGKILL
after 2 seconds); a process that detached itself with `setsid` or `nohup` keeps
running, as it would after an SSH logout.
- **An agent that runs in a container** (the development stack, a compose setup)
cannot see the host, so it starts a short-lived **helper container** from its
own image: `--privileged --pid=host`, running
`nsenter -t 1 -m -u -i -n -p -- /bin/sh -l`, which enters the namespaces of the
host's PID 1. The helper is named `nebulactrl-hostshell-<session8>`, labelled
`nebulactrl.managed=hostshell`, and **always removed**:
- when the session ends (the shell exits, the person closes it, an admin
switches the setting off, the connection drops or the session idles out);
- if the agent itself dies mid-session, the Docker daemon removes it once its
input is gone (`--rm` semantics with stdin closed on disconnect), and the
agent's **orphan sweep** removes whatever is left on its next reconcile and
every minute (a helper whose session no agent process knows).
The agent's image must contain `nsenter` (util-linux or BusyBox). Docker's own
cleanup never touches these containers.
## Limits and non-goals
- No recording or replay of sessions, no sharing of a live session, no file
transfer (use the [file browser](https://nebulactrl.com/docs/operations/terminal) for container
files).
- The shell is not a sandbox: the person can do anything root can do on the
machine, including stopping the agent and Docker.
- The team boundary applies: another team's servers are a 404. The host shell
does not reach the control plane's database or secret key, unless the control
plane happens to run on the same machine (then it does).
See also: [web terminal](https://nebulactrl.com/docs/operations/terminal), [login methods](https://nebulactrl.com/docs/security/login-methods)
(two-factor authentication), [threat model](https://nebulactrl.com/docs/security/threat-model).
---
# Monitoring and retention
Source: https://nebulactrl.com/docs/operations/monitoring
## Health endpoints
| Endpoint | Meaning |
|---|---|
| `GET /healthz` | process is up (always `{"status":"ok"}`) |
| `GET /readyz` | process is up **and the database answers** (`200 {"status":"ready"}`, otherwise 503) |
| `nebulactrl-server healthcheck` | the same readiness probe run inside the container (used by the image's `HEALTHCHECK` and Compose, so the check needs no curl). Reads `NEBULACTRL_LISTEN`; `--url` overrides. |
Use `/readyz` for load balancers and uptime monitors.
## Logs
The control plane writes **JSON, one object per line, to stdout**: `docker compose logs -f nebulactrl`. Level via `NEBULACTRL_LOG_LEVEL`. Every HTTP request is logged with method, path, status, duration and a request id. Lines produced while handling a deployment carry **`correlation_id`** = the deployment ID, so one deployment is traceable end to end:
```sh
docker compose logs nebulactrl | jq -c 'select(.correlation_id=="<deployment-uuid>")'
```
Secrets are never logged. Values of secret variables (6+ characters) are replaced with `••••••` before build/deploy/runtime log lines are stored or streamed.
Agent logs: `journalctl -u nebulactrl-agent -f` on the host (also JSON).
### In the UI
| What | Where | Retention |
|---|---|---|
| Build and deploy output | Deployment row → logs | stored per deployment, **capped at 10,000 lines** (a final marker line says so) |
| Runtime logs | Service → Logs, Environment → Observability | **stored on each server by its agent**, 7 days and 500 MB per service by default (configurable per server); **Live** streams, **History** searches the stored lines. Not available while the server is offline. [Log history and drains](https://nebulactrl.com/docs/operations/logs) |
| Audit log | Settings → Audit log (`GET /api/v1/audit-events`) | kept indefinitely (not pruned) |
## Resource metrics
Collected from agent heartbeats (host: CPU, memory, disk) and container stats (per service: CPU, memory), sampled about every 10 seconds, the way Coolify's Sentinel does. Container memory is reported as usage minus inactive file cache.
| Setting (per server) | Default | Range | Effect |
|---|---|---|---|
| `metricsRetentionDays` | 7 | 1–90 | Samples older than this are pruned by an hourly job. A `30d` chart range only shows the history that is retained ("History kept for N days"). |
| `diskAlertThresholdPercent` | 80 | 50–99 | Root-filesystem usage above this shows a warning on the server card and server page. |
Chart ranges: `5m`, `10m`, `30m`, `1h`, `6h`, `12h`, `24h`, `7d`, `30d`; the response is bucketed to at most about 300 points, and the short ranges refresh every 5 s, the others every 15 s. Older data is downsampled into coarser buckets (the API reports `stepSeconds`).
Change the settings on the server page, or with `PATCH /api/v1/servers/{serverId}` (scope `servers:write`).
## Traffic analytics
Request analytics (requests, bandwidth, status classes, p95 latency, unique visitors, top paths, referrers, browsers, devices) are collected **from the agent-managed Traefik's access log, on each host, as aggregates only**. Raw access logs are never shipped or stored by the control plane. Unique visitors are hashed with a salt that rotates daily; raw IP addresses are never sent.
| Setting (per server) | Default | Effect |
|---|---|---|
| `trafficAnalyticsEnabled` | on | Turn collection off per server. Like Coolify, this recreates the proxy container (a brief proxy restart), because the access log is a static Traefik setting. |
| `trafficHourlyRetentionDays` | 7 | Hourly buckets (fine-grained charts) kept this long. |
| `trafficDailyRetentionDays` | 90 | Daily roll-ups kept this long. |
Views: service inspector → Analytics, server page → Analytics, environment → Observability. Each has a "Where requests come from" map (country, optionally city); see [Traffic analytics and request origins](https://nebulactrl.com/docs/operations/traffic-analytics) for the GeoIP databases, licences and settings. Runtime logs can be shipped to Loki, OTLP, syslog or an HTTP intake: [log drains](https://nebulactrl.com/docs/operations/logs#log-drains).
## Liveness semantics
- A server is **offline** after 30 seconds without a heartbeat (the agent sends one every 10 s). The UI marks observations older than 30 s as *stale* instead of showing them as current.
- Each state change in the UI also appears as text or a glyph, not only as animation.
- Live updates use Server-Sent Events. If the connection drops the UI shows "Live updates paused — retrying" and falls back to polling every 15 s.
## Capacity planning
- Metrics volume is roughly (servers + running containers) samples per 10 s per retention window; the default 7 days is small. Raising retention to 90 days multiplies it by ~13.
- The agent keeps the 5 newest built images per service (`imageRetention`) and prunes the builder cache older than 7 days daily. Watch the disk gauge on the server card.
---
# Logs and log drains
Source: https://nebulactrl.com/docs/operations/logs
Runtime logs (what your containers write to stdout and stderr) are **stored on the server that runs them** and can be **shipped from that server** to Loki, OpenTelemetry, syslog or any HTTP intake. The control plane stores no runtime log line: it asks the server's agent.
## Log history
Every agent captures the output of every managed service container, continuously. A viewer is not needed: closing the Logs tab discards nothing. Build and deploy output are separate: they are stored per deployment (10,000 lines), see [deployments](https://nebulactrl.com/docs/operations/deployments).
| | |
|---|---|
| Where | `<state-dir>/logs/<serviceId>/` on the server (`/var/lib/nebulactrl-agent/logs` by default), compressed, nowhere else |
| How long | **7 days** by default (1–90) |
| How much | at most **500 MB per service** and **5 GB per server**, compressed on disk; the **oldest lines go first** (whole segments of at most 8 MB / 6 hours) |
| What | stdout and stderr of replicas of services; not task containers, helper containers or the proxy |
| Configurable | **per server**: server page → Settings → Log history, or `PUT /api/v1/servers/{serverId}/log-settings` (scope `servers:write`) |
Limits take effect at once; lowering them deletes the oldest segments within a minute. Turning **capture off** stops storing new lines; what is stored stays until it expires, and turning capture on again continues from where Docker still has the lines.
A container's output from before the agent first saw it is taken from Docker's own log (at most the retention window), so history starts with what Docker still has. Docker's own log rotation (`max-size` on the container) is independent of this store.
### Reading it
**Service → Logs** and **Environment → Observability** have a **Live | History** switch and a time range (last 15 minutes … 7 days, or a custom range). History uses the same search box as the live view: the query is sent to the server and **evaluated there**, so searching 7 days does not move 7 days of log over the wire. The query language is the one documented in the search box: free text, `"phrases"`, `level:error`, `-service:web`, `@status:[400 TO 499]`, `@path.to.field:value` for JSON and logfmt lines, `OR`/`NOT`/parentheses, wildcards. The Go implementation on the agent is checked against the web one with a shared fixture (`tests/fixtures/logquery/cases.json`), so the same query matches the same lines in both.
API: `GET /api/v1/services/{serviceId}/logs/history` and `GET /api/v1/environments/{environmentId}/logs/history` (scope `projects:read`):
| Parameter | |
|---|---|
| `from`, `to` | RFC 3339; `from` inclusive, `to` exclusive. Default: everything the server keeps. |
| `q` | the search query |
| `limit` | 1–2000, default 500 |
| `order` | `desc` (default, newest first) or `asc` |
| `cursor` | `nextCursor` of the previous page |
The answer has `items` (`ts`, `line`, `serviceId`, `serviceName`, `replica`, `deploymentId`, `stream`), `nextCursor` when more may follow, `oldest`/`newest` (what the server still has), `capturing`, and `partial: true` when the agent stopped scanning before the page was full (a rare query over a lot of log; keep paging with the cursor). Secret values are masked like in the live view.
**History is unavailable while the server is offline.** The service endpoint answers `503 server_offline` and the UI says "History is stored on <server> and is unavailable while it is offline"; the environment endpoint lists the affected services in `unavailable` and still returns the others. An agent older than this feature answers `409 agent_outdated`.
A service that moved to another server keeps its older history on the old one; the API reads the server the service is placed on now.
### Downloading logs
To attach a log to a bug report or keep a copy, download it as a file. Secret values are masked exactly as in the log views, and every download is audited (`service.logs.download`, `deployment.logs.download`); the audit entry records the range and format, never the content.
| | Service runtime log | Deployment build and deploy log |
|---|---|---|
| API (scope `logs:read`) | `GET /api/v1/services/{serviceId}/logs/download` | `GET /api/v1/deployments/{deploymentId}/logs/download` |
| UI | **Download** in the service's Logs tab (Live and History; History sends its time range and search) | **Download** above the log of an opened deployment |
| CLI | `nebulactrl services download-logs <service>` | `nebulactrl deployments download-logs <deployment-id>` |
| Source | the server's stored history (agent; 503 `server_offline` while it is offline, 409 `history_disabled` when capture is off) | the persisted lines (at most 10,000; `streams=all` adds the 500-line debug trace) |
Query parameters: `format` (`txt`, default, one `<time> <line>` per line, with `[build]`, `[deploy]` or `[debug]` added to deployment lines (runtime lines carry none); or `ndjson`, one `{ts, stream, line, …}` object per line), `from` and `to` (RFC 3339, `from` inclusive, `to` exclusive), `q` (the log search language, service logs only), `component` (Compose stacks, service logs only), `streams` (deployment logs only). Service lines come oldest first. A service download holds at most 200,000 lines: a longer range ends with a `# download truncated …` line (text) or `{"note": …}` object (NDJSON), so narrow the range. A read error after the first page ends the file with the same kind of note, because the bytes already sent cannot become an error response.
```sh
nebulactrl services download-logs acme/production/web --from 24h -o web.log
nebulactrl services download-logs web --format ndjson --grep 'level:error' > errors.ndjson
nebulactrl deployments download-logs 6f1c0d5e-… --streams all -o deploy.log
```
`--from`/`--to` take an RFC 3339 time, a date, or a duration ago (`90m`, `24h`). Without `-o` the file goes to stdout; with it the file is created with mode 0600, and only once the server answered, so a refusal leaves nothing behind.
## Log drains
A drain ships lines **from each agent, from its own host**, to a destination. Drains are configured **per team** (Settings → Log drains, scope `admin`), optionally narrowed to projects, services and a query.
| Kind | Sends | Credentials |
|---|---|---|
| **Loki** | the push API (`…/loki/api/v1/push`), JSON, grouped into streams by `service`, `project`, `environment`, `server`, `level` plus your static labels; `X-Scope-OrgID` for a tenant | basic auth or bearer token, extra headers |
| **OpenTelemetry (OTLP/HTTP)** | logs in OTLP JSON (`…/v1/logs`), resource attributes `service.name`, `service.namespace`, `deployment.environment.name`, `host.name` | bearer token and/or headers |
| **Syslog** | RFC 5424 over UDP, TCP or TCP+TLS (octet-counting framing), the service labels in structured data | none |
| **HTTP JSON** | batches of records (`timestamp`, `message`, `level`, `service`, `project`, `environment`, `server`, …) as a JSON array or NDJSON, to any intake. Field names and static fields are configurable, which covers **Datadog** (`https://http-intake.logs.datadoghq.com/api/v2/logs`, header `DD-API-KEY`, field `ddsource`), **Better Stack** (bearer token) and **Axiom** (bearer token, dataset in the URL) | headers, bearer token, or basic auth |
- **Credentials are send-time secrets**: sealed in the database, never returned by the API (a mask and the header *names* only), delivered to the agents with the configuration and never persisted in a command. They are masked from every error message.
- **Secrets in lines**: before a line leaves the host, the agent masks the secret values of that service (variables, git token, deploy key, registry credentials; 6+ characters, also their URL-escaped and base64 forms) with `••••••`. The values come with the configuration and are refreshed at every deploy; a secret you changed and have not deployed yet is not known to the agent. A drain's query is matched against the masked line.
- **Start**: a new drain ships what is written after it was saved, not the stored history.
- **Delivery**: batches of up to 500 lines or 1 MB, at least once (a crash can repeat the last two seconds). A failing destination is retried with exponential backoff (1 s to 5 min, honouring `Retry-After`). The log store is the buffer: the drain keeps a position per service and reads from the disk, so a destination that is down for hours loses nothing as long as the lines are still stored and the backlog fits the **buffer bound** (default 256 MB per server and drain; the oldest lines are skipped and counted as *dropped* past it). A destination that **rejects** a batch for good (HTTP 400, 413, 422, other non-retryable 4xx) has that batch dropped and counted, so one bad record cannot block the drain; 401, 403, 408, 425, 429 and 5xx are retried.
- **Health**: Settings → Log drains shows, per server, `Delivering`, `Retrying` (the last delivery failed), `Failing` (five failures in a row) or `Disabled`, the last error (masked), lines sent and dropped, and the backlog on disk. Agents report every 30 seconds.
- **Test**: *Send a test line* delivers one line from a connected server (or the one you pick), which is where reachability and credentials count.
- A drain ships from every server of the team that hosts matching services. A server that is offline ships nothing until it reconnects; its backlog waits on its disk.
### Quick reference
```sh
# Loki with basic auth, only errors of one project
curl -X POST $URL/api/v1/log-drains -H "Authorization: Bearer $TOKEN" -d '{
"name":"grafana","kind":"loki",
"config":{"loki":{"url":"https://logs-prod.grafana.net/loki/api/v1/push","tenantId":"12345"}},
"secrets":{"username":"12345","password":"glc_…"},
"allProjects":false,"projectIds":["<project-id>"],"query":"level:error"}'
# Datadog through the generic HTTP drain
curl -X POST $URL/api/v1/log-drains -H "Authorization: Bearer $TOKEN" -d '{
"name":"datadog","kind":"http",
"config":{"http":{"url":"https://http-intake.logs.datadoghq.com/api/v2/logs","extraFields":{"ddsource":"nebulactrl"}}},
"secrets":{"headers":{"DD-API-KEY":"…"}}}'
curl -X POST $URL/api/v1/log-drains/<id>/test -H "Authorization: Bearer $TOKEN" -d '{}'
```
Changes are audited (`log_drain.create|update|delete|test`, `server.log_settings.update`); credentials appear only as "changed".
## Limits to know
- Only stdout/stderr of containers Docker still has: a container that logs to a file inside itself is not captured. Stdout and stderr share one stream (no separate `stderr` facet).
- Lines are ordered by arrival at the agent; lines of two replicas can be a few milliseconds out of time order. A page is sorted by time, paging is by arrival.
- History and drains need an agent of this version: an older one answers history requests with `409 agent_outdated`. Update agents with the install script.
- There is no log search across servers in one scan beyond the environment view (one request per service, merged by time) and no full-text index: a query scans the compressed blocks of its time range (blocks outside the range are skipped by the time index; a scan is bounded to about 256 MB of decompressed log or 4 seconds, then returns a cursor to continue).
- HTTP access lines of the proxy are not stored here: see [traffic analytics](https://nebulactrl.com/docs/operations/traffic-analytics).
---
# Traffic analytics
Source: https://nebulactrl.com/docs/operations/traffic-analytics
NebulaCtrl counts requests at each host's proxy. The agent reads the Traefik access log, folds it into **per-minute aggregates** and ships only those. Raw log lines, client IPs, user agents, referrer URLs and query strings never leave the host and are never stored by the control plane. Retention and the other traffic settings are in [Monitoring](https://nebulactrl.com/docs/operations/monitoring#traffic-analytics).
Views: service inspector → Analytics, server page → Analytics, environment → Observability.
## Where requests come from
The "Where requests come from" card shows a dot map with the busiest places marked (size and brightness follow their share of requests), beside the top countries and, at the city level, the top cities. It appears in the service, server and environment views, and in the Observability summary.
### How it works
1. For every access-log line the agent looks the client IP (the same field the visitor count uses) up in a **local GeoIP database** inside the traffic aggregator.
2. Per minute and host it keeps counts per country (ISO 3166-1 alpha-2) and, at the city level, the busiest 20 cities with their coordinates rounded to 0.1° plus an "other" counter. Requests the database cannot place (private, loopback or unlisted addresses) go in an "unknown" counter.
3. Only those counts are sent. **The IP is never stored, logged or sent**: it is used for the lookup and dropped.
4. The control plane stores the counts next to the other traffic aggregates, in the same minute, hour and day tiers and under the same retention, and rolls them up by addition (exact for countries; cities keep the busiest 100 per row).
Reports gain a `geo` section: `level`, `source`, `attribution`, `countries` (`iso2`, `name`, `requests`, `share`) and `cities` (`name`, `country`, `lat`, `lon`, `requests`, `share`), plus `locatedRequests` and `unknownRequests`. `share` is the fraction of the located requests. When geolocation is off for every server in scope, `geo` is absent and `geoReason` says why (`geo_disabled` or `traffic_disabled`). The MCP tool `service_analytics` returns the same report, countries included, for a service.
### Settings (per server)
Server page → Settings → "Where requests come from", or `PATCH /api/v1/servers/{serverId}` (scope `servers:write`).
| Field | Default | Effect |
|---|---|---|
| `geoEnabled` | on | Count origins. Needs `trafficAnalyticsEnabled`. Turning it off deletes the database on the server. |
| `geoSource` | `dbip` | `dbip` = DB-IP Lite, no key. `maxmind` = MaxMind GeoLite2 with the team's account (below). |
| `geoLevel` | `country` | `country` needs a ~8 MB database on the server. `city` adds city pins but needs ~130 MB of disk **per server**. Changing the level or source deletes the old database and downloads the new one. |
| `geoRefreshDays` | 7 | How often the agent checks for a newer release (1–90). It downloads only when a newer one exists. |
Settings reach the agent with its next reconcile (sent at once to online agents).
### Databases and attribution
| Source | Licence | Download |
|---|---|---|
| **DB-IP Lite** (default) | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); credit "IP Geolocation by DB-IP" with a link to <https://db-ip.com> | `https://download.db-ip.com/free/dbip-{country\|city}-lite-YYYY-MM.mmdb.gz`, one file per month. Early in a month the new file may not exist yet; the agent then uses the previous month. |
| **MaxMind GeoLite2** | [GeoLite2 EULA](https://www.maxmind.com/en/geolite2/eula); credit "This product includes GeoLite2 data created by MaxMind" with a link to <https://www.maxmind.com> | `https://download.maxmind.com/geoip/databases/GeoLite2-{Country\|City}/download?suffix=tar.gz` with HTTP basic auth (account ID and licence key); the agent checks the release date with a `HEAD` request first (these do not count against MaxMind's download limit of 30 per day for GeoLite). |
The card shows the attribution wherever the data appears. Downloads happen **on each agent**, over HTTPS, straight from the vendor; the control plane never proxies the database. A server needs outbound HTTPS to `download.db-ip.com`, or to `download.maxmind.com` and MaxMind's presigned-URL host (`*.r2.cloudflarestorage.com`).
**MaxMind key.** Create a free MaxMind account, generate a licence key and enter the account ID and key once under any server's settings (or `PUT /api/v1/geo/maxmind`). They are stored **encrypted per team** with the instance secret key, are write-only (only the account ID is ever returned) and are delivered to agents as send-time secrets that are never persisted in a command or on the agent's disk. Removing the account is refused while a server still uses MaxMind.
### What the agent does with the database
- Downloads into `<state-dir>/geoip/`, **verifies** the file by opening it with the MaxMind DB reader (metadata, edition matches the level, the whole search tree and data section), then swaps it in atomically. A corrupt or wrong-edition download is discarded and the previous database keeps serving.
- Keeps only the current database. The old file is deleted after the swap; everything is deleted when geolocation is turned off, and the database is deleted when the level or source changes.
- Re-opens the stored database after an agent restart, so counting continues before the agent reconnects.
- Retries a failed download with backoff (1 min doubling to 1 h) and logs the reason (for example `MaxMind rejected the account ID or licence key`).
### Troubleshooting
| Symptom | Cause |
|---|---|
| Card says "No located requests yet" right after enabling | The agent is still downloading (the city database is ~130 MB), or the requests came from private addresses (for example a proxy or load balancer in front of the server records its own address as the client). |
| Everything is "unlisted or private" | The client address Traefik logs is the address of an upstream proxy. Traefik must see the real client address (no extra hop, or a trusted forwarded-headers setup). |
| MaxMind selected but the card stays empty | Wrong account ID or key, or no outbound access to MaxMind. Check the agent log (`component=geoip`). |
| Cities missing | The server is at the country level, or the database edition lacks a city for that address (the request still counts for its country). |
---
# Notifications
Source: https://nebulactrl.com/docs/operations/notifications
NebulaCtrl tells you when something needs attention: a failed or rolled-back deployment, a crashed, unhealthy or restart-looping service, an offline server, a full disk, a failed or missed backup or a failed task, a blocked scale-out, an expiring certificate. **Settings → Notifications** manages the channels; every channel has its own rule for which events it receives.
All endpoints and the UI need the `admin` scope (channels hold credentials, and the delivery log shows events of every project).
## Channels
| Kind | You provide | Notes |
|---|---|---|
| Email | SMTP host, port, connection (`starttls` default 587, `implicit` TLS 465, `none` 25), user name and password (optional), From, recipients | Standard library SMTP. `none` sends clear text: trusted relays only. "Accept a self-signed certificate" keeps encryption but skips identity checks. Multipart message: plain text plus HTML in the NebulaCtrl palette. |
| Slack | Incoming-webhook URL | Block Kit message with a coloured attachment, field grid and an "Open in NebulaCtrl" button. `<`, `>`, `&` are escaped so event text cannot ping `@channel`. |
| Discord | Webhook URL | One embed; `allowed_mentions` is empty so event text cannot ping anyone. |
| Telegram | Bot token, chat id or `@channelname` | `sendMessage` with MarkdownV2; every reserved character is escaped; text is kept under Telegram's 4096 limit. |
| Webhook | URL, signing secret (generated if omitted) | JSON body, signed (below). |
Webhook URLs, bot tokens, SMTP passwords and signing secrets are sealed with `NEBULACTRL_SECRET_KEY` (same box as variables), **never returned by any read** (the API shows a masked hint such as `https://hooks.slack.com/••••••`), never written to the audit log (which records "secrets changed", not values), and scrubbed from stored delivery errors and logs. Editing a channel with an empty credential field keeps the stored value. Losing the secret key makes stored credentials undecryptable: deliveries then fail with a clear message and you re-enter them.
Receivers on LAN addresses and `localhost` are allowed (homelab services live there). Link-local addresses, which hold the cloud metadata services (`169.254.0.0/16`, `fe80::/10`, `fd00:ec2::254`), are refused. Redirects are not followed.
## Rules
A channel subscribes to **event types**, with a **minimum severity** (`info`, `warning`, `error`) and a **scope**: the whole workspace, or selected projects. Server events (offline, disk, certificates) belong to no project, so only whole-workspace channels receive them.
A new channel starts with failures and alerts on and successes off:
| On by default | Off by default |
|---|---|
| `deploy.failed`, `deploy.rolled_back`, `server.offline`, `server.disk_alert`, `server.cleanup`, `server.reboot_required`, `backup.failed`, `backup.missed`, `task.failed`, `scale.blocked`, `cert.expiring`, `dns.sync_failed`, `cloudflare.ips_stale`, `service.crashed`, `service.unhealthy`, `service.restart_loop`, `service.recovered` | `deploy.succeeded`, `server.online`, `backup.succeeded`, `preview.created`, `preview.deleted`, `maintenance.on`, `maintenance.off`, `cloudflare.ips_changed` |
`GET /api/v1/notifications/event-types` returns the catalogue (label, group, default severity, default on/off).
Where events come from: a deployment reaching `succeeded`, `failed`/`rollback_failed` (both are `deploy.failed`) or `rolled_back` (cancellations are not announced); a server going offline (30 s without heartbeat), connecting or coming back, and a disk crossing its alert threshold (once per crossing); a server's [Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup) that failed (error) or freed more than 1 GB (info), as `server.cleanup`; a database backup or restore finishing (a failed restore and a skipped scheduled backup are `backup.failed`, the latter as a warning); task failures (a failed post-deploy command is a `task.failed` warning; the deployment still succeeded); preview created/deleted; the autoscaler being blocked (once per block); a certificate crossing 30 days, 7 days or expiry (once per threshold); a Cloudflare DNS record that could not be updated, or a Cloudflare token that was rejected (`dns.sync_failed`, once per failure and at most every six hours; see [Cloudflare](https://nebulactrl.com/docs/operations/cloudflare)); maintenance mode switched on or off.
Service health and backups: a crash, a failing health check or a restart loop is announced once per incident (`service.crashed`, `service.unhealthy`, `service.restart_loop`) and closed with one `service.recovered`; a flapping service is rate-limited; a scheduled backup that did not succeed within its window is `backup.missed`, once per streak. Thresholds are set in **Settings → Notifications → Service alerts**. See [Service alerts](https://nebulactrl.com/docs/operations/service-alerts).
Server patching: when a server's host starts needing a reboot (the first [check](https://nebulactrl.com/docs/operations/server-patching) that sees the flag, or an upgrade that sets it), one `server.reboot_required` warning is sent; it is not repeated while the flag stays set, and a new streak is announced again. Pending updates alone send nothing.
## Delivery
Delivery is asynchronous: the event is recorded per matching channel and a River job sends it, so a slow or broken channel never delays a deployment.
- **Retries:** up to 6 attempts, 1, 3, 8, 15 and 30 minutes apart (about an hour). A permanent failure is not retried: HTTP 4xx other than 408/425/429, SMTP 5xx (bad password, unknown recipient), STARTTLS demanded but not offered, undecryptable credentials.
- **Rate limit:** per channel, a burst of 10 then one message every 3 seconds. Over the limit a delivery waits (it does not use an attempt and is not dropped).
- **Deduplication:** an identical event (same type, subjects, text and fields) is not sent again to the same channel within 5 minutes.
- **Delivery log:** per channel, 30 days: status (`pending`, `retrying`, `delivered`, `failed`), attempts, response code (HTTP status or SMTP reply), last error, next attempt. Open it with **Delivery log** on the card or `GET /api/v1/notifications/channels/{id}/deliveries?status=&cursor=`.
- **Test send:** `POST /api/v1/notifications/channels/{id}/test` (the **Send test** button) sends now, bypassing the queue, retries, rate limit and dedupe, and returns the delivery with the result. It works on a paused channel and updates the card's "last test".
## Signed webhooks
Every request carries:
```
X-NebulaCtrl-Signature: t=1793023655,v1=<hex>
X-NebulaCtrl-Event: deploy.failed
X-NebulaCtrl-Delivery: <delivery id>
```
`v1` is the hex HMAC-SHA256, keyed with the channel's signing secret, of `"<t>." + raw request body`. Verify the **raw bytes**, compare in constant time and reject timestamps more than five minutes off. The body:
```json
{"id":"…","type":"deploy.failed","severity":"error","title":"api failed to deploy","body":"…",
"url":"https://ctrl.example.com/p/…","fields":{"Project":"Shop","Service":"api"},
"projectId":"…","serverId":"","serviceId":"…","timestamp":"2026-10-26T14:30:05Z","test":false}
```
Node.js:
```js
import { createHmac, timingSafeEqual } from "node:crypto";
export function verify(secret, header, rawBody, toleranceSec = 300) {
const parts = Object.fromEntries(header.split(",").map((p) => p.split("=", 2)));
const t = Number(parts.t);
if (!parts.v1 || !Number.isFinite(t)) return false;
if (Math.abs(Date.now() / 1000 - t) > toleranceSec) return false;
const expected = createHmac("sha256", secret).update(`${t}.`).update(rawBody).digest("hex");
const a = Buffer.from(expected), b = Buffer.from(parts.v1);
return a.length === b.length && timingSafeEqual(a, b);
}
```
Python:
```python
import hashlib, hmac, time
def verify(secret, header, raw_body, tolerance=300):
parts = dict(p.split("=", 1) for p in header.split(","))
try:
t = int(parts["t"])
except (KeyError, ValueError):
return False
if abs(time.time() - t) > tolerance:
return False
expected = hmac.new(secret.encode(), f"{t}.".encode() + raw_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, parts.get("v1", ""))
```
Go: `notifications.VerifySignature(secret, header, body, now, tolerance)` (`internal/notifications/webhook.go`) implements the same check; the UI shows Go, Node.js and Python snippets.
The signing secret is shown **once**, on creation (when generated) and when rotated (`POST …/rotate-secret`, **Rotate secret** on the card). After a rotation the old secret stops signing immediately: update the receiver first.
## For other domains
Code that wants to announce something calls `notifications.Emit(ctx, emitter, notifications.Event{…})` (`internal/notifications/emit.go`). It is nil-safe, never fails and never blocks. Fill the type, the project/service/server ids and a title and body; names, the environment and the link are looked up. A `URL` starting with `?` is added to the service link (`?tab=deployments`). Titles and bodies go to third parties: no secrets, no log excerpts.
## Troubleshooting
- **Test fails with `HTTP 404: no_service` / `invalid_token`:** the Slack/Discord URL or Telegram token is wrong or revoked; edit the channel and paste a new one.
- **Email `authentication failed` (535):** wrong user/password, or the relay wants an app password. `the server does not offer STARTTLS`: choose `TLS` (465) or `None` for a trusted relay.
- **Nothing arrives, log is empty:** the event type or severity is not in the rule, the channel is paused, or (project-scoped channels) the event belongs to no project.
- **`address is link-local or a cloud metadata service`:** the target resolves to a blocked address.
---
# Service alerts
Source: https://nebulactrl.com/docs/operations/service-alerts
NebulaCtrl tells you when a running service goes wrong, once, and tells you when it is well again. Four events carry it, plus one for backups:
| Event | Severity | Sent when |
|---|---|---|
| `service.crashed` | error | A container exited and stayed down, or Docker restarted it after a crash |
| `service.unhealthy` | warning | The container runs but its health check keeps failing |
| `service.restart_loop` | error | Docker restarted the container 3 times in 10 minutes, it sits in the `restarting` state, or the service keeps failing again and again (see [Flapping](#flapping)) |
| `service.recovered` | the severity of the alert it closes | The incident ended: the service has been healthy for 2 minutes |
| `backup.missed` | error | A scheduled database or volume backup did not succeed within its window ([below](#missed-backups)) |
They are ordinary notification events. A [channel](https://nebulactrl.com/docs/operations/notifications) receives the ones it selects in its event matrix (the **Services** and **Backups** groups), at or above its minimum severity, for the projects it covers. All five are on by default for new channels. On an existing install the upgrade adds them to every channel that already receives `deploy.failed`; other channels stay as you configured them.
## What is watched
The agent reports every service's state every 15 seconds: per replica its state, health and Docker's restart counter. The control plane feeds each report to a small state machine per service (`internal/svcalerts`). Nothing new runs on your servers.
A service is **not** watched while
- a deployment of it is active (the container is being replaced on purpose);
- you stopped it (`desiredState: stopped`). Stopping a service with an open incident closes the incident without a recovery message;
- its server is offline (no reports arrive: [`server.offline`](https://nebulactrl.com/docs/operations/notifications) covers that). When reports resume after more than 2 minutes of silence, the evidence gathered before (how long it was down, restart counters) is discarded.
A new release starts with a clean slate: its containers are new, so their restart counters begin at 0.
A compose stack component that finished cleanly (a one-shot job that exited with code 0) is healthy, not crashed.
## One incident, announced once
A stretch of trouble is one **incident**. It has a kind, escalates but never de-escalates, and produces:
1. **One alert** when it opens, with the cause and, for a crash or restart loop, the last runtime log lines.
2. **One more alert** only if it gets worse: unhealthy, then crashed, then restart loop (each kind at most once per incident).
3. **One recovery** (`service.recovered`) after the service has been healthy for the *healthy-for* time. A blip during that time restarts the clock, so a service that bounces never recovers early.
Nothing repeats while the incident stays open, however long it lasts. A restart of the control plane does not re-announce an open incident: incidents are stored (`service_incidents`) and resumed.
Every message carries an `Incident` field (the first 8 characters of the incident id), so two incidents of the same service are never merged by the 5-minute duplicate filter.
### Debounce
A short blip is not an incident:
- **Down:** a container must stay down (exited, or missing) for the *down-for* time (30 s) before it is announced. A restart by Docker after a crash is the exception: the counter proves the crash happened, so it is announced at once.
- **Unhealthy:** the health check must fail for the whole *unhealthy-for* time (2 min). `starting` is not unhealthy, so a slow start never alerts.
- **Restart loop:** *restart-loop-at* restarts (3) inside the *restart window* (10 min).
### Flapping
A service that fails, recovers and fails again would otherwise send an alert and a recovery every few minutes. When the *flapping-at* th incident (3) opens inside the *flap window* (1 hour), that incident is announced once as `service.restart_loop` ("keeps failing") and **marked flapping**: from then on it is silent, absorbing every further failure and recovery. It closes, with one `service.recovered`, only after the service has been stable for a whole flap window. The next failure after that is a normal incident again. So the most a flapping service can send in an hour is two alert/recovery pairs and one flapping notice.
## Thresholds
**Settings → Notifications → Service alerts** (admin scope; `GET/PUT /api/v1/notifications/alert-settings`) sets them for the workspace. They apply to every channel; which channel hears about which event is each channel's own event selection.
| Setting | Default | Range | Meaning |
|---|---|---|---|
| Down for | 30 s | 0 to 3600 | How long a container stays down before `service.crashed` |
| Unhealthy for | 120 s | 0 to 3600 | How long the health check fails before `service.unhealthy` |
| Restart loop at | 3 | 2 to 50 | Docker restarts inside the restart window |
| Restart window | 10 min | 1 to 1440 | How far back restarts are counted |
| Healthy for | 120 s | 10 to 3600 | Stable time before an incident closes |
| Flapping at | 3 | 2 to 20 | Incidents inside the flap window |
| Flap window | 60 min | 5 to 1440 | How far back incidents are counted, and the stable time a flapping service needs |
| Backup grace | 0 (automatic) | 0 to 10080 min | Extra time a backup gets after its due time |
| Log lines | 5 | 0 to 20 | Runtime log lines attached to a crash or restart-loop message |
A change takes effect within a minute and is audited (`notifications.alert_settings_update`).
### Log lines
A crash or restart-loop alert attaches the last runtime lines of the service so the cause is in the message. The lines are masked for the service's own secrets (the same redaction as the log viewer), control characters are removed and each line is cut at 200 characters. Notifications go to third parties (Slack, Discord, email), and application output can still hold data you would not want there, so set *Log lines* to 0 if your logs are sensitive. An unhealthy alert never carries logs. If the log tail cannot be read (the agent is busy or the container is gone) the alert is sent without it.
## Where it shows
- **Notifications:** the five events above, in the channel's event matrix and the delivery log.
- **Audit log:** `service.incident_opened`, `service.incident_escalated` and `service.incident_resolved` (actor `system`; the metadata has the kind, the detail, the incident id and, when closed, the resolution `recovered` or `stopped`).
- **Service Overview:** a *Health incidents* card with the last 5 incidents, newest first (shown once the service had one).
- **API:** `GET /api/v1/services/{id}/incidents` (`projects:read`). The MCP tool catalogue exposes it and the settings read through `call_operation`; changing the thresholds is not available over MCP.
## Missed backups
`backup.missed` closes the gap left by `backup.failed`: a schedule that silently stops (the control plane or the server was down, the destination is unreachable, every run is skipped) raised nothing until now.
An enabled schedule, database or volume, is **missed** when no backup succeeded by its *deadline*:
> deadline = the next due time after the last success + a grace
The grace is the *backup grace* setting; with 0 it is automatic, a quarter of the interval to the following run, at least 15 minutes and at most 6 hours (so 15 min for an hourly schedule, 6 h for a daily or weekly one). The next due time comes from the cron expression, so irregular schedules (`0 3 * * 1-5`) are handled.
- The last success is the newest succeeded backup of that schedule (for a database also a manual backup of the database; for volumes any non-restore backup). A failed or skipped run does not count.
- A schedule is tracked from the first check (every 15 minutes) that sees it enabled, so a new or re-enabled schedule gets a whole interval before it can be missed.
- A run that is still queued or running is not a miss.
- A database you stopped is not checked (its scheduled runs are *skipped* and already raise `backup.failed` as warnings).
- It is sent **once per streak**. The next successful backup ends the streak; missing the next window announces again. The success itself is the recovery: it raises `backup.succeeded` if a channel selected it.
The message names the service, the destination and the schedule, says when the last backup succeeded (or that none has since the schedule was enabled) and when one was due.
## Not included
- **No restart cap.** Coolify can stop a container after a maximum restart count. NebulaCtrl only announces the loop; a crash-looping container keeps restarting under Docker's `unless-stopped` policy until you stop or fix it.
- Detection runs on the 15-second observed reports, so the shortest crash alert is about 15 s after the crash (plus the down-for time).
- Thresholds are per workspace, not per service or per channel.
---
# Security overview
Source: https://nebulactrl.com/docs/security
- **[`threat-model.md`](https://nebulactrl.com/docs/security/threat-model)**: trust boundaries (spec §11.2), what the platform protects and what it does not, teams, secret handling, agent identity, MCP, and a hardening checklist.
- **[`secrets-at-rest.md`](https://nebulactrl.com/docs/security/secrets-at-rest)**: how stored secrets are sealed (key ring, ciphertext format, binding to table/column/row), the key rotation runbook with `nebulactrl-server rekey`, release secret retention, process hygiene, and what sealing does not protect.
- **[`login-methods.md`](https://nebulactrl.com/docs/security/login-methods)**: password, passkeys (WebAuthn), TOTP two-step verification and recovery codes, requiring 2FA per team, OpenID Connect and GitHub single sign-on (setup, account matching, sign-up rules, trusting a provider's MFA).
- **[`sso-setup.md`](https://nebulactrl.com/docs/security/sso-setup)**: where to find Single sign-on (Settings → Instance), the five setup steps, the redirect URI you can copy before saving, **Test connection**, and a recipe with the issuer format and console link for Google, Microsoft Entra ID, Okta, Auth0, authentik, Keycloak, Zitadel, Pocket ID, GitLab, GitHub and any other OpenID Connect provider.
- **[`host-shell.md`](https://nebulactrl.com/docs/security/host-shell)**: the opt-in root shell on a server (off by default; Owners and Admins with two-factor authentication; production confirm; 2 sessions per person, 15 minutes idle; audited, I/O never recorded), how the agent runs it (a pty on the host, or a helper container that is always removed) and its limits.
- **[`host-hardening.md`](https://nebulactrl.com/docs/security/host-hardening)**: the Lynis audit of each server and its Security tab, the curated safe fixes (Lynis test ID, check, fix, rollback, downside), what is deliberately never changed (SSH, the firewall, accounts, Docker's forwarding), who may apply them (Owners and Admins, audited), the weekly audit, the `nebulactrl-agent hardening` command and what was verified on a real Debian host.
Also see:
- [`SECURITY.md`](): how to report a vulnerability; the Docker-is-root warning and the tenancy boundary.
- [ADR 0002 Agent transport](https://nebulactrl.com/docs/decisions/0002-agent-transport): agent identity, replay protection, revocation.
- [ADR 0004 Build on the target host](https://nebulactrl.com/docs/decisions/0004-build-on-target-host): what building untrusted code on the target host implies.
- [Backup and restore](https://nebulactrl.com/docs/operations/backup-restore): the secret key and the database.
## The two things to remember
1. **Docker access is root on the host.** A NebulaCtrl administrator is a root administrator of every enrolled server.
2. **Teams are an authorization boundary, not isolation between hostile tenants.** Everyone in a team is trusted to deploy to the team's hosts; teams share one control plane, database and secret key. Projects and Docker networks are not isolation between hostile tenants.
---
# Teams, roles and invites
Source: https://nebulactrl.com/docs/operations/teams
Glossary: **Team (UI) = workspace (code)**. The `workspaces` table, `workspace_id` columns and `policy.WorkspaceID(ctx)` keep the internal name.
## Model
- A user belongs to any number of teams through `memberships(user_id, workspace_id, role, project_ids)`. Each team has its own projects, servers, integrations, API tokens, notification channels, backup destinations and audit log. Servers are never shared.
- Setup creates the first team (optional `teamName`, default "NebulaCtrl") and its owner, who is also the **instance admin** (the first user created).
- The browser session stores the current team (`teamId`). `POST /teams/{id}/switch` re-checks membership (404 for non-members). If the stored team is gone the session falls back to the user's oldest team.
- Any user can create a team and becomes its owner, unless the instance admin turns off `allowTeamCreation` (`PATCH /instance`); the instance admin may always.
## Roles
| Role | Scopes | Notes |
|---|---|---|
| Owner | all | Deletes the team, manages owners (there must always be one) |
| Admin | all | Everything except managing owners |
| Developer | projects:read/write, deployments:write/rollback, servers:read | Deploys, variables, terminal/files, tasks. No members, integrations, servers or other people's tokens |
| Viewer | projects:read, servers:read | Read-only; never sees secrets, reveal, terminal or files |
**Host access is an instance-admin decision.** Only the instance admin can make a server *unrestricted* (devices, GPUs, a custom runtime, privileged mode and the other options that hand a container the host, and unrestricted Compose stacks: [Docker options](https://nebulactrl.com/docs/operations/docker-options#the-unrestricted-switch-per-server)). A team owner, an admin or a developer cannot, whatever their scopes, so nobody below the instance admin can gain root on a host unless the instance admin allowed it for that server.
`policy.ScopesForRole` maps roles to scopes; every existing `policy.Require` keeps working. `policy.RequireRole(ctx, projectID, roles...)` covers role-only checks. Owners and admins always see every project; developers and viewers can be restricted to a project list (outside it: 404).
## API tokens
A token belongs to one team (`api_tokens.workspace_id`). Its scopes are intersected with the creator's role at creation and **again on every use**: demoting the creator demotes the token, removing them from the team makes it answer 401. `admin` on a token means "everything my role holds". Project lists are intersected with the member's restriction. Admins see all tokens of the team; other members manage only their own (browser session). MCP follows tokens.
## Invites
`POST /teams/{id}/invites`: with `email` the link is mailed (the team's enabled email notification channel, else the instance SMTP relay from `PATCH /instance`, else link-only; the response's `delivery` says which) and works once for that address. Without `email` it is a link invite with `maxUses` (default 1) and an expiry (default 7 days, max 90). The token (`nci_…`) is stored hashed and shown once. `GET /invites/{token}` previews it publicly (404 invalid, 410 expired/used); `POST /invites/{token}/accept` joins a signed-in user or creates an account (name + password) and signs it in. Single use is enforced atomically. Emailed links need `NEBULACTRL_PUBLIC_URL`.
## CLI
## Password reset links
`POST /teams/{teamId}/members/{userId}/password-reset` (Owners and Admins, signed-in users only; admins cannot reset an Owner and nobody can reset themselves) returns a single-use link, valid for 30 minutes, shown once. In the web app: **Settings → Members → Reset password**. It needs no SMTP setup, so it is also the way to help someone when the instance cannot send email. Using the link signs the member out of every device. Members who only sign in through an identity provider have no password: `409 no_password`. Audited as `member.password_reset_link`. Details: [forgot password](https://nebulactrl.com/docs/security/login-methods#forgot-password).
`nebulactrl teams ls|create|switch`, `nebulactrl members ls|invite`. A token belongs to one team, so `login` saves each token under its team slug and `teams switch <slug>` selects the saved one.
---
# Login methods
Source: https://nebulactrl.com/docs/security/login-methods
NebulaCtrl signs people in with a password, with a passkey, with a single-sign-on provider, or with a password or provider followed by a passkey or one-time code. This page covers what each does, how to set it up, and what it does and does not protect.
(The product calls a tenant a **team**; the code calls it a workspace. Everything below applies per user, per team or per instance as stated.)
## Summary
| | Who configures | Default |
|---|---|---|
| Password (argon2id) | each person | on |
| Forgot password (emailed reset link) | the instance administrator sets up SMTP; see [Forgot password](#forgot-password) | off until email can be sent |
| Passkeys (WebAuthn): sign-in without a password, or the second step | each person, in **Settings → Security** | none registered |
| Two-step verification (TOTP) | each person, in **Settings → Security** | off |
| **Require 2FA** for a team | an Owner or Admin of that team | off |
| OpenID Connect / GitHub sign-in | the instance administrator (the first user), in **Settings → Instance → Single sign-on** | no provider |
| Open sign-up through a provider | the instance administrator, same place | **off** (invite only) |
| Trust a provider's MFA | the instance administrator, per provider | **off** |
## Forgot password
The login page offers **Forgot your password?** when the instance can send email: it needs `NEBULACTRL_PUBLIC_URL` (the link in the email is absolute) and either the instance SMTP relay (**Settings → Instance**) or an enabled email notification channel in one of the person's teams. The instance relay is tried first, then the person's teams in the order they joined them. Without that the login page says to ask a team admin, and the request endpoint answers `503 password_reset_unavailable`.
- **The link:** `…/reset-password?token=ncr_…`. It works **once** and expires after **30 minutes**. Only its SHA-256 is stored. Asking for another link does not cancel the first one, but using any link ends all the others of that account. A weak password (under 10 characters) is refused without spending the link.
- **Same answer for everyone:** `POST /api/v1/auth/password-reset/request` answers `202 {"status":"requested"}` whether or not the address belongs to an account, and the email is sent in the background so response time does not tell them apart. Accounts that sign in only through an identity provider (no password) get no email, and an admin cannot generate a link for them either (`409 no_password`).
- **Rate limits:** 10 requests per hour per client IP and 3 per hour per email address (answered `429` the same way for real and unknown addresses), and at most 3 emails per hour per account (further requests are dropped silently). Using a link is limited to 10 attempts per minute per IP.
- **Everything ends with the reset:** every browser session of the account is destroyed, including a sign-in waiting for its second factor, and every other open link of the account stops working. API tokens are **not** revoked. Two-step verification still applies at the next sign-in: a reset never bypasses it.
- **Admin-generated link:** an Owner or Admin opens **Settings → Members → Reset password** (`POST /api/v1/teams/{teamId}/members/{userId}/password-reset`). The link is shown once and works without any SMTP setup. Admins cannot reset an Owner, nobody can generate one for themselves, and API tokens cannot generate links (a token must not be able to take over an account).
- **Operator link from the host:** the only Owner who is locked out (no SMTP, no other admin) runs `nebulactrl-server reset-password <email>` on the control-plane host. It prints the same one-time link (30 minutes, hash stored, no `created_by`), or with `--generate` sets a random password shown once, which also works for an account that has no password. It ends the account's sessions and audits `operator.password_reset_link` / `operator.password_set` as **system** actions (never the link or password). See [Troubleshooting → Locked out](https://nebulactrl.com/docs/operations/troubleshooting#locked-out).
- **Audit:** `user.password_reset` (written in each of the person's teams, with the person as actor) and `member.password_reset_link` (the admin who generated one). Neither records the token. Requests for a link are not audited.
- **Test it:** the unit and database tests use the in-process fake SMTP server of the notifications package; see `internal/notifications/passwordreset_test.go`.
## Two-step verification (TOTP)
- Standard time-based one-time passwords: RFC 6238, HMAC-SHA1, 30-second steps, 6 digits, one step of clock-drift tolerance either way. Any authenticator app works (Google Authenticator, 1Password, Authy, Aegis …). There is no QR code: the setup screen shows the key to type or paste, and an `otpauth://` link for the phone's own app.
- **Enrol:** Settings → Security → Two-step verification → Turn on. The secret is generated on the server and becomes active only after the first code verifies, so abandoning the dialog changes nothing. The secret is stored encrypted with the instance secret key.
- **Sign-in:** after a correct password the server answers `401 two_factor_required` and keeps the half-finished login in the session; nothing is signed in until the code is accepted (`POST /auth/2fa`). The pending login lasts 5 minutes and 5 wrong codes.
- **Replay and guessing:** an accepted time step can never be accepted again. Five wrong codes in a row lock the account's second-factor check for 5 minutes, from any session, in addition to the per-IP limit of 10 attempts a minute.
- **Recovery codes:** turning 2FA on issues **10 single-use codes** (`xxxx-xxxx-xxxx-xxxx`, 80 bits each). Only SHA-256 hashes are stored. Each is burned the moment it is used. They are shown once; Settings → Security can replace all of them (that needs a current code). When three or fewer are left the page says so.
- **Turning it off** needs a current code (or recovery code) and, for accounts with a password, the password.
- **A lost phone and no recovery codes** cannot be fixed from the UI: there is deliberately no "reset my 2FA by email". An operator on the control-plane host clears it with `nebulactrl-server reset-2fa <email>` (it removes the authenticator app, passkeys and recovery codes after a typed confirmation, ends the account's sessions and audits `operator.2fa_reset` as a system action); see [Troubleshooting → Locked out](https://nebulactrl.com/docs/operations/troubleshooting#locked-out).
## Passkeys (WebAuthn)
A passkey is a WebAuthn credential held by a device, a password manager or a security key. NebulaCtrl uses it two ways:
- **Passwordless sign-in.** The login page offers **Sign in with a passkey** (shown only where the browser supports WebAuthn). The browser lists the passkeys it holds for this site, the person verifies on the authenticator (fingerprint, face, PIN) and the session starts. Because the authenticator verified the person, the sign-in is already two factors: no code follows and the session counts as second-factor verified for "require 2FA".
- **Second factor.** After a correct password (or an identity-provider sign-in that is not trusted for MFA), an account with passkeys is asked for one on the second step (`401 two_factor_required`, then `POST /auth/2fa/passkey/options` and `POST /auth/2fa/passkey`). A code from an authenticator app or a recovery code still works from the same screen. `GET /auth/2fa` says which methods the pending login can use. The pending login lasts 5 minutes and 5 wrong answers, like the code step.
Rules:
- **Register several and name them** (Settings → Security → Passkeys): a laptop, a phone, a security key. Up to 20 per account. They can be renamed and removed. The list shows whether each is **synced** (backed up by a platform password manager) or **this device only**, when it was added and last used.
- **What is required of the authenticator.** The credential is discoverable (a resident key, so passwordless sign-in works) and user verification is required at registration and at passwordless sign-in. Attestation is not requested (`none`): NebulaCtrl trusts any authenticator and records no hardware model. For the second step after a password, verification is "preferred", so a security key without a PIN still works.
- **Adding or removing** needs the **password** again when the account has one (so a stolen session cannot plant a passkey). The last passkey of an account that has no password and no linked identity cannot be removed (`409 last_sign_in_method`).
- **A passkey satisfies "require 2FA".** Members of a team that requires 2FA can enrol a passkey instead of an authenticator app, in the setup screen or in Settings → Security. The team page counts them as enrolled.
- **Recovery codes still apply.** Adding the first passkey issues 10 recovery codes (shown once) when the account has no unused code left; they also exist for an account that has only passkeys. Settings → Security can replace them: with a current code, or with the password for an account that has passkeys and no authenticator app. Removing the last passkey (with no authenticator app) deletes the codes.
- **Relying party.** The RP ID is the host of `NEBULACTRL_PUBLIC_URL` (without port) and the allowed origin is its scheme and host (and port). Without a public URL (development only) both come from the request. Passkeys are bound to that host: **changing the public hostname makes existing passkeys unusable** (they are not deleted; the sign-in falls back to password and a code). WebAuthn needs **HTTPS or `localhost`** and a **domain name**: on an install opened by IP address (`http://<ip>:8000`) the passkey screens say so (`422 passkeys_unavailable`) and the buttons are hidden or disabled.
- **Clone detection.** A passkey whose signature counter goes backwards is refused. Synced passkeys keep the counter at zero, which is accepted.
- **Limits.** Registration and sign-in ceremonies expire after 5 minutes and work once. The sign-in endpoints share the 10-per-minute-per-IP limit with the password login. The management endpoints need a browser session, never an API token or MCP.
- **Not yet:** conditional UI (autofilling a passkey in the email field), attestation or an allow-list of authenticator models, and an admin-initiated passkey reset from the dashboard (an operator on the host can run `nebulactrl-server reset-2fa <email>`, which also removes the account's passkeys).
## Requiring 2FA of a team
Settings → Security → **Require two-step verification** (Owners and Admins; `PUT /teams/{id}/security`).
- Members of that team who have not turned on 2FA see a setup screen instead of the team's pages, and the API answers `403 two_factor_enrollment_required` to everything except what enrolment needs: `/me*`, `/auth/*`, listing and switching teams, and invites. They can still switch to another team.
- The person flipping the switch must already have 2FA (or a session verified by a trusted provider): `409 enrol_first`. This prevents locking the only administrator out.
- **API tokens follow their creator.** A token whose creator has not enrolled stops working in the team. The `/mcp` endpoint authenticates tokens separately and is **not** gated by this rule today.
- A session counts as verified when the user has TOTP or a passkey, or signed in through a provider that is trusted and asserted MFA (below).
## Single sign-on
Settings → **Instance** → **Single sign-on** (first block; visible only to the instance administrator, who also gets a **Manage single sign-on** link in Settings → Security). Two kinds of provider:
### OpenID Connect (any issuer with discovery)
Keycloak, Authentik, Okta, Microsoft Entra ID, Google, Auth0, Zitadel, Pocket ID, GitLab …
1. At the identity provider create a confidential **web** client for NebulaCtrl.
2. Register the **callback URL** NebulaCtrl shows (it is `https://<your public URL>/api/v1/auth/oidc/<provider id>/callback`). The dialog shows it **before you save**: it reserves the new provider's ID (the optional `id` of `POST /auth-providers`), so the address never changes between registering and saving. Set `NEBULACTRL_PUBLIC_URL` first: without it NebulaCtrl derives the URL from the request's `Host` header, which is only acceptable for local development.
3. In NebulaCtrl choose **Connect your identity provider**, pick the provider (or **Any OpenID Connect provider**): name (button text), **issuer URL** (the base of `/.well-known/openid-configuration`; the preset fills the pattern and highlights the `{placeholders}` to replace, and saving is blocked until none is left), client ID, client secret (encrypted; never shown again; leave empty on edit to keep it), optional extra scopes (`openid email profile` are always requested), optional allowed email domains. **Test connection** reads the issuer's discovery document without saving (`POST /instance/sign-in-providers/test`, instance admin only) and says what is wrong: unreachable host, untrusted certificate, no discovery document, an issuer that differs from the one entered, missing endpoints. Saving checks the same thing.
The flow is authorization code with **PKCE (S256)**, a per-attempt `state` and `nonce` bound to the browser session, and the ID token's signature, issuer, audience, expiry and nonce are verified against the issuer's keys. If the ID token carries no email, the userinfo endpoint is consulted.
### GitHub
Register an OAuth App on GitHub (or GitHub Enterprise Server: put its base URL in the provider's URL field), with the callback URL above, and add the client ID and secret. NebulaCtrl asks for `read:user user:email` and uses the **primary verified** email. A GitHub account's two-factor status cannot be verified, so "trust MFA" is never available for GitHub.
Step-by-step recipes for each provider (with the exact issuer format, console link and the field that takes the callback URL) are in [Single sign-on setup](https://nebulactrl.com/docs/security/sso-setup).
### How a person is matched to an account
1. By a previous link: the provider's stable subject (`sub`; numeric id for GitHub) is looked up first.
2. Otherwise by **verified email** only: a provider-asserted email that is *not* marked verified never links to, or creates, an account. (Linking on an unverified email would let anyone who can claim an address at a lax provider take over the account that owns it.) "Verified" means the ID token's `email_verified` is true, or, for Microsoft Entra ID, which never sends that claim, the optional claim `xms_edov` (the tenant admin has verified the email's domain) is true. The link is recorded and shown under **Linked sign-in accounts**, where the person can unlink it (never their last way in).
3. Otherwise a new account is created, but **only** when
- the person presents a team **invite** (the invite link carries a token through the sign-in round trip; an invite bound to an email must match the provider's verified email), or
- the instance administrator turned on **Allow open sign-up** (off by default); the new user then gets a team through the teams module.
Otherwise sign-in ends with `signup_disabled`.
4. **Allowed email domains** on the provider apply to every sign-in and sign-up through it; an account outside the list is refused with `domain_not_allowed`.
Accounts created through a provider have no password (the stored hash is empty and matches nothing), so the provider is their only way in; there is no "set a password" screen for them yet. For that reason the last sign-in method of such an account cannot be unlinked.
### Two-step verification after single sign-on
By default a person who has TOTP enabled still gets NebulaCtrl's code step after the provider signs them in (the provider's own MFA is not verifiable by NebulaCtrl). Per provider you can switch on **Trust the provider's MFA**: when the ID token's `amr` claim contains `mfa`, `otp` or `hwk`, the code step is skipped and the session counts as verified for team requirements. Only enable it for an issuer that enforces MFA and sets `amr` honestly. It is off by default and unavailable for GitHub.
## What this does not do
- **No QR codes, SMS or email codes.** Authenticator-app codes and passkeys only.
- **No SCIM, group-to-role mapping or just-in-time role assignment from claims.** Roles are assigned in the team.
- **No admin-initiated 2FA reset from the dashboard** (an operator on the host has `nebulactrl-server reset-2fa`, see the lost-phone note above).
- **The `/mcp` endpoint is not subject to the team 2FA requirement** (tokens authenticate there without the gate).
- **Logging out of NebulaCtrl does not log out of the provider**, and a provider session ending does not end a NebulaCtrl session (sessions last up to 30 days, 7 idle).
## Audit and storage
Audited, in the same transaction as the change: `auth.passkey.add`, `auth.passkey.rename`, `auth.passkey.remove` (name and id only, never key material), `auth.2fa.enable`, `auth.2fa.disable`, `auth.recovery_codes.regenerate`, `auth.identity.unlink`, `auth.provider.create|update|delete` (never the secret; `secretChanged` only), `auth.settings.update`, `team.require_2fa`. Tables: `user_passkeys` (credential id, public key and sign counter as JSON, name; no secret), `user_totp` (sealed secret, last accepted step, failure counter), `user_recovery_codes` (hashes), `auth_providers` (sealed client secret), `user_identities`, `auth_settings`, and `workspaces.require_two_factor`.
---
# Single sign-on setup
Source: https://nebulactrl.com/docs/security/sso-setup
Where: **Settings → Instance → Single sign-on** (instance administrator only; Settings → Security links to it). What it does and does not protect is in [login methods](https://nebulactrl.com/docs/security/login-methods); this page is the recipe per identity provider.
Provider facts below were checked against each provider's documentation on 2026-10-26; the source is linked on each entry. They change: if a console has moved, the issuer and the redirect-URI rules still hold.
## The same five steps everywhere
1. In NebulaCtrl click **Connect your identity provider** and pick the provider. The dialog fills in the name and the issuer pattern. Parts in `{braces}` are yours to replace (the first one is selected, so type over it); **saving is blocked while one is left**.
2. Copy the **Redirect URI** the dialog shows. It is `https://<your public URL>/api/v1/auth/oidc/<provider id>/callback` and you can register it **before you save**: the dialog reserves the new provider's ID, so the address does not change when you save. Set `NEBULACTRL_PUBLIC_URL` first; without it NebulaCtrl uses the address your browser is on, which is only right for local development.
3. At the provider create a **confidential web client** (it has a client secret), paste the redirect URI into the field named below, and copy the client ID and secret.
4. Back in NebulaCtrl paste the client ID and secret, and click **Test connection**. It reads `<issuer>/.well-known/openid-configuration` without saving anything and answers in a sentence:
| Answer | Meaning |
|---|---|
| The host name does not resolve | The domain in the issuer is misspelt. |
| Could not connect to … | Nothing answers there from the control plane (firewall, wrong port, private address). |
| … did not answer in time | Same, slower. |
| … presented a TLS certificate this server does not trust | A private CA or a self-signed certificate; the control plane does not accept it. |
| … answered 404 for /.well-known/openid-configuration | The issuer is not the base address of an OpenID Connect provider. |
| … did not return a JSON OpenID configuration | You entered a login page or a proxy answered. |
| The provider says its issuer is "…", but you entered "…" | The issuer must be **identical**, including any trailing slash. Copy the value it quotes. |
| The OpenID configuration has no … | The provider is not usable for sign-in (missing endpoints or key set). |
5. Set **Allowed email domains** (an empty list lets **any** verified email through), save, and sign in from a private window to try it. People are matched to accounts by verified email; a person with no account needs an invite or **Allow open sign-up** ([login methods](https://nebulactrl.com/docs/security/login-methods#how-a-person-is-matched-to-an-account)).
## Providers
### Google
- Issuer: `https://accounts.google.com`
- Console: [Clients page](https://console.developers.google.com/auth/clients) → Create client → **Web application**. Field: **Authorized redirect URIs**.
- Any Google account can reach the sign-in; put your Workspace domain under allowed email domains.
- Source: <https://developers.google.com/identity/openid-connect/openid-connect>
### Microsoft Entra ID
- Issuer: `https://login.microsoftonline.com/{tenant}/v2.0`, with `{tenant}` your **Directory (tenant) ID** (a GUID, on the app's Overview page). Not `common`, `organizations` or `consumers`: their discovery document reports a tenant-specific issuer, so the exact-match check fails (NebulaCtrl warns about it).
- Console: [Entra admin center](https://entra.microsoft.com) → Entra ID → App registrations → New registration. Add a **Web** platform; field: **Redirect URIs**. Create a client secret under Certificates & secrets.
- Entra never sends `email_verified`. Under **Token configuration → Add optional claim → ID** add `email` and `xms_edov` ("the email's domain owner is verified"). NebulaCtrl treats a true `xms_edov` as a verified email; without it every sign-in ends with `email_not_verified`.
- Sources: <https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc>, <https://learn.microsoft.com/en-us/entra/identity-platform/optional-claims-reference>
### Okta
- Issuer: `https://{domain}/oauth2/default` (the pre-configured custom authorization server; `{domain}` is your Okta domain, for example `dev-123456.okta.com`). To use the org authorization server instead, the issuer is just `https://{domain}`.
- Admin Console → Applications → Applications → **Create App Integration** → OIDC → **Web Application**. Field: **Sign-in redirect URIs**.
- Sources: <https://developer.okta.com/docs/concepts/auth-servers/>, <https://help.okta.com/oie/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm>
### Auth0
- Issuer: `https://{domain}/` with `{domain}` the tenant domain from the application's Settings (or your custom domain). **Keep the trailing slash**: Auth0 reports its issuer with one.
- [Dashboard → Applications](https://manage.auth0.com/#/applications) → Create Application → **Regular Web Applications**. Field: **Allowed Callback URLs**.
- Source: <https://auth0.com/docs/get-started/applications/application-settings>
### authentik
- Issuer: `https://{domain}/application/o/{slug}/`, per application, **with the trailing slash** (the default issuer mode).
- Admin interface → Applications → Applications → New Application, with an OAuth2/OpenID Provider. Field: **Redirect URIs** (an exact URL is fine; authentik also accepts regular expressions).
- Source: <https://docs.goauthentik.io/add-secure-apps/providers/oauth2/>
### Keycloak
- Issuer: `https://{domain}/realms/{realm}`. A Keycloak older than 17, or one started with a relative path, serves it under `/auth/realms/{realm}`.
- In the realm: Clients → Create client → OpenID Connect, **Client authentication on**, standard flow on. Field: **Valid redirect URIs** (compared as an exact string).
- Source: <https://www.keycloak.org/securing-apps/oidc-layers>
### Zitadel
- Issuer: `https://{domain}`, your instance domain (for example `my-instance-abc123.zitadel.cloud`, or a custom domain), no path.
- Console → a project → New application → **Web**, authentication method **Code** (client ID and secret). Field: **Redirect URIs**.
- Sources: <https://zitadel.com/docs/apis/openidoauth/endpoints>, <https://zitadel.com/docs/guides/manage/console/applications>
### Pocket ID
- Issuer: `https://{domain}`, the address you open Pocket ID at, no path.
- Add an OIDC client in Pocket ID and paste the redirect URI under **Callback URLs**. The `email` scope returns `email` and `email_verified`.
- Sources: <https://pocket-id.org/docs/client-examples>, <https://pocket-id.org/docs/guides/scopes-and-claims>
### GitLab (gitlab.com or self-managed)
- Issuer: `https://gitlab.com`, or the base URL of your GitLab.
- Profile → Edit profile → Access → [Applications](https://gitlab.com/-/user_settings/applications) → Add new application, **Confidential** on, scopes **openid, profile, email**. Field: **Redirect URI**.
- Sources: <https://docs.gitlab.com/integration/openid_connect_provider/>, <https://docs.gitlab.com/integration/oauth_provider/>
### GitHub (OAuth, not OpenID Connect)
- There is nothing to discover, so no **Test connection** and no issuer: leave the address empty for github.com, or enter the base URL of GitHub Enterprise Server.
- Settings → [Developer settings](https://github.com/settings/developers) → OAuth Apps → New OAuth App. Field: **Authorization callback URL**.
- Only the account's primary **verified** email signs in, and GitHub's own 2FA cannot be verified, so "Trust the provider's MFA" is never offered.
- Source: <https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app>
### Any other OpenID Connect provider
Pick **Any OpenID Connect provider** and enter the issuer exactly as the provider's discovery document states it (<https://openid.net/specs/openid-connect-discovery-1_0.html>: the issuer in the document must be identical to the one used to fetch it). Requirements: a confidential web client, the authorization-code flow with PKCE, and an ID token that carries `email` and `email_verified: true` (or the userinfo endpoint does).
## If sign-in fails afterwards
The sign-in page shows the code; the control-plane log has the detail.
| Code | Usual cause |
|---|---|
| `oidc_failed` | The provider is disabled or its discovery document is no longer reachable. Run **Test connection** from Edit. |
| `oidc_exchange_failed` | Wrong client secret, or the redirect URI at the provider differs from the one NebulaCtrl sent. |
| `oidc_invalid_token` | The ID token's audience, issuer or nonce did not check out (a wrong client ID). |
| `email_not_verified` | The provider sent no verified email (Entra: add `xms_edov`; GitHub: verify the primary email). |
| `domain_not_allowed` | The email's domain is not in the provider's allowed list. |
| `signup_disabled` | The person has no account and no invite; turn on **Allow open sign-up** or invite them. |
---
# Secret managers and team variables
Source: https://nebulactrl.com/docs/operations/secret-managers
A variable can point at a secret that lives in your secret manager instead of holding it. NebulaCtrl reads the value **when a deploy starts**, sends it to the server with the command and forgets it. The variable stores only the reference.
```text
DATABASE_PASSWORD = ${{vault:bao/apps/shop/db#password}}
DATABASE_URL = postgres://shop:${{urlencode(vault:bao/apps/shop/db#password)}}@db/app
API_KEY = ${{doppler:dp/backend/prd#API_KEY}}
```
Supported: **HashiCorp Vault and OpenBao** (KV v2, token or AppRole sign-in), **Infisical** (machine identity, universal auth) and **Doppler** (service token, or a service account or personal token). Bitwarden Secrets Manager is not supported: its SDK licence forbids redistribution and its protocol is undocumented; revisit if that changes.
NebulaCtrl's own sealed store (the encrypted variables in its database) stays the default. A secret manager is for teams that already keep their secrets elsewhere; nothing here replaces `NEBULACTRL_SECRET_KEY`.
## Reference syntax
| Manager | Reference |
|---|---|
| Vault, OpenBao, Infisical | `${{vault:<integration>/<path>#<key>}}` |
| Doppler | `${{doppler:<integration>/<project>/<config>#<key>}}` |
| Doppler, with a default or a service token | `${{doppler:<integration>#<key>}}` |
Usable anywhere a reference is: service variables, shared and team variables, inside `urlencode()` and `base64()`.
| Part | Vault / OpenBao | Infisical | Doppler |
|---|---|---|---|
| `<integration>` | the name you gave the integration (lowercase letters, digits, dashes) | same | same |
| `<path>` | the secret's path under the KV v2 mount, e.g. `apps/shop/db` (the mount is set on the integration) | the environment slug, then the folder: `prod/backend`; `prod` alone is the root folder | `<project>/<config>`, e.g. `backend/prd`; left out when the integration's default (or its service token) fixes it |
| `<key>` | the field to read, e.g. `password`. Strings, numbers and booleans are read as text | the secret's name, e.g. `DATABASE_URL` | the secret's name, e.g. `DATABASE_URL` |
A Vault or Infisical path may contain letters, digits and `. _ - / ~ @ + = :`, at most 512 characters, with no empty, `.` or `..` segment. A Doppler project or config is 1 to 64 letters, digits and `. _ -`. The key is letters, digits and `. _ - @ :`. To write the text literally, escape it: `$${{vault:…}}`.
A `vault:` reference needs a Vault, OpenBao or Infisical integration and a `doppler:` reference a Doppler one. Anything else is refused when you save the variable and again when you deploy: `the integration "bao" is a vault integration, which does not take doppler: references`. A service called `doppler` is unaffected: `${{doppler.KEY}}` (a dot) is an ordinary service reference.
An external value is always a secret: variables that contain one are secret too, are never returned by the API and are masked in logs.
## Add an integration
Team settings → Integrations → Secret managers → Add secret manager (admin scope; owners and admins). Each form has the credential help: where to create the credential and the exact commands or clicks.
**Vault / OpenBao.** Address, KV v2 mount (default `secret`), optional namespace, optional CA certificate for a private CA, and one of:
- **Token.** A token with a policy that can read the mount:
```hcl
path "secret/data/*" {
capabilities = ["read"]
}
```
Tokens expire and NebulaCtrl does not renew them: use a periodic token (`vault token create -policy=nebulactrl -period=720h`) or, better, AppRole.
- **AppRole.** Role id plus secret id (and the AppRole mount, default `approle`). NebulaCtrl signs in once per deploy. Create the role with `token_type=batch` so tokens expire on their own:
```sh
vault policy write nebulactrl nebulactrl.hcl
vault auth enable approle
vault write auth/approle/role/nebulactrl token_policies=nebulactrl token_type=batch token_ttl=10m secret_id_ttl=0
vault read auth/approle/role/nebulactrl/role-id
vault write -f auth/approle/role/nebulactrl/secret-id
```
Narrow the policy path to what the team needs. In KV v2 the data lives under `<mount>/data/…`.
**Infisical.** Infisical URL (empty is Infisical Cloud US; `https://eu.infisical.com` or your own instance otherwise), the machine identity's client id and client secret, the project id and optionally an organization slug. Create the identity under Access Control → Machine Identities with universal auth, add a client secret, and add the identity to the project with a role that can read secrets (Viewer is enough).
**Doppler.** The token, optionally the Doppler API URL (empty is Doppler Cloud; only a private Doppler deployment sets it) and optionally a **default project and config** (both or neither).
- **Service token (recommended).** In Doppler open the project, choose a config, Access → Generate. The token is read-only and bound to that one config, so references need no project or config (`${{doppler:dp#API_KEY}}`) and it can reach nothing else. Create one integration per config you need. See [Service Tokens](https://docs.doppler.com/docs/service-tokens).
- **Service account token.** One token for several projects, with project access granted per environment (Team and Enterprise plans). Name the project and config in each reference, or set a default. See [Service Accounts](https://docs.doppler.com/docs/service-accounts).
- **Personal or CLI token.** It works, but it carries the write access of the person who made it, which Doppler advises against in live environments. If you must, set the project and config as above.
Doppler is read through its REST API v3 ([API reference](https://docs.doppler.com/reference/api)): `Authorization: Bearer <token>` over HTTPS. For each `(project, config)` NebulaCtrl makes **one** `GET /v3/configs/config/secrets/download?format=json&include_dynamic_secrets=false` per operation ([Download](https://docs.doppler.com/reference/secrets-download)) and serves every reference to that config from it, so a deploy with many references stays far below Doppler's secret read limit (120 a minute on the Developer plan, 240 on Team, 480 on Enterprise, per the API reference). Dynamic secrets are never requested, so no lease is issued by accident. The values are the config's *computed* values (Doppler's own references expanded). A secret that has no text value is treated as missing. The connection test uses `GET /v3/me` ([Me](https://docs.doppler.com/reference/auth-me)).
The credential is sealed with `NEBULACTRL_SECRET_KEY` and is write-only: the API returns `secretSet` only. Changing the address, auth method or identity (for Doppler, the API URL) requires entering the credential again, so a stored credential can never be redirected to another server. The default project and config and the scope are not an identity and can change freely. The name and kind cannot change (references use the name).
**Test** signs in and, if you give a path and key, reads that secret to check the path and permissions (for Doppler the path is the optional `project/config`). The value is discarded and never shown. The result is stored on the integration, and the reference that was read is recorded in the audit log.
Plain `http://` addresses work but send the credential unencrypted; the form warns about every address that is not `localhost`. Redirects are never followed (a standby node would otherwise receive the token): point the integration at the active node.
## Who may use an integration
Every integration is either **available to the whole team** (the default, and what integrations created before this setting keep) or **limited to selected projects and environments**. Without a limit, any member who can write variables can reference anything the integration's credential can read. Limit it when the credential reaches more than one project needs.
In Team settings → Secret managers → Add or Edit, choose **Available to → Selected projects** and tick:
- a **project**: every environment of it, including environments created later, such as previews;
- or only some of its **environments**. Ticking every environment folds back into the whole project.
At least one must be chosen; changing the scope back to the whole team drops the list. The API takes `scope` (`team` or `selected`) and `grants` (`{projectId, environmentId?}`) on create and update; omitting `scope` on update keeps the stored one.
The limit is enforced in two places, before any request leaves for the manager:
1. When a variable is saved, planned or resolved, a reference to an integration the project and environment may not use is an unresolved reference: `reference ${{vault:locked/apps/db#password}}: the secret manager integration "locked" is not available to this project and environment (an admin can allow it under Team settings → Secret managers)`.
2. When a command is sent, the same check runs again. That is the check that matters: it also catches a variable that was valid before the integration was narrowed. The command is then not delivered and the deployment fails with `integration_not_allowed`, naming the reference and the project and environment. Nothing was read.
A scope with no project and environment, such as a caller that did not say which service it reads for, is refused for a limited integration (it fails closed). Narrowing an integration does not touch existing variables: those in projects that lost access fail at the next plan or deploy until someone changes the reference or the scope. Scope changes are audited (`secret_manager.update`, with the previous and new scope and the grants' ids).
The admin test button is not subject to scoping: it checks the credential itself.
## What happens at deploy time
1. Saving a variable, the deploy plan and every listing handle only the reference. Nothing contacts the secret manager.
2. A deploy stores the reference in the release (`env_enc` holds `${{vault:…}}`, never the value) and the command payload carries no value either.
3. When the server's command is **sent** (the first time and again after a reconnect), the control plane checks each integration's scope, signs in to each integration once, reads each secret once and attaches the values to the command frame as send-time secrets, like every other secret. They are never written to the database, logs or audit trail.
4. The values stay in memory only: per operation for the fetch itself, and for log masking in the last three deploys of the service. After a control-plane restart they are fetched again, at most once a minute per service, to keep masking running.
Because the value is read at send time, a **rollback** or **redeploy** of an old release reads the secret's current value, not the value of the original deploy. A change in the secret manager reaches a service on its next deploy; nothing is pushed.
If a secret cannot be read, the command is **not delivered** and the deployment fails with "Not delivered: could not resolve the command's secrets: …. No changes were made." The message names the reference and the reason, never a value:
```text
reference ${{vault:bao/apps/shop/db#password}}: permission denied
reference ${{vault:bao/apps/shop/db#password}}: no secret at apps/shop/db in mount "secret" (or the token cannot see it)
reference ${{doppler:dp/backend/prd#API_KEY}}: no secret "API_KEY" in Doppler project "backend", config "prd" (or the token cannot see it)
reference ${{doppler:dp#API_KEY}}: authentication failed (invalid or expired credential): Invalid Auth token
reference ${{vault:other/a#b}}: no secret manager integration named "other" (add it under Team settings → Secret managers)
```
A reference to an integration that does not exist, or that the project may not use, is shown as a warning when you save it and blocks the deploy (`unresolved_reference`). Database credentials and backup hooks that use an external value read it the same way, when the operation starts.
An integration cannot be removed while a variable references it (`409 integration_in_use`, with the variables listed).
## The secret access log
Every operation that reads external secrets leaves **one** audit event, written when the operation ends and kept whatever its outcome:
| Action | When | Metadata (never a value) |
|---|---|---|
| `secret.fetch` | a deploy command is sent, a task starts, a database credential or backup hook is resolved, or logs are re-masked after a restart | `service`, `project`, `environment`, `purpose` (`deploy`, `task`, `database`, `backup`, `log-mask`), `references` (the references read, sorted) and `count`; for a deploy also `deployment` and `startedBy` (the user, token or "System" that started it); `failed`: `{reference, error}` for the ones that could not be read |
| `secret.fetch.denied` | the same, when an integration's scope refused at least one reference | the same, with `denied`: the refused references |
| `secret_manager.test` | an admin tests an integration | `name`, `ok`, `read` and the `reference` that was read |
| `secret.reveal` | someone reveals a database connection string (private or public) | the service and key, `public` for the external one |
| `platform.reveal` | someone reveals the generated credentials of a platform service | the service |
| `volume_backup.recovery_key.reveal` | an admin reveals a volume backup's repository password | the service |
The actor is the signed-in user or token when the operation runs for one (a database reveal, a test). A deploy command is sent by the control plane, so its actor is `system` and `startedBy` carries who started the deployment. The resource is the service, so the event also shows under that project.
**Volume.** A deploy that reads N references writes one row, not N, and only deploys of services that use external secrets write one: one extra row next to the `deployment.*` events a deploy already writes. A log-masking re-fetch happens at most once a minute per service, and only after a control-plane restart. There is no separate access-log table: the rows live in the audit log, which the team already lists and filters by prefix.
Where to look: Team settings → Audit log → **Secrets** (every `secret.` and `secret_manager.` action), or `GET /audit-events?type=secret`. A `secret.fetch.denied` event is the signal that a reference points at an integration its project may not use.
What is *not* in the log: values, credentials, and the fetched computed text. Reveals that return a value are always audited; every other read of a secret variable returns `{key, isSecret: true, set: true}` and is not a reveal.
## Run OpenBao yourself
NebulaCtrl does not run OpenBao for you as a platform service: it starts sealed, a single node is what upstream calls highly discouraged, and it would not remove the need for `NEBULACTRL_SECRET_KEY`. Bring your own, or install the **OpenBao** template from the gallery (Security), which starts one node on a volume.
The template is an ordinary service: `openbao/openbao` pinned by digest, integrated Raft storage on a single node (the `file` storage backend was development-only and is removed in OpenBao 2.7, so it cannot be used with a current release), the listener on port 8200 behind the platform proxy's TLS, and the data on the volume `data`.
- **Unsealing is manual after every restart.** OpenBao starts sealed. After the first start, open the service's terminal and run `bao operator init` once: it prints the unseal keys and the initial root token. Keep them somewhere safe and separate; NebulaCtrl never sees them and nobody can recover them. Then `bao operator unseal` with three of the five keys.
- **Every restart seals it again.** A redeploy, a server reboot or a crash leaves OpenBao sealed until someone unseals it. While it is sealed, a deploy whose variables reference its secrets is not delivered (`Not delivered … No changes were made.`); running services keep the values they have.
- Take snapshots (`bao operator raft snapshot save <file>`) and back the volume up. A lost volume or lost unseal keys cannot be recovered, even from backups.
- To read from it, create a KV v2 mount, a read-only policy and an AppRole (see above), then add a Vault / OpenBao integration. The control plane calls OpenBao itself, so use the service's public https address, not its private one, and limit the integration to the projects that need it.
The health check is "the listener is up", because a sealed or uninitialized OpenBao answers `/v1/sys/health` with 503 or 501. The unseal state is yours to watch.
## Team variables
`${{team.KEY}}` is a shared variable of the whole team, next to the project and environment layers (see [variables](https://nebulactrl.com/docs/operations/variables#team-variables)). A team variable may itself point at a secret manager: `API_KEY = ${{vault:bao/shared/api#key}}`, referenced from every service with `${{team.API_KEY}}`. The integration's scope is checked against the project and environment of the service that is being deployed, not against the team variable.
## API and audit
`GET|POST /secret-managers`, `PUT|DELETE /secret-managers/{id}`, `POST /secret-managers/{id}/test`. Reading needs `projects:read`; everything else the `admin` scope. Audit actions: `secret_manager.create|update|delete|test` (name, kind, which fields changed, the scope and the grants' ids; never the credential), and the access log above.
## Limits
- Vault / OpenBao KV **v2** only (not KV v1, dynamic secrets or transit).
- Doppler: secrets of one config at a time, no dynamic secrets, no secret versions.
- No secret versions: the latest version is read.
- Values are read as text. A secret that is a JSON object must be split into fields.
- A secret that changes in the manager is not pushed: deploy to apply it.
- No further providers are planned. Bitwarden Secrets Manager is not supported (SDK licence forbids redistribution, undocumented protocol; revisit if that changes).
---
# Threat model
Source: https://nebulactrl.com/docs/security/threat-model
Scope: NebulaCtrl P0 as built from this repository (control plane, agent, Traefik proxy, web UI, REST API, CLI, optional MCP endpoint). It follows the trust boundaries of spec §11.2 and states what the platform does and, as importantly, what it does not protect against. Report vulnerabilities as described in [`SECURITY.md`]().
## Security boundary in one paragraph
**Teams are an authorization boundary inside one shared trust domain.** An instance hosts one or more teams (workspaces, [teams](https://nebulactrl.com/docs/operations/teams)). *Inside a team* everyone who can sign in or holds a token with `deployments:write` is trusted to run arbitrary code in containers on the team's enrolled hosts, and Docker is root, so the platform's own allow-lists are what keep that from being root on the host: **nobody below the instance admin can gain root on a host through the platform unless the instance admin made that server _unrestricted_** (a per-server switch: [Docker options](https://nebulactrl.com/docs/operations/docker-options#the-unrestricted-switch-per-server), [Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks#unrestricted-per-server)). On an unrestricted server any deployer can take the host over. Builds are a separate, explicitly trusted path (below). *Between teams* the control plane enforces the separation in code: every lookup joins through the team, an inaccessible resource is a 404, and projects, servers, integrations, tokens, notification channels, backup destinations and the audit log are per team. A server is enrolled by exactly one team, so one team's containers never share a Docker daemon with another team's. What teams do **not** get is separate cryptography or separate infrastructure: one control-plane process, one Postgres and **one `NEBULACTRL_SECRET_KEY` serve every team**, so an authorization bug, a database dump together with the key, or a compromised control-plane host exposes all teams at once, and the instance admin (the first user) administers instance-wide settings. Projects and environments within a team organize work and per-environment Docker networks keep their traffic apart on a host ([what is and is not isolated](#4-deployed-apps-and-build-inputs-are-untrusted-workloads)), but they do **not** isolate hostile tenants from each other: they share a kernel, a Docker daemon and root. Do not host mutually untrusting customers on one NebulaCtrl installation; give each its own instance, database and key.
## Assets
| Asset | Why it matters | Where it lives |
|---|---|---|
| Docker daemon on each agent host | Docker access = root on the host | agent host |
| `NEBULACTRL_SECRET_KEY` (and any `NEBULACTRL_SECRET_KEY_PREVIOUS`) | decrypts every stored secret of every team | operator's key file (`NEBULACTRL_SECRET_KEY_FILE`, mode 0400) or `.env` / vault, control-plane process memory; as an environment variable it is also visible to anyone with Docker access on the control-plane host |
| Secret variables, git tokens, SSH deploy keys, registry credentials | access to your apps, repos, registries | encrypted in Postgres (AES-256-GCM); deploy keys are write-only and reach the agent as send-time secrets ([SSH deploy keys](https://nebulactrl.com/docs/operations/ssh-deploy-keys)); registry push and pull credentials are write-only and reach agents the same way |
| Container images in a registry | the code that runs on your hosts; a registry write is a code-execution path to every server that pulls from it | the registry you chose (a BYO provider or the managed registry); targets pull by **digest**, never by tag |
| Build server (agent identity, git credentials, build variables, registry push credential) | a privileged host that handles the build inputs of **every project** of the team | the enrolled build server |
| External secret manager credentials (Vault token or AppRole secret id, Infisical client secret) and the secrets read with them | read access to everything the credential's policy allows | the credential is encrypted in Postgres and write-only; the secrets themselves are fetched at deploy time, held in memory and sent to the agent as send-time secrets, never stored ([secret managers](https://nebulactrl.com/docs/operations/secret-managers)) |
| Agent identity (Ed25519 private key) | impersonating a host to the control plane | `agent.db` on the host, mode 0600 |
| API tokens, sessions | act as a user | token hashes / session store in Postgres |
| Postgres | the whole desired state | the control plane's Compose stack |
| Proxy certificates and keys (ACME) | TLS for your domains | `nebulactrl-traefik-acme` on each agent host |
## Trust boundaries
```
Browser / CLI / MCP client ──HTTPS──▶ Control plane ◀──outbound WSS── Agent ──▶ Docker ──▶ your containers
(untrusted) (trusted coordinator) (privileged on its host) (untrusted workloads)
│
Postgres
```
### 1. Browsers, CLI and MCP clients are untrusted
- **Authentication:** local admin account created through the one-time setup page (works only while no user exists; afterwards `409`). Passwords are hashed with **argon2id**; login is limited to **10 attempts per minute per IP**. Sessions use `alexedwards/scs` with a Postgres store; the cookie `nebulactrl_session` is **HttpOnly, SameSite=Lax, and Secure when `NEBULACTRL_PUBLIC_URL` is https**. Run behind HTTPS.
- **CSRF:** Go's `http.CrossOriginProtection` rejects cross-origin unsafe requests that rely on cookies. Requests with an `Authorization: Bearer` header are exempt, since a cross-site page cannot attach one.
- **API tokens:** `nct_…`, 32 random bytes; only a SHA-256 hash is stored; shown once with an 8-character prefix for identification. Scopes: `projects:read`, `projects:write`, `deployments:write`, `deployments:rollback`, `servers:read`, `servers:write`, `admin`. Optional project restriction, expiry, revocation. **Bearer header only**; never accepted in query strings, so tokens do not end up in access logs.
- **Authorization:** checked inside every domain service method (`policy.Require(scope, project)`), not in HTTP handlers, so REST, CLI and MCP share it. Every lookup joins through workspace and project, and an inaccessible resource returns **404, not 403** (no IDOR oracle). Session users hold every scope.
- **Audit:** project/environment/service create/update/delete, variable set/delete (**key only, never the value**), domain add/remove, deploy, rollback, restart, stop, server enroll/revoke, token create/revoke, reconciler limit events, in the same transaction as the mutation. Actor type is `user`, `token` or `mcp`.
- **Idempotency:** mutating deploy-like requests honor `Idempotency-Key` (24 h), so retries do not double-deploy.
- **Residual risk:** a stolen session or `admin`/`deployments:write` token can deploy code to your hosts. Use short expiries, project restrictions and the read-only presets for automation that only reads.
### 2. The control plane is a trusted coordinator that assumes nothing about repositories or agents
- It validates input at the boundary: image references, hostnames (RFC 1123, lowercase, ≤ 253 chars), environment variable keys, relative Dockerfile/context paths with no `..`, server placement.
- It never builds a shell string from user input: commands to agents are typed JSON from a **closed set** (`deploy`, `build_image`, `registry_check`, `restart_service`, `stop_service`, `remove_service`, `sync_routes`, `logs_subscribe`, `logs_unsubscribe`). The agent rejects anything else.
- Agent messages are treated as untrusted data: results are matched to known commands; log lines are redacted and bounded (10,000 stored lines per deployment) before storage and fan-out.
- **Secret handling.** Variables are encrypted with **AES-256-GCM**; ciphertext carries a key-version byte and authenticates (tampering fails to decrypt). Secret values are never returned by any read endpoint (`{key, isSecret:true, set:true}`), never written to audit metadata, and never persisted in `agent_commands.payload`: the dispatcher decrypts them at send time and attaches them to the outgoing frame only. Log redaction replaces every secret value of 6+ characters with `••••••` before persisting or streaming. Short secrets (under 6 characters) are **not** redacted: use real secrets. Stored runtime logs live **unmasked** on the server's disk (the agent cannot know which values are secret when it captures); the control plane masks them when it reads history, and the agent masks each service's secrets in every line **before a log drain ships it** off the host ([log history and drains](https://nebulactrl.com/docs/operations/logs)). Drain credentials are sealed with the instance key and travel only in the send-time configuration; a drain sends your application's logs to the destination you configure, so treat it as a trust decision.
- **The one deliberate exception: database connection strings.** A managed database's `DATABASE_URL` and friends are masked like any secret; the *reveal* action (`POST /services/{id}/database/reveal`) returns one in plain text, uncached (`Cache-Control: no-store`), only for sessions and tokens with `deployments:write` (or `admin`), and is audited as `secret.reveal` (service and key, never the value). MCP has no reveal tool. Credential rotation sends the new password to the agent in the command frame (never in `agent_commands.payload`), runs it through the exec's stdin/environment (never argv), and redacts it from results; see [managed databases](https://nebulactrl.com/docs/operations/databases).
- **The web terminal and file browser are a deliberate exec surface.** Owners, Admins and Developers (never Viewers; tokens need `deployments:write` and a creator who still holds the role) can run a shell and read or write files **inside a replica container**, relayed over the agent's existing connection as ephemeral stream frames (`exec_*`, `fs_*`; nothing is persisted or redelivered). Production needs an explicit confirmation checked server-side; the WebSocket handshake enforces the `Origin` (no cross-site terminal riding a session cookie); at most 3 sessions per person, 15 minutes idle timeout; every session, download and upload is audited (the executable of a command, never its arguments; never the I/O), and a session that cannot be audited is not opened. The agent only ever reaches containers carrying the service's managed labels, through the Docker API: the host filesystem is out of reach. Residual risk: a shell sees the container's environment, including secrets, and can reach whatever the container's network can; treat the roles that hold it like SSH access. File uploads are read-only in production unless an admin allows them per project. See [web terminal and file browser](https://nebulactrl.com/docs/operations/terminal).
- **The host shell is a second, much stricter exec surface** ([host-shell.md](https://nebulactrl.com/docs/security/host-shell)): a root shell on the server itself, **off by default and opt-in per server**. Only Owners and Admins with a signed-in session (never a token or MCP principal) whose account has **two-factor authentication** can open one; production servers need the guarded confirm and `confirmProduction` checked server-side; at most 2 sessions per person and 15 minutes idle; `server.shell.enable`, `server.shell.disable`, `server.shell.open` and `server.shell.close` are audited, the I/O never. An agent on the host spawns the pty itself with a minimal environment (never the agent's own variables); an agent in a container uses a privileged `--pid=host` helper container that is always removed, with an orphan sweep as backstop.
- **Host hardening is a third surface that changes the host** ([host-hardening.md](https://nebulactrl.com/docs/security/host-hardening)): a Lynis audit (read-only) and a curated catalog of reversible fixes, applied only by Owners and Admins, audited with who, which fixes and the index before and after. The agent runs them as root on the host, directly or through a helper container like the host shell. There is no fix for SSH, the firewall or user accounts, none touches Docker's IP forwarding, every fix is verified after it is made and undone by itself if it did not take effect, and rollback only undoes what NebulaCtrl applied.
- **Single point of trust:** the control plane process holds the decrypted key and can deploy to every host. Anyone with its database *and* the key can read all secrets and impersonate the platform to agents (but not to a host that has no enrolled identity). Protect the key, the host and the Postgres volume as one trust domain.
- **Availability:** a single instance; a compromised or stopped control plane cannot stop running workloads, but cannot manage them either.
### 3. The agent is privileged on its host
- The agent holds the Docker socket: **Docker access is root-equivalent on the host**. Anyone who can make the agent run a container can read the host filesystem, other containers' data and the agent's own state. Therefore: **a control-plane administrator is a root administrator of every enrolled host.** A compromised control plane (database + key + code) is a compromise of all agent hosts.
- It only dials **outbound** to `wss://<control-plane>/agent/v1/connect`; no inbound port for management. `ws://` is refused unless `--insecure-dev`.
- **Identity:** at enrollment the agent generates an Ed25519 keypair; only the public key is sent. Each connection runs a challenge: the server sends a fresh nonce, the agent signs `nonce || agentId || serverTime`; stale nonces (> 30 s) and revoked identities are rejected (close 4401). The private key never leaves `agent.db` (mode 0600). Enrollment tokens are 32 random bytes, stored hashed, single use, 1 hour lifetime, revocable.
- **Commands are idempotent and durable**; replays return the stored result instead of re-running.
- **Revocation** is immediate for the control-plane link. It does not remove containers or secrets already on the host: treat a revoked, suspected host as compromised and rotate its secrets.
- **Residual risk:** root on an agent host reveals the agent's private key and the secrets of the services deployed there (they are present in container environments). It does not reveal other hosts' secrets or the control plane's key; one agent cannot read another's secrets.
### 4. Deployed apps and build inputs are untrusted workloads
- **Containers** are created without `--privileged`, without host networking, and without the Docker socket or arbitrary host bind mounts; the platform never adds those, and the API has no field to request them. Volumes are named Docker volumes. Memory and CPU limits are applied from the service config; there are no default limits when none are set.
- **Network:** the Docker network layout is described in full in [networking](https://nebulactrl.com/docs/operations/networking). In short:
- **Isolated now.** Every environment has its own bridge network `nebulactrl-env-<id>`; app, task and tailnet-sidecar containers no longer join the shared `nebulactrl-edge`, so containers of different projects, environments and teams on one host cannot reach each other by name or address. Only Traefik, the error-pages server and `cloudflared` stay on `nebulactrl-edge`; Traefik joins the environment network of each routed service while the route exists and leaves it afterwards. Agent health probes use the container's address on its environment network.
- **Shared networks are opt-in.** A team **shared network** (`nebulactrl-shared-<id>`) is created by someone with `servers:write`, lives on exactly one server, and only services attached to it (`deployments:write` on the service; staged and deployed like any other setting) can join. Each gets the alias `<service>.<env>.<project>`. A network of another team is a 404 for everyone; attaching a service that runs on another server is `network_wrong_server` (422). Create, delete, attach and detach are audited (`network.*`).
- **Block outbound internet.** The setting is part of the release (`config.blockEgress`), staged and rolled back with it. A blocked service joins only `nebulactrl-env-<id>-int`, a Docker `internal` network with no gateway, so it has no route to the internet, the LAN or the host; every other service of the environment joins `-int` too, with the same alias, so it still reaches it. It defaults to **on for managed databases** and off for apps. A blocked service cannot join shared networks (a shared bridge would give its internet back).
- **Not isolated.** Containers of one environment trust each other: they share a network and every port. All containers of a host share its kernel, its Docker daemon and its root: a container escape, or root on the host, crosses every network boundary. Docker networks are not a hostile-tenant sandbox (see the boundary paragraph above). Outbound filtering is all-or-nothing per service: there are no per-destination egress rules, and a service that is allowed out can reach anything the host can route to, including the LAN and a cloud provider's metadata address.
- **By default builds run on the target host** with its Docker daemon (ADR 0004). A Dockerfile executes arbitrary commands during `docker build`, and BuildKit may read build context files. Git repositories are cloned to a per-deployment directory and removed afterwards; the build has a 30 minute timeout; `dockerfilePath` and `buildContext` cannot escape the clone. Without a Dockerfile, Railpack (ADR 0006) runs the project's own install and build commands (`npm install`, `pip install`, …) the same way; on hosts without the containerd image store the agent also runs a privileged BuildKit container (`nebulactrl-buildkit`) for those builds, which adds nothing a Docker-socket holder lacked. Build variables reach builds as BuildKit secrets via the docker CLI's environment, never as arguments. **Only deploy repositories you trust.** Git tokens are passed only inside the command frame and used for the clone as HTTPS BasicAuth (`x-access-token`); they are not passed into the build's environment.
- **Build servers and registries are optional and widen the blast radius** ([ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers), [build servers](https://nebulactrl.com/docs/operations/build-servers)). A build server is an enrolled server with `build_enabled`; it runs the same untrusted build code as a target would (Dockerfiles, Railpack steps), in a **privileged BuildKit container**, but away from your running workloads. What it gains and loses:
- **Better:** a malicious or compromised build cannot reach production data, the runtime secrets or other running services; **runtime variables never go to a build server** (only the git credential, build variables and the registry push credential do); the target never sees the repository credential.
- **Worse:** one host now handles the build inputs of **every project** of the team (git tokens and SSH keys of all of them, build variables, and a push credential), so an escape from a build container is a compromise of all of them and of the agent's identity. Treat it as a high-value, **dedicated** host: role *Builds*, no workloads, nothing else on it. Per-project BuildKit builders (`nebulactrl-buildkit-<project8>`, each with its own cache) stop one project's build from reading or poisoning another's **cache** (cache mounts are shared by id inside a builder); they do not stop a container escape. A Developer restricted to project A can still run code on the host that builds project B, which is inside the trusted-team boundary above and outside what NebulaCtrl isolates.
- **Pull-request previews do not build on a shared build server** unless the owner turns on **Allow previews** for that server; fork code is untrusted, so leave it off or dedicate a build server to previews.
- **Registry credentials are never ambient.** The push credential is sealed in Postgres, attached to the `build_image` command at send time only (never in `agent_commands.payload`), written by the build agent to a mode-0700 temporary `DOCKER_CONFIG` that exists for one build and is always removed, and redacted from build logs. The target receives a separate, optionally read-only pull credential for each deploy. No `docker login` ever leaves a long-lived credential in a host's `~/.docker/config.json`.
- **Managed registry:** the control plane is its token server, so credentials are per command, scoped to one repository and the needed actions, and expiring (15 minutes by default, at most 6 hours for a build), exchanged for tokens valid at most 15 minutes; no long-lived registry password exists on any host. The registry host holds every image of the team and is a high-value target: keep it patched and its volume private. Docker only accepts TLS for non-loopback registries, so it needs a real certificate.
- **BYO registry:** the credential cannot be scoped finer than the provider allows (a GHCR classic token is account-wide), and the provider's staff and its own breaches are outside NebulaCtrl. Image layers and the image configuration are readable by anyone with pull access, and plain `--build-arg` values persist in the image: put secrets in build **secret** variables, which reach BuildKit as secrets.
- **Digest pinning.** Targets pull `repo@sha256:…`, so a tag overwritten or a repository tampered with after the build cannot change what a release runs: a different image has a different digest and fails to pull. The first trust decision is the push: whoever holds a push credential can push any content to that repository.
- **Plain-HTTP registries** are accepted for loopback addresses only.
- **Images** come from whatever registry you name. Pin digests where it matters; an image tag can change. No vulnerability scanning is included (recommended in spec §11.3; use external scanners).
- **Container escape** is a host compromise: Docker is not a hostile-tenant sandbox.
- **Proxy:** Traefik is the only component publishing ports. Routes come from the database's domain table only and are pushed by the agent as the full dynamic configuration through Traefik's REST provider on the admin entrypoint. That entrypoint is protected by basic auth with a per-host random secret (kept in the agent's local store; only its bcrypt hash is given to Traefik). It is published only on `127.0.0.1:9000` of the host, or, when the agent itself runs in a container, not published at all and reachable only on the edge network. Containers you deploy join the edge network, but without the secret they get `401` from the admin entrypoint. The probe entrypoint (`:8099`) is not published, but it is reachable from containers on the edge network; it serves only the same routes as the public entrypoints plus liveness. Any local root user can still read the secret and reconfigure the proxy; this is part of the trusted-team boundary.
- **Per-service tailnet sidecar** ([service tailnet](https://nebulactrl.com/docs/operations/service-tailnet)): a pinned `tailscale/tailscale` container in userspace mode (`cap_drop: ALL`, `no-new-privileges`, no TUN device, no published ports, no Docker socket) on the service's environment network only; it advertises no routes, so a tailnet peer reaches only the ports named in the serve config, and **your tailnet policy (ACLs, tag ownership) is the access boundary**. A sidecar is as trusted as any container on that environment network and can reach its siblings. The OAuth client secret (or Headscale API key) stays in the control plane, sealed with `NEBULACTRL_SECRET_KEY`; sidecars only ever receive a **single-use, tagged auth key minted at send time** (never persisted, never logged; masked by the agent). An integration's allowed-tag list stops a project member from assigning a more privileged tag. Funnel is public by design: it is opt-in with a typed confirmation, audited (`tailnet.funnel_enabled`) and bypasses NebulaCtrl's proxy. A Headscale *pre-auth key* credential is reusable and remains visible to anyone with Docker access on the hosts.
- **Server mesh nodes and cross-server links** ([mesh](https://nebulactrl.com/docs/operations/mesh)): two kinds of agent-created containers, from a fixed spec that no user-supplied Docker option can change and that does not depend on the server's `unrestricted` switch. A **server node** runs in userspace mode (`cap_drop: ALL`, `no-new-privileges`, no TUN device, no host network, no Docker socket) and forwards only the TCP ports of the services you expose. A **link node** adds `NET_ADMIN`, `NET_RAW` and `/dev/net/tun` **inside its own network namespace only** (the same shape as the proxy's managed tailnet container): not privileged, no Docker socket, no host network, and it changes no host firewall rule. Both carry one tag per team; keys are single-use, pre-authorized, non-ephemeral, tagged and valid for an hour, delivered in the command frame and never persisted or logged. **The access policy is the boundary**, and it is generated: one rule per link, with the link's own tailnet IP as the source and the granted server-node ports as the destination; a link belongs to one environment and a grant cannot join it to another environment's service, so environments are separated at the tailnet layer without a tag each. Replication links and service links are different nodes, so a consumer cannot reach a replication port. Other services of the same environment on the consuming server can use the environment's link (the environment is the trust boundary, as everywhere else). On **Headscale** NebulaCtrl owns the policy (database mode) and overwrites hand edits. On **Tailscale** the default is advisory (you paste the rules; with an advisory policy and an allow-all tailnet a link node can reach any exposed port of the remote server node, which the Mesh page warns about). The opt-in **managed push needs the `policy_file` scope: a token with it can rewrite the entire tailnet policy, and the control plane holds it**, so a compromised control plane could open the tailnet; the mode is audited, refused when another team shares the tailnet, and edits only rules whose endpoints are all mesh-node IPs. Managed Headscale falls back to Tailscale's **public DERP relays** when there is no direct path: traffic stays end-to-end encrypted, but relays see ciphertext, endpoint addresses and volume, and the control server needs outbound internet. Mesh traffic is encrypted by WireGuard between nodes, not on the local hop to the provider container, and the mesh adds no authentication beyond the provider's own.
- **Postgres read replicas** ([database replicas](https://nebulactrl.com/docs/operations/database-replicas)): the replication role (`nebula_r_<id>`) has the REPLICATION privilege only, a per-replica scram password (generated, sealed, delivered at send time, never in a payload, the audit log, the API or the logs) and a managed `pg_hba.conf` entry; the primary's WAL retention for a dead replica is capped (the smaller of 4 GiB and 10 % of the disk). A replica keeps **Block outbound internet** and reaches its primary only through the environment network or the mesh link. **Promote is manual and authorised like a production deploy**: `confirm` with the replica's slug, the production approval where it applies, and an explicit `acceptDataLoss` for a forced Promote. The old primary is **fenced** (stopped, restart policy `no`, never started again by a deploy) so two writers cannot run through the platform, and a fenced service has no "unfence". Residual risk: replication is asynchronous, so a forced Promote loses what the replica had not received, and if the old primary's host returns and Docker restarts it before the agent reconnects, writes in that window are lost and diverge. A replica is a copy of the data: anyone who can read the replica's service can read the whole database.
### 5. MCP (optional)
Enable with `NEBULACTRL_MCP_ENABLED=true`; otherwise `/mcp` returns 404. When on:
- **Auth:** bearer API token only; **60 requests per minute per token**; every call is audited with actor type `mcp` and the client's name.
- **Authorization in the server, not the prompt:** each tool maps to the scopes of the REST operations it calls (`policy.Require` applies in the domain service, on every call), and a token **lists only the tools it holds every scope for**. Toolsets only reduce what is listed: a client is offered `core` and `observe` by default, the URL (`/mcp?toolsets=…`) opts in to more, and an explicit list also fences `find_operation` / `call_operation` to those toolsets. They never grant anything a token's scopes do not. Reading logs needs its own scope, `logs:read`. Create tokens for agents with the *MCP read-only* preset or *MCP deploy*, and restrict them to specific projects. See the [MCP reference](https://nebulactrl.com/docs/reference/mcp#scopes-and-token-presets).
- **Production guard (server-side, fail-closed):** when an MCP principal changes a **production** environment (deploy or apply, rollback, restart, stop, delete, restore, one-off command, deleting a volume, domain, environment or project, and the other operations listed in the [MCP reference](https://nebulactrl.com/docs/reference/mcp#production-approval)), the domain service refuses unless a person has approved that exact action; if the approval store is unavailable it refuses too. The client is asked through MCP elicitation, or gets a plan and a link to `/approvals/{id}` in the web app (session users only: Owners, Admins and Developers with deploy rights decide; the approval endpoints are denied over MCP). An approval is single-use and bound to the token, the tool, the arguments and the generations that were reviewed, lives at most 10 minutes, and any change in between voids it. A model-supplied `confirm` is never accepted for production. The only bypass is the per-token setting `allowProductionWithoutApproval` (**Let MCP clients change production without approval**): off by default, settable only by an Owner or Admin, honoured only while the creator still holds that role, and audited. Plain REST and CLI tokens are **not** guarded unless created with `requireProductionApproval`, so CI and deploy hooks keep working. Every outcome (requested, approved, declined, expired, refused, error, used) is audited. Non-production environments need no approval.
- **Staged changes:** MCP edits to services, variables, domains, volumes and links only stage; `apply_changes` is the only path to a running container and it requires the generation the client reviewed.
- **Bounded output:** `tail_logs` returns at most 500 lines, redacted; any tool result is capped at 64 KB with `truncated: true`. Log text is wrapped as untrusted with a per-call marker.
- **Deliberate exec surface: `run_command`.** It is **shell-equivalent** (root-equivalent on that service's data): it runs a command you name in a throwaway container of the service's own image, with the service's environment variables and network, so a token that may call it can read every secret that service holds and do anything the service can. It needs `deployments:write`, is subject to the production guard, waits at most `waitSeconds`, returns bounded redacted output and is audited with the command line (clipped). Do not give it to a token an agent uses on untrusted input.
- **Denied surface (the catalog's deny list):** no sign-in or account management, token, enrollment-token, invite and reset-link minting, deploy hook enabling or rotation, secret or credential reveal, integration credential writes, instance and SSO settings, team and role changes, the `unrestricted` switch, host shell and host hardening, approving an approval; no streaming follow modes. Everything else the API offers is available, behind its scopes, through curated tools or `find_operation` / `call_operation` (denied operations are never found nor callable). There is no raw Docker API or arbitrary SQL, and the web terminal and file upload are not exposed. No tool returns a secret: `service_get` returns configuration with secrets redacted, and a variable set through MCP is never echoed back.
- **Errors** are structured `{code, message}`; an acknowledged command whose agent is offline is `outcome_uncertain`.
- **Prompt injection:** deployment/runtime logs and repository content are attacker-controlled text that an LLM client may read. The server cannot stop a client from acting on injected instructions. The production guard puts a person between an injected instruction and production, but it does not protect other environments: **limit what the token can do** (read-only, project restricted), and keep deploy tokens and `run_command` away from agents that ingest untrusted logs.
## Data in transit and at rest
| Data | Protection |
|---|---|
| Browser ↔ control plane | TLS terminated by the host's own Traefik (the dashboard's system route, Let's Encrypt) with `--domain`, or by your proxy; without a domain the dashboard is plain HTTP and the installer says so |
| Agent ↔ control plane | TLS (`wss://`) plus Ed25519 challenge identity at the application layer (not mTLS) |
| Secrets in Postgres | AES-256-GCM; the key ring has key ids in the ciphertext, every value is bound to its table, column and row (a copied ciphertext does not open), and `nebulactrl-server rekey` rotates keys; releases keep their sealed environment only inside the rollback window. Details: [secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest) |
| Everything else in Postgres | **not encrypted by NebulaCtrl**: use disk encryption/storage-level encryption if required |
| Secrets in agent memory/containers | environment variables of the running container (visible to anyone with Docker access on the host) |
| Registry credentials | sealed with the key in Postgres; reach agents only in the command frame; a temporary 0700 `DOCKER_CONFIG` for the length of one build; managed registry: an expiring credential scoped to one repository, exchanged for a JWT valid at most 15 minutes |
| Images between build server, registry and target | TLS to the registry (required for non-loopback hosts); pulls verify the digest |
| Backups | contain ciphertext plus the rest of the state; keep the key elsewhere ([backup](https://nebulactrl.com/docs/operations/backup-restore)) |
## Threats and mitigations
| Threat | Mitigation | Residual |
|---|---|---|
| Stolen admin password | argon2id, login rate limit, HTTPS-only cookie, optional passkeys (WebAuthn, phishing-resistant, bound to the public hostname) and TOTP two-step verification with recovery codes, per-team "require 2FA" ([login methods](https://nebulactrl.com/docs/security/login-methods)) | a person who never enrols is protected by the password alone; `/mcp` is not subject to the team 2FA rule |
| Stolen API token | hashed storage, scopes, project restriction, expiry, revocation, bearer-only; over MCP the production approval applies even to a stolen token | token is a bearer secret; a token with `allowProductionWithoutApproval`, or `run_command`, is as powerful as the role that created it; REST tokens are not production-guarded by default |
| An AI client steered by injected text (a log line, a README) into a harmful action | scopes and toolsets, project restriction, staged edits that need a reviewed generation, server-side human approval for production, deny list for credentials and shells, untrusted-text wrapping of logs, audit | non-production environments take changes without approval; `run_command` is shell-equivalent; the client can still act on injected text within the token's scopes |
| Cross-site request forgery | `SameSite=Lax`, cross-origin protection | none known |
| IDOR across projects and teams | 404 on policy mismatch, workspace/project joins | teams share one database and one key: a bug here is a cross-team leak, not a contained one |
| Rogue/compromised agent | identity per host, closed command set, revocation, cannot read other hosts' secrets | root on the host |
| Cross-project or cross-team traffic on a host | each environment has its own Docker network and only Traefik, the pages server and cloudflared are on the shared `nebulactrl-edge`; a service reaches only its environment, the shared networks it was explicitly attached to, and (unless outbound is blocked) the internet; shared networks are team-level, one server each, and never cross teams (404 across teams, 422 `network_wrong_server`) | same host, same kernel and same Docker daemon: a container escape or root on the host defeats every network boundary; a service that may reach the internet can reach any address the host can route to (LAN, cloud metadata) |
| A compromised app calls out or exfiltrates data | per-service **Block outbound internet**: the container joins only an `internal` Docker network that has no gateway (on by default for managed databases) | all-or-nothing per service, no per-destination rules; a service that is allowed out is not filtered at all; traffic through the proxy, a tailnet sidecar or the agent (backups) is deliberate |
| Cross-environment or cross-server reach through the mesh | one link node per environment and server pair, with its own source IP; generated deny-by-default rules that list only granted (server-node IP, port) pairs; a grant that joins a link to another environment is refused (`mesh_environment_mismatch`); replication and service links are separate nodes (`mesh_purpose_mismatch`); links and devices are added only by explicit actions (deploy of the consumer, a confirmed canvas edge, a replica or Promote) and removed after a grace period | an advisory Tailscale policy that is looser than the generated rules still allows more; the connectivity check proves intent, not exclusivity; a Tailscale policy token (`policy_file`) can rewrite the whole policy; managed Headscale uses Tailscale's public DERP relays as the fallback |
| A wrong or unwanted Promote | manual only, `confirm` by slug, production approval, a plan that lists the data at risk, the consumers per server and what will be rewritten; a switchover freezes, stops and fences the old primary and waits for the replica to catch up (and rolls back if it cannot); consumer rewrites are staged and can be discarded | forced mode loses up to the replication lag by design; the old primary can restart before it is fenced on a forced Promote; there is no automatic detection of a lost primary |
| Platform services exposed to the internet (S3, Loki, Headscale, sign-in) | each is a normal service behind the server's proxy with a generated hostname and Let's Encrypt certificate (an ACME email is required, or the set-up refuses); images pinned by digest and run **unmodified**; outbound access blocked except Headscale; credentials generated, sealed, delivered as secret variables and never in configuration files; S3 requires a signed request (a wrongly signed one is refused), Loki sits behind basic auth, Authelia's configuration holds no secret (they are read from the environment) and its users have argon2id hashes; all of it is created by an administrator (scope `admin`, and an instance administrator for single sign-on) and audited | the endpoints are reachable from the internet like any domain (restrict by IP in the domain's access rules if you can); the S3 store on the databases' own server is not an off-site backup; an `http://` public URL carries S3 traffic in clear text; Loki and Grafana are AGPL-3.0 ([platform services](https://nebulactrl.com/docs/operations/platform-services)) |
| Replay of agent auth | fresh nonce per connection, 30 s window | none known |
| Malicious or stale one-click template | templates ship inside the binary after review (no runtime fetch); images pinned by tag **and** digest, checked against the registry without pulling; strict schema (no shell strings, no host mounts/privileged); file contents are plain text and rejected inside volume paths; generated secrets are stored encrypted and never returned | the app's own licence and security are the upstream project's; source-available and proprietary templates are labelled |
| Database leak (without key) | secrets are ciphertext | other state readable: variable names, secret flags, topology, domains, git URLs, audit log |
| Database write access (SQL injection, a bad restore script) without the key | a sealed value is bound to its table, column and row: copying one secret's ciphertext into another row or column fails to decrypt instead of delivering that secret somewhere else | the attacker can still delete or corrupt values (denial of service) |
| Database leak + key | none | full secret disclosure of every team; rotate everything. The key and the database sit on the same host in the default install, so a compromised control-plane host is this case |
| Old secret values kept in releases | each release stores its sealed environment; an hourly job drops it from releases outside the rollback window (the newest `imageRetention` succeeded ones); a rollback to a pruned release uses the current variables and says so | values inside the window, and in database backups taken before they were pruned, stay until the backup expires |
| Secrets in control-plane memory | no core dumps, process not dumpable (Linux), key bytes not kept after start-up, external secret-manager values expire from memory after 30 minutes | root or a debugger on the control-plane host can read the process; Go strings cannot be wiped |
| Secrets in logs/audit/payloads | redaction (6+ chars), key-only audit, send-time secret attachment, no secret in logs | short secrets, secrets printed in encoded forms |
| Malicious Dockerfile/image | host-level builds are *explicitly trusted* | host compromise possible |
| Compromised or malicious build server | closed command set, dedicated host (role *Builds*) with no workloads and no runtime secrets, per-project builders, revocation of the agent identity, per-build temporary registry credentials; images are deployed by digest, so what a build server pushed later cannot replace a release already recorded | the build host sees the build inputs and push credential of every project; a privileged-builder escape compromises all of them; a malicious build can push a bad image that the **next** deployment will run |
| Untrusted pull-request code on a shared build server | previews do not build on a shared build server unless **Allow previews** is on for it ([previews](https://nebulactrl.com/docs/operations/previews#where-previews-build)) | with it on, fork code runs next to other projects' builds and the push credential |
| Registry leak, deletion or outage | credentials sealed, short-lived for the managed registry; images pinned by digest; the target keeps `imageRetention` local images; a rollback fails before touching the running release; a deploy falls back to building on the target when no registry is usable | a BYO lifecycle rule or an out-of-band delete removes rollback targets; a registry compromise is a code-execution path to every server that pulls from it |
| Malicious or over-privileged Compose file (stack) | unless the instance admin made the server **unrestricted**, the agent normalises the file with `docker compose config` and refuses privileged mode, host network/PID/IPC/UTS/user namespaces, devices, unsafe capabilities, security options other than `no-new-privileges`, host bind mounts (including the Docker socket), external or renamed volumes and networks, volume/network `driver_opts` that mount host paths, `file:` secrets and configs and reserved labels; the control plane runs the same check on save; compose runs in a clean environment (no agent secrets), with project, container names and networks fixed by NebulaCtrl's override; ports are never published on the host ([Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks), [ADR 0007](https://nebulactrl.com/docs/decisions/0007-compose-stacks)) | Docker is still root: the check is a deny list on the normalised model, not a sandbox, and a new Compose key that grants host access is allowed until the list learns it; on a server an instance admin made **unrestricted** any deployer of a stack can take over the host (audited, warned) |
| Over-privileged Docker options on a plain service (devices, GPUs, runtime, privileged, host PID, unconfined seccomp/AppArmor, `NET_ADMIN`, `SYS_ADMIN`) | typed fields only (no free-text flags); these options work **only on a server the instance admin made unrestricted** (instance admin only, name typed to confirm, audited as `server.unrestricted.enable`/`.disable`); refused with 422 `requires_unrestricted_server` on save, on a move to another server, on template install and at every deploy, rollback, reconcile and scale-up; the agent re-validates every option against the server policy carried in the deploy payload; never possible: host networking, `--ip`, restart and log options, host IPC, host binds, the Docker socket; hardening options (`capDrop`, `noNewPrivileges`, `readOnlyRootfs`, `pidsLimit`) and a curated list of namespaced sysctls are available everywhere ([Docker options](https://nebulactrl.com/docs/operations/docker-options)) | on an unrestricted server any deployer can take the host over; a device node, GPU driver or toolkit bug is a way out of the container; hardening is opt-in per service; host sysctls such as `vm.max_map_count` are only warned about in the plan |
| SSRF via git URL / registry | URLs are only used by the control plane for `git ls-remote` of the configured repository and by the agent for pulls; a registry host is contacted by the build and target agents (and plain-HTTP only on loopback) | the control plane connects to whatever host the URL names; restrict egress if needed |
| Command injection | typed commands, argument vectors, no shell | none known |
| Hostname hijack/conflict | global unique hostnames, 409 on conflict, DNS checks | DNS correctness is operator-owned |
| Bypassing Cloudflare by connecting to a proxied domain's server IP (skips its WAF, rate limits, bot rules and IP rules) | the **origin lock** ([Cloudflare](https://nebulactrl.com/docs/operations/cloudflare#origin-lock)), on by default for every proxied domain: Traefik answers `403` unless the TCP peer is in Cloudflare's published ranges, and, while the zone's Authenticated Origin Pulls is on, a TLS handshake without Cloudflare's client certificate (pinned CA) fails. Applied per route in Traefik, not by host firewall rules (Docker-published ports bypass `ufw`) | both layers prove "comes from Cloudflare", not "from **your** Cloudflare account": anyone who learns the IP can point their own zone at it and pass; hide the IP or use a Cloudflare Tunnel (no open port). Opting a domain out, DNS-only domains, tunnel domains, domains whose DNS NebulaCtrl does not manage, wildcard hosts and TCP/UDP routes are not locked. AOP is IP-only when the token lacks *Zone Settings: Edit* or the zone is Flexible/Off (shown in the panel) |
| A forged visitor address (`X-Forwarded-For`, `CF-Connecting-IP`) to slip past IP allow/deny rules, rate limits or to pollute analytics | the entrypoints trust forwarded headers only from Cloudflare's ranges (Traefik discards them from every other peer); on a locked domain the visitor is the **last** `X-Forwarded-For` entry (the address Cloudflare saw connect), entries to its left are never read, and a user-set `trustedProxyDepth` is ignored; the Cloudflare lock checks the TCP peer, the user's rules check the visitor, as two separate middlewares | on a domain that is **not** locked (DNS-only, opted out) the forwarded header is untrusted: rules see the connecting address, and a depth chosen by the operator is only as safe as the CDN in front |
| A stale, poisoned or unreachable Cloudflare IP list (lock too wide, or visitors blocked) | the list is fetched every 3 hours from Cloudflare's API with the published text files as fallback; an answer is refused unless it has valid CIDRs of the right family, none wider than `/8` (v4) or `/16` (v6), at least 10 v4 and 5 v6 ranges and no more than a 50% shrink; the last good list is kept on failure; a binary-embedded list covers first start and offline installs; every change is audited (`cloudflare.ips.updated`) and announced, and 24 hours of failures raise a notification | a range Cloudflare adds while fetching is down is blocked until the next successful fetch; a hostile answer that passes the plausibility checks would be accepted (the fetch is TLS to Cloudflare's hosts only, no credentials are sent) |
| A tampered or expired pinned origin-pull CA | embedded in the binary with its source URL and SHA-256 fingerprint pinned in code; start-up fails when they differ, and a unit test fails 180 days before the certificate expires (2029-11-01) | the CA is refreshed with a release, not at runtime |
| Over-reaching Cloudflare token (Zone Settings: Edit to turn on AOP) | the permission is optional and requested only for the origin lock's AOP layer; NebulaCtrl changes exactly one zone setting (`tls_client_auth`), only to turn it on and never off, and audits it (`cloudflare.zone.aop.enable`) | the token can do whatever its permissions allow in the Cloudflare account; limit its Zone Resources |
| Lost/changed secret key | a key ring (previous keys keep old values readable), `rekey`, documented backup of the key apart from the database | losing every key that sealed a value loses that value; no recovery |
| Control plane outage | agents keep workloads running, reconcile on reconnect | no HA |
## Hardening checklist
1. Serve the control plane only over HTTPS; keep `NEBULACTRL_PUBLIC_URL` `https://`.
2. Keep Postgres unpublished (the compose file does); encrypt the volume at rest if your threat model needs it.
3. Deliver `NEBULACTRL_SECRET_KEY` as a file (`NEBULACTRL_SECRET_KEY_FILE`, mode 0400) or from a vault rather than an environment variable, and back it up apart from the dumps. Rotate it with `rekey` ([secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest#rotating-the-key)).
4. Create tokens with the narrowest scopes and a project restriction, with an expiry; revoke unused ones.
5. Keep MCP off unless needed; if on, start with the *MCP read-only* preset, leave **Let MCP clients change production without approval** off, and withhold `deployments:write` from agents that read untrusted logs or repositories.
6. Restrict who can reach the control plane (firewall/VPN) when possible; agents only need outbound access to it.
7. Update Docker, the host OS and the agent regularly; use hosts dedicated to NebulaCtrl workloads.
8. Review the audit log (**Settings → Audit log**).
9. Deploy only code and images you trust; pin image digests for critical services.
10. Rotate affected secrets after revoking a suspected host.
11. Turn on two-step verification for your own account and require it for the team (**Settings → Security**); keep the recovery codes offline. Leave open sign-up off and set allowed email domains on every single-sign-on provider ([login methods](https://nebulactrl.com/docs/security/login-methods)).
12. Block outbound internet for every service that does not need it (managed databases start with it on) and keep shared networks few and named for a purpose ([networking](https://nebulactrl.com/docs/operations/networking)).
13. If you use build servers: dedicate them (role *Builds*), keep them patched and free of other workloads, give the push credential the least scope the registry allows (a bot account, a repository prefix), use a separate read-only pull credential, leave **Allow previews** off, do not enable "delete untagged" lifecycle rules in a BYO registry, and revoke a suspect build server like any host ([build servers](https://nebulactrl.com/docs/operations/build-servers)).
14. If you use the [server mesh](https://nebulactrl.com/docs/operations/mesh): leave Tailscale on the **advisory** ACL mode unless you accept that the control plane then holds a token that can rewrite your whole policy, tighten an allow-all tailnet, keep a dedicated tag (`tag:nebula-mesh`) and review the generated rules (**Settings → Mesh → Preview rules**); remember that managed Headscale uses Tailscale's public DERP relays. For [read replicas](https://nebulactrl.com/docs/operations/database-replicas), keep backups of the primary (a replica is not one) and rehearse a Promote on a non-production environment first.
---
# Secrets at rest
Source: https://nebulactrl.com/docs/security/secrets-at-rest
NebulaCtrl stores every secret it can read back (variable values, git tokens, registry and cloud credentials, SSH deploy keys, TOTP seeds, backup passwords) encrypted in Postgres with AES-256-GCM. This page says what that protects, how the format works, how to rotate the key, and what the sealing does **not** do. The wider picture is in the [threat model](https://nebulactrl.com/docs/security/threat-model); external secret managers (`${{vault:…}}`, Doppler, Infisical) that keep values out of the database altogether are in [secret managers](https://nebulactrl.com/docs/operations/secret-managers).
## What it protects, and what it does not
| Situation | What an attacker gets |
|---|---|
| A Postgres dump, replica, backup or volume snapshot **without** the key | Ciphertext for every sealed column. Not protected: variable names, secret flags, topology, domains, git URLs, the audit log, argon2id password hashes |
| The same **plus** the key | Everything. Rotate every secret |
| Write access to Postgres without the key (SQL injection, a bad restore script) | Cannot read secrets, and cannot move one: a ciphertext is bound to its table, column and row, so copying it into another row or column fails to decrypt (below) |
| Root on the control-plane host, or anyone with Docker access there | The key (`docker inspect` shows an environment variable; a key file does not appear there) and the database. The key and the database live on the same host in the default install, so sealing protects against **off-host** leaks, not against a compromised control-plane host |
| Root, or Docker access, on an app host | The plaintext environment of every container there. Environment variables are how containers receive secrets; they are visible in the Docker daemon on that host |
**One key serves the whole instance**, every team included. Teams are an authorization boundary, not a cryptographic one ([threat model](https://nebulactrl.com/docs/security/threat-model#security-boundary-in-one-paragraph)).
## Ciphertext format
Every sealed value starts with a version byte.
```
v2 (written today) 0x02 ‖ keyID (4 bytes) ‖ nonce (12 bytes) ‖ AES-256-GCM(ciphertext ‖ tag)
AAD = 0x02 ‖ keyID ‖ "<table>:<column>:<rowID>"
v1 (legacy, readable) 0x01 ‖ nonce (12 bytes) ‖ AES-256-GCM(ciphertext ‖ tag)
AAD = 0x01
```
- **Key id.** The first four bytes of an HKDF-SHA-256 derivation of the key (label `nebulactrl secret-key-id`). It tells `Open` which key of the ring to use and cannot be reversed to the key. Two keys with the same id are refused at start-up.
- **Context binding (AAD).** The additional authenticated data contains the version, the key id and `table:column:rowID`. The row id is the row's uuid, or for tables keyed otherwise a composite (`service_id/KEY` for service variables, `domain_id/username` for domain basic-auth users, `workspace_id` for the per-team integrations, a constant for the instance-wide singletons). Move a ciphertext to another row, column or table and decryption fails with the same `decryption failed` error every failure returns.
- **v1 values** (written before this format) still open: under any key of the ring, with no context check. `rekey` rewrites them as v2.
- **Nonces** are 96 random bits per seal. At the volumes of a control plane that is far below the invocation limit NIST SP 800-38D sets per key.
- Fingerprints and digests (variable change detection, the build-reuse hash) use a **separate** HKDF-derived HMAC key, never the encryption key itself. After a rotation, stored fingerprints are matched against every key of the ring; the build-reuse hash changes once with the key (one extra build).
## The sealed columns
A central registry lists every sealed column (`internal/secretbox/registry.go`); `rekey` walks exactly that list, and a test fails when a migration adds a `bytea` `*_enc` column that is not registered.
| Group | Columns |
|---|---|
| Variables | `service_variables.value_enc`, `.template_enc`; `shared_variables.value_enc`, `.template_enc`; `variable_versions.value_enc` (non-secret versions only) |
| Releases | `releases.env_enc` (the resolved environment a release was deployed with) |
| Service credentials | `services.git_token_enc`, `.registry_auth_enc`; `service_ssh_keys.private_key_enc`; `registries.push_auth_enc`, `.pull_auth_enc` |
| TLS and proxy | `certificates.key_enc`; `domain_auth_users.hash_enc`; `dns_credentials.secrets_enc` |
| Integrations | `cloudflare_connections.token_enc`; `git_connections.credentials_enc`, `.webhook_secret_enc`; `tailnet_integrations.secret_enc`; `secret_integrations.credentials_enc`; `notification_channels.secrets_enc`; `log_drains.secrets_enc`; `platform_services.secrets_enc`; `geo_credentials.maxmind_key_enc` |
| Backups | `backup_destinations.access_key_enc`, `.secret_key_enc`; `instance_backup_settings.access_key_enc`, `.secret_key_enc`; `volume_repos.password_enc` (the restic repository password) |
| Other | `auth_providers.client_secret_enc`; `instance_settings.smtp_password_enc`; `database_rotations.new_password_enc`; `user_totp.secret_enc` |
`NEBULACTRL_SECRET_KEY` is therefore also the key to your restic backup repositories, every user's TOTP seed and the single sign-on client secret, not only to app variables.
## Configuring keys
| Variable | Meaning |
|---|---|
| `NEBULACTRL_SECRET_KEY` | The **current** key: base64 of 32 random bytes (`nebulactrl-server keygen`). New values are sealed with it |
| `NEBULACTRL_SECRET_KEY_FILE` | Path to a file holding the current key (a Docker or systemd secret; surrounding whitespace is ignored). Setting both this and `NEBULACTRL_SECRET_KEY` is an error |
| `NEBULACTRL_SECRET_KEY_PREVIOUS` | Optional list of earlier keys (comma, space or newline separated). They only **open** existing values |
| `NEBULACTRL_SECRET_KEY_PREVIOUS_FILE` | The same, read from a file |
A key file keeps the key out of `docker inspect` and `/proc/<pid>/environ` of the control-plane container. Mount it read-only, mode 0400, owned by the container user; with Compose use a [`secrets:`](https://docs.docker.com/compose/how-tos/use-secrets/) entry and set `NEBULACTRL_SECRET_KEY_FILE=/run/secrets/nebulactrl_secret_key`. It does not protect against root on the host. The installer still writes the key to `/opt/nebulactrl/.env` (0600); switching an installed instance to a file is a manual step.
At start-up the control plane logs the ring (`secret key ring`, with the current key id) and, when values are not on the current key or still in the v1 format, a warning with their number and the hint to run `rekey`.
## Rotating the key
```
nebulactrl-server keygen # 1. a new key
# 2. NEBULACTRL_SECRET_KEY=<new> NEBULACTRL_SECRET_KEY_PREVIOUS=<old> → restart the control plane
nebulactrl-server rekey --dry-run # 3. what would change, and whether everything decrypts
nebulactrl-server rekey # 4. re-encrypt everything to the new key
# 5. remove NEBULACTRL_SECRET_KEY_PREVIOUS → restart
```
Run `rekey` with the same environment as the server (it reads the same variables, plus `NEBULACTRL_DATABASE_URL`). In the standard install: `docker compose exec control-plane nebulactrl-server rekey`.
What `rekey` does:
- Walks every registered column and rewrites each value that is not already a v2 value on the current key. That includes **v1 values**: running it once after upgrading to this format upgrades them under the existing key even if you never rotate.
- Works in batches (`--batch`, default 200 rows), each in its own transaction that locks its rows. It **resumes**: rows already on the current key are skipped, so an interrupted run is finished by running it again, and a second run changes nothing. It can run next to a live control plane (rows are locked while rewritten, and the update re-checks the old ciphertext). A session advisory lock stops two runs at once.
- Rebuilds the AAD from the row's own key columns, so a ciphertext someone copied into the wrong row is **not** re-sealed (that would launder it): it is reported and left untouched.
- Finishes with a verify pass that opens every value with the **current key alone**. Only when that passes does it print that the previous keys can be removed.
- `--dry-run` decrypts everything that needs rewriting and writes nothing.
- Exit code: non-zero when any value could not be decrypted with the keys in the ring (a key was removed too early, or data is corrupt). The report lists the columns and row ids, never values.
Things to know:
- **Keep the old key** as long as database backups from before the rotation might be restored: those backups hold ciphertext on the old key. The backup of the control plane never contains a key ([backup and restore](https://nebulactrl.com/docs/operations/backup-restore)).
- Old keys remain valid forever in `NEBULACTRL_SECRET_KEY_PREVIOUS`; the list only costs a few microseconds per open. Remove them when you are done.
- Rotating the key does **not** rotate the secrets themselves. If the old key leaked together with a database copy, every secret in that copy must be rotated at its source.
- Variable version history stores fingerprints only for secrets (never values); old fingerprints stay valid because they are matched against every key of the ring.
- If the key is lost and no backup exists, the sealed values are gone: variables, git and registry credentials, SSH keys, integration tokens, TOTP seeds and the restic password of volume backups. There is no recovery. Back the key up first.
## Release secret retention
Every release stores the sealed environment it was deployed with (`releases.env_enc`), secret values included, so a rollback restores exactly what ran. Without limits, a variable you deleted or rotated would stay in the database, and in every dump, as long as the service exists.
An hourly job therefore clears `env_enc` from every release **outside its service's rollback window**: the newest `imageRetention` succeeded releases (default 5, the set a rollback can target), plus the release of the running deployment and the release of any deployment in flight. The release row, its image reference, source and configuration stay.
Rolling back to a release whose environment was cleared **does not start the old image with an empty environment**. The deployment resolves the service's **current variables** instead, records that in the deployment timeline, and the rollback response carries the same notice. If the current variables do not resolve (an unresolved reference, an unreachable secret manager), the rollback fails with the reason and changes nothing. After a successful rollback the release records the environment it was deployed with and is inside the window again until it ages out.
Consequence: rolling back past the window restores the old code and configuration, **not** old secret values. Values inside the window are restored as they were. Database backups taken before a release was pruned still contain its sealed environment until they expire.
## Process hygiene
- The control plane sets `RLIMIT_CORE=0` and (Linux) `PR_SET_DUMPABLE=0` at start-up: no core dumps, and no `ptrace` or `/proc/<pid>/mem` access for other processes of the same user. `rekey` does the same.
- Key bytes read from the environment or a key file are zeroed after the key ring is built. Decrypted plaintext buffers are zeroed after use where the code holds them as bytes. Go strings cannot be wiped, so a decrypted value that becomes a string lives until it is collected.
- Values fetched from an external secret manager are kept in memory only to mask them in logs, for at most 30 minutes (re-fetched on demand), never longer than the process.
- Secrets are never written to logs: the structured logs name keys, ids and counts, not values; log lines from your apps are redacted for every secret of six or more characters (plus its URL-escaped and base64 forms) before they are stored or shown.
- There is no `mlock` and no TPM or cloud KMS sealing of the key; these were weighed and are not built.
## Who can read secrets, and what is audited
Secret variables are write-only: no API, UI, CLI or MCP read returns them, and a Viewer never sees any value. The paths that do reveal a value are audited without it: a managed database's connection string (`secret.reveal`), platform service credentials (`platform.reveal`) and a volume backup's recovery key (`volume_backup.recovery_key.reveal`). Reads from an external secret manager are audited per operation as `secret.fetch` / `secret.fetch.denied` (references, service and who started it, never values), and an integration can be limited to selected projects and environments ([secret managers](https://nebulactrl.com/docs/operations/secret-managers)). Sealing changes record the key name only.
## See also
- [Threat model](https://nebulactrl.com/docs/security/threat-model): trust boundaries, including teams.
- [Variables](https://nebulactrl.com/docs/operations/variables): secret flags, write-only values, send-time delivery.
- [Backup and restore](https://nebulactrl.com/docs/operations/backup-restore): the key is never in a backup.
- [Secret managers](https://nebulactrl.com/docs/operations/secret-managers): keep values out of the database.
---
# Platform services
Source: https://nebulactrl.com/docs/operations/platform-services
Some features need a service you may not have: somewhere to store backups, a log store, a tailnet control server, a single sign-on provider, a container registry for build servers. **Set up for me** runs one for you, as a managed service NebulaCtrl owns, and connects it to the feature that asked for it with generated credentials. You can still point the same features at your own S3, Loki, Tailscale, identity provider or registry; nothing here is required. For single sign-on, connecting the identity provider you already have (Settings, Instance, Single sign-on, **Connect your identity provider**) is the main path; **Set up for me** sits underneath it for teams that have none.
| You are on | The button creates | Wired into | Product |
|---|---|---|---|
| Settings, Backups | **S3 storage** | a backup destination named "NebulaCtrl S3" (bucket created, tested from the server) | [SeaweedFS](https://github.com/seaweedfs/seaweedfs) 4.51, Apache-2.0 |
| Settings, Log drains | **Log store** | a Loki log drain named "NebulaCtrl Loki" (basic auth, tested from the server) | [Loki](https://github.com/grafana/loki) 3.7.9, **AGPL-3.0** |
| Settings, Log drains (optional) | **Grafana** | Loki added as a data source | [Grafana](https://github.com/grafana/grafana) 13.2.4, **AGPL-3.0** |
| Settings, Integrations, Tailscale | **Tailnet control server** | the Tailscale integration, provider Headscale (a user and an API key, connection tested) | [Headscale](https://github.com/juanfont/headscale) v0.29.6, BSD-3-Clause |
| Settings, Instance, Single sign-on | **Single sign-on** (your choice of product) | a sign-in provider (OpenID Connect) | [Pocket ID](https://github.com/pocket-id/pocket-id) v2.19.1, BSD-2-Clause (preferred, passkeys only), or [Authelia](https://github.com/authelia/authelia) 4.39.28, Apache-2.0 (passwords + 2FA) |
| Settings, Registries | **Container registry** | a registry named after the service, kind *managed*, used by [build servers](https://nebulactrl.com/docs/operations/build-servers#set-up-for-me-managed-registry) | [CNCF Distribution](https://github.com/distribution/distribution) `registry:3.1.2`, Apache-2.0 |
Why these images, how they compare with the alternatives and which rule picked them: platform-services-research.
The page that lists, repairs and removes them is **Settings, Platform services**. Owners and Admins only (scope `admin`); single sign-on is instance-wide, so it needs an **instance administrator**.
## Where they live
Every team gets one hidden project, **NebulaCtrl platform**, with one environment, `platform`. It is not in the projects list or the switcher; the page links to its canvas ("Open in canvas") when you need to look at a service, its logs or its deployments. It cannot be renamed or deleted (409 `system_project`), and it is **visible to administrators only**: people without the `admin` scope get 404 for the project, its environment and its services (and the actions on them). The services in it are ordinary services: they show up on the server's page and in the fleet overview, count towards the server's capacity, are deployed by the normal engine and can be stopped, restarted or inspected like any other. Do not delete them there: the platform page would show them as removed outside it. Remove them from the platform page.
Each kind exists **once per team**. A server flagged "does not accept workloads" cannot host them, like any service.
## What a set-up does
`POST /platform-services` stages everything in one transaction (the service pinned by digest, its volume, its variables, its public hostname, an audit entry) and starts a job. The job then:
1. **configure** (Loki: protects the public route with basic auth);
2. **deploy** the staged service through the normal deploy path (so cancel, logs and the deployment history work) and **await** it;
3. **wire**: wait until the public URL answers, create the integration with the generated credentials, and **test** it the way the feature itself would (the destination test and the drain test run from the agent; the Tailscale test mints and expires a key). A test that fails is retried for up to 15 minutes (certificates and DNS take a while) and then the service is **failed** with the reason.
A failed service keeps what it created; **Retry** resumes at the step that failed (a failed deployment is deployed again). **Reconnect** runs the wiring again on a ready service (Headscale mints a new API key).
### Hostname, TLS and the public URL
Every one of them is reached through the server's proxy at a public hostname, because the things that talk to it run elsewhere: the agents (backups, log shipping), the control plane (bucket creation, sign-in) and your people's browsers and Tailscale clients. The hostname is generated like any generated domain (`<name>-platform-nebulactrl-platform.<base domain>`, else `<public ip>.sslip.io`); enter your own to override it. Certificates are Let's Encrypt, so the server needs an **ACME email** (422 `acme_email_required` otherwise: without it the proxy serves a self-signed certificate that clients refuse). `.localhost` names are exempt, for local use.
A **public URL** that is `http://` (or that names a non-standard port, such as `https://s3.example.com:8444` for a proxy that is not on 443) is accepted as typed. An `http://` URL is served without the redirect to HTTPS; use it only on a trusted network or for local tests, because S3 requests would travel in clear text (signed, but not encrypted).
### Credentials
NebulaCtrl generates every credential (S3 key pair, Loki basic-auth password, Grafana admin password, Authelia secrets and signing key, Pocket ID's encryption key and API key, the OpenID client secret). They are stored sealed with the instance key, the service ones as secret variables (masked everywhere, delivered to the agent only at deploy time) and never appear in a configuration file, a view, the audit log or a command row. The **Show credentials** action on the page reveals S3, Loki and Grafana credentials (audited as `platform.reveal`); Headscale's API key is created inside the container and kept only by the Tailscale integration; the sign-in provider's client secret is kept only by the sign-in provider. With Authelia, the first person gets a one-time password shown once; with Pocket ID you create a one-time sign-in link (see below). **Back up the instance key** (`NEBULACTRL_SECRET_KEY`): it protects every sealed credential, including Pocket ID's `ENCRYPTION_KEY`, without which a restored Pocket ID data volume cannot be read.
### Outbound access
Everything except Headscale starts with **Block outbound internet** on (see [networking](https://nebulactrl.com/docs/operations/networking)); they only talk to the other services of their environment and to the proxy. Pocket ID needs no outbound access either: its analytics heartbeat, release check and icon library are switched off, and GeoLite2 is only downloaded when a MaxMind key is set (it is not). This was checked by running the pinned image on a Docker network without a route to the internet. Headscale needs the internet to fetch the DERP relay map, so it is not blocked. Everything runs from the pinned digests listed above; there is no pull of "latest".
## S3 storage
SeaweedFS runs as one process (`weed server -s3`: master, volume server, filer and S3 gateway) with the key pair from the environment, one bucket (default `nebulactrl-backups`, `bucket` at set-up) and path-style access. The destination uses the prefix `nebulactrl/`.
**Same server, not off-site.** If the store runs on the server that also runs the databases it backs up, a backup there does not survive the loss of that server. The page and the destination row say so ("Same server, not off-site"); place the store on **another** server to make it off-site, or add an off-site destination as well. The store's own data volume is not scheduled for backup (it holds your backups; backing up backups into the same place is circular): if you want a copy, back its volume up to another destination from the service's **Backups** tab.
## Log store and Grafana
Loki runs as one process with filesystem storage (`tsdb`, schema v13), retention through the compactor (default **14 days**, `retentionDays` at set-up; the retention applies to what the drains ship, independent of the 7 days each server keeps), usage reporting off. Loki has no authentication of its own, so the public route carries basic auth and the drain sends the same credentials. Logs are searchable in Grafana (below) or any Loki client; the in-product log views keep reading each server's own history.
**Grafana is optional and separate**: it needs Loki to be ready and always runs next to it (a different server would not reach it: there is no cross-host networking). It starts with the admin user `admin` and a generated password and with Loki as its default data source. Signing in to Grafana with your single sign-on provider is not configured.
**AGPL.** Loki and Grafana are licensed under the AGPL-3.0. NebulaCtrl runs the **unmodified upstream images** as separate containers and talks to them over HTTP; it does not link, bundle or modify them, and the licence applies to those programs, not to NebulaCtrl. If you modify and offer them to others over a network, the AGPL's source-offer terms apply to your modified version. The set-up dialog and the card show the licence.
## Tailnet control server
Headscale runs with SQLite on its volume, no embedded DERP, MagicDNS under `tailnet.internal`, and a **database-backed policy** (`policy.mode: database`) that NebulaCtrl owns: the user `nebulactrl` owns `tag:nebula-svc` and `tag:nebula-mesh`, members reach tagged services, and the [server mesh](https://nebulactrl.com/docs/operations/mesh#access-rules-acl) adds the rules it generates. Anything you change in that policy by hand is overwritten. The wiring creates the user, an **API key valid for 365 days** (`headscale apikeys create`, run in the container), writes the base policy before any node can join, and creates the Tailscale integration with provider Headscale, credential "API key" and `tag:nebula-svc`. The page shows the key's expiry; **Reconnect** mints a new one before it runs out. If a Tailscale integration already exists the set-up stops with `integration_exists` unless you confirm **Replace it**. On removal the integration is deleted only if it is the one this service created.
**Existing installs are migrated once, with the mesh paused.** A Headscale set up before the mesh existed used a policy file. A background job pauses the mesh with the reason "Headscale is being migrated", redeploys Headscale with the database policy mode, waits for it to be healthy, writes the base policy and the mesh rules, reads them back and un-pauses. While it runs, no key is minted and no node is added or removed, replica and Promote operations wait, and a deploy that needs a new link fails with "Not deployed: mesh paused. No changes were made."; **established tunnels, existing nodes and per-service sidecars keep working**. Between the restart and the first policy write Headscale has no policy, which allows everything, for seconds and with nothing able to join. If a step fails the mesh stays paused with the reason; **Retry** on this service resumes it (the job also looks again every 10 minutes and is safe to run more than once). See [the paused migration window](https://nebulactrl.com/docs/operations/mesh#the-paused-migration-window).
**Relays are Tailscale's public DERP servers.** Headscale has no relay of its own here; it hands out Tailscale's public DERP map as the fallback for peers that cannot connect directly. Traffic stays end-to-end encrypted (a relay sees ciphertext and endpoint addresses), but it does leave your infrastructure, and Headscale needs outbound internet to fetch the map. This is shown in the "Set up for me" dialog ([DERP](https://nebulactrl.com/docs/operations/mesh#derp-public-relays-are-the-fallback)). An embedded DERP server is not built.
Tailscale clients need HTTPS on the control URL (the ACME email requirement above). Joining your own devices to this tailnet is not automated: create a pre-auth key with `headscale preauthkeys create --user nebulactrl` from the service's terminal.
## Container registry
CNCF Distribution `registry:3.1.2` (pinned by index digest; an owner exception to the two-year rule, see [ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers)) runs as one process with filesystem storage on a named volume, behind the server's proxy at a public hostname with a Let's Encrypt certificate (the ACME email requirement above applies: Docker daemons only accept a non-loopback registry over TLS). It exists for [build servers](https://nebulactrl.com/docs/operations/build-servers#set-up-for-me-managed-registry): a build server pushes the images it builds here and the servers that run the services pull them by digest. Setting it up creates a registry of kind *managed* under **Settings, Registries**, so nothing has to be wired by hand.
Authentication is **token authentication with the control plane as the token server** (`/registry/token`, Docker token authentication, ES256 tokens signed with a key generated per registry and sealed in the platform service; the registry only holds the public key), not a shared password. There is no stored credential: for each build and each pull the control plane mints a sealed, expiring credential (15 minutes by default, longer for a build, never more than 6 hours) limited to **one repository and the actions needed**, and the token endpoint exchanges it for a token valid at most 15 minutes. A credential for another repository gets no access, a wrong or expired one gets 401, and an address that fails 30 times a minute gets 429. There is nothing to reveal, so **Show credentials** does not apply to this kind. Set-up needs `NEBULACTRL_PUBLIC_URL` (422 `public_url_required`) and an `https` address (422 `https_required`, `http` only for localhost).
NebulaCtrl also owns the registry's retention. An **hourly sweep** deletes the digests of images outside every service's live set through the registry API (the registry runs with deletion enabled). A **weekly window** (starting between 02:00 and 05:00 UTC) collects the blobs nothing references with `registry garbage-collect`, never with `--delete-untagged`: builds for this registry queue, running pushes get up to 2 hours to finish, the registry is set read-only (a short redeploy), the collection runs, and the registry is set back to read-write (a second redeploy). Pulls keep working during the window and pushes get `405`. A window always ends with the restore step; if the registry cannot be redeployed read-write for over an hour the platform service goes `failed` and the registry stays read-only until you retry. Details in [build servers: rollback and retention](https://nebulactrl.com/docs/operations/build-servers#rollback-and-retention). The registry's domain has **large uploads** on: the proxy's request-read timeout (60 seconds by default, long enough to cut a big layer over a slow link) is raised to 30 minutes on that server's entrypoints, which recreates the proxy container once, like an ACME email change; other servers keep 60 seconds. Place the registry on the build server (pushes stay local) or on the control-plane server. A server flagged "does not accept workloads" cannot host it.
## Single sign-on
The dialog asks which identity provider to run. **Pocket ID is preselected and recommended**; Authelia is the alternative ("Need passwords + 2FA instead? Use Authelia"). Only one of them can exist per team; set-up needs an **instance administrator** and `NEBULACTRL_PUBLIC_URL`. Sign-in providers are instance-wide: everyone sees the button (default label "NebulaCtrl SSO"). Accounts are matched the normal OpenID Connect way: by verified email to an existing NebulaCtrl user, else a new account only if open sign-up is on or an invitation is used.
### Pocket ID (preferred)
**Users sign in with passkeys only, no passwords.** Pocket ID runs with SQLite on its volume (`/app/data`), configured entirely by environment: `APP_URL` (the public URL), a generated `ENCRYPTION_KEY` and a generated `STATIC_API_KEY`, `TRUST_PROXY`, analytics, release check and icon library off. Its hostname must be a **domain name** (passkeys are bound to it): an IP address or a single word is refused with `invalid_hostname`, and passkeys need HTTPS in browsers (a `.localhost` name is the exception).
The job then wires it through Pocket ID's REST API with the static API key:
1. **test the discovery endpoint** (`/.well-known/openid-configuration`): the issuer must be the public URL and PKCE S256 offered;
2. **create the first administrator** from your account (username from your email, your name and email, marked as verified) and check that **nobody else has an account**: until then Pocket ID's setup page is open to whoever knows the address, so if any other account exists the set-up **fails** with "its setup page was used before NebulaCtrl could" and wires nothing. Remove it and set it up again;
3. **create the OpenID client** (confidential, PKCE on, no consent screen, callback `<NebulaCtrl URL>/api/v1/auth/oidc/<provider id>/callback`) with a generated secret that NebulaCtrl supplies, so a retried set-up always ends with exactly the stored one;
4. **add the sign-in provider**.
**The one-time sign-in link.** There is no password, so once the service is ready, **Create my sign-in link** (in the set-up dialog, and later **Sign-in link** on the card) asks Pocket ID for a one-time access token for the first administrator and shows the link `<URL>/lc/<token>` **once**. Open it, sign in, then add a passkey on your account page: that is how you sign in from then on. The link works once and expires after an hour. **It is never stored, logged or audited**: it lives only in the response to that request (`Cache-Control: no-store`) and in the open dialog; the audit entry (`platform.admin_link`) says who asked, never the link. Lost it? Create another. Any instance administrator can create one, for the first administrator's account.
**People are managed in Pocket ID**, not here: add, change and remove them in its admin page (the card links to it: **Open Pocket ID**). The platform page has no People dialog for it, and the people endpoints answer 409 `users_not_managed`.
### Authelia (alternative)
Authelia runs in file-backend mode, so **NebulaCtrl manages its people**: users and their argon2id password hashes live in the service's configuration, never in a database you have to administer. Set-up creates the first person from your own account (username from your email, your name and email, a generated password **shown once**), generates the signing key and secrets, registers NebulaCtrl as the only OpenID Connect client (authorization code with PKCE S256, client secret by HTTP basic, no consent screen, one factor) and adds the provider.
On the platform page **Add person**, **Reset password** and **Remove** rewrite the users file and redeploy the service: the sign-in page restarts for a few seconds (recreate deploy; people already signed in to NebulaCtrl are not affected). Passwords need at least 10 characters; leave the field empty to generate one. The last person cannot be removed (409 `last_user`). Authelia's own second factor is not used (it needs email delivery for enrolment, which this setup does not have); NebulaCtrl's own two-factor requirement still applies after sign-in.
### Removing it
**Removing it** (either product) deletes the provider and every identity linked through it. If people have no other way in (no password) the removal is refused with `sso_only_users` and the count; **Remove anyway** forces it. Give those people a password first (Settings, Members or the password reset link). The Pocket ID data volume stays unless you also delete the data; its passkeys and users are then gone with it.
## Backups of the data
Headscale, Pocket ID, Authelia and Grafana hold state. When a service becomes ready and the team already has an S3 destination that is **not** the platform S3, NebulaCtrl schedules the [volume backup](https://nebulactrl.com/docs/operations/backups) of its data there (daily 03:17 UTC, keeping 7 daily, 4 weekly, 3 monthly). Otherwise the page says "Not backed up" and offers **Back up daily** with a destination picker (any destination, including the platform S3). Loki holds logs bounded by its retention and is not scheduled; S3 holds backups (above).
## Removing
**Remove** needs the kind typed as confirmation and asks whether to **also delete the data** (the volumes; off by default: they stay on the server and can be removed in Docker or by removing the volumes through the server's cleanup). The integration is removed **first**; if it is still in use nothing changes and you get the reason:
| Kind | Refused while |
|---|---|
| S3 | backup schedules or stored backups use the destination (`destination_in_use`): move or delete them first |
| Loki | never (the drain is removed); Grafana must go first (`grafana_depends_on_loki`) |
| Headscale | servers use the managed tailnet (`integration_in_use`): this includes a [mesh](https://nebulactrl.com/docs/operations/mesh) that is enabled or has server nodes |
| Single sign-on | people could only sign in through it (`sso_only_users`), unless forced |
| Container registry | the team's build settings, a project or a service override still references it (`409 registry_in_use`): switch them to another registry or to building on the target first. Its stored images are deleted if you also delete the data. |
Then the services are deleted through the normal path (stop, remove the container, optionally the volumes) and the row is gone; you can set the kind up again. A removal that stops half-way (a deployment in progress, an offline server) leaves the service in `removing`/`failed` with the reason; repeat the removal.
## API
| Method and path | |
|---|---|
| `GET /platform-services` | the catalog (image, licence, AGPL flag, footprint, notice) with the team's instance of each kind |
| `POST /platform-services` | set one up (202): `kind`, `serverId`, optional `hostname`, `publicUrl`, `bucket`, `retentionDays`, `providerName`, `variant` (`pocketid` default, or `authelia`; single sign-on), `replace` |
| `GET /platform-services/{id}` | status, step, message, wiring, warnings, details, people |
| `POST /platform-services/{id}/retry` · `/reconnect` | resume a failed one; run the wiring again |
| `POST /platform-services/{id}/reveal` | S3, Loki and Grafana credentials (audited) |
| `PUT /platform-services/{id}/backup` | schedule the data backup to a destination |
| `GET/POST /platform-services/{id}/users`, `POST .../users/{username}/reset-password`, `DELETE .../users/{username}` | people of the Authelia provider (409 `users_not_managed` for Pocket ID) |
| `POST /platform-services/{id}/admin-link` | a one-time sign-in link for the first Pocket ID administrator; returned once, never stored (instance administrators) |
| `DELETE /platform-services/{id}?confirm=<kind>&deleteData=&force=` | unwire and remove |
Audit actions: `platform.provision`, `platform.remove`, `platform.rewire`, `platform.reveal`, `platform.backup`, `platform.user.add`, `platform.user.reset_password`, `platform.user.remove`, `platform.admin_link` (who asked; never the link), plus the usual entries of what the job creates (destination, drain, provider, deployments).
## Upgrading the images
The versions are constants in `internal/platform/catalog.go`, each pinned by tag and digest of the multi-platform index. To bump one: read the product's release notes, `docker buildx imagetools inspect <image>:<tag>` for the digest, change the constant, and run `go test ./internal/platform` (it refuses an unpinned image and re-renders every configuration). Existing platform services keep their pinned version until you deploy a newer one: the service's image is staged config, so change it on the service in the platform project and deploy. Authelia and Headscale read their configuration at start, so a new minor version may need a configuration change: check the release notes first. For Pocket ID use `docker buildx imagetools inspect ghcr.io/pocket-id/pocket-id:<tag>` (or a registry `HEAD` on the manifest); its database migrates itself on start, and it refuses to start an older version on a newer database.
## Limits
- One of each kind per team; no second S3 or a second Loki through this page (add your own destinations and drains as usual).
- The registry holds every image your build servers built; its volume is not scheduled for backup (images can be rebuilt from their commits). Losing it breaks rollbacks of releases built on a build server until you rebuild them (**Rebuild commit**).
- Single server per service; Grafana is on Loki's server. Moving a platform service between servers is a normal service move (the volume is not migrated), which breaks the public URL if the hostname's DNS does not follow.
- The public URL is fixed at set-up. Changing the hostname means removing and setting up again.
- Pocket ID is the owner's exception to the "stable for two years" rule for new products (its 1.0 is 17 months old, 2025-05); the pinned release is v2.19.1.
- The sign-in callback is registered with this instance's **public URL** (`NEBULACTRL_PUBLIC_URL`) at set-up. If that URL changes later, remove single sign-on and set it up again.
- Not verified against real third parties: single sign-on was verified end to end against the pinned Authelia image (login, authorization code with PKCE, token, userinfo) outside NebulaCtrl, and the Pocket ID wiring (discovery, first administrator, OpenID client and secret, one-time token, token endpoint authentication) against the pinned Pocket ID image on a network without internet access (`go test -tags pocketid_smoke`), but not a passkey registration in a browser; the browser sign-in through NebulaCtrl on a real public hostname, and Headscale with real Tailscale clients, were not exercised. The mesh's policy flow was checked in a design spike against Headscale 0.29.6 on one machine, not across real networks or with Tailscale SaaS ([mesh limits](https://nebulactrl.com/docs/operations/mesh#limits)). The S3 round trip (set-up, bucket, destination test, database backup into it, removal) runs in the full integration tier (`TestPlatformS3BackupRoundTrip`).
---
# Operations index
Source: https://nebulactrl.com/docs/operations
## Start here
| Page | Read it when |
|---|---|
| [Backup and restore](https://nebulactrl.com/docs/operations/backup-restore) | **Before anything else.** The scheduled control-plane backup (daily 03:30 UTC, 14 kept, S3 or a local directory), `NEBULACTRL_SECRET_KEY` (never in a backup), `nebulactrl-server restore --from <dest> --at <timestamp>`, what is *not* backed up |
## Applications and deployments
| Page | Read it when |
|---|---|
| [Deployments](https://nebulactrl.com/docs/operations/deployments) | rolling vs recreate, health check semantics (Coolify-style), rollback, image retention, `rollback_failed` recovery |
| [Builds: Dockerfile, Railpack, starters](https://nebulactrl.com/docs/operations/builds) | a repository without a Dockerfile, the build-plan preview, pinned Railpack and how to upgrade it, buildx fallback, disk use |
| [SSH deploy keys](https://nebulactrl.com/docs/operations/ssh-deploy-keys) | cloning over `ssh://` or `git@host:org/repo.git`, generated or uploaded keys, rotation, known_hosts pinning (trust on first use), "Accept new host key", limits |
| [Import a Docker Compose project](https://nebulactrl.com/docs/operations/compose-import) | turning a `docker-compose.yml` into staged services: the review, what maps and what does not, rewritten references, managed-database opt-in, secrets, blockers, the CLI |
| [Import from Coolify](https://nebulactrl.com/docs/operations/import-coolify) | migrating a Coolify v4 instance: the token it needs (`read` + `read:sensitive`), what maps (applications, Dockerfile/Nixpacks builds, domains, health checks, custom Docker run options, managed databases with the same engine and version, variables and rewritten references), what is never copied (data) and stays manual, the CLI `nebulactrl import coolify` |
| [Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks) | running a `docker-compose.yml` as one service with `docker compose`: the generated override, routing through domains, deploy and rollback, the allow-list (no privileged mode, host network, devices, Docker socket or host bind mounts), the per-server Unrestricted switch, limits |
| [Docker options](https://nebulactrl.com/docs/operations/docker-options) | letting a service add a capability, a ulimit, a bigger `/dev/shm`, extra hosts, labels, an init process, a stop signal or tmpfs mounts; what is never allowed; the instance switch for `NET_ADMIN` / `SYS_ADMIN` |
| [Replicas and autoscaling](https://nebulactrl.com/docs/operations/scaling) | running a stateless service on several replicas, CPU autoscaling rules, capacity guard |
| [Pull-request previews](https://nebulactrl.com/docs/operations/previews) | a preview environment per open PR: what is cloned, fresh databases and seed commands, domains, the 5-running / 7-idle-day limits, pausing and resuming, fork PRs and why they get no secrets |
| [Clone and move](https://nebulactrl.com/docs/operations/clone-and-move) | a staging copy of production: cloning a service or a whole environment (secrets copied sealed, generated values regenerated, new empty volumes and databases, domains dropped or suffixed), moving a service to another environment with its references and links checked, the dry run, the CLI, what is not copied |
| [Web terminal and file browser](https://nebulactrl.com/docs/operations/terminal) | opening a shell or browsing files in a replica container, roles and the production confirmation, what is audited (never the I/O), limits, the CLI `services shell`, proxy settings for WebSockets |
| [Maintenance mode and error pages](https://nebulactrl.com/docs/operations/maintenance-and-error-pages) | putting a service into maintenance (UI, CLI, API), what visitors see (503 + Retry-After), the automatic 404/502/503 pages, the `nebulactrl-pages` container and its pinned nginx image |
| [Deploy hooks](https://nebulactrl.com/docs/operations/deploy-hooks) | a URL per service that deploys it from CI or `curl` (`POST /hooks/deploy/<id>/<secret>`): the show-once secret and rotation, `?tag=` / `?digest=` as a one-off release, Docker Hub and GHCR push payloads, queueing behind a running deployment, the 6/min rate limit, audit, curl / GitHub Actions / GitLab CI snippets |
## Domains, networking and certificates
| Page | Read it when |
|---|---|
| [Networking](https://nebulactrl.com/docs/operations/networking) | what can reach what on a server, team shared networks (same server, hostnames `<service>.<env>.<project>`), **Block outbound internet** (default on for databases), deploying the siblings, what is and is not isolated |
| [Proxy routes: route services, domain options, TCP/UDP](https://nebulactrl.com/docs/operations/proxy-routes) | publishing a LAN/host/tailnet upstream, exposing a TCP/UDP port, redirects, path prefixes, basic auth and other per-domain options, the proxy restart on a new port, why ufw does not protect Docker ports |
| [Custom certificates, upstream policy, tailnet](https://nebulactrl.com/docs/operations/proxy-tailnet-certificates) | uploading a certificate, the SSRF upstream policy, routing to tailnet devices (managed or host Tailscale), the Tailscale OAuth client and ACL setup |
| [Per-service tailnet (Tailscale / Headscale)](https://nebulactrl.com/docs/operations/service-tailnet) | making one service reachable on your tailnet (HTTPS, raw TCP, optional Funnel), "Tailnet only", the sidecar and how its key is minted, re-provisioning, device removal, Headscale limits |
| [Server mesh and cross-server links](https://nebulactrl.com/docs/operations/mesh) | joining servers to your tailnet (server and link nodes), letting services of one environment on different servers reach each other, the generated ACL (managed on Headscale, advisory or opt-in managed on Tailscale), the device count, public DERP relays, the paused migration window, failure modes, the connectivity check |
| [DNS-01 and wildcard certificates](https://nebulactrl.com/docs/operations/dns01-wildcard-certificates) | Cloudflare, Route 53, Hetzner and DigitalOcean resolvers, credentials per team or server, wildcard domains, generated-domain coverage, certificate status and expiry |
| [Cloudflare: DNS records, proxy toggle, live DNS status](https://nebulactrl.com/docs/operations/cloudflare) | connecting a scoped API token, the records NebulaCtrl creates and how it recognises them (ownership marker, adopting existing records), staying in sync on IP change / move / drain, DNS only vs proxied, the SSL-mode redirect-loop warning, the per-domain status and fix, tunnels |
## Databases and backups
| Page | Read it when |
|---|---|
| [Managed databases](https://nebulactrl.com/docs/operations/databases) | engines and versions, `DATABASE_URL` exports, the audited reveal action, credential rotation, version upgrades (pg_upgrade) |
| [Postgres standby replicas and manual Promote](https://nebulactrl.com/docs/operations/database-replicas) | advanced, Postgres only: when to use a standby (another server, read offload) and when not, adding a replica on another or the same server, `DATABASE_WRITE_*` / `DATABASE_READ_*`, lag, Promote (switchover or forced, data loss, fencing, consumers rewritten), rebuild, backups, restore and upgrades, limits |
| [Database backups](https://nebulactrl.com/docs/operations/database-backups) | scheduled and on-demand dumps streamed to S3-compatible storage or a server directory, retention, restore into a new database or in place |
| [Volume backups](https://nebulactrl.com/docs/operations/backups) | file-level backups of a service's volumes: pre-hook, freeze (`docker pause`), staged copy, background restic upload, stop and live modes, retention, restore into a new volume with a staged swap, file browser and single-file download |
## Servers
| Page | Read it when |
|---|---|
| [Servers and agents](https://nebulactrl.com/docs/operations/servers) | offline servers, revoke and re-enroll, in-flight deployments, reconciler |
| [Build servers and registries](https://nebulactrl.com/docs/operations/build-servers) | building on a dedicated server instead of the target: roles (Deployments, Builds, Both), where builds run (team, project, service), bring-your-own registries (GHCR, Docker Hub, GitLab, Harbor) and the managed "Set up for me" registry, cross-architecture builds and emulation, per-project builders and preview isolation, fallback to the target, rollback and what to keep in a registry, the failure table, limits |
| [Draining servers and the fleet overview](https://nebulactrl.com/docs/operations/draining-servers) | taking a server out for maintenance or retirement, capacity and overcommit, drain failure and cancel semantics, required scopes |
| [Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup) | the daily and disk-alert cleanup of images, build cache and networks per server, what is never removed (volumes, rollback images, infrastructure images), the optional stopped-container step, run history and the explicit unused-volume action |
| [Upgrades: agents and the control plane](https://nebulactrl.com/docs/operations/upgrades) | agents replacing their own binary (checksum, atomic swap, rollback when the new one does not start), automatic and staged upgrades, version skew per server, the control-plane backup taken before an upgrade and when it blocks one, `--force-upgrade` |
| [Server patching](https://nebulactrl.com/docs/operations/server-patching) | pending OS updates and reboot-required per server (apt and dnf, checked every 6 hours), "N updates (M security)" on the server card, the instance admin's Apply updates (plan, typed confirmation, production acknowledgement, log) and Reboot (waits for the agent to reconnect), the `server.reboot_required` notification, limits |
| [Host shell](https://nebulactrl.com/docs/security/host-shell) | the opt-in root shell on a server: who may use it (Owners and Admins with two-factor authentication), the production confirm, what is audited (never the I/O), how the agent runs it |
## Observability
| Page | Read it when |
|---|---|
| [Monitoring, logs and retention](https://nebulactrl.com/docs/operations/monitoring) | health endpoints, JSON logs, `correlation_id`, metrics and traffic analytics retention, disk alerts |
| [Log history and log drains](https://nebulactrl.com/docs/operations/logs) | runtime logs stored on each server (7 days, size caps), History view and query language, drains to Loki, OTLP, syslog and HTTP JSON (Datadog, Better Stack, Axiom), retries, backlog bound, health |
| [Traffic analytics and request origins](https://nebulactrl.com/docs/operations/traffic-analytics) | what the proxy access log is turned into, the "Where requests come from" map, DB-IP Lite vs MaxMind GeoLite2, country vs city (~8 MB vs ~130 MB), attribution, privacy |
| [Notifications](https://nebulactrl.com/docs/operations/notifications) | email, Slack, Discord, Telegram and signed-webhook channels, event rules, retries and the delivery log, verifying `X-NebulaCtrl-Signature` |
| [Service alerts](https://nebulactrl.com/docs/operations/service-alerts) | crash, unhealthy, restart-loop and missed-backup alerts: once per incident, recovery messages, flapping, thresholds, what is not watched |
## Secrets and platform services
| Page | Read it when |
|---|---|
| [Secret managers and team variables](https://nebulactrl.com/docs/operations/secret-managers) | `${{vault:<name>/<path>#<key>}}` references to HashiCorp Vault / OpenBao (KV v2, token or AppRole) and Infisical (machine identity), fetched at deploy time and never stored, the policy to create, errors that name the reference, redaction, the team variable layer `${{team.KEY}}` |
| [Platform services ("Set up for me")](https://nebulactrl.com/docs/operations/platform-services) | one-click S3 storage (SeaweedFS), log store (Loki, optional Grafana), tailnet control server (Headscale) and single sign-on (Authelia) run by NebulaCtrl in a hidden system project, wired into backup destinations, log drains, the Tailscale integration and sign-in providers with generated credentials; same-server-not-off-site, AGPL notice, people management, backups of their data, removal and what blocks it |
## When something is wrong
| Page | Read it when |
|---|---|
| [Troubleshooting](https://nebulactrl.com/docs/operations/troubleshooting) | proxy/port 80, certificates, DNS, agent will not connect, deployment stuck |
| [Known limitations](https://nebulactrl.com/docs/operations/limitations) | what this release does **not** do |
| [Advanced settings](https://nebulactrl.com/docs/operations/configuration) | you were told to change a control-plane timer (`NEBULACTRL_MESH_LINK_GRACE`, `NEBULACTRL_REPLICA_BROKEN_AFTER`); the defaults and what shortening them risks |
Installation and upgrade live in [`docs/installing`](https://nebulactrl.com/docs/installing); the security model in [`docs/security`](https://nebulactrl.com/docs/security); design background in [`docs/architecture`]() and [`docs/decisions`]().
---
# CLI reference
Source: https://nebulactrl.com/docs/reference/cli
`nebulactrl` is a command line client for the control plane REST API. It authenticates with an API token; see [API tokens and scopes](https://nebulactrl.com/docs/operations/teams) and [the API reference](https://nebulactrl.com/docs/reference/api).
## Global options
Manage NebulaCtrl projects, services, deployments and servers through the control plane REST API.
A <service> is a service ID or project/env/slug. Configuration lives in ~/.config/nebulactrl/config.json (override with NEBULACTRL_CONFIG; NEBULACTRL_URL and NEBULACTRL_TOKEN override its fields).
| Flag | Type | Default | Description |
|---|---|---|---|
| `--idempotency-key` | string | | Idempotency-Key for mutating calls (default: random per invocation) |
| `--json` | bool | | print machine-readable JSON |
Exit codes: `0` success, `1` failure, `2` wrong flags or arguments.
## Command list
| Command | Description |
|---|---|
| [`nebulactrl backups`](#nebulactrl-backups) | Back up and restore managed databases |
| [`nebulactrl backups create`](#nebulactrl-backups-create) | Back up a database now |
| [`nebulactrl backups destinations`](#nebulactrl-backups-destinations) | Manage backup destinations (S3-compatible buckets, server directories) |
| [`nebulactrl backups destinations add`](#nebulactrl-backups-destinations-add) | Add a backup destination |
| [`nebulactrl backups destinations ls`](#nebulactrl-backups-destinations-ls) | List backup destinations |
| [`nebulactrl backups destinations rm`](#nebulactrl-backups-destinations-rm) | Remove a backup destination |
| [`nebulactrl backups destinations test`](#nebulactrl-backups-destinations-test) | Write, read and delete a probe object through a server's agent |
| [`nebulactrl backups lock`](#nebulactrl-backups-lock) | Keep a backup forever (never pruned or deleted) |
| [`nebulactrl backups ls`](#nebulactrl-backups-ls) | List a database's backups |
| [`nebulactrl backups restore`](#nebulactrl-backups-restore) | Restore a backup into a new database (default) or in place |
| [`nebulactrl backups rm`](#nebulactrl-backups-rm) | Delete a backup and its stored object |
| [`nebulactrl backups schedules`](#nebulactrl-backups-schedules) | Manage recurring backups |
| [`nebulactrl backups schedules add`](#nebulactrl-backups-schedules-add) | Schedule recurring backups |
| [`nebulactrl backups schedules ls`](#nebulactrl-backups-schedules-ls) | List a database's backup schedules |
| [`nebulactrl backups schedules rm`](#nebulactrl-backups-schedules-rm) | Remove a backup schedule |
| [`nebulactrl backups unlock`](#nebulactrl-backups-unlock) | Allow retention to prune a backup again |
| [`nebulactrl build-settings`](#nebulactrl-build-settings) | Show or set the team's default build location and registry |
| [`nebulactrl databases`](#nebulactrl-databases) | Manage managed databases |
| [`nebulactrl databases create`](#nebulactrl-databases-create) | Create a managed database (staged until deployed) |
| [`nebulactrl databases engines`](#nebulactrl-databases-engines) | List the database engines and versions |
| [`nebulactrl databases info`](#nebulactrl-databases-info) | Show a database's engine, exports and last reported state |
| [`nebulactrl databases promote`](#nebulactrl-databases-promote) | Promote a read replica to primary (manual) |
| [`nebulactrl databases rebuild`](#nebulactrl-databases-rebuild) | Rebuild a replica, or a fenced old primary as a replica of the new primary |
| [`nebulactrl databases replicas`](#nebulactrl-databases-replicas) | Manage PostgreSQL read replicas |
| [`nebulactrl databases replicas add`](#nebulactrl-databases-replicas-add) | Add a read replica of a PostgreSQL primary |
| [`nebulactrl databases replicas ls`](#nebulactrl-databases-replicas-ls) | List the read replicas of a database and their lag |
| [`nebulactrl databases replicas rm`](#nebulactrl-databases-replicas-rm) | Remove a read replica |
| [`nebulactrl databases restore`](#nebulactrl-databases-restore) | Restore a database from a dump file on this machine |
| [`nebulactrl databases reveal`](#nebulactrl-databases-reveal) | Print a connection string (audited as secret.reveal) |
| [`nebulactrl databases rotate`](#nebulactrl-databases-rotate) | Rotate the database password and stage its consumers |
| [`nebulactrl databases upgrade`](#nebulactrl-databases-upgrade) | Plan (default) or run a database version upgrade |
| [`nebulactrl deployments`](#nebulactrl-deployments) | Inspect and roll back deployments |
| [`nebulactrl deployments cancel`](#nebulactrl-deployments-cancel) | Cancel a running deployment |
| [`nebulactrl deployments download-logs`](#nebulactrl-deployments-download-logs) | Download a deployment's build and deploy log as a file |
| [`nebulactrl deployments ls`](#nebulactrl-deployments-ls) | List the deployments of a service |
| [`nebulactrl deployments rollback`](#nebulactrl-deployments-rollback) | Redeploy the release of a previous deployment |
| [`nebulactrl deployments status`](#nebulactrl-deployments-status) | Show a deployment and its step timeline |
| [`nebulactrl domains`](#nebulactrl-domains) | Manage service domains |
| [`nebulactrl domains add`](#nebulactrl-domains-add) | Attach a hostname to a service |
| [`nebulactrl domains generate`](#nebulactrl-domains-generate) | Generate a domain from the server's base domain |
| [`nebulactrl domains ls`](#nebulactrl-domains-ls) | List a service's domains |
| [`nebulactrl domains rm`](#nebulactrl-domains-rm) | Remove a domain from a service |
| [`nebulactrl domains www`](#nebulactrl-domains-www) | Turn the www/apex redirect of a domain on or off |
| [`nebulactrl envs`](#nebulactrl-envs) | Manage environments |
| [`nebulactrl envs clone`](#nebulactrl-envs-clone) | Clone an environment into a new, non-production one |
| [`nebulactrl envs ls`](#nebulactrl-envs-ls) | List the environments of a project |
| [`nebulactrl import`](#nebulactrl-import) | Import a project from another tool |
| [`nebulactrl import compose`](#nebulactrl-import-compose) | Import a Docker Compose project as staged services |
| [`nebulactrl import coolify`](#nebulactrl-import-coolify) | Import projects, applications and databases from a Coolify v4 instance as staged services |
| [`nebulactrl instance`](#nebulactrl-instance) | Settings of the whole installation (instance admin only) |
| [`nebulactrl instance dashboard-access`](#nebulactrl-instance-dashboard-access) | Who may reach the dashboard and how its certificate is issued |
| [`nebulactrl instance dashboard-access get`](#nebulactrl-instance-dashboard-access-get) | Show the dashboard's allow list and certificate method |
| [`nebulactrl instance dashboard-access set`](#nebulactrl-instance-dashboard-access-set) | Change the dashboard's allow list and certificate method |
| [`nebulactrl login`](#nebulactrl-login) | Store the control plane URL and API token |
| [`nebulactrl members`](#nebulactrl-members) | Manage the members of your team |
| [`nebulactrl members invite`](#nebulactrl-members-invite) | Invite someone by email, or create a link invite |
| [`nebulactrl members ls`](#nebulactrl-members-ls) | List the members of the token's team |
| [`nebulactrl mesh`](#nebulactrl-mesh) | Manage the server mesh (a tailnet of your servers) |
| [`nebulactrl mesh check`](#nebulactrl-mesh-check) | Test whether the server's mesh node reaches the other servers |
| [`nebulactrl mesh disable`](#nebulactrl-mesh-disable) | Remove a server from the mesh |
| [`nebulactrl mesh enable`](#nebulactrl-mesh-enable) | Join a server to the mesh (adds one tagged tailnet device) |
| [`nebulactrl mesh links`](#nebulactrl-mesh-links) | Show the cross-server links of an environment: links, grants and routes |
| [`nebulactrl mesh reconcile`](#nebulactrl-mesh-reconcile) | Create (add) or clean up (repair) the cross-server links of an environment now |
| [`nebulactrl mesh reprovision`](#nebulactrl-mesh-reprovision) | Re-create the server's mesh node (new device and tailnet IP) |
| [`nebulactrl mesh service`](#nebulactrl-mesh-service) | Show what a service exposes on the mesh, which providers it reaches and who reaches it |
| [`nebulactrl mesh status`](#nebulactrl-mesh-status) | Show the mesh: nodes, links and the ACL state |
| [`nebulactrl projects`](#nebulactrl-projects) | Manage projects |
| [`nebulactrl projects create`](#nebulactrl-projects-create) | Create a project (with a production environment) |
| [`nebulactrl projects ls`](#nebulactrl-projects-ls) | List projects |
| [`nebulactrl projects update`](#nebulactrl-projects-update) | Rename a project or set where its services are built |
| [`nebulactrl servers`](#nebulactrl-servers) | Manage Docker hosts |
| [`nebulactrl servers enroll-token`](#nebulactrl-servers-enroll-token) | Create a one-time agent enrollment token |
| [`nebulactrl servers ls`](#nebulactrl-servers-ls) | List servers |
| [`nebulactrl servers revoke`](#nebulactrl-servers-revoke) | Revoke a server's agent identity |
| [`nebulactrl servers unrestricted`](#nebulactrl-servers-unrestricted) | Show, enable or disable the Unrestricted switch of a server (instance admin) |
| [`nebulactrl servers update`](#nebulactrl-servers-update) | Set a server's role and build settings |
| [`nebulactrl services`](#nebulactrl-services) | Manage services |
| [`nebulactrl services build-plan`](#nebulactrl-services-build-plan) | Preview how a git service would be built |
| [`nebulactrl services clone`](#nebulactrl-services-clone) | Clone a service (staged copy; secrets are copied sealed) |
| [`nebulactrl services create`](#nebulactrl-services-create) | Create a service (staged until deployed) |
| [`nebulactrl services deploy`](#nebulactrl-services-deploy) | Deploy a service |
| [`nebulactrl services deploy-hook`](#nebulactrl-services-deploy-hook) | Show, enable, disable or rotate a service's deploy hook |
| [`nebulactrl services docker`](#nebulactrl-services-docker) | Show or set a service's Docker options (staged until the next deployment) |
| [`nebulactrl services download-logs`](#nebulactrl-services-download-logs) | Download a service's stored runtime logs as a file |
| [`nebulactrl services logs`](#nebulactrl-services-logs) | Show runtime logs of a service |
| [`nebulactrl services ls`](#nebulactrl-services-ls) | List the services of an environment |
| [`nebulactrl services maintenance`](#nebulactrl-services-maintenance) | Put a service into maintenance mode, or take it out |
| [`nebulactrl services move`](#nebulactrl-services-move) | Move a service to another environment (staged until the next deploy) |
| [`nebulactrl services restart`](#nebulactrl-services-restart) | Restart a service's container |
| [`nebulactrl services run`](#nebulactrl-services-run) | Run a one-off command in a container of the service's image |
| [`nebulactrl services scale`](#nebulactrl-services-scale) | Set the number of replicas or turn on autoscaling |
| [`nebulactrl services shell`](#nebulactrl-services-shell) | Open an interactive shell in a service's container |
| [`nebulactrl services update`](#nebulactrl-services-update) | Set where and how a service is built |
| [`nebulactrl tasks`](#nebulactrl-tasks) | Cron tasks and their runs |
| [`nebulactrl tasks create`](#nebulactrl-tasks-create) | Schedule a recurring command |
| [`nebulactrl tasks logs`](#nebulactrl-tasks-logs) | Print the output of a run (follows a run that is still active) |
| [`nebulactrl tasks ls`](#nebulactrl-tasks-ls) | List a service's cron tasks |
| [`nebulactrl tasks rm`](#nebulactrl-tasks-rm) | Delete a cron task and its run history |
| [`nebulactrl tasks run`](#nebulactrl-tasks-run) | Run a cron task now and follow its output |
| [`nebulactrl tasks runs`](#nebulactrl-tasks-runs) | Run history of a service's tasks and one-off commands |
| [`nebulactrl tasks update`](#nebulactrl-tasks-update) | Change a cron task's schedule or time zone |
| [`nebulactrl teams`](#nebulactrl-teams) | Manage teams |
| [`nebulactrl teams create`](#nebulactrl-teams-create) | Create a team (you become its owner; needs an admin-scope token) |
| [`nebulactrl teams ls`](#nebulactrl-teams-ls) | List your teams and the saved token per team |
| [`nebulactrl teams switch`](#nebulactrl-teams-switch) | Use the saved token of another team |
| [`nebulactrl templates`](#nebulactrl-templates) | Browse and install one-click templates |
| [`nebulactrl templates install`](#nebulactrl-templates-install) | Install a template as staged services (or deploy right away) |
| [`nebulactrl templates installed`](#nebulactrl-templates-installed) | List the template stacks of an environment |
| [`nebulactrl templates ls`](#nebulactrl-templates-ls) | List catalog templates |
| [`nebulactrl templates plan`](#nebulactrl-templates-plan) | Preflight an install (changes nothing) |
| [`nebulactrl templates rm`](#nebulactrl-templates-rm) | Delete a template stack |
| [`nebulactrl templates show`](#nebulactrl-templates-show) | Show a template: services, inputs and licence |
| [`nebulactrl templates update`](#nebulactrl-templates-update) | Show what a newer template revision changes, or stage it (--apply) |
| [`nebulactrl vars`](#nebulactrl-vars) | Manage service environment variables |
| [`nebulactrl vars history`](#nebulactrl-vars-history) | Show a variable's versions (secrets show a fingerprint only) |
| [`nebulactrl vars ls`](#nebulactrl-vars-ls) | List variables (secret values are never shown) |
| [`nebulactrl vars rm`](#nebulactrl-vars-rm) | Delete variables (staged until the next deploy) |
| [`nebulactrl vars rotate`](#nebulactrl-vars-rotate) | Replace a generated variable's value (staged, never deployed) |
| [`nebulactrl vars set`](#nebulactrl-vars-set) | Set variables (staged until the next deploy) |
| [`nebulactrl vars shared`](#nebulactrl-vars-shared) | Manage shared variables of the team (@team), a project (<project>) or an environment (<project>/<env>) |
| [`nebulactrl vars shared ls`](#nebulactrl-vars-shared-ls) | List shared variables |
| [`nebulactrl vars shared rm`](#nebulactrl-vars-shared-rm) | Delete shared variables |
| [`nebulactrl vars shared rotate`](#nebulactrl-vars-shared-rotate) | Replace a generated shared variable (staged, never deployed) |
| [`nebulactrl vars shared set`](#nebulactrl-vars-shared-set) | Set shared variables (restages the services that use them) |
| [`nebulactrl vars shared share`](#nebulactrl-vars-shared-share) | Stage KEY=${{shared.KEY}} on services |
## nebulactrl backups
Back up and restore managed databases.
```
nebulactrl backups
```
### nebulactrl backups create
Back up a database now.
```
nebulactrl backups create <database> --destination <name|id> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--destination` | string | | destination name or id |
| `--timeout` | duration | `2h0m0s` | give up waiting after this long (with --wait) |
| `--wait` | bool | | wait for the backup to finish; exit non-zero unless it succeeded |
### nebulactrl backups destinations
Manage backup destinations (S3-compatible buckets, server directories).
```
nebulactrl backups destinations
```
#### nebulactrl backups destinations add
Add a backup destination.
```
nebulactrl backups destinations add --name <n> --kind s3|local [flags]
```
The S3 secret key is never taken as an argument (it would show in process lists): pass --secret-key-file
(`-` reads standard input) or set NEBULACTRL_BACKUP_SECRET_KEY.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--access-key` | string | | S3 access key id |
| `--bucket` | string | | S3 bucket |
| `--endpoint` | string | | S3 endpoint URL (empty: AWS) |
| `--kind` | string | | s3 or local |
| `--local-dir` | string | | local destinations: sub-directory of the agent's backup directory |
| `--name` | string | | destination name |
| `--path-style` | bool | | path-style addressing (MinIO, SeaweedFS, Garage) |
| `--prefix` | string | | key prefix |
| `--region` | string | | S3 region |
| `--secret-key-file` | string | | file holding the S3 secret key (`-` for stdin) |
#### nebulactrl backups destinations ls
List backup destinations.
```
nebulactrl backups destinations ls
```
#### nebulactrl backups destinations rm
Remove a backup destination.
```
nebulactrl backups destinations rm <name|id>
```
#### nebulactrl backups destinations test
Write, read and delete a probe object through a server's agent.
```
nebulactrl backups destinations test <name|id> --server <server-id> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--server` | string | | server whose agent runs the test |
### nebulactrl backups lock
Keep a backup forever (never pruned or deleted).
```
nebulactrl backups lock <backup-id>
```
### nebulactrl backups ls
List a database's backups.
```
nebulactrl backups ls <database>
```
### nebulactrl backups restore
Restore a backup into a new database (default) or in place.
```
nebulactrl backups restore <backup-id> [flags]
```
By default the backup is restored into a NEW sibling database with fresh credentials; the original is
untouched. With --in-place the database is overwritten after a safety backup; --confirm must equal its slug.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--confirm` | string | | the database's slug (required with --in-place) |
| `--in-place` | bool | | overwrite the database instead of restoring into a new one |
| `--name` | string | | name of the new database |
| `--server` | string | | server id for the new database |
| `--timeout` | duration | `2h0m0s` | give up waiting after this long (with --wait) |
| `--wait` | bool | | wait for the restore to finish; exit non-zero unless it succeeded |
### nebulactrl backups rm
Delete a backup and its stored object.
```
nebulactrl backups rm <backup-id>
```
### nebulactrl backups schedules
Manage recurring backups.
```
nebulactrl backups schedules
```
#### nebulactrl backups schedules add
Schedule recurring backups.
```
nebulactrl backups schedules add <database> --destination <name|id> --cron "0 3 * * *" [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--cron` | string | | 5-field cron expression or @daily/@hourly/… |
| `--destination` | string | | destination name or id |
| `--keep` | int | `7` | keep the newest N backups (0 = no limit) |
| `--keep-days` | int | | delete backups older than N days (0 = never) |
| `--timezone` | string | `UTC` | IANA timezone |
#### nebulactrl backups schedules ls
List a database's backup schedules.
```
nebulactrl backups schedules ls <database>
```
#### nebulactrl backups schedules rm
Remove a backup schedule.
```
nebulactrl backups schedules rm <schedule-id>
```
### nebulactrl backups unlock
Allow retention to prune a backup again.
```
nebulactrl backups unlock <backup-id>
```
## nebulactrl build-settings
Show or set the team's default build location and registry.
```
nebulactrl build-settings [--build-location target|build-server] [--registry <id|none>] [flags]
```
Without flags, prints the team default. With flags (administrators only), replaces it. Projects and services can
override it. When no usable build server or registry exists a build falls back to the target server.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--build-location` | string | | target or build-server |
| `--registry` | string | | default registry ID (none clears) |
## nebulactrl databases
Manage managed databases.
```
nebulactrl databases
```
### nebulactrl databases create
Create a managed database (staged until deployed).
```
nebulactrl databases create --project <project> --env <env> --engine <engine> [--version <v>] [--name <name>] [--flavor postgis] [--database <db>] [--server <server>] [--accept-license] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--accept-license` | bool | | acknowledge the licence of a source-available engine (MongoDB, Redis) |
| `--database` | string | | database name (default: app) |
| `--engine` | string | | postgres, mysql, mariadb, valkey, redis, mongodb or clickhouse |
| `--env` | string | | environment ID, slug or name |
| `--flavor` | string | | image variant, e.g. postgis |
| `--name` | string | | display name (default: the engine name) |
| `--project` | string | | project ID, slug or name |
| `--server` | string | | target server ID or name |
| `--version` | string | | major version (default: the engine's default) |
### nebulactrl databases engines
List the database engines and versions.
```
nebulactrl databases engines
```
### nebulactrl databases info
Show a database's engine, exports and last reported state.
```
nebulactrl databases info <service>
```
### nebulactrl databases promote
Promote a read replica to primary (manual).
```
nebulactrl databases promote <replica> [--yes] [--accept-data-loss] [flags]
```
Prints the promotion plan. With --yes it promotes: a switchover while the old primary is reachable (it is frozen, stopped and fenced; no data is lost) or a forced promote otherwise (needs --accept-data-loss: what the replica had not received is lost). The old primary is fenced and never runs as a primary again. Variables that reference it are rewritten as staged changes. Manual: NebulaCtrl does not detect failures or switch automatically.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--accept-data-loss` | bool | | accept losing what the replica had not received (forced promote) |
| `--yes` | bool | | promote |
### nebulactrl databases rebuild
Rebuild a replica, or a fenced old primary as a replica of the new primary.
```
nebulactrl databases rebuild <replica-or-fenced-primary> [--follow <new-primary>] [flags]
```
Clones the primary again into a new data volume; the old volume is kept until it is deleted.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--follow` | string | | the new primary a fenced old primary should follow (default: the replica that was promoted) |
### nebulactrl databases replicas
Manage PostgreSQL read replicas.
```
nebulactrl databases replicas
```
#### nebulactrl databases replicas add
Add a read replica of a PostgreSQL primary.
```
nebulactrl databases replicas add <primary> --server <server> [--name <name>] [--dry-run] [--accept-mesh-changes] [flags]
```
Creates a read replica service in the primary's environment and clones the primary into it. The replica runs on the primary's server or, with the server mesh enabled on both, on another server. Replication is asynchronous and manual: nothing is detected or switched automatically. --dry-run prints the plan only.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--accept-mesh-changes` | bool | | confirm that the replica adds tailnet devices |
| `--dry-run` | bool | | only print the plan |
| `--name` | string | | display name (default: <primary>-replica-<n>) |
| `--server` | string | | server ID or name the replica runs on |
#### nebulactrl databases replicas ls
List the read replicas of a database and their lag.
```
nebulactrl databases replicas ls <primary-or-replica>
```
#### nebulactrl databases replicas rm
Remove a read replica.
```
nebulactrl databases replicas rm <replica> --yes [flags]
```
Deletes the replica's container and data volume, drops its slot, role and pg_hba.conf line on the primary and releases its mesh link.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--yes` | bool | | confirm the removal |
### nebulactrl databases restore
Restore a database from a dump file on this machine.
```
nebulactrl databases restore <database> --file <dump> [--name <name>] [--server <server>] [--in-place --confirm <slug> [--destination <dest>]] [flags]
```
Uploads a dump file and restores it into a managed database: a pg_dump file (custom -Fc, tar -Ft or plain SQL, optionally
.gz), a mysqldump / mariadb-dump SQL file (optionally .gz) or a mongodump --archive file. The format is detected from the file.
The file is streamed through the control plane to the server's agent; nothing is stored on the way, and the default limit is 5 GiB.
By default the dump is restored into a NEW sibling database with fresh credentials (the original is untouched): point your
apps at it when you are satisfied. With --in-place the database's data is replaced after a safety backup; --confirm must
equal its slug, and --destination names the backup destination of the safety backup (default: the one of its schedule).
Examples:
```sh
nebulactrl databases restore shop/production/postgres --file shop.pgdump
nebulactrl databases restore shop/production/postgres --file shop.sql.gz --in-place --confirm postgres --destination s3
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--confirm` | string | | the database's slug (required with --in-place) |
| `--destination` | string | | backup destination (id or name) of the safety backup (--in-place) |
| `--file` | string | | the dump file to restore (required) |
| `--in-place` | bool | | replace the database's data instead of restoring into a new database |
| `--name` | string | | name of the new database |
| `--server` | string | | server id for the new database |
| `--timeout` | duration | `2h0m0s` | give up waiting for the new database, and again for the restore, after this long |
| `--wait` | bool | `true` | wait for the restore to finish; exit non-zero unless it succeeded (--wait=false returns once the file is uploaded) |
### nebulactrl databases reveal
Print a connection string (audited as secret.reveal).
```
nebulactrl databases reveal <service> [--key DATABASE_URL] [flags]
```
Prints one derived connection variable in plain text to stdout. The action is recorded in the audit log and needs the deployments:write scope.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--key` | string | | export to reveal (default: the connection URL) |
### nebulactrl databases rotate
Rotate the database password and stage its consumers.
```
nebulactrl databases rotate <service> [--dry-run] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--dry-run` | bool | | only print the plan |
### nebulactrl databases upgrade
Plan (default) or run a database version upgrade.
```
nebulactrl databases upgrade <service> <version> [--yes] [flags]
```
Prints the upgrade plan. With --yes it runs the upgrade: in-place engines stage the new image; PostgreSQL majors are converted with pg_upgrade into a new volume and then staged. Take a backup first.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--yes` | bool | | apply the upgrade |
## nebulactrl deployments
Inspect and roll back deployments.
```
nebulactrl deployments
```
### nebulactrl deployments cancel
Cancel a running deployment.
```
nebulactrl deployments cancel <deployment-id>
```
Cancels a deployment at whatever step it has reached. Before the agent started it the deployment ends as canceled at once.
Once the agent runs it, the agent is asked to abort: a build is killed, the new containers are removed, routes are restored and a stopped previous release is started again, so the current release keeps serving.
The deployment ends as canceled when the agent reports back (check with `deployments status`). Once the new release is live it is too late and the deployment finishes.
### nebulactrl deployments download-logs
Download a deployment's build and deploy log as a file.
```
nebulactrl deployments download-logs <deployment-id> [flags]
```
Writes the deployment's persisted lines (build, deploy and runtime streams; at most 10,000) in the order
they were recorded, as plain text (`<time> [<stream>] <line>`) or NDJSON. Secret values are masked on the
server. --streams all adds the agent's verbose debug trace.
Examples:
```sh
nebulactrl deployments download-logs 6f1c0d5e-… -o build.log
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--format` | string | `txt` | file format: txt or ndjson |
| `--from` | string | | only lines at or after this time: RFC 3339, a date, or a duration ago (24h) |
| `-o, --output` | string | | write to this file instead of stdout (created with mode 0600) |
| `--streams` | string | | all, or a comma-separated list of build, deploy, runtime, debug (default: build, deploy, runtime) |
| `--to` | string | | only lines before this time (same forms as --from) |
### nebulactrl deployments ls
List the deployments of a service.
```
nebulactrl deployments ls <service>
```
### nebulactrl deployments rollback
Redeploy the release of a previous deployment.
```
nebulactrl deployments rollback <deployment-id> [flags]
```
Creates a new deployment that points at the release of the given deployment.
Application image and configuration only. Databases, volumes and external services are not rolled back.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--timeout` | duration | `30m0s` | give up waiting after this long (with --wait) |
| `--wait` | bool | | wait for the rollback to finish; exit non-zero unless it succeeded |
### nebulactrl deployments status
Show a deployment and its step timeline.
```
nebulactrl deployments status <deployment-id>
```
## nebulactrl domains
Manage service domains.
```
nebulactrl domains
```
### nebulactrl domains add
Attach a hostname to a service.
```
nebulactrl domains add <service> <hostname> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--component` | string | | Compose stacks: the component (compose service) the domain routes to |
| `--port` | int | | target port (default: the service port; Compose stacks: the container port, optional when the component declares one) |
| `--www-redirect` | bool | | also route the www/apex counterpart and 301-redirect it to this hostname (needs proxy:write; both hostnames get certificates) |
### nebulactrl domains generate
Generate a domain from the server's base domain.
```
nebulactrl domains generate <service> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--component` | string | | Compose stacks: the component the generated domain routes to |
### nebulactrl domains ls
List a service's domains.
```
nebulactrl domains ls <service>
```
### nebulactrl domains rm
Remove a domain from a service.
```
nebulactrl domains rm <service> <hostname|domain-id>
```
### nebulactrl domains www
Turn the www/apex redirect of a domain on or off.
```
nebulactrl domains www <service> <hostname|domain-id> <on|off>
```
`on` routes the www/apex counterpart of the domain (www.<host>, or the host without its www.) and answers it with
a 301 redirect to the domain, keeping the path and query. Both hostnames get certificates and each needs its own
DNS record. `off` removes the redirect. The change is staged until the next deploy. Needs proxy:write; the domain
must be a whole-host serve domain without a path prefix.
Examples:
```sh
nebulactrl domains www acme/production/web example.com on
```
## nebulactrl envs
Manage environments.
```
nebulactrl envs
```
### nebulactrl envs clone
Clone an environment into a new, non-production one.
```
nebulactrl envs clone <env> --project <project> --name <name> [--domains drop|suffix] [--dry-run] [flags]
```
Create a new environment with a staged copy of every service of <env>: configuration, variables, the environment's shared variables, volume definitions, links and the canvas layout. Slugs are kept, so ${{postgres.DATABASE_URL}} in a cloned service points at the cloned postgres. Secrets are copied sealed and generated values are generated again. Databases are new and empty (restore a dump into them); volumes are new and empty. Nothing runs until the services are deployed.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--domain-suffix` | string | | suffix for --domains suffix (default: -copy in the same environment, -<environment slug> otherwise) |
| `--domains` | string | `drop` | domains of the clone: drop (attach new ones later) or suffix (suffix the first label of each hostname) |
| `--dry-run` | bool | | show what the clone would do and change nothing |
| `--name` | string | | name of the new environment |
| `--project` | string | | project ID, slug or name |
| `--server` | string | | place the clone on this server (ID or name; default: the source's server) |
### nebulactrl envs ls
List the environments of a project.
```
nebulactrl envs ls --project <project> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--project` | string | | project ID, slug or name |
## nebulactrl import
Import a project from another tool.
```
nebulactrl import
```
### nebulactrl import compose
Import a Docker Compose project as staged services.
```
nebulactrl import compose <docker-compose.yml|-> --project <p> --env <e> --server <s> [flags]
```
Reads a Compose file (a local file, stdin with -, or a file in a git repository with --repo) and stages one NebulaCtrl service per
Compose service in the environment, grouped on the canvas and linked in deploy order. Nothing is deployed unless you pass --deploy.
Files next to the Compose file (.env, env_file entries, bind-mounted config files) are read from disk. Host names of other
services in variables are rewritten to references, and official database images become managed databases unless you turn that off.
Use --plan to review without changing anything. Per-service decisions are written as <service>=<value> or <service>.<KEY>.
With --as-stack the whole file becomes ONE service (a native Compose stack): the file is kept as written, the .env entries become the
stack's variables (used for ${VAR} interpolation only), and domains are suggested per component. A stack runs under the safe
allow-list (no privileged mode, host network, devices, Docker socket or host bind mounts, no build:) unless an instance admin made
the server unrestricted; violations are listed and block the import. In stack mode --secret, --not-secret and
--skip-var take the variable name (<KEY>); --domain and --port take the component name.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--as-stack` | bool | | run the whole file as one native Compose stack (one service, the file kept as written) instead of splitting it into services |
| `--branch` | string | | branch of --repo or --repo-url (default: the default branch) |
| `--connection` | string | | git connection ID or name for --repo |
| `--convert-bind` | stringArray | | <service>=<container path>: turn a directory bind mount into a named volume (repeatable) |
| `--deploy` | bool | | deploy the new services right away |
| `--domain` | stringArray | | <service>=<hostname>\|auto\|off: set the domain (repeatable) |
| `--env` | string | | environment ID, slug or name |
| `--env-file` | string | | the .env file for ${VAR} interpolation (default: .env next to the Compose file) |
| `--exclude` | stringArray | | <service>: do not import this service (repeatable) |
| `--file` | stringArray | | supply a referenced file: <path as written in Compose>=<local file> (repeatable) |
| `--git` | stringArray | | <service>=<https url>[#branch]: build a service from this repository (repeatable) |
| `--image` | stringArray | | <service>=<image>: pull this image instead of building (repeatable) |
| `--include` | stringArray | | <service>: also import a service that Compose gates behind a profile (repeatable) |
| `--managed` | stringArray | | <service>: use a managed database (repeatable; needed to accept the licence of Redis and MongoDB) |
| `--name` | string | | canvas group title (default: the Compose project name) |
| `--no-managed` | stringArray | | <service>: keep an official database image as a plain image service (repeatable) |
| `--no-rewrite` | stringArray | | <service>.<KEY>: keep the host names in the variable as they are (repeatable) |
| `--not-secret` | stringArray | | <service>.<KEY>: store the variable in plain text (repeatable) |
| `--path` | string | | path of the Compose file in the repository (default: docker-compose.yml, compose.yaml, ...) |
| `--plan` | bool | | show what would be imported and change nothing |
| `--port` | stringArray | | <service>=<port>: the service port (repeatable) |
| `--project` | string | | project ID, slug or name |
| `--publish` | stringArray | | <service>=<port>/<tcp\|udp>: publish a container port through a TCP/UDP route (repeatable) |
| `--repo` | string | | read the Compose file from owner/name of a git connection (with --connection) |
| `--repo-url` | string | | read the Compose file from a public github.com or gitlab.com repository URL |
| `--secret` | stringArray | | <service>.<KEY>: store the variable as a secret (repeatable) |
| `--server` | string | | server ID or name the services run on |
| `--skip-var` | stringArray | | <service>.<KEY>: do not import the variable (repeatable) |
### nebulactrl import coolify
Import projects, applications and databases from a Coolify v4 instance as staged services.
```
nebulactrl import coolify --url <coolify url> --server <server> [flags]
```
Reads a Coolify instance through its API and stages the same projects and environments here: applications become image or git services
(Dockerfile builds stay Dockerfile builds, Nixpacks becomes Railpack), databases become managed databases of the same engine and version,
domains, health checks, resource limits, custom Docker options, deployment commands and variables come along. Variables that point at a Coolify
database or application are rewritten to ${{service.EXPORT}} references; credentials, locked values and secret-looking keys are stored as secrets.
Nothing is deployed unless you pass --deploy, and database and volume DATA is never copied (the plan prints the follow-up commands).
The Coolify API token needs the `read` and `read:sensitive` abilities (Keys & Tokens → API tokens in Coolify); read:sensitive is what returns
variable values and custom Docker options. The token is read from --token-env (default COOLIFY_TOKEN), --token-file or --token, sent to your
NebulaCtrl server for this one request, and never stored or printed. Registry pull credentials cannot be read from Coolify: give them with
--registry-user and --registry-token-env.
Use --plan (or --dry-run) to review without changing anything: it lists every resource, its variables by key (never values), the rewritten
references and everything that is not imported. Per-resource decisions name the resource by its Coolify name or UUID, variables as <resource>.<KEY>.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--deploy` | bool | | deploy the new services right away (default: everything stays staged) |
| `--domain` | stringArray | | <resource>/<coolify host>=<new host>\|off: change or drop a domain (repeatable) |
| `--dry-run` | bool | | same as --plan |
| `--engine-version` | stringArray | | <database>=<version>: managed database version (repeatable) |
| `--exclude` | stringArray | | <resource>: do not import this application or database (repeatable) |
| `--git-connection` | string | | git connection ID or name used to clone private GitHub/GitLab repositories |
| `--include` | stringArray | | <resource>: import it although it is off by default; for Redis and MongoDB this accepts their source-available licence (repeatable) |
| `--into` | stringArray | | <coolify project>=<project>: import into an existing project instead of creating one (repeatable) |
| `--kind` | stringArray | | <resource>=<app\|worker> (repeatable) |
| `--no-rewrite` | stringArray | | <resource>.<KEY>: keep host names and credentials of Coolify resources in the value as they are (repeatable) |
| `--not-secret` | stringArray | | <resource>.<KEY>: store the variable in plain text (repeatable) |
| `--plan` | bool | | show what would be imported and change nothing |
| `--port` | stringArray | | <resource>=<port>: service port; 0 makes it a worker without a port (repeatable) |
| `--registry-token-env` | stringArray | | <registry host>=<ENV_VAR>: environment variable with that registry's token or password (repeatable) |
| `--registry-user` | stringArray | | <registry host>=<username>: pull credential user, for example ghcr.io=octocat (repeatable) |
| `--rename` | stringArray | | <resource>=<name>: service name (repeatable) |
| `--rename-project` | stringArray | | <coolify project>=<name>: name of the new project (default: the same name, plus " (Coolify)" when it exists) |
| `--secret` | stringArray | | <resource>.<KEY>: store the variable as a secret (repeatable) |
| `--server` | string | | server ID or name the services run on (required to import) |
| `--set` | stringArray | | <resource>.<KEY>=<value>: supply the value of a locked variable (visible in the process list: prefer --set-env) (repeatable) |
| `--set-env` | stringArray | | <resource>.<KEY>=<ENV_VAR>: supply the value of a locked variable from an environment variable (repeatable) |
| `--skip-project` | stringArray | | <coolify project>: do not import this project (repeatable) |
| `--skip-var` | stringArray | | <resource>.<KEY>: do not import the variable (repeatable) |
| `--token` | string | | Coolify API token (visible in the process list: prefer --token-env or --token-file) |
| `--token-env` | string | `COOLIFY_TOKEN` | environment variable that holds the Coolify API token |
| `--token-file` | string | | read the Coolify API token from this file (- for stdin) |
| `--url` | string | | address of the Coolify instance (default: $COOLIFY_URL) |
## nebulactrl instance
Settings of the whole installation (instance admin only).
```
nebulactrl instance
```
### nebulactrl instance dashboard-access
Who may reach the dashboard and how its certificate is issued.
```
nebulactrl instance dashboard-access
```
#### nebulactrl instance dashboard-access get
Show the dashboard's allow list and certificate method.
```
nebulactrl instance dashboard-access get
```
#### nebulactrl instance dashboard-access set
Change the dashboard's allow list and certificate method.
```
nebulactrl instance dashboard-access set [--allow <cidr>]... [--tailnet-only | --clear-allow] [--dns-resolver <provider> | --http-01] [--confirm-lockout] [flags]
```
Restricts who may reach the dashboard's hostname (an IP allow list on its route) and chooses how its certificate is issued.
Settings you do not name stay as they are. --allow replaces the list (repeat it or separate CIDRs with commas; IPv4 and IPv6).
--tailnet-only is the list 100.64.0.0/10, fd7a:115c:a1e0::/48. Agents on other servers and webhooks reach the control plane
through the same hostname, so keep their addresses in the list.
The change is refused (409 dashboard_lockout) when the list leaves out the address you are connecting from, unless --confirm-lockout is set.
--dns-resolver needs a resolver of that provider on the control-plane server (Servers, Proxy). Run from the host with an SSH tunnel to recover from a lock-out.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--allow` | stringSlice | | CIDR or address allowed to reach the dashboard; replaces the list (repeatable, comma separated) |
| `--clear-allow` | bool | | lift the restriction: everybody may reach the dashboard |
| `--confirm-lockout` | bool | | save even though the list excludes the address you are connecting from |
| `--dns-resolver` | string | | issue the certificate through this DNS-01 provider (for example cloudflare) |
| `--http-01` | bool | | issue the certificate automatically over HTTP-01 |
| `--tailnet-only` | bool | | allow only Tailscale addresses (100.64.0.0/10, fd7a:115c:a1e0::/48) |
## nebulactrl login
Store the control plane URL and API token.
```
nebulactrl login --url <url> --token <token> [flags]
```
Verifies the token against GET /me and writes ~/.config/nebulactrl/config.json (mode 0600).
Pass --token - to read the token from stdin; NEBULACTRL_TOKEN is used when --token is omitted.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--token` | string | | API token (nct_…); - reads stdin |
| `--url` | string | | control plane URL, e.g. https://nebula.example.com |
## nebulactrl members
Manage the members of your team.
```
nebulactrl members
```
### nebulactrl members invite
Invite someone by email, or create a link invite.
```
nebulactrl members invite [--email <address>] --role <role> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--email` | string | | mail the invite to this address (single use); omit for a link invite |
| `--expires-in-days` | int64 | | validity in days (default 7) |
| `--max-uses` | int64 | | link invites: how many people may use it (default 1) |
| `--role` | string | | owner, admin, developer or viewer |
### nebulactrl members ls
List the members of the token's team.
```
nebulactrl members ls
```
## nebulactrl mesh
Manage the server mesh (a tailnet of your servers).
```
nebulactrl mesh
```
### nebulactrl mesh check
Test whether the server's mesh node reaches the other servers.
```
nebulactrl mesh check <server>
```
### nebulactrl mesh disable
Remove a server from the mesh.
```
nebulactrl mesh disable <server>
```
### nebulactrl mesh enable
Join a server to the mesh (adds one tagged tailnet device).
```
nebulactrl mesh enable <server> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--accept-mesh-changes` | bool | | confirm that this adds tailnet devices |
| `--dry-run` | bool | | print the plan without changing anything |
### nebulactrl mesh links
Show the cross-server links of an environment: links, grants and routes.
```
nebulactrl mesh links <project/env>
```
### nebulactrl mesh reconcile
Create (add) or clean up (repair) the cross-server links of an environment now.
```
nebulactrl mesh reconcile <project/env> [flags]
```
Mode add creates every link the environment's variable references and canvas links need. It adds tailnet devices (tagged devices may be billed by Tailscale), so it needs --accept-mesh-changes.
Mode repair only removes grants whose source is gone; it never adds a device.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--accept-mesh-changes` | bool | | confirm the tailnet devices mode add creates |
| `--mode` | string | `add` | add or repair |
### nebulactrl mesh reprovision
Re-create the server's mesh node (new device and tailnet IP).
```
nebulactrl mesh reprovision <server>
```
### nebulactrl mesh service
Show what a service exposes on the mesh, which providers it reaches and who reaches it.
```
nebulactrl mesh service <service>
```
### nebulactrl mesh status
Show the mesh: nodes, links and the ACL state.
```
nebulactrl mesh status
```
## nebulactrl projects
Manage projects.
```
nebulactrl projects
```
### nebulactrl projects create
Create a project (with a production environment).
```
nebulactrl projects create <name> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--description` | string | | project description |
### nebulactrl projects ls
List projects.
```
nebulactrl projects ls
```
### nebulactrl projects update
Rename a project or set where its services are built.
```
nebulactrl projects update <project> [--name <n>] [--description <d>] [--build-location inherit|target|build-server [--build-server <server>] [--registry <id>]] [flags]
```
--build-location overrides the team default for every git service of the project (a service can override it again).
The placement is replaced as a unit: --build-server pins a build server and --registry names the registry; an
omitted one is cleared (the control plane picks a build server, the team default registry applies).
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--build-location` | string | | inherit, target or build-server |
| `--build-server` | string | | pin the build server (ID or name) |
| `--description` | string | | new description |
| `--name` | string | | new name |
| `--registry` | string | | registry ID the builds are pushed to |
## nebulactrl servers
Manage Docker hosts.
```
nebulactrl servers
```
### nebulactrl servers enroll-token
Create a one-time agent enrollment token.
```
nebulactrl servers enroll-token --name <name> [flags]
```
Registers a pending server and prints the one-hour, single-use enrollment token together with the
command that installs the agent on the host. The token is shown once.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--acme-email` | string | | email for Let's Encrypt registration |
| `--base-domain` | string | | base domain for generated hostnames |
| `--name` | string | | server name |
| `--public-ip` | string | | public IP, used for DNS checks and sslip.io domains |
### nebulactrl servers ls
List servers.
```
nebulactrl servers ls
```
### nebulactrl servers revoke
Revoke a server's agent identity.
```
nebulactrl servers revoke <server-id|name>
```
Disconnects the agent and cancels its pending commands. The agent must re-enroll to come back.
### nebulactrl servers unrestricted
Show, enable or disable the Unrestricted switch of a server (instance admin).
```
nebulactrl servers unrestricted <server-id|name> [--on --confirm <name> | --off] [flags]
```
By default Compose stacks and services run under safe allow-lists. An unrestricted server lifts them for everything on it:
privileged mode, host networking, devices, GPUs, a custom runtime, the host PID namespace, unconfined seccomp or AppArmor,
NET_ADMIN, SYS_ADMIN, the Docker socket and host bind mounts. Anyone who can deploy there can take over the server, so only
the instance admin can switch it, and enabling needs --confirm with the server's exact name. Disabling does not stop running
containers; the next deploy of something that needs it fails. Without --on or --off, prints the current state.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--confirm` | string | | the server's exact name (required with --on) |
| `--off` | bool | | make the server restricted again |
| `--on` | bool | | make the server unrestricted |
### nebulactrl servers update
Set a server's role and build settings.
```
nebulactrl servers update <server-id|name> [--role deployments|builds|both] [--max-builds <n>] [--allow-emulation[=false]] [--allow-previews[=false]] [flags]
```
Roles: `deployments` runs services (the default), `builds` only builds images for other servers and pushes them
to a registry, `both` does both. A build server needs an agent that can build and push (`build.push`).
--max-builds caps parallel builds (1-16, default 2; more queue). --allow-emulation lets it build for another
CPU architecture (slow). --allow-previews lets pull request previews build on it (off: previews build on their own server).
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--allow-emulation` | bool | | allow cross-architecture builds through emulation |
| `--allow-previews` | bool | | let pull request previews build here |
| `--max-builds` | int | | parallel builds on this server (1-16) |
| `--role` | string | | deployments, builds or both |
## nebulactrl services
Manage services.
```
nebulactrl services
```
### nebulactrl services build-plan
Preview how a git service would be built.
```
nebulactrl services build-plan <service>
```
Asks the service's server to clone the branch head and report the build method (Dockerfile, Railpack or a
starter Dockerfile), the detected providers and the start command. Nothing is built or deployed.
### nebulactrl services clone
Clone a service (staged copy; secrets are copied sealed).
```
nebulactrl services clone <service> [--env <env> [--project <project>]] [--name <name>] [--domains drop|suffix] [--dry-run] [flags]
```
Clone a service into the same or another environment. The clone is staged: nothing runs until it is deployed.
It gets the configuration, variables (secrets stay secret, generated values are generated again), volume definitions (new, empty volumes) and, in the same environment, the source's links. A managed database is created new and empty: restore a dump into it. Domains are dropped by default or cloned with a suffix.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--domain-suffix` | string | | suffix for --domains suffix (default: -copy in the same environment, -<environment slug> otherwise) |
| `--domains` | string | `drop` | domains of the clone: drop (attach new ones later) or suffix (suffix the first label of each hostname) |
| `--dry-run` | bool | | show what the clone would do and change nothing |
| `--env` | string | | environment of the clone (ID, slug or name; default: the source's environment) |
| `--name` | string | | name of the clone (default: "<name> copy" in the same environment) |
| `--project` | string | | project of --env (default: the source's project) |
| `--server` | string | | place the clone on this server (ID or name; default: the source's server) |
### nebulactrl services create
Create a service (staged until deployed).
```
nebulactrl services create --project <project> --env <env> --name <name> (--image <ref> | --git <url> [--branch <b>]) [--server <server>] [--port <n>] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--branch` | string | | Git branch (default: the repository default) |
| `--build-location` | string | | where the git source is built: inherit (default), target or build-server |
| `--build-server` | string | | pin the build server (ID or name); with --build-location build-server |
| `--build-type` | string | | how a git source is built: auto (default), dockerfile, railpack or starter:<node\|python\|go\|ruby\|php\|static> |
| `--env` | string | | environment ID, slug or name |
| `--git` | string | | Git repository URL (built on the target server, or on a build server) |
| `--image` | string | | Docker image, e.g. traefik/whoami:latest |
| `--name` | string | | service name |
| `--port` | int | | HTTP/internal port the service listens on (0 = none) |
| `--project` | string | | project ID, slug or name |
| `--registry` | string | | registry ID the build is pushed to (default: the project or team registry) |
| `--server` | string | | target server ID or name |
### nebulactrl services deploy
Deploy a service.
```
nebulactrl services deploy <service> [flags]
```
Starts a deployment and returns once it is accepted (HTTP 202). With --wait, polls until the
deployment reaches a terminal status and exits 0 only when it is `succeeded`.
With --no-cache a git service builds this one deployment without any cache (docker build --no-cache --pull,
no reuse of an image a build server already pushed); later deployments build normally.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--no-cache` | bool | | build this deployment without cache (git services; docker build --no-cache --pull) |
| `--timeout` | duration | `30m0s` | give up waiting after this long (with --wait) |
| `--wait` | bool | | wait for the deployment to finish; exit non-zero unless it succeeded |
### nebulactrl services deploy-hook
Show, enable, disable or rotate a service's deploy hook.
```
nebulactrl services deploy-hook <service> [--enable|--disable|--rotate] [flags]
```
A deploy hook is a URL (POST /hooks/deploy/<service-id>/<secret>) that deploys the service like a manual deploy,
for CI pipelines and container registries (Docker Hub, GHCR). Without a flag the command shows the hook's state.
--enable creates a secret and prints the URL; --rotate replaces the secret of an enabled hook and prints the new URL;
--disable turns the hook off and deletes the secret. The URL is shown once and cannot be read again.
Managing a hook needs the deployments:write scope.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--disable` | bool | | turn the hook off and delete its secret |
| `--enable` | bool | | turn the hook on with a new secret and print the URL once |
| `--rotate` | bool | | replace the secret of an enabled hook and print the new URL once |
### nebulactrl services docker
Show or set a service's Docker options (staged until the next deployment).
```
nebulactrl services docker <service> [flags]
```
Without flags, prints the service's Docker options. With flags, changes them and stages the service. The options are typed:
there is no free-text docker run string. Hardening options (--cap-drop ALL, --no-new-privileges, --read-only, --pids-limit),
--hostname, --dns, curated --sysctl, --user and --group-add work on every server.
--device, --gpus, --runtime, --privileged, --pid-host, --seccomp-unconfined, --apparmor-unconfined (and the NET_ADMIN and
SYS_ADMIN capabilities, set in the web UI) work only on a server an instance admin made unrestricted
(`nebulactrl servers unrestricted`): they can hand over the host. Other servers answer 422 requires_unrestricted_server.
Capabilities to add, ulimits, shm size, extra hosts, labels, tmpfs and the entrypoint are edited in the web UI.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--apparmor-unconfined` | bool | | [unrestricted server] run without the default AppArmor profile |
| `--cap-drop` | stringSlice | | capabilities to drop, e.g. ALL or NET_RAW (comma-separated; none clears) |
| `--clear` | bool | | remove every Docker option of the service |
| `--device` | stringArray | | [unrestricted server] host device host[:container][:perms], repeatable; none clears |
| `--dns` | stringSlice | | DNS resolver IPs, at most 3 (none clears) |
| `--gpus` | string | | [unrestricted server] NVIDIA GPUs: all or a count (none clears) |
| `--group-add` | stringSlice | | supplementary numeric group IDs (none clears) |
| `--hostname` | string | | container hostname (default: the service slug; "default" resets) |
| `--no-new-privileges` | bool | | set the no-new-privileges security option (--no-new-privileges=false turns it off) |
| `--pid-host` | bool | | [unrestricted server] share the host's PID namespace |
| `--pids-limit` | int64 | | limit the processes and threads of the container (0 = no limit) |
| `--privileged` | bool | | [unrestricted server] privileged mode: root on the host |
| `--read-only` | bool | | mount the root filesystem read-only (config files must then sit on a volume) |
| `--runtime` | string | | [unrestricted server] OCI runtime, e.g. runsc (default resets) |
| `--seccomp-unconfined` | bool | | [unrestricted server] run without the default seccomp profile |
| `--sysctl` | stringArray | | namespaced kernel parameter key=value, repeatable (key= removes it) |
| `--user` | string | | run as uid[:gid] ("default" resets to the image's user) |
### nebulactrl services download-logs
Download a service's stored runtime logs as a file.
```
nebulactrl services download-logs <service> [flags]
```
Writes the service's stored runtime log (the server keeps 7 days by default), oldest first, as plain text
(`<time> <line>`) or NDJSON. Secret values are masked on the server. At most 200,000 lines: a longer range
ends with a note, so narrow it with --from/--to. Needs the logs:read scope and the service's server online.
Examples:
```sh
nebulactrl services download-logs acme/production/web --from 24h -o web.log
nebulactrl services download-logs web --format ndjson --grep 'level:error' > errors.ndjson
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--component` | string | | Compose stacks: only this component's lines |
| `--format` | string | `txt` | file format: txt or ndjson |
| `--from` | string | | only lines at or after this time: RFC 3339, a date, or a duration ago (24h) |
| `--grep` | string | | only lines matching this log search query (free text, "phrases", level:error) |
| `-o, --output` | string | | write to this file instead of stdout (created with mode 0600) |
| `--to` | string | | only lines before this time (same forms as --from) |
### nebulactrl services logs
Show runtime logs of a service.
```
nebulactrl services logs <service> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `-f, --follow` | bool | | stream new lines until interrupted |
| `--tail` | int | `200` | number of recent lines to show |
| `--timestamps` | bool | | prefix each line with its timestamp |
### nebulactrl services ls
List the services of an environment.
```
nebulactrl services ls --project <project> --env <env> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--env` | string | | environment ID, slug or name |
| `--project` | string | | project ID, slug or name |
### nebulactrl services maintenance
Put a service into maintenance mode, or take it out.
```
nebulactrl services maintenance <service> on|off [--title <t>] [--message <m>] [--eta <rfc3339|duration>] [flags]
```
While on, every domain of the service answers HTTP 503 with a maintenance page and a Retry-After header instead of
reaching the service. It takes effect immediately without a deployment, and it is never staged or rolled back.
--eta takes an RFC 3339 time (2026-11-02T09:30:00Z) or a duration from now (45m, 2h). The page text and ETA are
only used with `on`; `off` keeps the stored text for next time.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--eta` | string | | expected return: an RFC 3339 time or a duration from now (45m, 2h) |
| `--message` | string | | page message |
| `--title` | string | | page title (default: Down for maintenance) |
### nebulactrl services move
Move a service to another environment (staged until the next deploy).
```
nebulactrl services move <service> --env <env> [--project <project>] [--force] [--dry-run] [flags]
```
Move a service to another environment. It keeps running where it is until its next deployment, which joins the new environment's network and variables. References are checked first: variables that point at services staying behind, and other services' variables that point at the moved one, break. Without --force a move that breaks references is refused; --dry-run prints the plan. Canvas links of the service are removed.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--dry-run` | bool | | print the plan (references, links, slug) and change nothing |
| `--env` | string | | target environment (ID, slug or name) |
| `--force` | bool | | move even though references from or to the service break |
| `--project` | string | | project of --env (default: the service's project) |
### nebulactrl services restart
Restart a service's container.
```
nebulactrl services restart <service>
```
### nebulactrl services run
Run a one-off command in a container of the service's image.
```
nebulactrl services run <service> -- <command...> [flags]
```
Runs the command (argv, no shell; use `-- sh -c '...'` for shell syntax) in a throwaway container of the service's
current image, with its environment, networks and resource limits, on its server. Prints the output and exits
with the command's exit code. Needs the deployments:write scope and is audited.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--detach` | bool | | start the run and return without waiting for it |
| `--timeout` | duration | `0s` | stop the command after this long (default 30m) |
### nebulactrl services scale
Set the number of replicas or turn on autoscaling.
```
nebulactrl services scale <service> [<replicas>] [--autoscale min:max[:target]|off] [flags]
```
Runs a stateless service (no volumes, an HTTP port) on 1-10 replicas on its server. The change applies
immediately without a deployment and without a rebuild. --autoscale min:max[:target] scales on CPU
(target percent of the replica's CPU limit, default 70); --autoscale off returns to manual scaling.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--autoscale` | string | | autoscale on CPU between min and max replicas: min:max[:target%] or off |
### nebulactrl services shell
Open an interactive shell in a service's container.
```
nebulactrl services shell <service> [-- command args…] [flags]
```
Opens a terminal in a running replica of the service (bash if the image has it, else sh), or runs the command after `--` with a terminal attached. Needs the Owner, Admin or Developer role and a token with deployments:write. Production services need --production. Sessions are audited (never their input or output) and close after 15 minutes without input.
Examples:
```sh
nebulactrl services shell acme/production/web
nebulactrl services shell web --replica 1 -- psql -U postgres
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--production` | bool | | confirm opening a terminal in a production environment |
| `--replica` | int | | replica index to open (0-based) |
### nebulactrl services update
Set where and how a service is built.
```
nebulactrl services update <service> [--build-location inherit|target|build-server] [--build-server <server|none>] [--registry <id|none>] [--submodules auto|on|off] [flags]
```
The service's own override of the project and team build settings; `inherit` follows them. `none` clears a pin or
registry. `--submodules` sets the git submodule checkout (auto: recursive when the repository has a .gitmodules file; on: always, a failure fails the build; off). The change is staged until the next deployment, like every setting. Compose stacks are edited in the web UI.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--build-location` | string | | inherit, target or build-server |
| `--build-server` | string | | pin the build server (ID or name; none clears) |
| `--registry` | string | | registry ID (none clears) |
| `--submodules` | string | | git submodules: auto, on or off |
## nebulactrl tasks
Cron tasks and their runs.
```
nebulactrl tasks
```
### nebulactrl tasks create
Schedule a recurring command.
```
nebulactrl tasks create <service> --name <name> --schedule <cron> -- <command...> [flags]
```
Runs the command (argv, no shell; use `-- sh -c '...'` for shell syntax) in a throwaway container of the service's
current image with its environment. The schedule is a 5-field cron expression or @hourly/@daily/@weekly/@monthly,
evaluated in --timezone (an IANA name such as Europe/Berlin; default UTC). Daylight-saving changes keep the wall-clock
time: a run in the skipped hour is skipped for that day, a repeated hour runs once.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--concurrency` | string | | forbid (default: skip a tick while the previous run is active) or allow |
| `--disabled` | bool | | create the task disabled |
| `--name` | string | | task name, unique per service |
| `--schedule` | string | | cron expression, e.g. "*/15 * * * *" or @daily, evaluated in --timezone |
| `--timeout` | duration | `0s` | stop a run after this long (default 30m) |
| `--timezone` | string | | IANA time zone of the schedule, e.g. Europe/Berlin (default UTC) |
### nebulactrl tasks logs
Print the output of a run (follows a run that is still active).
```
nebulactrl tasks logs <run-id>
```
### nebulactrl tasks ls
List a service's cron tasks.
```
nebulactrl tasks ls <service>
```
### nebulactrl tasks rm
Delete a cron task and its run history.
```
nebulactrl tasks rm <service> <task>
```
### nebulactrl tasks run
Run a cron task now and follow its output.
```
nebulactrl tasks run <service> <task> [flags]
```
Starts a run immediately (whatever the schedule), prints its output and exits with the command's exit code.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--detach` | bool | | start the run and return without waiting for it |
### nebulactrl tasks runs
Run history of a service's tasks and one-off commands.
```
nebulactrl tasks runs <service> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--limit` | int | | how many runs (default 50) |
| `--task` | string | | only the runs of this task (name or id) |
### nebulactrl tasks update
Change a cron task's schedule or time zone.
```
nebulactrl tasks update <service> <task> [--schedule <cron>] [--timezone <zone>] [flags]
```
Changing the schedule or the time zone recomputes the next run from now; the new next run is printed.
The time zone is an IANA name such as Europe/Berlin (UTC when never set).
Examples:
```sh
nebulactrl tasks update acme/production/web nightly --timezone Europe/Berlin
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--schedule` | string | | new cron expression |
| `--timezone` | string | | new IANA time zone |
## nebulactrl teams
Manage teams.
```
nebulactrl teams
```
### nebulactrl teams create
Create a team (you become its owner; needs an admin-scope token).
```
nebulactrl teams create <name>
```
### nebulactrl teams ls
List your teams and the saved token per team.
```
nebulactrl teams ls
```
### nebulactrl teams switch
Use the saved token of another team.
```
nebulactrl teams switch <slug>
```
## nebulactrl templates
Browse and install one-click templates.
```
nebulactrl templates
```
### nebulactrl templates install
Install a template as staged services (or deploy right away).
```
nebulactrl templates install <template> --project <p> --env <e> --server <s> [--input KEY=VALUE]... [--deploy] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--deploy` | bool | | deploy the new services right away |
| `--env` | string | | environment ID, slug or name |
| `--input` | stringArray | | install input KEY=VALUE (repeatable) |
| `--name` | string | | canvas group title |
| `--project` | string | | project ID, slug or name |
| `--server` | string | | server ID or name the stack runs on |
### nebulactrl templates installed
List the template stacks of an environment.
```
nebulactrl templates installed --project <p> --env <e> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--env` | string | | environment ID, slug or name |
| `--project` | string | | project ID, slug or name |
### nebulactrl templates ls
List catalog templates.
```
nebulactrl templates ls [--category c] [--license class] [--search text] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--category` | string | | homelab, media, storage, games, devtools, ai, data, web, analytics, automation, productivity or security |
| `--license` | string | | licence class: permissive, copyleft, source-available or proprietary |
| `--search` | string | | search words (name, summary, tags, licence, database engines) |
### nebulactrl templates plan
Preflight an install (changes nothing).
```
nebulactrl templates plan <template> --project <p> --env <e> [--server <s>] [--input KEY=VALUE]... [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--env` | string | | environment ID, slug or name |
| `--input` | stringArray | | install input KEY=VALUE (repeatable) |
| `--project` | string | | project ID, slug or name |
| `--server` | string | | server ID or name (enables capacity and port checks) |
### nebulactrl templates rm
Delete a template stack.
```
nebulactrl templates rm <instance-id> --confirm <template-id> [--delete-volumes] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--confirm` | string | | the template id, to confirm |
| `--delete-volumes` | bool | | also remove the stack's Docker volumes (data is lost) |
### nebulactrl templates show
Show a template: services, inputs and licence.
```
nebulactrl templates show <template>
```
### nebulactrl templates update
Show what a newer template revision changes, or stage it (--apply).
```
nebulactrl templates update <instance-id> [--apply] [--input KEY=VALUE]... [flags]
```
Without --apply this prints the diff between the template as installed and the catalog's current revision and changes nothing.
With --apply the changes marked 'apply' are staged as ordinary edits (nothing is deployed). Fields you changed are kept; databases, routes and services are never touched.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--apply` | bool | | stage the changes (default: only show them) |
| `--input` | stringArray | | value for an input the new revision added, KEY=VALUE (repeatable) |
## nebulactrl vars
Manage service environment variables.
```
nebulactrl vars
```
### nebulactrl vars history
Show a variable's versions (secrets show a fingerprint only).
```
nebulactrl vars history <service> KEY
```
### nebulactrl vars ls
List variables (secret values are never shown).
```
nebulactrl vars ls <service> [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--exports` | bool | | also list the virtual variables other services can reference |
| `--platform` | bool | | also list the platform-provided variables |
### nebulactrl vars rm
Delete variables (staged until the next deploy).
```
nebulactrl vars rm <service> KEY...
```
### nebulactrl vars rotate
Replace a generated variable's value (staged, never deployed).
```
nebulactrl vars rotate <service> KEY [--dry-run] [flags]
```
Re-runs the generators of a variable created with ${{secret(..)}} and similar. --dry-run prints the plan only:
which services consume the value, in which order to deploy them, and any blockers. Without it the new
value is staged on this service and on every consumer; nothing is deployed. Database credentials
cannot be rotated this way once deployed.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--dry-run` | bool | | show the plan without staging anything |
### nebulactrl vars set
Set variables (staged until the next deploy).
```
nebulactrl vars set <service> KEY=VALUE... [--secret] [--scope build] [flags]
```
Upserts variables on a service. Changes are staged: run `services deploy` to apply them.
Values are never echoed. Use KEY=- to read a value from stdin, which keeps secrets out of the
shell history and process list. A variable that is already secret stays secret.
A value may contain generators that run once, on the server: KEY='${{secret(32)}}' stores a random
secret. Running the same command again keeps the generated value (use `vars rotate` to replace it).
Other forms: ${{service-slug.KEY}}, ${{KEY}}, ${{shared.KEY}}; write $${{ for a literal ${{.
--scope build passes the variable to image builds of git services (--build-arg, or a BuildKit secret
when it is secret); --scope both also keeps it in the container.
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--comment` | string | | a note stored with the variable |
| `--scope` | string | | where the variable is injected: runtime (default), build or both |
| `--secret` | bool | | mark the variables as secret (write-only) |
### nebulactrl vars shared
Manage shared variables of the team (@team), a project (<project>) or an environment (<project>/<env>).
```
nebulactrl vars shared
```
#### nebulactrl vars shared ls
List shared variables.
```
nebulactrl vars shared ls <@team|project[/env]>
```
#### nebulactrl vars shared rm
Delete shared variables.
```
nebulactrl vars shared rm <@team|project[/env]> KEY...
```
#### nebulactrl vars shared rotate
Replace a generated shared variable (staged, never deployed).
```
nebulactrl vars shared rotate <@team|project[/env]> KEY [--dry-run] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--dry-run` | bool | | show the plan without staging anything |
#### nebulactrl vars shared set
Set shared variables (restages the services that use them).
```
nebulactrl vars shared set <@team|project[/env]> KEY=VALUE... [--secret] [flags]
```
Options:
| Flag | Type | Default | Description |
|---|---|---|---|
| `--secret` | bool | | mark the variables as secret (write-only) |
#### nebulactrl vars shared share
Stage KEY=${{shared.KEY}} on services.
```
nebulactrl vars shared share <@team|project[/env]> KEY <service>...
```
A shared variable reaches a service only through an explicit reference row. This stages
KEY=${{shared.KEY}} on each service (services given as ID or project/env/slug).
---
# API reference
Source: https://nebulactrl.com/docs/reference/api
NebulaCtrl's dashboard, CLI and MCP server all use the same REST API under `/api/v1`. Everything you can click, you can call. The pages below list every operation, grouped by area. They are generated from `openapi/openapi.json`; a running control plane also serves the spec at `/api/v1/openapi.json`.
## Authentication
There are two ways to authenticate:
| Method | Use it for |
|---|---|
| `Authorization: Bearer nct_…` | Scripts, CI and the CLI. Create a token under **Settings → API tokens**. Tokens are accepted only in this header, never in a query string. |
| The `nebulactrl_session` cookie | The dashboard. Cross-origin writes are rejected, so a browser page on another site cannot act as you. |
A token's **scopes** limit what it can do, and its project list can limit where. Scopes are intersected with the creator's role when the token is created and again on every use. See [API tokens](https://nebulactrl.com/docs/operations/teams#api-tokens).
| Scope | Allows |
|---|---|
| `projects:read` | Read projects, environments, services, deployments and logs |
| `projects:write` | Create and change projects, environments, services, variables and domains |
| `deployments:write` | Deploy, restart and stop services |
| `deployments:rollback` | Roll back a deployment |
| `servers:read` | Read servers and their metrics |
| `servers:write` | Enroll, change and revoke servers |
| `proxy:write` | Publish routes, expose TCP/UDP ports, certificates and tailnet egress |
| `admin` | Everything the creating member's role allows |
An operation marked **Public** needs no credentials, for example the first-run setup and sign-in calls.
## Errors
Errors are [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) `application/problem+json` with an extra stable `code` field. Match on `code`, not on the message.
```json
{
"status": 409,
"code": "deployment_in_progress",
"title": "Conflict",
"detail": "another deployment of this service is active"
}
```
| Status | Meaning |
|---|---|
| `401` | No valid credentials: no session, or a missing, revoked or expired token |
| `403` | Credentials are valid but lack the scope or role, or the project is not allowed |
| `404` | Not found, **or not visible to you**. NebulaCtrl does not reveal whether a resource in another project exists |
| `409` | A conflict, such as `deployment_in_progress` or a name already taken |
| `412` | Stale `If-Match`: the resource changed since you read it (`generation_conflict`) |
| `422` | Valid JSON that fails validation. `errors` lists the fields |
## Conventions
- **Lists** are paginated with `?cursor=` and `?limit=` (default 50, maximum 200). The response carries `nextCursor` while more items exist.
- **Idempotency.** Mutating calls that create things accept an `Idempotency-Key` header (up to 255 characters). A retry with the same key replays the original response for 24 hours, so a flaky network cannot create two deployments.
- **Optimistic concurrency.** Updating a service takes `If-Match` with the generation you read. A mismatch returns `412` with `generation_conflict`.
- **Changes are staged.** Writing configuration, variables or domains stages the change and returns immediately. Nothing restarts until you call a deploy endpoint.
- **Deploy and rollback return `202`** with the deployment. They run asynchronously and never report success before the agent has finished; poll the deployment, or use `nebulactrl services deploy --wait`.
- **Secrets are write-only.** A secret variable is returned as `{key, isSecret: true, set: true}`, never its value.
- **Live updates** are Server-Sent Events: `GET /events` for resource changes, and the log endpoints with `follow=true`.
## Endpoints by area
Pick an area below, or use the search.
- [Analytics](https://nebulactrl.com/docs/reference/api/analytics.md): 3 operations.
- [Approvals](https://nebulactrl.com/docs/reference/api/approvals.md): 3 operations. Production approvals for AI clients
- [Audit](https://nebulactrl.com/docs/reference/api/audit.md): 1 operation. Audit log
- [Auth](https://nebulactrl.com/docs/reference/api/auth.md): 21 operations. Sessions
- [Backups](https://nebulactrl.com/docs/reference/api/backups.md): 21 operations.
- [Builds](https://nebulactrl.com/docs/reference/api/builds.md): 2 operations.
- [Canvas](https://nebulactrl.com/docs/reference/api/canvas.md): 2 operations. Canvas layouts
- [Certificates](https://nebulactrl.com/docs/reference/api/certificates.md): 3 operations.
- [Compose](https://nebulactrl.com/docs/reference/api/compose.md): 2 operations.
- [Databases](https://nebulactrl.com/docs/reference/api/databases.md): 17 operations.
- [Deployments](https://nebulactrl.com/docs/reference/api/deployments.md): 13 operations. Deployments and releases
- [Dns01](https://nebulactrl.com/docs/reference/api/dns01.md): 12 operations.
- [Domains](https://nebulactrl.com/docs/reference/api/domains.md): 10 operations. Service domains
- [Environments](https://nebulactrl.com/docs/reference/api/environments.md): 5 operations. Environments
- [Events](https://nebulactrl.com/docs/reference/api/events.md): 1 operation. Live invalidation feed
- [Fleet](https://nebulactrl.com/docs/reference/api/fleet.md): 6 operations.
- [Git](https://nebulactrl.com/docs/reference/api/git.md): 18 operations.
- [Imports](https://nebulactrl.com/docs/reference/api/imports.md): 2 operations.
- [Instance](https://nebulactrl.com/docs/reference/api/instance.md): 8 operations.
- [Integrations](https://nebulactrl.com/docs/reference/api/integrations.md): 12 operations.
- [Links](https://nebulactrl.com/docs/reference/api/links.md): 4 operations. Service links
- [Logs](https://nebulactrl.com/docs/reference/api/logs.md): 16 operations. Logs
- [Me](https://nebulactrl.com/docs/reference/api/me.md): 14 operations. Current principal
- [Mesh](https://nebulactrl.com/docs/reference/api/mesh.md): 9 operations.
- [Metrics](https://nebulactrl.com/docs/reference/api/metrics.md): 2 operations. Metrics
- [Networks](https://nebulactrl.com/docs/reference/api/networks.md): 6 operations.
- [Notifications](https://nebulactrl.com/docs/reference/api/notifications.md): 11 operations.
- [Platform](https://nebulactrl.com/docs/reference/api/platform.md): 13 operations.
- [Previews](https://nebulactrl.com/docs/reference/api/previews.md): 4 operations.
- [Projects](https://nebulactrl.com/docs/reference/api/projects.md): 5 operations. Projects
- [Proxy](https://nebulactrl.com/docs/reference/api/proxy.md): 6 operations.
- [Registries](https://nebulactrl.com/docs/reference/api/registries.md): 7 operations.
- [Routes](https://nebulactrl.com/docs/reference/api/routes.md): 5 operations.
- [Secret managers](https://nebulactrl.com/docs/reference/api/secret-managers.md): 5 operations.
- [Servers](https://nebulactrl.com/docs/reference/api/servers.md): 37 operations. Servers and agents
- [Services](https://nebulactrl.com/docs/reference/api/services.md): 23 operations. Services
- [Setup](https://nebulactrl.com/docs/reference/api/setup.md): 2 operations. First-run bootstrap
- [Shared variables](https://nebulactrl.com/docs/reference/api/shared-variables.md): 18 operations.
- [Stacks](https://nebulactrl.com/docs/reference/api/stacks.md): 1 operation.
- [System](https://nebulactrl.com/docs/reference/api/system.md): 1 operation. Control-plane info
- [Tasks](https://nebulactrl.com/docs/reference/api/tasks.md): 9 operations.
- [Teams](https://nebulactrl.com/docs/reference/api/teams.md): 19 operations. Teams, members, invites and instance settings
- [Templates](https://nebulactrl.com/docs/reference/api/templates.md): 8 operations.
- [Terminal](https://nebulactrl.com/docs/reference/api/terminal.md): 6 operations.
- [Tokens](https://nebulactrl.com/docs/reference/api/tokens.md): 3 operations. API tokens
- [Variables](https://nebulactrl.com/docs/reference/api/variables.md): 11 operations. Service variables
- [Volume backups](https://nebulactrl.com/docs/reference/api/volume-backups.md): 14 operations.
- [Volumes](https://nebulactrl.com/docs/reference/api/volumes.md): 3 operations. Service volumes
The complete machine-readable API is the OpenAPI 3.1 document at https://nebulactrl.com/api/v1/openapi.json.
---
# Advanced settings
Source: https://nebulactrl.com/docs/operations/configuration
The everyday control-plane settings (database URL, secret key, public URL, MCP, log level) are in [the control plane](https://nebulactrl.com/docs/installing/control-plane). This page lists the **advanced timers**: product timeouts that exist for safety and that you almost never need to change. They exist so the integration suite does not have to wait ten real minutes for a grace period to expire. **Leave them unset in production**: the defaults are the tested, documented behavior.
## Advanced timers
Every timer is read by the control plane at start (a restart applies a change) from an environment variable holding a Go duration such as `30s`, `90s`, `10m` or `1h30m`. An unset or empty variable keeps the default. A value that is not a duration, or is shorter than `1s`, stops the start with a message that names the variable.
| Variable | Default | What it controls |
|---|---|---|
| `NEBULACTRL_MESH_LINK_GRACE` | `10m` | How long a cross-server link grant whose variable reference is gone stays before the repair removes it. A reference that is removed and added back, or a redeploy in between, keeps its link within this window. A canvas edge's grant is removed at once, whatever this says. The grace is remembered in memory only, so a control-plane restart starts the window again. See [Mesh](https://nebulactrl.com/docs/operations/mesh) |
| `NEBULACTRL_REPLICA_BROKEN_AFTER` | `2m` | How long a database replica may be out of streaming before it is marked `broken`. The agent polls a replica every 30 seconds, so keep this above `30s`, or a single missed poll marks a healthy replica broken. See [Database replicas](https://nebulactrl.com/docs/operations/database-replicas) |
Rules of thumb when you do change one:
- **Shorten with care.** A grace that is shorter than the interval at which its condition is re-checked removes things that would have recovered by themselves. `NEBULACTRL_MESH_LINK_GRACE` is re-checked by the repair job every 5 minutes, so on a real install a value below `5m` only makes the removal happen at the next repair run, not sooner.
- **Do not lengthen to hide a problem.** A replica that is `broken` for two minutes is not streaming; waiting longer does not fix the cause.
- The values are not stored in the database and do not appear in the settings screens.
## What the test suite sets
`tests/integration` starts every control plane with short values (`tests/integration/harness_timers_test.go`): `NEBULACTRL_MESH_LINK_GRACE=30s` and `NEBULACTRL_REPLICA_BROKEN_AFTER=60s`. The same variable in the environment of `tests/run.sh` overrides the short value, for example `NEBULACTRL_MESH_LINK_GRACE=10m ./tests/run.sh full -run '^TestMeshCrossServer$'` runs the mesh test with the production timing.
---
# MCP server
Source: https://nebulactrl.com/docs/reference/mcp
NebulaCtrl can expose a Model Context Protocol (MCP) server so an AI client, such as a coding agent, can do most of what you do in the dashboard: find out what is wrong, change a service, add a database, set up a domain, deploy, roll back. It is **off by default**.
The MCP server is a layer over the same services the dashboard, CLI and REST API use. Scopes, project restrictions, roles, audit entries and the deployment rules apply unchanged. An agent can do nothing a token with the same scopes could not do through the [API](https://nebulactrl.com/docs/reference/api), and a few things you can do by hand it can never do at all (see [what it cannot do](#what-it-cannot-do)).
## What an AI can do
| You want to… | The client uses |
|---|---|
| Know the state of everything, or what is broken | `get_overview`, `find_issues`, `get_recent_changes`, `search_resources` |
| Find out why a service or a deployment failed | `diagnose_service`, `explain_failed_deployment`, `diagnose_server`, `tail_logs` |
| Change a service (image, port, health check, variables, domains, volumes) | `stage_changes`, then `review_changes` and `apply_changes`. With the `config` toolset on, single edits such as `service_update`, `variable_set`, `domain_add` and `volume_add` are listed too. |
| Add a database or a domain in one step | `add_database`, `setup_domain` |
| Check that a deployment really works | `verify_service` |
| Go back to the previous release | `rollback_service` |
| Back up and restore, run a one-off command, install a template | `create_backup`, `restore_backup`, `run_command`, `plan_template`, `install_template` |
| Reach anything else the API offers | `find_operation`, then `call_operation` |
The tools come in three layers:
1. **Workflow tools (15).** One call does what takes several clicks: `get_overview`, `find_issues`, `diagnose_service`, `explain_failed_deployment`, `diagnose_server`, `tail_logs`, `search_resources`, `review_changes`, `stage_changes`, `apply_changes`, `rollback_service`, `add_database`, `setup_domain`, `verify_service`, `get_recent_changes`. They fetch in parallel, and when one part fails they return the rest with an `errors` list instead of failing everything.
2. **Single-purpose tools (about 60).** Named `noun_verb` (`service_list`, `variable_set`, `server_get`, `database_create`), with a short description and the four MCP hints (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`) set per tool, so a client can tell reading from deleting. Only the `observe` ones are listed by default; the rest are [opt-in toolsets](#toolsets). Large inputs, such as a service's `config`, point to `find_operation("updateService")` for the full field reference.
3. **The long tail through two tools.** `find_operation` (`query`, optional `tag`, `limit` up to 15) lists the operations the client may call, with name, summary, input schema and scopes. `call_operation` runs one by `operationId`: path, query and body arguments are one flat object validated against the operation's schema, `ifMatch` is an integer, and names such as `projectId` or `serviceId` also accept a selector. Operations on the [deny list](#what-it-cannot-do) are refused with `operation_denied` and their reason; missing scopes answer `forbidden` and name them; the production approval applies to everything.
Every result has structured content and a short text version, and lists are paged by cursor (`{items, nextCursor}`). `detail: "concise"` (the default) or `"full"` controls how much a read returns. Results are capped at 64 KB and say `truncated` when cut. A client can name things instead of passing ids: a project by slug or name, an environment as `project/env`, a service as `project/env/service`, a hostname, or a bare slug when it is unique. An ambiguous name fails with `ambiguous_selector` and the `choices` instead of guessing. Selectors need `projects:read`, so a deploy-only token must pass ids.
The server also publishes four prompts (`troubleshoot_service`, `explain_failed_deployment`, `estate_health`, `ship_a_change`) and three resources (`nebula://overview`, `nebula://service/{id}`, `nebula://deployment/{id}`), and its `instructions` tell the client where to start, how the staged workflow goes and the production rule.
## Turn it on
Set `NEBULACTRL_MCP_ENABLED=true` on the control plane and restart it. While it is off, `/mcp` answers `404`. **Settings → MCP** shows whether it is on, the endpoint URL and a client configuration you can copy.
## Connect a client
The endpoint is `/mcp` on your control plane, using the Streamable HTTP transport. It accepts an API token only, as `Authorization: Bearer nct_…`. Sessions and tokens in a query string are refused.
1. In **Settings → API tokens**, create a token. Pick a preset (see [scopes](#scopes-and-token-presets)) and restrict it to the projects the agent should see.
2. Put the token in an environment variable on your machine, so it is not written into a config file you may commit:
```sh
export NEBULACTRL_TOKEN=nct_…
```
3. Add the server to your client. Replace `nebula.example.com` with your control plane's address.
### Claude Code
```sh
claude mcp add --transport http nebulactrl https://nebula.example.com/mcp \
--header "Authorization: Bearer $NEBULACTRL_TOKEN"
```
Or commit a project-scoped `.mcp.json` that reads the token from the environment:
```json
{
"mcpServers": {
"nebulactrl": {
"type": "http",
"url": "https://nebula.example.com/mcp",
"headers": { "Authorization": "Bearer ${NEBULACTRL_TOKEN}" }
}
}
}
```
### Cursor
In `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (all projects):
```json
{
"mcpServers": {
"nebulactrl": {
"url": "https://nebula.example.com/mcp",
"headers": { "Authorization": "Bearer ${env:NEBULACTRL_TOKEN}" }
}
}
}
```
### Codex
In `~/.codex/config.toml`:
```toml
[mcp_servers.nebulactrl]
url = "https://nebula.example.com/mcp"
bearer_token_env_var = "NEBULACTRL_TOKEN"
```
### Any other client
Clients that read an `mcpServers` block with a URL and headers take the same shape as the Cursor example, with the token pasted in place of the variable. **Settings → MCP** generates this snippet with your real endpoint.
### Offering more tools
A client is offered the `core` and `observe` tools by default, which is enough to look, diagnose, stage a change, apply it and roll back (the editing happens through `stage_changes`). For an agent that edits configuration, add `?toolsets=core,config` to the URL, for example in Claude Code:
```sh
claude mcp add --transport http nebulactrl "https://nebula.example.com/mcp?toolsets=core,config,observe" \
--header "Authorization: Bearer $NEBULACTRL_TOKEN"
```
and the same URL in the `url` field of the Cursor and Codex configs. List every toolset you want: an explicit list is exactly those (see [toolsets](#toolsets)). Add `data` for databases and backups, `servers` for server management.
## Toolsets
A long tool list costs the model attention and context, so tools are grouped into toolsets. By default a client is offered **`core` and `observe` only** (about 32 KB of tool descriptions for an admin token); `config`, `data` and `servers` are opt-in:
| Toolset | Contains |
|---|---|
| `core` | The 15 workflow tools, `find_operation`, `call_operation` and `whoami`. Always the best place to start. |
| `config` | Projects, environments, services, variables, domains, volumes, links, previews, tasks, `run_command`, templates, Compose import, and clone and move (`cloneService`, `cloneEnvironment`, `moveService` through `call_operation`; a move into or out of production needs approval). |
| `data` | Databases, backups and restores (including volume backups and schedules). |
| `servers` | Servers, the fleet overview, and draining a server. |
| `observe` | Deployments, service and server metrics and analytics, the audit log and the notification channel list (the last two need `admin`). |
Three things decide what a client sees:
- **Scopes.** A tool is listed only when the token holds every scope it needs. A read-only token never sees `apply_changes`.
- **Toolsets, set in the URL.** There is no per-token setting. Without `?toolsets=` the client lists `core` and `observe`. To opt in to more, name all you want: `https://nebula.example.com/mcp?toolsets=core,config,data,servers,observe`. An explicit list is exactly those toolsets (`core`, `config`, `data`, `servers`, `observe`; an unknown name answers `400 invalid_toolsets`), so `?toolsets=core` lists about 25 KB of core tools only. The URL cannot add anything the token's scopes do not allow.
- **The long tail.** Tools that are not listed stay reachable. With no `?toolsets=`, `find_operation` and `call_operation` reach every operation the token may use that is not denied, with the same scope and approval checks. With an explicit `?toolsets=`, they are restricted to those toolsets too, so the URL doubles as a fence.
## The staged workflow
Changes to a service, its variables, domains, volumes and links **only stage**. Nothing reaches a running container until the changes are applied, the same as in the dashboard, where staged changes wait behind the "N changes to apply" bar.
1. **Stage.** `stage_changes` takes the `service`, the `ifMatch` generation the client last saw, and any of `config`, `variables`, `domains` and `volumes`. A variable can be generated (`generate: {length, charset}`) or deleted. Each edit returns what changed: `{staged, generation, appliedGeneration, change, pendingCount, next}`. A service with `generation` above `appliedGeneration` has staged changes. If someone else edited the service in between, the `ifMatch` no longer matches and nothing is staged.
2. **Review.** `review_changes` (a `service` or a whole `environment`) shows every pending change field by field, with secrets masked.
3. **Apply.** `apply_changes` is the only way from staged to running. It takes `services` (each a `service` selector plus the `expectedGeneration` the client reviewed) or a whole `environment` (`project/env`) together with `confirmPending`, the number of staged services the client reviewed (what `review_changes` lists for that environment). `waitSeconds` (at most 50) waits for the result. `noCache: true` builds the git services of that call without cache (`--no-cache --pull`, no reuse of a build server's earlier image) for these deployments only ([deploying without cache](https://nebulactrl.com/docs/operations/builds#deploying-without-cache)); it applies to services with staged changes, like the rest of the call. If someone changed a service after the review, the generation no longer matches; if the number of staged services differs, the call fails with `plan_changed`. Either way nothing deploys and the client reviews again.
4. **Verify.** `verify_service` requests the service's own hostnames through the proxy and reports whether it answers. It refuses loopback, private and link-local addresses (unless it is the service's own server), follows no cross-host redirects and times out after 8 seconds. `rollback_service` (`service`, `to`: `previous`, a release id, a deployment id or a commit's first 7 or more characters; `dryRun` previews it) returns to an earlier release. A rollback restores the application image and configuration only; data is not rolled back.
A deployment that outlasts `waitSeconds` is still running. The result says so and names the next tool to call; it never reports success it has not seen. `outcome_uncertain` means a command was accepted but its server went offline before reporting, so check before retrying.
Secrets: `variable_set` can generate a value (`generate: {length, charset}`; `length` 1 to 1024, default 32; `charset` one of `alnum`, `hex`, `base64`, `base64url`, `digits`, `lower`), or take a value you typed. A secret value is never returned by any tool, never logged and masked in audit entries and in log output. `service_update` merges the `config` fields you give into the current config, so you send only what changes.
## Production approval
An AI client that changes a **production** environment needs a person to approve it. The check lives in the domain services, not in the tools, so a model cannot route around it (not even through `call_operation`), and it fails closed: if the approval store cannot be reached, the action is refused.
What is guarded, when the environment is production: deploy and apply, deploy staged changes, rollback, restart, stop, deleting a service, restoring a backup, an in-place volume restore, running a task or one-off command, deleting a volume, domain, environment or project, un-flagging an environment as production, upgrading a database, rotating credentials, enabling maintenance mode, changing scaling, deleting a template instance, importing a Compose file with deploy, cancelling an unfinished deployment, and draining a server that runs production services. Non-production environments need no approval.
When a guarded action is requested:
1. **Elicitation.** If the client supports MCP elicitation (the 2026-07-28 protocol's multi round-trip flow), NebulaCtrl asks it to show you a form with an **Approve** checkbox that states exactly what will happen: the service, the environment, the changes and the strategy. You approve or decline in the client, and the call continues.
2. **The approval URL.** Older clients, and clients that cannot elicit, get the tool error `confirmation_required` with the plan and a link, `https://nebula.example.com/approvals/{id}`. Open it signed in to the web app: the page shows the same plan with **Approve** and **Decline**. After you approve, the client repeats the same call and it goes through. Owners, Admins and Developers with deploy rights on the project can decide; Viewers can only read. An agent can never approve: the approval operations are not available over MCP.
3. **Nothing a model can type.** The model cannot pass a `confirm: true` for production. An approval is single-use and bound to the token, the tool, the exact arguments and the generations that were reviewed. It lives at most 10 minutes from the request. If the arguments differ on the retry, or the service's generation moved in between, it is void and the client has to ask again. A request you decline keeps refusing the same call for one minute.
Over plain HTTP the same refusals appear as `409 approval_required`, `403 approval_declined` and `503 approval_unavailable`. Every outcome is written to the [audit log](https://nebulactrl.com/docs/operations/teams) as `approval.requested`, `approved`, `declined`, `expired`, `refused`, `error` or `used`.
### Token settings
When you create a token in **Settings → API tokens**, the **Production changes** section has two settings:
- **Let MCP clients change production without approval** (`allowProductionWithoutApproval`) is for an unattended job such as a nightly deploy. It is **off by default**, only an Owner or Admin can turn it on, and turning it on is audited. It is honoured only while the token's creator is still an Owner or Admin. Keep it off for any token a model uses interactively.
- **Also ask for approval on REST calls** (`requireProductionApproval`) applies the same guard to the token's REST and CLI calls, for any member who creates it. REST tokens are not guarded by default, so CI and deploy hooks keep working. Signed-in browser sessions are never guarded by this.
## Scopes and token presets
Each tool needs the scopes of the REST operations it calls, and the service re-checks them on every call. `logs:read` is new: it gates reading runtime, build and task logs, both in `tail_logs` and in the diagnostic tools that include log excerpts, because logs can hold data a person would not paste into a chat. Existing tokens that could read projects received it when you upgraded, so nothing stops working.
| Scope | Lets the client |
|---|---|
| `projects:read` | Read projects, environments, services, deployments, variables (never secret values) and templates. |
| `servers:read` | Read servers, their metrics and proxy state. |
| `logs:read` | Read logs (`tail_logs`), and log-bearing diagnoses. `get_task_run` needs it together with `projects:read`. |
| `projects:write` | Stage changes to projects, environments, services, variables, domains and volumes; install templates (`deploy: true` also needs `deployments:write`); restoring a backup needs it together with `deployments:write`. |
| `deployments:write` | Apply changes, restart, stop, back up, run commands (`run_command`; a selector in it needs `projects:read`, so a deploy-only token passes ids). |
| `deployments:rollback` | Roll back. |
| `servers:write` | Change server settings. |
| `proxy:write` | Change proxy routes, certificates and ports. Never in an MCP preset. |
| `admin` | Everything the token's creator's role holds. |
The presets in **Settings → API tokens**:
- **MCP read-only:** `projects:read`, `servers:read`, `logs:read`. Look and diagnose; change nothing. Start here.
- **MCP deploy:** adds `deployments:write`: apply changes that are already staged (say, from the dashboard), restart, run commands. It cannot stage edits itself.
- **MCP configure:** adds `projects:write` as well, so the client can stage edits to services, variables, domains and volumes and then apply them. Production changes still need approval.
No preset includes `deployments:rollback` or `proxy:write`; add `deployments:rollback` under Custom if you want the client to roll back.
A token is also limited to its projects and by its creator's role, which is checked again on every use. See [API tokens](https://nebulactrl.com/docs/operations/teams#api-tokens).
## What it cannot do
Some operations are never available over MCP, whatever the token holds, and `find_operation` does not list them. They would let a model (or whoever steers it through a prompt injection) take over the account, read a secret, or reach the host itself. You can still do all of them in the dashboard. Each one is denied for one of these reasons:
| Not available | Why |
|---|---|
| Sign-in, two-factor, passkeys, password reset, OIDC, first-run setup (`auth-plumbing`) | They exist for people signing in. A token never authenticates a person. |
| Your own account: profile, password, two-factor and passkey management, switching or listing teams, accepting an invite, approving a production approval (`session-only`) | Browser account management. An agent must not change how you sign in, and a principal must never approve its own production change: only a person in the web UI can. |
| Create API tokens, enrollment tokens, team invites and member password reset links; platform admin links; enable or rotate deploy hooks and webhook secrets; start a GitHub App manifest (`mints-credential`) | They return a credential, link or secret URL once. An agent holding one could escalate beyond its own scopes or add a machine to your fleet. |
| Reveal a database export or public connection string, a volume backup recovery key, platform service credentials (`reveals-secret`) | Secret values stay write-only. Copy them in the dashboard, which audits it. |
| Write Cloudflare, Tailscale, DNS, registry, git, secret manager, backup destination, log drain, notification channel or MaxMind credentials; upload a TLS key; instance backup and instance settings; SSO provider setup; teams, member roles and removal, team security; platform services (`instance-admin`) | A credential would pass through the model's context, and these settings change the platform for everyone or decide who may do what. Reading most of them stays available. |
| The server `unrestricted` switch, the host shell switch, applying or reverting host hardening, upgrading a server's packages and rebooting it (`host-root`) | They give or change root-level access to the host or widen what containers may do on it. The web terminal and file upload are not exposed over MCP at all. |
| Following a log stream (`streaming`) | MCP calls return one result. `tail_logs` returns a bounded, redacted tail instead. |
| An HTML page, such as a maintenance-page preview (`binary`) | A tool result is JSON or text. |
| Raw deploy, deploy staged, rollback, log history endpoints (`superseded-by-composite`) | `apply_changes`, `rollback_service` and `tail_logs` do the same job with the review, the production approval and the result handling the raw endpoint lacks. |
One exec path remains on purpose: `run_command` runs a command you name in a throwaway container of the service's image, with the service's environment. That is **shell-equivalent** for that service: it can read the service's secrets and change its data. It needs `deployments:write`, is subject to the production approval and is audited with the command line. Leave `deployments:write` off any token an agent uses on untrusted input.
Everything else is available, through a tool or through `call_operation`, behind its scopes and the production approval.
## Limits and safety
- **Rate limit.** 60 requests per minute per token. Beyond that the call fails with a rate-limit error; wait and retry.
- **Output.** A result is capped at 64 KB. `tail_logs` returns at most 500 lines, redacted.
- **Logs are untrusted text.** A container can print anything, including text that looks like an instruction. Log and command output comes back as `{untrusted: true, nonce, source, notice, text, lines}`, with the text between `<untrusted-log nonce=…>` markers and a fresh random `nonce` on every call, so the client can tell it from tool output. It stays redacted.
- **Audit.** Every call is written to the audit log as `mcp.tool_call` with the arguments redacted (anything that may hold a secret, a value or a command is replaced) and the client's name. The service's own audit rows carry the actor type `mcp` and the label "token via client". A network client cannot claim the MCP actor: it is set in-process, never by a header.
- **Retries.** Mutating calls are idempotent: a repeated call returns the first result rather than deploying twice.
- **Errors** carry a `code` and a message.
## How it compares to Coolify's MCP
Coolify ships a first-party MCP server too, and its newest versions cover a lot: around 77 tools for servers, applications, databases, services, environment variables, deployments and scheduled tasks, plus prompts and resources. Both are scoped to a token and audited.
Where NebulaCtrl differs:
- **Staging.** Edits stage and `apply_changes` ships them after a review at a specific generation. Coolify changes a variable and then you deploy, without a diff to review.
- **Production approval.** Ours is a person approving a plan that is bound to the exact arguments, over elicitation or a web page. Destructive tools in Coolify rely on a `confirm` argument the model fills in.
- **Hints.** Every tool states whether it reads, writes, deletes or repeats safely, tool by tool.
- **Diagnosis.** Workflow tools return the cause with the evidence (failing health check, last log lines, recent changes), including deployment logs, capped, redacted and marked untrusted.
- **One surface, filtered.** Scopes and toolsets decide what each token lists, and the rest is reachable through `find_operation` rather than being absent.
Where Coolify is ahead: its server has been in use longer, and it also manages notification channels, S3 storage and git sources from MCP. We deliberately keep credential writes, token minting and host shells out of MCP, so those stay in the dashboard. (Based on Coolify's published source and docs at v4.5.0, October 2026.)
## Renamed tools
If a client or prompt still uses an older tool name, these are the replacements. The old names are gone, with no aliases.
| Old name | Now |
|---|---|
| `list_projects`, `list_environments`, `list_services` | `project_list`, `environment_list`, `service_list` |
| `get_service`, `get_deployments`, `get_deployment_status` | `service_get`, `deployment_list`, `deployment_get` (adds `terminal` and `outcome_uncertain`) |
| `list_servers`, `get_server_status` | `server_list`, `server_get` |
| `get_service_traffic`, `restart_service` | `service_analytics`, `service_restart` |
| `get_service_logs` | `tail_logs` (needs `logs:read`) |
| `plan_deployment` | `review_changes` |
| `deploy_service` | `apply_changes` |
| `rollback_deployment` (with `confirmReleaseId`) | `rollback_service` |
`run_command`, `get_task_run`, the four backup tools and the three template tools keep their names.
## Machine-readable docs
These docs are also available to agents as plain text: see [llms.txt](https://nebulactrl.com/llms.txt) for an index, [llms-full.txt](https://nebulactrl.com/llms-full.txt) for everything in one file, and the OpenAPI document at `/api/v1/openapi.json`.
---
# Troubleshooting
Source: https://nebulactrl.com/docs/operations/troubleshooting
Start with: which part is failing, the control plane (`docker compose logs nebulactrl`), the agent (`journalctl -u nebulactrl-agent`), or the proxy (`docker logs nebulactrl-traefik` on the agent host)?
## The agent does not connect
| Symptom | Cause / fix |
|---|---|
| Installer: "refusing plain http:// control plane URL" | The control plane URL is `http://`. Use an `https://` URL (see [serving the dashboard](https://nebulactrl.com/docs/installing/installer#two-ways-to-serve-the-dashboard)) or `--insecure-dev` on a trusted dev network. |
| Installer: "Docker Engine … too old" / "daemon is not reachable" | Install or start Docker 24+ (`systemctl start docker`). |
| Journal: enrollment `401` / "invalid or expired token" | Tokens are single use and live 1 hour. Create a new one. |
| Journal: TLS / certificate errors | The control plane's certificate is not valid for the hostname in `--url`, or the host's clock is wrong (`timedatectl`). |
| Journal: "agent revoked; re-enroll" | The server was revoked. [Re-enroll](https://nebulactrl.com/docs/operations/servers#re-enroll-a-host). |
| Connects, drops, reconnects | A reverse proxy is closing idle WebSockets. Raise its read/idle timeout for `/agent/v1/connect`; the agent heartbeats every 10 s. |
| Server shows `offline` but the host is fine | Outbound HTTPS blocked, DNS broken, or the control plane URL changed. `curl -v <url>/healthz` from the host. |
## Proxy: port 80 or 443 in use
The server card shows **Proxy error** and adding a domain fails with `422 proxy_unavailable`. Find the occupant:
```sh
sudo ss -ltnp 'sport = :80 or sport = :443'
docker logs nebulactrl-traefik 2>&1 | tail -20
```
Typical culprits: a host nginx/Apache, or another reverse-proxy container. (The control plane's own compose file has no proxy: the host's agent's Traefik serves the dashboard.) Stop it, **or** move NebulaCtrl's proxy: `sudo systemctl edit nebulactrl-agent`, add `--proxy-http-port` / `--proxy-https-port` to `ExecStart` (or put `NEBULACTRL_PROXY_HTTP_PORT=8080` into `/etc/nebulactrl-agent.env`) and restart. Public HTTPS certificate issuance needs the real 80/443 reachable from the internet.
The proxy also needs **127.0.0.1:9000** for its admin entrypoint. If something else (Portainer, for example) uses that port, the proxy fails to start the same way; move it with `--proxy-admin-port` (or `NEBULACTRL_PROXY_ADMIN_PORT`).
The agent creates `nebulactrl-traefik` when it starts, so after freeing the port run `sudo systemctl restart nebulactrl-agent`; the heartbeat then reports the proxy as running.
## A domain does not work
1. **DNS**: the domain row shows DNS status. A records must point at the server's public IP; a job rechecks every 10 minutes (`dns_ok` is empty when the server has no public IP configured). Wildcard CNAMEs to a proxy such as Cloudflare in orange-cloud mode make the check say "mismatch" although it may work.
2. **Certificate**: Traefik needs port 80 reachable from the internet for the ACME HTTP-01 challenge and an ACME email on the server; only non-local names get Let's Encrypt certificates. Look at `docker logs nebulactrl-traefik 2>&1 | grep -i acme`. Let's Encrypt rate limits apply to repeated failures; wait or use another hostname.
3. **502 / "no upstream"**: the service is not healthy or not on the edge network. `docker ps --filter name=nc-<slug>`; check the deployment succeeded.
4. **Hostname conflict**: `409 domain_conflict`: a hostname belongs to exactly one service in the whole installation.
5. **Local testing**: `*.localhost`, `*.local`, bare names and IPs get Traefik's default self-signed certificate: use `curl -k`. Unknown hosts get Traefik's default 404.
## A deployment fails
Open the deployment: the step timeline says *where*, the log says *why*.
| Message | Meaning |
|---|---|
| "Not delivered: server offline. No changes were made." | The agent never received the command within 15 min. Safe to retry once the server is online. |
| "Waiting for <server> (offline since …)" | The command was **acknowledged**; its outcome is unknown until the agent returns. Do not retry blindly; check the host or revoke the server. |
| pull error (unauthorized, not found) | Image name/tag or registry credentials wrong. |
| "Previous release is still serving" + container logs | The new container failed its health check; fix the app or the health check settings (path, port, start period, deadline). The old release is untouched. |
| `rolled_back` | The route verification failed; the old release was restored. Check the route/health path and that the app answers `<500` for the first hostname. |
| `rollback_failed` | See [recovering from rollback_failed](https://nebulactrl.com/docs/operations/deployments#recovering-from-rollback_failed). |
| `unresolved_reference` / `reference_cycle` | A `${{service.KEY}}` variable points at an unknown service/key in the same environment, or variables reference each other in a loop. |
| `409 deployment_in_progress` | Another deployment of this service is still active; wait for it to reach a terminal state. |
| build fails | The full `docker build` output is in the deployment log. Builds run on the target host with its Docker daemon (timeout 30 min). |
## Disk full on an agent host
Images, the build cache and volumes share Docker's data root. Check the disk gauge (warns at the per-server threshold, default 80 %). The server's [Docker cleanup](https://nebulactrl.com/docs/operations/docker-cleanup) runs daily and starts by itself when the alert is active; press *Run cleanup now* on the server page to reclaim at once, or lower a service's *image retention*. Never `docker volume prune` blindly: it removes service data (the cleanup's *Remove unused volumes* keeps every volume of your services).
## The UI looks stale
If the pill "Live updates paused — retrying" is visible, the event stream (`/api/v1/events`, Server-Sent Events) is blocked, usually by a proxy that buffers or has a short timeout. Data still refreshes every 15 s. Disable buffering for `text/event-stream` in your reverse proxy.
## Control plane will not start
| Log message | Fix |
|---|---|
| `NEBULACTRL_SECRET_KEY is required` / invalid | The key must be base64 of exactly 32 bytes (set `NEBULACTRL_SECRET_KEY` or `NEBULACTRL_SECRET_KEY_FILE`, not both). |
| `sealed values are not on the current key or format; run nebulactrl-server rekey` (warning) | Values are still on an older key or in the v1 format. Run `nebulactrl-server rekey` ([secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest#rotating-the-key)). Everything keeps working while the old key is in `NEBULACTRL_SECRET_KEY_PREVIOUS`. |
| `decryption failed` for a stored credential, `rekey` reports `cannot decrypt` | The value was sealed with a key that is not in the ring (a previous key was removed too early), or its row was tampered with. Put the old key back into `NEBULACTRL_SECRET_KEY_PREVIOUS`, or re-enter the secret. |
| `database unreachable` | Postgres is not healthy or the password in `NEBULACTRL_DATABASE_URL` does not match `POSTGRES_PASSWORD` (the password is only applied when the database volume is first created). |
| `goose up` errors | A migration failed; the database may be on a newer schema than the binary (downgrade). Restore a backup or run the newer version. |
| Exits immediately under Compose | `docker compose logs nebulactrl`; missing required `.env` values are reported by Compose itself. |
## Locked out
Lost the password of the only Owner, or the phone with the authenticator app? Both are fixed from the control-plane host; nothing needs SMTP or another admin. Run the commands with the server's environment, in the standard install (`/opt/nebulactrl`; the Compose service is `nebulactrl`, or `control-plane` in `deploy/compose`):
```sh
cd /opt/nebulactrl
docker compose exec nebulactrl nebulactrl-server reset-password you@example.com
docker compose exec nebulactrl nebulactrl-server reset-2fa you@example.com
```
- **`reset-password <email>`** prints a one-time link (`<dashboard>/reset-password?token=ncr_…`, works once, valid 30 minutes). Open it and choose a new password. If `NEBULACTRL_PUBLIC_URL` is not set the output shows the path only: put the address you open the dashboard at in front of it. `--generate` sets a random password instead and prints it once; use it for an account that has no password (passkey or single sign-on only). It asks you to type the account email first (`--confirm <email>` answers in advance).
- **`reset-2fa <email>`** removes the authenticator app (also a half-finished enrolment and its lockout), every passkey and every recovery code, after you type the account email (`--confirm <email>`). Afterwards the account signs in with its password alone until it enrols again in **Settings → Security**; a team that requires 2FA asks for that at the next sign-in. It refuses for an account with no password and no linked provider whose only way in is its passkeys: run `reset-password --generate` first.
- A lost password **and** a lost second factor: run both. A reset link never bypasses 2FA.
- Both end every browser session of the account and are audited as **system** actions (`operator.password_reset_link`, `operator.password_set`, `operator.2fa_reset`, in each of the account's teams, with the OS user that ran them; never the link or the password). API tokens are not revoked: revoke them in **Settings → Tokens** if you suspect a leak.
- No accounts yet (`error: this instance has no accounts yet…`): first-run setup is still pending. Open the setup URL the installer printed (`<dashboard>/setup?token=…`, also in `/opt/nebulactrl/INSTALL-SUMMARY.txt`).
- Whoever can run these has the database and the secret key already; see [login methods](https://nebulactrl.com/docs/security/login-methods#forgot-password) for the by-email flow and the team-admin reset.
## Reporting a problem
Include the NebulaCtrl version, the deployment ID, the relevant `correlation_id` log lines, and `journalctl -u nebulactrl-agent` around the time. Remove secrets first. Security problems: see [`SECURITY.md`]().
---
# Known limitations
Source: https://nebulactrl.com/docs/operations/limitations
This is a first release. These are the real remaining gaps, not bugs; the list is what the product does **not** do today. Teams and roles, SSO, passkeys and TOTP two-factor sign-in, git-provider webhooks, notifications, TCP/UDP routes, the web terminal, templates and managed databases all exist: see [teams](https://nebulactrl.com/docs/operations/teams), [login methods](https://nebulactrl.com/docs/security/login-methods), [git providers](https://nebulactrl.com/docs/operations/git-providers), [notifications](https://nebulactrl.com/docs/operations/notifications), [proxy routes](https://nebulactrl.com/docs/operations/proxy-routes), [terminal](https://nebulactrl.com/docs/operations/terminal), [templates](https://nebulactrl.com/docs/operations/templates) and [databases](https://nebulactrl.com/docs/operations/databases).
## Scope
- **Compose runs as a stack, with limits.** A Compose file can run as one **stack** service ([Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks)): recreate-only deploys, one rollback for the whole stack, HTTP routed through domains (published ports are never bound on the host), no `build:`, no replicas, per-component backups, maintenance mode or tasks, and the safe allow-list unless an instance admin makes a server unrestricted. The alternative is to **split** a Compose file into services with the one-time [Compose import](https://nebulactrl.com/docs/operations/compose-import) (not kept in sync; build args, bind-mounted directories and the other keys listed there are not carried over). Without the Docker Compose plugin (2.24 or newer) on the server, stacks cannot be deployed.
- **Templates are a curated catalog of 300+.** They install as ordinary services on one server ([templates](https://nebulactrl.com/docs/operations/templates)); there is no user-facing "save as template" and no runtime import of other catalogs. A newer template revision is offered as an [update](https://nebulactrl.com/docs/operations/templates#versions-and-updates) that stages image, configuration, variable, file and volume changes (your own edits are kept); it never adds or removes services, upgrades a database, changes published ports or deploys by itself, and installs from before updates existed cannot be updated in place. Game-server templates publish TCP/UDP ports on the server's public interface; hosts with a firewall must open them.
- **Managed databases are single-container**, with one exception: PostgreSQL can have up to 5 asynchronous **read replicas** and a **manual Promote** ([database replicas](https://nebulactrl.com/docs/operations/database-replicas)). There is no HA, no automatic failover, no point-in-time recovery, no synchronous or cascading replication, no replicas for the other engines, and no connection pooling or reader proxy; engine settings, init scripts and TLS are not part of the database panel, and `dump_restore` upgrades (PostGIS, MongoDB) are guided, not automatic ([databases](https://nebulactrl.com/docs/operations/databases#limits)). ClickHouse and MongoDB are lightly exercised.
- **A public database port is an IP allow-listed TCP route, not a managed TLS endpoint.** It restarts the proxy briefly when the port is new, and the managed databases do not enable TLS on their own ([databases](https://nebulactrl.com/docs/operations/databases#public-port)).
- **Backups:** managed databases (logical dumps), volumes of other services (restic, with a short freeze, snapshot or stop per [backups](https://nebulactrl.com/docs/operations/backups)) and the control plane's own database (scheduled, one instance destination; see [backup and restore](https://nebulactrl.com/docs/operations/backup-restore)). The secret key is never part of a backup and must be stored separately. Volume backups need the service on one replica (services with volumes always are) and restic-compatible storage. A Docker volume is not a backup.
- **The web terminal is an exec into a replica container**; a shell on the server itself is the separate, opt-in [host shell](https://nebulactrl.com/docs/security/host-shell) (Owners and Admins with two-factor authentication). There is no session recording or replay (by design: I/O is never stored), no `docker exec` through MCP, no host-shell file transfer and no sharing of a live session between people. The file browser is read-only in production. See [web terminal and file browser](https://nebulactrl.com/docs/operations/terminal).
- **Platform services are one of each kind per team** (S3 storage, log store, optional Grafana, tailnet control server, single sign-on, container registry), each on one server with the images pinned in the release; the S3 store placed on a server that also runs the databases is not off-site; Loki and Grafana are AGPL-3.0 (unmodified images, separate containers); the identity provider's people are managed by NebulaCtrl and every change redeploys it; changing the public hostname or the control plane's public URL means removing and setting up again; browser sign-in through a real public hostname and Headscale with real Tailscale clients were not exercised ([platform services](https://nebulactrl.com/docs/operations/platform-services#limits)).
- **No Kubernetes, no Swarm**, no scheduler: you pick the server for each service. Builds run on the target host by default; an optional [build server](https://nebulactrl.com/docs/operations/build-servers) pushes to a registry and the target pulls by digest (see [Build servers: limits](https://nebulactrl.com/docs/operations/build-servers#limits): one platform per build, no multi-arch manifests, no ECR/GAR/ACR token refresh, no registry-free transport).
- **Docker Engine only** on Linux hosts (Debian 12/13, Ubuntu 22.04, 24.04 and 26.04, amd64/arm64 are the supported targets).
- **OS patching is visibility plus two instance-admin actions** ([server patching](https://nebulactrl.com/docs/operations/server-patching)): apt and dnf only, a check at most every 6 hours, no automatic upgrades or reboots, `apt-get upgrade` semantics (held-back packages are left alone), and the upgrade log is stored when it ends, not streamed.
- **Trusted-team model.** Teams have roles (Owner, Admin, Developer, Viewer, plus per-project access), but the Docker socket is root-equivalent on every host ([threat model](https://nebulactrl.com/docs/security/threat-model)). API tokens have scopes and optional project restrictions.
- **Sign-in:** password reset by email needs `NEBULACTRL_PUBLIC_URL` and an SMTP relay or a team email channel; otherwise an Owner or Admin generates a reset link for the member ([forgot password](https://nebulactrl.com/docs/security/login-methods#forgot-password)). Accounts that sign in only through an identity provider have no password to reset. Two-factor authentication is an authenticator app or WebAuthn passkeys, with recovery codes; passkeys need HTTPS (or localhost) and a domain name, and are bound to the public hostname.
- **Git sources:** generic Git takes HTTPS, HTTP and `git://` URLs with a write-only token, and `ssh://` / `git@host:org/repo.git` URLs with a per-service SSH deploy key and a pinned host key ([SSH deploy keys](https://nebulactrl.com/docs/operations/ssh-deploy-keys)). Submodules are checked out recursively (Automatic by default); the repository's token or deploy key is used only for submodules on the same host, other hosts are fetched anonymously over https, and Git LFS and shallow clones are not supported ([submodules](https://nebulactrl.com/docs/operations/builds#git-submodules)). A one-off build without cache exists ([deploying without cache](https://nebulactrl.com/docs/operations/builds#deploying-without-cache)). A pre-deploy and a post-deploy command exist ([tasks](https://nebulactrl.com/docs/operations/tasks)); a failing post-deploy command only warns.
- **Production guard:** changes an AI client makes to a production environment through [MCP](https://nebulactrl.com/docs/reference/mcp#production-approval) are enforced **by the server**: a person must approve the exact action (MCP elicitation, or a page in the web app) and the check fails closed. REST and CLI tokens are **not** guarded unless the token was created with "Also ask for approval on REST calls", so CI and deploy hooks keep working; the dashboard's guarded Deploy key is a confirmation in the web app only (the web terminal is the other place the server enforces `confirmProduction`). Non-production environments never need approval, and a token with "Let MCP clients change production without approval" skips it.
- **Cancel** works at any step until the new release is live; after that it is too late and the deployment finishes ([canceling](https://nebulactrl.com/docs/operations/deployments#canceling-a-deployment)). A stop of the previous release in a recreate deployment is not interrupted: the cancel takes effect when it has finished (at most the stop grace period).
- **Log history lives on each server.** The agent keeps runtime logs for 7 days (at most 500 MB per service, 5 GB per server; configurable) and queries run there, so history is unavailable while a server is offline, and a service that moves to another server leaves its older lines behind. Drains (Loki, OTLP/HTTP, syslog, HTTP JSON) ship from the agents; there is no control-plane log store, no full-text index and no cross-server log search beyond an environment's merged view. See [log history and drains](https://nebulactrl.com/docs/operations/logs). Request origins by country/city are available; see [traffic analytics](https://nebulactrl.com/docs/operations/traffic-analytics).
- **Secrets and variables:** external secret managers are Vault / OpenBao (KV v2) and Infisical only; Bitwarden Secrets Manager is not supported (its SDK licence forbids redistribution and its protocol is undocumented; revisit if that changes). Shared variables exist at team, project and environment level, not server level ([variables](https://nebulactrl.com/docs/operations/variables), [secret managers](https://nebulactrl.com/docs/operations/secret-managers)).
- **Docker options are typed fields**, not free-form flags: capabilities to add (safe list) and drop, `no-new-privileges`, read-only root filesystem, pids limit, hostname (default: the service slug), DNS IPs, curated namespaced sysctls, user and groups, `nofile`/`memlock` ulimits, shm size, extra hosts, non-reserved labels, init, stop signal, tmpfs and entrypoint on every server; devices, GPUs, a runtime, privileged mode, the host PID namespace, unconfined seccomp or AppArmor and `NET_ADMIN`/`SYS_ADMIN` only on a server an instance admin made unrestricted. Never `--ip`, host networking on a plain service, restart or log options, the host IPC namespace, host bind mounts or the Docker socket ([Docker options](https://nebulactrl.com/docs/operations/docker-options)). Host kernel parameters such as `vm.max_map_count` cannot be set from a container: the plan only warns.
## Networking
- **Cross-server networking is the opt-in [mesh](https://nebulactrl.com/docs/operations/mesh), not an overlay.** Each host has its own Docker networks and its own Traefik proxy. Services in one environment on the **same host** reach each other by service name (`${{service.NEBULA_PRIVATE_HOST}}`). Services of one environment on **different hosts** reach each other through **mesh links** once the mesh is enabled on both servers: tagged tailnet devices (server nodes and link nodes) that count against your Tailscale plan, TCP only, same environment only (no cross-environment links, no UDP, no Compose stacks), added only by explicit actions, with the ACL generated by NebulaCtrl (managed on Headscale; advisory on Tailscale unless you opt in to a managed push whose token can rewrite the whole policy). Without the mesh, provide a reachable address yourself (VPN, a [per-service tailnet](https://nebulactrl.com/docs/operations/service-tailnet), tunnel or public domain). Managed Headscale falls back to Tailscale's public DERP relays (end-to-end encrypted; see [mesh](https://nebulactrl.com/docs/operations/mesh#derp-public-relays-are-the-fallback)). Tailscale SaaS, real NAT traversal, hosts with a firewall and `/dev/net/tun` inside docker-in-docker were not verified. The managed [Cloudflare Tunnel](https://nebulactrl.com/docs/operations/cloudflare) per server carries public HTTP traffic into Traefik only (Cloudflare terminates the public TLS); it does not connect services on different hosts, and it was verified against a fake of the Cloudflare API, not a real account.
- **Network isolation is per environment, opt-in beyond it** ([networking](https://nebulactrl.com/docs/operations/networking)). Environments, projects and teams on one host no longer share a network, and **Block outbound internet** cuts a service off (default on for databases), but it is all-or-nothing per service (no per-destination rules), a blocked service cannot join a shared network, shared networks span one server (a mesh link is the only cross-server path, and only within an environment), and a sibling deployed before another began blocking joins the internal network only at its next deploy. Docker networks are not a sandbox for hostile tenants.
- **TCP/UDP exposure goes through proxy routes**, which claim a listen port on the server (a clash returns `port_in_use`). IP rules are an allow list plus a deny list evaluated deny first (rendered as one allow list); there is no further rule ordering, and UDP cannot enforce IP rules at all. Typed proxy options only: there is no raw proxy configuration editor, by design ([proxy routes](https://nebulactrl.com/docs/operations/proxy-routes#ip-rules-allow-and-deny)). DNS-01 wildcard certificates work with any lego provider: 19 have a guided form, the rest a generic one; only the guided forms are documented against the lego catalog, and none is verified against a real DNS account ([DNS-01 certificates](https://nebulactrl.com/docs/operations/dns01-wildcard-certificates)).
- Hostnames are unique across the whole installation.
- **Per-service tailnet** ([service tailnet](https://nebulactrl.com/docs/operations/service-tailnet)): one tagged device per service (counts against your Tailscale device quota); Headscale gets plain HTTP and raw TCP only (no Serve HTTPS, no Funnel); Funnel bypasses NebulaCtrl's proxy; the web path balances per connection; no egress from the app to tailnet hosts; Tailscale SaaS paths (Serve HTTPS, Funnel, OAuth key minting, device deletion) are not verified against a real tailnet.
- **Service replicas run on one server only** (1-10, stateless services; not to be confused with Postgres [read replicas](https://nebulactrl.com/docs/operations/database-replicas), which can live on another server). Multi-host load balancing and automatic failover do not exist; a service runs on **one server**.
## Deployment behavior
- **No zero-downtime for `recreate` services**: services with volumes, or without an HTTP port, are stopped before the replacement starts. Only volume-less HTTP services use rolling updates.
- **Health semantics follow Coolify**: a failed check aborts a *deploying* release and keeps the old one; it does not kill or restart an already-running container beyond Docker's restart policy. `none` means "ran for 10 s".
- **Rollback is application image and configuration only**; volumes, databases and external services are never rolled back. A git release built on its server can only be rolled back while its built image still exists on the host (default: the newest 5); a release built on a [build server](https://nebulactrl.com/docs/operations/build-servers#rollback-and-retention) is pulled from the registry by digest, as long as the registry still holds it.
- A git source without a Dockerfile builds with **Railpack** (0.x, pinned), or a starter Dockerfile if you choose one ([builds](https://nebulactrl.com/docs/operations/builds)). Railpack needs outbound access to `ghcr.io` and either Docker's containerd image store or the `docker buildx` plugin on the server.
- Builds run **on the target host** with its Docker daemon by default (no registry, no build farm): a build competes for resources with running services and untrusted Dockerfiles execute on the host ([threat model](https://nebulactrl.com/docs/security/threat-model)). With a [build server](https://nebulactrl.com/docs/operations/build-servers) the build moves to a separate host, but that host is a shared privileged builder for every project of the team (per-project BuildKit builders isolate caches, not the kernel), pull-request previews build on their target unless a build server allows previews, and the registry becomes a dependency of those deployments (a deploy falls back to building on the target when no build server or registry is usable).
- Moving a service to another server does not move its volumes.
- **Clone and move copy configuration, not data.** A cloned service or environment gets new, empty volumes and new, empty databases (restore a dump into them); route services, Compose stacks and read replicas are not cloned, and services that are in a replica group or have cross-server links cannot be moved ([clone and move](https://nebulactrl.com/docs/operations/clone-and-move)).
- Replacing a service's container closes WebSocket connections still open to the old container: it is stopped after the 2 s drain and its stop grace period. A route change itself does not close existing connections (Traefik hot-reloads its configuration).
## Platform
- **One control-plane instance** (no high availability); it keeps agent connections in memory. Agents reconnect and reconcile after a restart, and workloads keep running, but there is no management while it is down.
- Agents **upgrade themselves** to the control plane's version (automatic, or on request, with a rollback when the new binary does not start), but only from an agent that already has the feature: an older agent is upgraded once by re-running the installer. The **control plane** is upgraded by re-running the installer; there is no dashboard "update" button ([upgrades](https://nebulactrl.com/docs/operations/upgrades)).
- **Forward-only migrations**: downgrading the control plane means restoring a backup.
- One secret key serves the whole instance (every team); there is no envelope encryption, KMS or TPM sealing of it. Rotation runs on a live instance but is manual: [secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest#rotating-the-key).
- Metrics and traffic analytics retention is per server and bounded ([monitoring](https://nebulactrl.com/docs/operations/monitoring#resource-metrics)); there is no long-term time-series store.
- The control plane's public image is not published yet; Compose builds from source.
- NebulaCtrl is AGPL-3.0-only ([ADR 0005](https://nebulactrl.com/docs/decisions/0005-license)). Anyone who runs a modified version as a network service must offer its users the modified source.
---
# 0001 Go modular monolith
Source: https://nebulactrl.com/docs/decisions/0001-go-modular-monolith
- Status: accepted
- Date: 2026-10-25
## Context
The spec (§5, §6, §14) asks for a lightweight self-hosted platform: one control-plane process (API, static web, optional MCP, background workers) plus one small agent per Docker host, backed by PostgreSQL. UI, CLI and MCP must call the same authorized domain services (§0.4). Operating it should not require a fleet of services.
## Decision
- The control plane is a **single Go binary** (`nebulactrl-server`), split into internal packages by domain (`projects`, `environments`, `services`, `variables`, `domains`, `deployments`, `servers`, `agents`, `reconciler`, `jobs`, `mcp`, …). Packages communicate through Go interfaces, not network calls. Domain service methods call `policy.Require` and `audit.Record`; HTTP handlers and MCP tools are thin adapters over them.
- The agent (`nebulactrl-agent`) and the CLI (`nebulactrl`) are separate static Go binaries in the same module and share `pkg/agentprotocol`.
- Libraries instead of frameworks: chi + huma (OpenAPI 3.1 and RFC 9457 errors generated from Go types), pgx + sqlc + goose, River for jobs in the same Postgres, scs sessions with argon2id, `coder/websocket`, the moby client, go-git, the official Go MCP SDK, cobra.
- Postgres is the only stateful dependency. Jobs, sessions, idempotency keys and the command queue live there; no Redis, no message broker.
- Live UI updates use SSE from an in-process event bus. A single control-plane instance is assumed for P0.
- The web app is built separately (React, Vite) and embedded into the server binary for release; in development Vite proxies `/api`, `/agent` and `/mcp` to the server.
## Consequences
- One artifact to deploy, back up and upgrade; transactional consistency between domain writes, audit rows and job inserts (River inserts in the same transaction).
- Module boundaries are enforced by convention and review only; keep packages free of import cycles and keep handlers free of business rules.
- The monolith is not horizontally scalable as written: the websocket hub and event bus are in-process. Splitting workers out later is possible because River and Postgres already mediate all work, but it is not designed for in P0.
- Go on both sides gives one toolchain and a shared protocol package, at the cost of the web stack being a second ecosystem.
## Alternatives considered
- **Laravel/PHP control plane** (spec §5 names it as a fallback if an advanced codebase existed; none does).
- **Rust/Axum**: strong, but more complexity across the whole system for no P0 product advantage (spec §5).
- **TypeScript full stack** (the `RobinGasi/nebulactrl` approach): separate language from the agent and no shared protocol package.
- **Microservices** (API, scheduler, hub as separate processes): rejected by spec §1 ("not a heavy microservices system").
---
# 0002 Agent transport
Source: https://nebulactrl.com/docs/decisions/0002-agent-transport
- Status: accepted
- Date: 2026-10-25
## Context
Spec §6.3 suggests agent-initiated, strongly authenticated connections with explicit identity, revocation, replay protection, protocol versioning, durable commands and honest handling of uncertain state. Homelab hosts are often behind NAT, so the control plane cannot dial them. Open question §19.5 asks gRPC stream vs WebSocket over TLS.
## Decision
**Transport.** The agent always dials outbound to `wss://<control-plane>/agent/v1/connect` using `github.com/coder/websocket`. A plain `ws://` URL is refused unless the agent runs with `--insecure-dev`. One socket per server carries commands, results, heartbeats, events, stats and log streams in both directions. A newer connection from the same server replaces the older one.
**Enrollment.**
1. The UI creates a one-time token `nct_enroll_<32 random bytes, base32>`. Only its SHA-256 hash is stored; it expires after 1 hour, is single-use and revocable. The server row is created at this point with status `pending`.
2. The agent generates an Ed25519 keypair and calls `POST /agent/v1/enroll {token, publicKey, hostname, facts}`; the response is `{serverId, agentId}`.
3. The agent stores its identity in bbolt at `<state-dir>/agent.db` (mode 0600). The `--token` is used only while no identity is stored.
**Connect handshake.**
1. The server sends `challenge{nonce, serverTime}`.
2. The agent replies `auth{agentId, signature}` where the signature covers `nonce || agentId || serverTime`.
3. The server verifies the signature against the stored public key, rejects a nonce older than 30 seconds (replay protection), and rejects any identity with `revoked_at` set. Then the agent sends `hello`.
4. A revoked agent is closed with status **4401**; the agent exits non-zero with "agent revoked; re-enroll". An unsupported protocol major version is closed with **4400**.
**Envelope.** Every frame is JSON `{"v":1,"type":…,"id":…,…}`. `pkg/agentprotocol` exports `ProtocolVersion = 1`. Agent to server: `hello`, `heartbeat` (every 10 s), `ack`, `result`, `event`, `log`, `log_end`, `stats`, `observed`. Server to agent: `challenge`, `command`, `result_ack`, `reconcile`. Command kinds are a closed set: `deploy`, `restart_service`, `stop_service`, `remove_service`, `sync_routes`, `logs_subscribe`, `logs_unsubscribe`. The agent rejects anything else; it never runs a shell string from the wire.
**Command durability and uncertainty.**
- The control plane inserts an `agent_commands` row (status `pending`) **before** sending, with a unique idempotency key and an `expires_at` 15 minutes ahead. It sends immediately when the server is online and on every connect.
- The agent ACKs on receipt and records the command ID in bbolt. A redelivered command returns the stored result instead of running again, which makes commands idempotent.
- The agent keeps each result until it receives `result_ack` and resends unacknowledged results in `hello.unackedResults` after a reconnect.
- A periodic job moves `pending`/`sent` commands past `expires_at` to `expired`; the deployment then fails with "Not delivered: server offline. No changes were made." This is safe because the agent never saw the command.
- **Acked commands never time out to a terminal state.** After an ACK the outcome is genuinely unknown while the agent is offline. The deployment stays active and the UI shows "Waiting for <server> (offline since …)" until the result arrives or an operator revokes the server.
- Secrets travel only inside the outgoing frame (`secrets`), attached at send time and never persisted in `agent_commands.payload`.
**Liveness.** A server is `offline` after 30 s without a heartbeat; observations older than 30 s render as stale. On `hello` the control plane sends `reconcile` with the active deployments for that server (see the reconciler in the plan).
## Consequences
- Works behind NAT and strict firewalls: nothing listens on the agent host for the control plane. Log streaming and commands share one authenticated, encrypted connection.
- Agent identity is a keypair rather than a bearer secret; compromise of the control-plane database does not yield agent credentials, and revocation is immediate (close 4401).
- Authentication happens at the application layer inside TLS, not mTLS. It gives equivalent identity properties without managing a CA; certificate rotation is replaced by re-enrollment.
- The control plane must keep one live socket per server in process memory (see ADR 0001 on single-instance scope).
- The protocol is hand-rolled JSON over WebSocket, so versioning and frame compatibility are our responsibility. The `v` field and close code 4400 are the compatibility gate.
## Alternatives considered
- **gRPC bidirectional stream**: strong typing and flow control, but heavier toolchain, more friction through HTTP proxies, and no extra capability needed for this message set.
- **Control plane dials agents (SSH or HTTPS)**: needs inbound ports on every host; breaks homelab NAT; SSH is a general shell, contradicting spec §6.2.
- **Polling agent over HTTPS**: simple, but no push for logs and higher command latency.
- **mTLS with a private CA**: rejected for P0 to avoid certificate lifecycle code; the Ed25519 challenge provides identity, replay protection and revocation.
---
# 0003 Traefik proxy
Source: https://nebulactrl.com/docs/decisions/0003-traefik-proxy
- Status: accepted (supersedes the original Caddy decision, see Alternatives)
- Date: 2026-10-26
## Context
Spec §9 requires domain routing with automatic HTTPS, failure-safe configuration updates, preserved old routes when a new target is unhealthy, WebSocket compatibility and hostname conflict detection, and asks for an ADR choosing Caddy or Traefik. Each host has its own proxy (spec §8.6); the proxy is managed infrastructure, not a user service.
## Decision
Every enrolled host runs **Traefik v3**, created and owned by the agent as a Docker container. Services never publish host ports; only Traefik does. Routes are pushed as one complete dynamic configuration through Traefik's **REST provider**.
**Bootstrap (at agent start, idempotent).**
- network `nebulactrl-edge`; volume `nebulactrl-traefik-acme` (mounted at `/data`, holds `acme.json`);
- container `nebulactrl-traefik` from `traefik:v3.7.16@sha256:c8a121e4bb1347e8fd52657fab12f2cb70b68f8a018895e2068f92c9e519b9c2` (v3.6.17 / v3.7.1 and later carry the fix for GHSA-96qj-4jj5-wcjc, the `rest@internal` exposure), restart `unless-stopped`, json-file logs rotated at 10 MB × 3;
- static configuration as command flags:
- entrypoints `web` `:80`, `websecure` `:443` (HTTP/3 on 443/udp), `probe` `:8099` (never published) and `ncadmin` `:9000`;
- `--providers.rest=true`, `--providers.file.filename=/etc/traefik/bootstrap.yml` (static bootstrap only), `--ping` with manual routing, the API without dashboard;
- `gracetimeout=10s` on the public entrypoints;
- the ACME resolver `le` (HTTP-01 on `web`, storage `/data/acme.json`) only when the server has an ACME email;
- the JSON access log on stdout (read by the agent's traffic collector) unless the server's traffic analytics setting is off; Traefik's own log is JSON on stderr;
- ports `<--proxy-http-port>:80`, `<--proxy-https-port>:443` and `…:443/udp`. The admin entrypoint is published as `127.0.0.1:<--proxy-admin-port>:9000` for a host-installed agent; an agent running in a container on the edge network reaches `http://nebulactrl-traefik:9000` and nothing is published;
- the bootstrap file is written into the container with `CopyToContainer` **before its first start**, so there is no window without the admin router. It defines the router `nc-admin` (entrypoint `ncadmin`, `rest@internal`, `basicAuth` user `nebulactrl-agent`), a read-only `nc-api` router (`api@internal`, same auth) and a liveness router on `probe`. The password is a random 32-byte secret the agent generates once and keeps in its bbolt store; only its bcrypt hash is in the container. `providers.rest.insecure` is never set: it would expose an unauthenticated write endpoint to every container on the edge network. Other containers get 401.
- If Traefik cannot start (for example port 80 is taken) the heartbeat reports `proxy.status="error"` with the reason, the server view shows it, and creating a domain on that server returns 422 `proxy_unavailable`.
**Configuration model.** The agent holds the whole route table and pushes the **full** dynamic configuration with `PUT /api/providers/rest`. No incremental patching. For each service it emits:
- router `nc-<serviceId>-https` on `websecure`, rule `Host(…) || Host(…)` for all hostnames, with TLS. Names that can be issued a public certificate use `certResolver: le` (only when an ACME email exists); `*.localhost`, `*.local`, bare names and IPs use Traefik's default certificate. A service that mixes both kinds gets a second router `…-https-local`;
- router `nc-<serviceId>-http` on `web`, a `redirectScheme` middleware to https (with the public HTTPS port when it is not 443). Traefik answers ACME HTTP-01 challenges before routers;
- router `nc-<serviceId>-probe` on `probe` with the same rule, no TLS and `observability.accessLogs=false`;
- service `nc-<serviceId>` as a load balancer with the single server `http://<containerName>:<port>` over the edge network and `passHostHeader`. The container name is unique per deployment, so old and new releases coexist;
- one generation marker router `nc-gen` on `probe`, rule ``Host(`gen-<n>.nc.invalid`)``, to `ping@internal`.
Hosts that match no router get Traefik's default `404`, so an unrouted host cannot pass verification.
**Synchronous apply on an asynchronous provider.** Traefik applies a pushed provider configuration asynchronously and a `200` from the REST endpoint only means "decoded". `Apply` therefore returns only when the new table is live or has failed:
1. `PUT` the configuration (basic auth). A non-2xx answer is an error.
2. Poll the probe entrypoint with `Host: gen-<n>.nc.invalid` until it answers 200 (at most 15 s). The generation number grows monotonically (millisecond clock).
3. Read all routers from the authenticated admin API. Traefik loads a config it can parse even when it cannot build part of it (a bad upstream URL, an invalid rule); such routers are `disabled` with an error. Any disabled router of this generation fails the apply.
4. On any failure the agent pushes the previous table again (a fresh generation) and returns the error. The in-memory and bbolt copies of the last-applied table change only after success.
The agent validates hostnames, upstreams and ports before building the configuration, so step 3 is a safety net rather than a normal path. Because the REST provider replaces the whole configuration, the routes of a rejected generation are absent until the restore completes (measured at roughly 0.3 s).
**Static versus dynamic settings.** Route changes never restart Traefik. The ACME email and the access-log toggle are static flags: when either changes, the agent recreates the container (and then pushes the table again). The container carries a spec label (a hash of image, flags, ports, log options and the admin secret) so an unchanged configuration is never recreated.
**Liveness.** The REST provider keeps its state in Traefik's memory only. The agent checks every 5 s that the marker of the live generation still answers; if Traefik restarted, the marker is gone and the agent bootstraps again and pushes the table.
**Verification before commitment.** After a new container is healthy and the table is live, the agent sends `GET http://<traefik-ip>:8099<healthPath>` with `Host: <first hostname>` and expects a status below 500 (3 tries, 2 s apart; the container IP on the edge network, falling back to the `nebulactrl-traefik` DNS name). The probe routers have the same rules and services as the public ones, so this tests the exact table production traffic uses without DNS or public ports. On failure the agent pushes the previous table and removes the new container.
**Conflicts.** Hostnames are globally unique in the control-plane database (409 `domain_conflict`); because services never bind host ports, a hostname clash is the only routing conflict.
## Consequences
- One whole-document apply is easy to reason about, makes the route table reproducible from the control plane after a restart, and lets the reconciler re-apply it on reconnect.
- The proxy is a Docker container the agent must be allowed to create; it needs ports 80/443 free and the admin port (default 9000 on 127.0.0.1). It restarts with the host through Docker's restart policy; the agent re-pushes the table.
- Config reloads never close existing connections (Traefik hot reload), so rolling swaps keep serving (measured: 0 errors over 2,281 requests with the old upstream stopped after each swap). WebSocket connections to the old container end when that container is stopped.
- The probe entrypoint shares Traefik's process, so a verify pass shows routing is correct but not that DNS or the public network path is.
- Unlike Caddy's `/load`, a bad configuration is not refused up front; the agent detects it through router status and restores the previous table, with a short gap for the affected routes.
- A change of ACME email or of the traffic-analytics setting restarts the proxy container briefly.
- Local names use Traefik's default self-signed certificate (browsers warn), there is no internal CA.
- Cross-host traffic is explicitly out of scope: each host's Traefik only routes to local containers.
## Amendment: proxy v2 (Phase 6a, agent side)
The agent renders more than host routers. The wire contract is `pkg/agentprotocol/proxy.go`; the agent code is `internal/agent/proxy`.
- **Per-domain routers.** A route carries `DomainRule`s. Each rule renders `nc-<serviceId>-d<domainId8>-https|http|probe` (redirect domains: `-redir` and `-http`) with the rule ``Host(`h`) [&& PathPrefix(`/p`)]``. Hosts without a rule keep the legacy names (`nc-<serviceId>-https|-http|-probe`), so a table without rules renders exactly as before. The middleware chain is ordered cheapest rejection first: `ipAllowList → rateLimit → basicAuth → redirectRegex → stripPrefix → addPrefix → headers → compress`. The probe router keeps only the prefix rewrites, so verification checks routing and the upstream, not access rules. A basic-auth rule without resolved users is a hard error: the table is never applied with an unprotected domain.
- **Deny lists are subtraction.** Traefik has no deny middleware (only `ipAllowList`), so `Access.DenyCIDRs` and `Route.DenyCIDRs` are rendered into the same one `ipAllowList` as (allow list, or every address when it is empty) minus the denied ranges (`agentprotocol.EffectiveAllow`, CIDR arithmetic on `netip`). That gives exactly "deny first, then allow, default deny when an allow list exists". Rule sets that admit nobody or expand beyond 4096 networks are refused by both the control plane and the agent. Without a deny list the allow list is rendered as written, so existing tables are unchanged.
- **Secrets stay out of payloads.** bcrypt lines (`auth:<domainId>`) and certificate keys (`cert:<id>`) arrive in `Command.Secrets`; `ResolveSecrets` fills the agent-side fields. The resolved table and the certificates (with keys) live in the agent's bbolt file (mode 0600), so a restart converges offline. A missing secret fails the command before anything is applied.
- **Custom certificates** go into the same REST document as inline PEM (`tls.certificates[].certFile/keyFile`): no files, no restart (verified on Traefik v3.7.16). A pair that does not parse is excluded and reported in `ProxyConfigureResult.Certificates`. A name covered by an uploaded certificate is served from the certificate store even on an `auto` domain; ACME stays for names no certificate covers.
- **TCP/UDP routes** need static entrypoints (`tcp-<port>`, `udp-<port>`) and published host ports, so the container is recreated when the set of listen ports changes (about one second of dropped connections; HTTP routes are rebuilt immediately after). A container that cannot start (a port already in use) is removed and the previous container is created again with the previous table and configuration; the error names the port. The host-gateway entry (`host.docker.internal:host-gateway`, upstream policy) and the tailnet resolver (`100.100.100.100`, MagicDNS) are static settings of the same kind. `Apply` waits for tcp and udp routers through `/api/tcp/routers` and `/api/udp/routers` as for http.
- **Upstream policy** is enforced by the agent as well as the control plane, for external routes and every tcp/udp route: loopback, link-local (cloud metadata) and the proxy's own containers are denied unless an administrator allows them per server. Names are also resolved best effort; a name that resolves to a denied address is rejected, a resolution failure is allowed. UDP routes cannot carry an allowlist (Traefik has no UDP middleware), and the agent refuses a udp route that has one.
- **Tailnet egress** runs as a second container, `nebulactrl-tailscale`, in kernel mode inside the proxy container's network namespace (`network_mode: container:nebulactrl-traefik`, `/dev/net/tun`, `NET_ADMIN`/`NET_RAW`, no privileged mode, no docker socket, identity on the `nebulactrl-tailscale-state` volume). It is recreated whenever the proxy container is, because the namespace belongs to the proxy; the identity survives on the volume, so no new auth key is needed. A one-time auth key is held in memory only and dropped after the apply that used it. Disabling removes the container and volume; an agent that merely lost its state never deletes a node. Status comes from `tailscale status --json` and travels in heartbeats.
Not verified against a real tailnet.
## Amendment: maintenance and error pages (Phase 6a)
Traefik cannot serve files, so a second managed container, `nebulactrl-pages`, serves the maintenance page and the automatic 404/502/503/504 pages. See [operations/maintenance-and-error-pages.md](https://nebulactrl.com/docs/operations/maintenance-and-error-pages).
- **Image.** The official `nginx` on Alpine, `slim` variant, stable branch (`1.30.6-alpine-slim`), pinned by index digest. Alternatives weighed: Caddy (~60 MB, a second ACME-capable proxy we would have to keep from touching certificates), `busybox httpd` (cannot set a status code with a file body or per-file headers), `traefik/whoami` or a purpose-built Go image (an image of our own to build, sign and publish). nginx answers a status with an `error_page` file and per-location `add_header ... always`, which is exactly the need, at about 5 MB.
- **Wiring.** An `errors` middleware (`502-504`, query `/err/{status}`) leads the chain of every service router; a `priority: 1` catch-all on `web`/`websecure` (`tls: {}`) rewrites unknown hosts to `/err/404`. Maintenance repoints a service's *serving* routers to the pages service behind a `replacePath` to `/m/<serviceId>`; probe routers stay on the real service, so verification keeps checking the app. The flag is `Route.Maintenance` and travels with the ordinary route table (`sync_routes`, reconcile).
- **Ordering.** Maintenance content is written before a push that needs it and removed after a push that no longer does.
- **Deploys inherit.** A deploy payload's route table is older than any maintenance toggle since dispatch, so the engine takes the flag from the proxy's live table.
## Amendment: who owns a service's upstreams
A route table from the control plane (`sync_routes`, a replacing `proxy_configure`) is built from its database when the command is queued. The database learns of a new deployment or replica count only when the agent's result arrives, so a table queued for any other reason (a domain, a maintenance toggle, another service) in that window is older than what the agent has done. Applied as sent it either names the *previous* containers of a service that a rolling deploy or scale has just switched (they go back into the upstreams and the retire step stops them right after: a sustained 502), or lacks the route of a service whose first deployment has just been routed (a 404 page, and the proxy leaves the service's network: 502 until the next push).
The agent therefore keeps the live upstreams, networks and health path (one unit with the replica set) of every managed route whose service has a command running, or was deployed or scaled here to a different replica set than the table names, and keeps such a service's live route when the table lacks it. Hosts, domains and maintenance in a route the table does carry are applied as sent. A deployed service's routes change through its own deploy, scale and remove commands, never by absence from a table; this is the rule Reconcile already applied to its own frames. External upstreams and services the agent knows nothing about stay the control plane's. See `keepLiveRoutes` in `internal/agent/engine/proxyconfigure.go`.
## Alternatives considered
- **Caddy 2** (the first version of this ADR): `POST /load` validates and applies a whole JSON document atomically and keeps the old config if it rejects the new one, with a built-in internal CA for `*.localhost`. It was replaced by Traefik on the owner's decision. The cost of the switch is the asynchronous apply described above and the default (self-signed) certificate for local names.
- **Traefik with the Docker provider (labels)**: needs the Docker socket inside the proxy container and gives no single whole-table apply or acknowledgement.
- **Traefik with the file provider**: needs a file write into the container and relies on filesystem watching with no apply acknowledgement; the REST provider gives a single authenticated write endpoint and the same marker technique works.
- **Traefik with the HTTP provider**: the agent would have to serve the configuration to Traefik and Traefik polls it, adding latency and a second moving part.
- **nginx or HAProxy with templated config files**: no built-in ACME, reload semantics are manual, and config validation means shelling out.
- **A proxy built into the agent**: violates the spec's "no custom proxies" rule (§0.3).
---
# 0004 Build on the target host
Source: https://nebulactrl.com/docs/decisions/0004-build-on-target-host
- Status: accepted. Amended by [0008](https://nebulactrl.com/docs/decisions/0008-build-servers): building on the target host and no registry is the **default**, not the only path. Builders beyond the Dockerfile: [0006](https://nebulactrl.com/docs/decisions/0006-railpack-default-builder).
- Date: 2026-10-25
## Context
Spec §8.1/§8.4 want pinned commits, immutable image digests, a build timeout and cleanup, scoped encrypted registry credentials, and no mandatory proprietary registry. Open questions §19.4 (build location) and §19.8 (image distribution) are unresolved. P0 supports a single explicit host per service (spec §8.5) and must also work for homelab hosts that have no registry.
## Decision
**By default, images are built where they run, with no registry.** A server can optionally be a build server, and a registry is then used to ship the image ([ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers)). Everything below describes the default path.
**Git builds.**
1. Before creating the release, the control plane resolves the branch to a commit SHA with go-git `remote.List` (ls-remote). The release stores `git URL + commit SHA`.
2. The agent clones with go-git into `<state-dir>/builds/<deploymentId>` and checks out that exact SHA. Private repositories use HTTPS BasicAuth with user `x-access-token` and the token from the frame's `secrets`.
3. The agent runs `docker build` through `exec.CommandContext` as an argument vector, never through a shell:
`docker build --progress=plain -f <dockerfile> -t nc-build/<service-slug>:<deploymentId8> --label nebulactrl.managed=true --label nebulactrl.service=<id> --iidfile <file> <contextDir>`. `DOCKER_HOST` is inherited.
4. Every output line streams as a `log` frame with stream `build`. The timeout is 30 minutes; the build directory is removed afterwards. The scheduled Docker cleanup (`docker builder prune -f --filter until=168h` among its steps) bounds the cache; see `docs/operations/docker-cleanup.md`.
5. `dockerfilePath` and `buildContext` must be relative paths without `..` and are validated both in the control plane and again in the agent.
**Image sources.** The agent pulls with optional decrypted registry auth (from `secrets`, never persisted). The digest is taken from `RepoDigests`.
**Digest.** For an image built **on the target** the **local image ID is the digest**; it is recorded in `releases.source.digest` on success. For pulled images it is `repo@sha256:…`. A rollback of an image release pulls `repo@digest` if the image is missing. An image built on a **build server** is identified by its registry **manifest digest** instead and is deployed as `repo@sha256:…` (recorded as `registryRef` in the release source); the local-image-ID rule applies to target-built images only.
**Retention and rollback.** After a successful deploy the agent keeps the newest `imageRetention` (default 5) `nc-build/<slug>` images and removes older ones. Rolling back to a git release whose image has been pruned returns 409 `image_pruned`; the UI offers "Rebuild commit <sha7>" which creates a normal git deployment for that SHA. Rollback restores application image and configuration only; it does not touch volumes or databases, and the UI says so. With a registry the retention is two-tier: the target still keeps `imageRetention` images locally, and the control plane owns the registry's **live set** (current release, releases in flight, the newest `imageRetention` succeeded releases of each service). A registry release rolls back by pulling `repo@digest` on the target without any build server, and `image_pruned` is then decided from the control plane's registry inventory (managed registries) or from a failed pull (BYO). Details in [ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers).
**Labels.** Every managed container carries `nebulactrl.managed=true`, `nebulactrl.service`, `nebulactrl.deployment`, `nebulactrl.release`, `nebulactrl.environment`, which is also what the orphan sweep and image retention match on.
## Consequences
- No registry to run, secure or pay for; works on air-gapped or NAT'd homelab hosts; build secrets stay on the host that will run the code.
- Builds consume CPU, memory and disk on the production host and compete with the running service. Bounds are the 30 minute timeout, one build per service at a time, and cache pruning; there are no per-build resource limits in P0.
- Untrusted repository content is built by the host Docker daemon (root-equivalent). This is documented in the threat model; a Dockerfile can run arbitrary commands at build time.
- A target-built image exists on one host only. Moving a service to another server means building or pulling again there; an image built on a build server and pushed to a registry can be pulled by any server.
- Build output is not reproducible bit-for-bit; the commit SHA plus image ID are what the release records.
- Single-platform builds for another architecture are in scope through build servers with emulation enabled ([ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers)); multi-arch manifests remain out. On the default path, BuildKit features that need a builder outside the host Docker daemon (multi-platform, remote cache) are still out.
## Alternatives considered
- **Registry (spec's stated preference)**: better for multi-host and build/run separation, but a mandatory registry conflicts with "no proprietary registry" and adds an operational burden. **Adopted as an optional distribution path in [0008](https://nebulactrl.com/docs/decisions/0008-build-servers)**, bring-your-own or managed.
- **Designated builder host**: needs an image transfer path and scheduling. **Adopted as optional in [0008](https://nebulactrl.com/docs/decisions/0008-build-servers)** as a build server that pushes to a registry; the `docker save | load` relay was not adopted.
- **Nixpacks/buildpacks**: out of scope; Dockerfile builds only.
- **Building through the Docker API (`ImageBuild`)**: workable, but the CLI gives BuildKit behavior, `--iidfile` and readable plain-progress logs with fewer moving parts.
---
# 0005 Licence
Source: https://nebulactrl.com/docs/decisions/0005-license
- Status: **accepted**
- Date: 2026-10-27
- Decided by: the project owner, 2026-10-27 (supersedes the proposal of 2026-10-25)
- Takes effect: **when the source is published.** The repository is private for now (owner decision, 2026-10-28). Until then NebulaCtrl is distributed free of charge as the installer, the public container images and the CLI served by the control plane, and is described as "free, self-hosted, source coming soon", not as open source. The licence decision, the `LICENSE` file and the rules below stay as written and apply from the day the source goes public. Publishing also means flipping the website and README wording and setting `GITHUB_URL`; the checklist is in `CONTRIBUTING.md`, "Release".
## Context
Spec §19.1 asks for a license choice before public release. Until this decision the repository had no `LICENSE` file, so the code was "all rights reserved" by default.
Relevant facts:
- NebulaCtrl is self-hosted software: users run the control plane and the agents themselves. The owner expects third parties may also offer it as a hosted service.
- `nebulactrl/nebula` (design-system source) is Apache-2.0; the ported web files keep compatible terms under the AGPL (Apache-2.0 is one-way compatible with GPLv3 and AGPLv3).
- The dependency tree was scanned on 2026-10-27: see [Third-party licences](https://nebulactrl.com/docs/security/third-party-licences).
## Options
| | Apache-2.0 | AGPL-3.0 |
|---|---|---|
| Use, modify, self-host | Unrestricted | Unrestricted |
| Closed-source forks / commercial hosting by others | Allowed | Hosting a modified version as a network service requires publishing the changes |
| Patent grant | Explicit | Included (GPLv3 terms) |
| Adoption friction in companies | Low | Many companies forbid AGPL |
| Contributor/CLA pressure | Low | Needs a CLA if you later want dual licensing |
| Protection against a competitor offering a hosted NebulaCtrl | None | Strong |
| Compatibility with embedding in other products | Easy | Hard |
## Decision
NebulaCtrl is licensed under the **GNU Affero General Public License, version 3 only**, SPDX identifier `AGPL-3.0-only`.
- The canonical, unmodified text from <https://www.gnu.org/licenses/agpl-3.0.txt> is in `LICENSE` at the repository root.
- "Only", not "or later": the project does not hand future relicensing decisions to the Free Software Foundation. Grafana and Mattermost make the same choice.
- `apps/web/package.json` carries `"license": "AGPL-3.0-only"` and stays `"private": true` because it is not published to npm. Every other manifest in the repository carries the same field.
- Dependencies must be compatible with the AGPL (permissive licences, LGPL, MPL-2.0 and similar). Proprietary, SSPL, BUSL, non-commercial and GPL-2.0-only dependencies are not allowed. The scan result and the rule are in [Third-party licences](https://nebulactrl.com/docs/security/third-party-licences).
- Contributions are accepted under the same licence (inbound = outbound). There is no CLA today. Adding one, or any dual licensing, is a later owner decision and would need that consent.
## Reasoning
- The owner's goal is to keep hosted derivatives open. The AGPL's network clause (§13) requires anyone who runs a modified NebulaCtrl as a network service to offer their users the corresponding source. Apache-2.0 gives no such protection.
- Self-hosting, modification and internal use stay unrestricted, so the adoption cost falls on the few organisations that ban the AGPL, not on typical self-hosters. The comparable products the owner follows (Grafana, Mattermost) made the same trade.
- The platform services NebulaCtrl deploys for users include AGPL software (Loki, Grafana). Using the AGPL for the platform itself is consistent with that and avoids a licence-mix story in the documentation.
- The platform is not a library. Embedding in other products, the main AGPL drawback, is not a use case.
## Consequences
- Public release and external contributions are no longer blocked by the licence question.
- Container images that NebulaCtrl **deploys** (the template catalog, database engines, platform services such as Loki, Grafana, SeaweedFS, Headscale and Pocket ID) are **separate works** under their own licences. NebulaCtrl runs them unmodified in separate containers and does not link them; the AGPL of NebulaCtrl does not extend to them and theirs does not extend to NebulaCtrl.
- Users who modify NebulaCtrl and offer it to others over a network must publish their modified source.
- New dependencies must pass the licence rule above before they are approved.
## Alternatives considered
Apache-2.0 (the earlier recommendation for maximum adoption; rejected because it does not keep hosted derivatives open), MIT/BSD (no patent grant), MPL-2.0 (file-level copyleft; weaker fit for a single-binary product), AGPL-3.0-or-later (cedes future relicensing to the FSF), a source-available license such as BSL (contradicts spec §1.11 "open-source-ready").
---
# 0006 Railpack builder
Source: https://nebulactrl.com/docs/decisions/0006-railpack-default-builder
- Status: accepted
- Date: 2026-10-26
- Amends: [0004](https://nebulactrl.com/docs/decisions/0004-build-on-target-host) (builds run on the target host **or on a build server** ([0008](https://nebulactrl.com/docs/decisions/0008-build-servers)); this adds how)
## Context
ADR 0004 builds a git source with the repository's Dockerfile only. Coolify and Dokploy both build repositories without one (Nixpacks, Railpack, buildpacks). The owner decided (phase 6 decisions, "Builders") that a repository without a Dockerfile builds with Railpack by default, that generated starter Dockerfiles remain as an alternative, that a Dockerfile always wins, and never Nixpacks (in maintenance mode).
Railpack is MIT, written in Go, and the default builder of Railway. At decision time it is 0.x and about 20 months old, so it is an explicit exception to the library rule (research: `parity-proxy-research.md` §3). The exception is contained by pinning and by being replaceable: the build method is one value in the service config.
## Decision
**Build types.** `services.config.buildType` is one of:
| Value | Behaviour |
|---|---|
| `auto` (default) | The repository's Dockerfile if it exists at `dockerfilePath`, otherwise Railpack. A `dockerfilePath` the user changed from the default and that is missing is an error, never a silent switch to Railpack. |
| `dockerfile` | The Dockerfile; its absence is an error. |
| `railpack` | Railpack, even when a Dockerfile exists. |
| `starter:<lang>` | A generated Dockerfile for `node`, `python`, `go`, `ruby`, `php` or `static`. It is written next to the clone (never into it), printed to the build log, and built like any Dockerfile. |
The agent decides after the clone, because only it can see the files. The control plane only carries the setting (`GitSource.BuildType`).
**Pinned Railpack: 0.43.0** (released 2026-10-19). Three constants in `pkg/agentprotocol/build.go` move together:
- `RailpackVersion = "0.43.0"`
- `RailpackFrontendImage = ghcr.io/railwayapp/railpack-frontend:v0.43.0@sha256:355ef61f…c676cc`, the digest of the multi-arch index (verified with `docker buildx imagetools inspect`)
- `BuildKitImage = moby/buildkit:v0.34.1@sha256:99368f3e…ec1da5c` (only for the buildx fallback below)
**How it runs: the BuildKit frontend, not the `railpack` CLI.** Railpack documents two ways to build: `railpack build` (the CLI talks to a BuildKit daemon, which the docs start as `docker run --privileged moby/buildkit`) and the BuildKit frontend through `docker build`. The frontend is what Railpack recommends for production and needs no extra daemon:
1. `railpack prepare` plans the build. It runs from the **same frontend image** (so one Railpack version plans and builds) in a throwaway container created with `docker create`. The source goes in and `railpack-plan.json` / `railpack-info.json` come out with `docker cp`, so it works when the agent itself runs in a container (no host bind mounts), and the container has no privileges, no mounts and no docker socket.
2. `docker build --build-arg BUILDKIT_SYNTAX=<frontend image> -f railpack-plan.json -t nc-build/<slug>:<dep8> --label … --iidfile …`. The result is a local image with the usual labels and image ID as digest, so deployment, retention and rollback are unchanged.
**The mergeop caveat (found by testing).** Railpack's build graph uses BuildKit `mergeop`, which the Docker daemon only enables with the **containerd image store**. Docker 29 enables it on fresh installs; upgraded installs and OrbStack keep the classic store and fail with `requested experimental feature mergeop has been disabled`. The agent therefore reads `docker info`:
- containerd image store → plain `docker build` (the daemon's own BuildKit and cache);
- otherwise → a managed `docker buildx` builder `nebulactrl-buildkit` (driver `docker-container`, image pinned above), created on first use, and `docker buildx build --load`.
If the daemon claims the containerd store and still refuses `mergeop`, the agent retries once through the managed builder. The managed builder's container is privileged (that is how the buildx `docker-container` driver works). It only exists on hosts that need it, and the agent already holds the Docker socket (root-equivalent), so it adds no capability a Dockerfile build lacked. A host with neither the containerd store nor the buildx plugin fails Railpack builds with a message that says so; Dockerfile and starter builds are unaffected.
**On a build server** ([ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers)) the same managed builder is reused, and it is always used: a build there is `docker buildx build --push` into a registry, never `--load`, so the classic-store question above does not arise. The builder is one container **per project**, `nebulactrl-buildkit-<project8>`, each with its own cache, instead of the single `nebulactrl-buildkit` of a target host. The Dockerfile, starter and Railpack paths all go through it, and the planner, variables and secrets behave exactly as described here. Railpack builds one platform per build, so a cross-architecture Railpack build runs every install and build step under emulation.
**Variables and secrets.** Railpack treats every variable as a BuildKit secret. Plain and secret build variables from the variables system are therefore both passed as `--secret id=KEY,env=NCBS_KEY` (values in the docker CLI's process environment, never in argv) plus a `secrets-hash` build arg so layers rebuild when a value changes. For planning, the values reach `railpack prepare` through a mode-0600 shell script copied into the planner container (bare `--env KEY` flags name them), which supports multi-line values and keeps them out of the container's argument vector. `RAILPACK_*` variables (for example `RAILPACK_START_CMD`) are ordinary build variables.
**Build-plan preview.** `POST /services/{id}/build-plan` queues a `build_plan` command; the agent clones the branch head, decides the method and, for Railpack, runs the planner and returns providers, resolved packages, steps and the start command. Nothing is built. The request waits for the agent (409 `server_offline` instead of queueing for an offline server, 504 after 150 s).
**Recorded per deployment.** The agent returns a `BuildInfo` with the deploy result; the control plane stores it in `deployments.build_info` and exposes it as `build` on the deployment.
**Start command.** Railpack images have the entrypoint `["/bin/bash","-c"]` and the start command as `CMD`. A service `startCommand` with several words is joined into one quoted script for such images, so `["node","server.js"]` works as expected.
## Consequences
- A repository without a Dockerfile deploys with no configuration; the plan preview says why Railpack chose what it chose.
- A Railpack build downloads its toolchain (mise) and pulls `ghcr.io/railwayapp/railpack-builder:*` images (hundreds of MB) on the build host on first use; those builder images are tagged by Railpack, not pinned by us. Hosts need outbound access to `ghcr.io`.
- Railpack's output depends on its version; bumping the pin is a deliberate change (see `docs/operations/builds.md`).
- Railpack is 0.x. If it stops being maintained, set affected services to `dockerfile` or a starter and remove three constants and `internal/agent/build/railpack.go`.
- Untrusted repository code still builds on the host Docker daemon; Railpack runs the project's install and build commands (`npm install`, `pip install`, …) just as a Dockerfile would.
## Alternatives considered
- **`railpack build` CLI.** Needs the CLI binary on every host and a privileged BuildKit container always; the frontend path needs neither except in the mergeop fallback.
- **Cloud Native Buildpacks (pack + Paketo).** Compliant with the library rule today, but `pack` exposes the Docker socket to the build container for trusted builders; ruled out by the owner in favour of Railpack.
- **Nixpacks.** In maintenance mode; never.
---
# 0007 Compose stacks
Source: https://nebulactrl.com/docs/decisions/0007-compose-stacks
- Status: accepted
- Date: 2026-10-27
## Context
Until Phase 7 a service was one container, and a Docker Compose file could only be imported once into ordinary services ([Compose import](https://nebulactrl.com/docs/operations/compose-import)). That import drops what Compose cannot express here (`entrypoint` in the first version, build args, directory bind mounts, `user`, Compose secrets and configs, devices) and is never kept in sync. Coolify and Dokploy run Compose files natively; for many self-hosters "my `docker-compose.yml` runs as it is" is the first question. The parity review (rev 3, §6) listed it as the largest remaining gap and said a runtime contradicts "a service is one container" and the staged-change model, so it needed a decision.
## Decision
**A stack is one service whose runtime is `docker compose`.** The owner chose native Compose stacks, with the safe allow-list on by default and an explicit per-server escape hatch.
- **Model.** `kind = stack`, `sourceType = compose`. The compose file is part of the service configuration (`config.stack.compose`), so everything that already works on a service keeps working on the whole stack as one unit: staged changes and the generation counter, one deployment, one release, rollback, audit, variables, domains. The compose services are *components*; they have no rows of their own and appear in the observed state, the metrics, the logs and the canvas card.
- **Runtime.** The agent writes the compose file and a generated **override** to `<state-dir>/stacks/<serviceId8>/` and runs `docker compose -p nc-<serviceId8> -f compose.yaml -f override.yaml up -d --remove-orphans --wait`. The override (pure code in `pkg/composestack`, shared by the control plane and the agent) adds the labels every managed container carries, fixed container names (`nc-<serviceId8>-<component>`), joins each component to its environment network with the component name as DNS alias (the same isolation model as ordinary services, including egress blocking and shared networks), removes host port publishing, applies per-component resource limits and, for rollbacks, pins images to the digests the release recorded.
- **Routing.** Compose `ports:` are never published on the host. HTTP traffic reaches a component through a domain, which names the component and a container port; Traefik routes to `nc-<serviceId8>-<component>:<port>` on the environment network, so the zero-exposure model of ordinary services carries over. Published TCP/UDP ports are not supported.
- **Strategy.** Always `recreate`: `docker compose up` replaces changed containers in place. Stacks do not get the rolling strategy; the plan says why. After the containers are healthy the agent applies the routes and verifies through the proxy like for ordinary services; a failed `up`, health check or verification re-applies the previous compose file and override (`rolled_back`), a failed restore is `rollback_failed`.
- **Safety.** Docker is root on the host, and a compose file can ask for almost anything. By default the agent normalises the file with `docker compose config --format json` and refuses it unless it passes the allow-list: no `privileged`, no host/container network, PID, IPC, UTS or user namespaces, no devices or GPUs, no capabilities beyond the safe list (`NET_ADMIN` and `SYS_ADMIN` need an unrestricted server), no security options other than `no-new-privileges`, no host bind mounts (which includes the Docker socket), no external volumes or networks or custom volume/network names (they could reach another project's data or break environment isolation), no driver options that mount host paths, no `file:` secrets and configs, no reserved labels. The control plane runs the same check on the file as written, so mistakes surface when you save. An **instance admin** can switch on **Unrestricted Compose** for a single server (type-the-name confirmation, audit entry); the allow-list is then not applied to the stacks of that server. A few things stay unsupported either way because the stack model cannot carry them (`build:`, `include`, `env_file`, `extends` with a file, `provider:` services, replicas above one).
- **Secrets.** A stack's variables are used for compose interpolation (`${VAR}`). They are sent as send-time secrets, never written to disk, and handed to compose in a clean environment (the agent's own environment, including its enrollment token, never reaches compose; `DOCKER_*` and `COMPOSE_*` keys from user variables are dropped except `COMPOSE_PROFILES`).
- **Requirement.** The Docker Compose plugin 2.24 or newer (the override uses `!reset` and `!override`). The agent reports its version in the facts and advertises the `compose` capability; the control plane refuses stack deploys on a server without it, and the installer installs the plugin.
## Consequences
- Compose files run with their own semantics (healthchecks, `depends_on`, `restart`, named volumes, `command`, `entrypoint`, `user`, anything the allow-list lets through) instead of an approximation.
- A stack is coarser than services: one deploy button, one rollback, no per-component staging, no rolling update, no replicas, no per-component volume backups or one-off tasks, no maintenance mode. Users who need those use "Split into services".
- Images are pulled by tag at deploy time; a rollback runs the digests recorded for the release. Data is never rolled back, as everywhere else.
- Named volumes live under the compose project name (`nc-<serviceId8>_<name>`); deleting a stack keeps them unless the user ticks "delete volumes".
- The trust boundary documented in the threat model does not move for restricted servers (Docker is still root, but the allow-list keeps a compose file from asking for the host); on an Unrestricted server any deployer can take over the host, and the UI says so.
- Compose evolves; the agent checks the normalised model rather than the YAML text so new syntaxes cannot slip past, but a new Compose key that grants host access would be allowed until the allow-list learns it. The check is a deny-by-key list for risky keys plus structural rules for volumes, networks and namespaces, not a sandbox.
## Alternatives considered
- **Keep import only.** Cheap, but not what the owner asked for and it leaves the biggest parity gap open.
- **One service row per compose service with a stack parent.** Gives per-component nodes, domains and variables, but every deployment-level feature (generation, rollback, staged changes, cancel) would have to be coordinated across rows, and a stack could end up half deployed. Components as observed state of one service keeps the unit of change atomic.
- **Translate the compose file into our own container creation** (what import does). That is the import path; it cannot keep Compose behaviour such as `depends_on` health conditions and init containers exact.
- **Free-form compose without an allow-list.** Rejected by the owner: the default must be safe.
- **Swarm or `docker stack`.** Out of scope (no cluster mode, ADR 0001).
## Update (Phase 9)
"Unrestricted Compose" became **Unrestricted**: one per-server switch (`servers.unrestricted`, `PUT /servers/{id}/unrestricted`, still instance admin only with the server's name typed and audited as `server.unrestricted.enable`/`.disable`) that stacks and plain-service [Docker options](https://nebulactrl.com/docs/operations/docker-options) share, because the two give identical power. The instance-wide advanced-capabilities switch was removed with it. Clean rename, no alias.
---
# 0008 Build servers and registries
Source: https://nebulactrl.com/docs/decisions/0008-build-servers
- Status: accepted
- Date: 2026-10-27
- Amends: [0004](https://nebulactrl.com/docs/decisions/0004-build-on-target-host) (building on the target stays the default; this adds an optional second path), [0006](https://nebulactrl.com/docs/decisions/0006-railpack-default-builder) (the managed builder is reused on build servers)
## Context
[ADR 0004](https://nebulactrl.com/docs/decisions/0004-build-on-target-host) builds every git service on the server that runs it and has no registry. That is the right zero-configuration path, and it stays. It has three costs that owners of larger fleets asked to remove:
- a build competes for CPU, memory and disk with the running service on the production host;
- untrusted Dockerfiles (and fork pull-request previews) execute next to production data and runtime secrets;
- a build exists on one host only, so a rollback outside the host's `imageRetention` window, a move to another server and a reconcile redeploy all need a rebuild.
Coolify and Dokploy solve this with a build server that pushes to a registry and deployment servers that pull. Both SSH into every host, so "build here, run there" needs only credentials on both ends. NebulaCtrl agents only **dial out** ([ADR 0002](https://nebulactrl.com/docs/decisions/0002-agent-transport)): there is no agent-to-agent path, and the control plane is the only rendezvous. Their registry handling is also their main bug source (an ambient `~/.docker/config.json` that differs between users, credentials that "log in" but cannot push, a mutable tag that the target does not re-pull), and neither product retains or collects registry storage.
The owner decided (phase 8, 2026-10-27):
- support **both** a bring-your-own (BYO) registry and a managed "Set up for me" registry now;
- the managed registry is CNCF Distribution `registry:3.1.2`, pinned by index digest, as an explicit exception to the "stable for two years" library rule (the project is 12 years old, the v3 line 19 months);
- **always fall back to the target** when no build server or no usable registry exists, with a visible event, never a blocked deploy;
- **one BuildKit builder per project** on a build server, and previews never build on a shared build server unless the owner opts in.
## Decision
**A build server builds and pushes to an OCI registry; the target pulls by digest.**
1. **Where a build runs is a placement decision**, resolved **service → project → team**, with `target` as the default at every level. A registry is needed only when something resolves to `build_server`. Compose stacks and image services never build, so they are unaffected.
2. **Server roles are two orthogonal flags.** `accept_workloads` (existing) decides placement of workloads; the new `build_enabled` decides build eligibility. *Deployments* (today's default) is `accept_workloads` on and `build_enabled` off, *Builds* is the reverse, *Both* sets both. Build eligibility never reads `accept_workloads`. Each build server has `build_max_concurrent` (default 2), `build_allow_emulation` and `build_allow_previews` (both off).
3. **Two durable commands per deployment.** The control plane writes the release first, sends a `build_image` command to the build server (clone the pinned commit, `docker buildx build --push`, return the manifest digest), records the pushed image, and then sends the ordinary `deploy` command to the target with `Image{Ref: repo@sha256:…}`. The target engine already pulls images by digest with send-time registry credentials, so everything after the pull (architecture guard, create, health, route, verify, retire) is the existing engine. A new step `push` maps to the `building` status; no new deployment status exists.
4. **Registry.** `registries` are team objects of two kinds: `external` (any OCI registry with a basic-auth push credential and an optional read-only pull credential: GHCR, Docker Hub, GitLab, Harbor, self-run) and `managed` (a `registry` platform service, [platform services](https://nebulactrl.com/docs/operations/platform-services)). ECR, Google Artifact Registry and Azure ACR need token refresh and are not supported in this version.
5. **Image identity.** The repository is `<host>/<namespace>/<project-slug>-<service-slug>` (an empty namespace is allowed), the tag is `r-<release8>`, and the release records the registry **manifest digest** and `registryRef = repo@sha256:…`. Targets deploy the digest, never the tag. Each pushed image is recorded in `registry_images` with its own repository, so renaming a service never breaks a rollback. Provenance and SBOM attestations are off (`--provenance=false --sbom=false`), so the pushed result is a single manifest rather than an index.
6. **One platform per build.** v1 builds one single-platform image for the **target's** architecture. A build server of another architecture is eligible only with `build_allow_emulation` on and an agent that reports the platform in `facts.emulation`. Multi-arch manifests are not built.
7. **Sticky selection with a concurrency cap.** An explicit pin wins; otherwise rendezvous hashing over eligible servers with the service id as the key keeps the BuildKit cache warm. Each build server has its own cap; further builds queue on the control plane and the deployment stays `preparing` with "Waiting for a build slot on <server>".
8. **Isolation.** The agent creates one BuildKit builder per project, `nebulactrl-buildkit-<project8>`, with its own cache and garbage-collection limits. Preview builds go to the target unless a build server sets `build_allow_previews`. Runtime environment variables **never reach a build server**; only the git credential, build variables and the registry push credential do.
9. **Credentials are send-time only and never ambient.** The push credential reaches the build agent in the command frame's `secrets` and is written to a mode-0700 temporary `DOCKER_CONFIG` that exists for one build and is always removed; pulls pass the credential per request. `agent_commands.payload` carries only booleans. For the managed registry the control plane is the token server (`/registry/token`, Docker token authentication): each command receives an expiring credential scoped to one repository and the needed actions (15 minutes by default, longer for a build, at most 6 hours), which the registry exchanges for a JWT valid at most 15 minutes. No long-lived registry credential rests on any agent host.
10. **Rollback needs no build server.** Rolling back a registry release is an ordinary deploy of `repo@digest` on the **current** target. `image_pruned` is decided from `registry_images.state` before dispatch; for a BYO registry, where NebulaCtrl deletes nothing, an image removed out of band shows up as a failed pull. "Rebuild commit" uses the current effective build location.
11. **Retention.** Targets keep `imageRetention` images locally, as before (fast rollback, works while the registry is down). The control plane owns the **live set** (current release, releases of deployments in flight, the newest `imageRetention` succeeded releases of each service). For a registry with **managed retention** (the managed registry) it marks other `registry_images` rows `expired`, deletes those explicit digests through the registry API and runs `registry garbage-collect` (without `--delete-untagged`) inside a read-only window. For BYO registries NebulaCtrl records images but never marks them expired and never deletes.
12. **Failure handling** is by situation, in [build servers: troubleshooting](https://nebulactrl.com/docs/operations/build-servers#troubleshooting): a build server offline at dispatch falls back to the target; one that goes offline after acknowledging the command never times out into a terminal state (the rule for every acknowledged command, [servers](https://nebulactrl.com/docs/operations/servers#what-happens-when-a-server-goes-offline)); a registry that cannot be reached fails fast in a pre-build check; a pull failure on the target leaves the old release serving.
Every frame that belongs to a deployment (events, results, logs, source scans, cancel) is now accepted from **any server that owns a command of that deployment**, looked up through `agent_commands`, instead of only from `deployments.server_id`. Cancel targets whichever command is active.
## Consequences
- A build server never runs production workloads (when set to *Builds*), so a build can no longer starve the service it is building, and an untrusted Dockerfile no longer executes on the host that holds runtime secrets. The cost is a second privileged host (the builder is privileged BuildKit) that holds the git credential, build variables and a push credential for **all projects** of the team; it is inside the documented trusted-team boundary ([threat model](https://nebulactrl.com/docs/security/threat-model)). Per-project builders isolate caches, not the kernel.
- A rollback of a registry release works with the build server offline or deleted and no longer depends on the target's local image cache. It depends on the registry being up and on the digest still existing there.
- The registry is a new availability dependency for build-server deployments only: a registry outage blocks the push and the pull, but a fallback keeps deploys working when none is configured.
- Two WAN hops per deploy replace zero (build server → registry, registry → target), mitigated by layer deduplication in the OCI protocol.
- The managed registry holds every image of the team, so its host is a high-value target; it needs a public hostname, port 443 and an ACME email (Docker daemons require TLS for non-loopback registries). Traefik's default 60 s entrypoint `readTimeout` can cut a large layer upload over a slow uplink, so the registry's route is given a larger timeout.
- BYO registries cannot be garbage-collected by NebulaCtrl. A lifecycle rule that deletes untagged images, or only keeps a few tags, can destroy rollback targets; the docs say what to keep.
- Old releases keep working: a release with a local `imageId` keeps the target-built semantics; a release with a `registryRef` takes precedence. Both representations coexist.
- Agents that predate the `build.push` capability are never chosen as build servers; the deployment falls back to the target.
- Multi-arch manifests, ECR/GAR/ACR token refresh, registry cache export and a registry-free transport are explicitly not included.
## Alternatives considered
- **Relay transport: `docker save | docker load` through the control-plane WebSocket.** The only registry-free option that works with outbound-only agents, and its one real merit. Rejected for the first version: the control plane sits in the image data path (it would see every layer in clear); there is **no layer deduplication** (`docker save` ships every parent layer, so a 300 MB image with 30 MB of changed layers moves ~600 MB through the control plane instead of ~60 MB with a registry); both agents must be online at once and a drop restarts the whole transfer; the bulk bytes would share the one socket that carries heartbeats, logs and commands; and rollback would still need each target's local image. It stays a possible later fallback for installs that cannot run a registry host.
- **Remote BuildKit driven by the target (`docker buildx create --driver remote tcp://…`).** Needs an inbound listener on the build server; agents never accept inbound connections. It still would need the source to flow to the builder, so it has no advantage over the builder the build agent drives locally (which is what this ADR does).
- **zot instead of `registry:3`.** Inline garbage collection and a retention policy remove the read-only window, and it has dedupe and scrub. Its default garbage collection deletes **untagged manifests** after a delay, which is a footgun for digest-pinned rollback targets unless every retained release keeps a tag, and it is a larger surface. `registry:3` is the reference implementation, smaller, and its token authentication fits a control plane that every agent can already reach. Both speak the OCI API, so swapping is a catalog change; the unique `r-<release8>` tag already makes zot safe.
- **Native multi-arch.** One buildx invocation cannot span two hosts without an inbound path, so native multi-arch would be two builds on two build servers plus `docker buildx imagetools create`. Valuable only for fleets that move one release between architectures; recorded as the extension path.
- **A registry credential stored in `~/.docker/config.json` on each server** (Coolify, Dokploy). Rejected: a global `docker login` leaves long-lived credentials on a shared builder and fails when the build and the pull run under different Docker configs.
- **Always random build-server choice.** Fragments the cache (1/N hit chance) and was a Coolify complaint; replaced by sticky selection.
- **Hosted builders (Docker Build Cloud and similar).** Contradicts the self-hosted, no-SaaS goal.
---
# Third-party licences
Source: https://nebulactrl.com/docs/security/third-party-licences
NebulaCtrl is licensed under **AGPL-3.0-only** ([ADR 0005](https://nebulactrl.com/docs/decisions/0005-license)). This page records the dependency licence scan run on 2026-10-27 and the rule new dependencies must meet.
**Result: no incompatible licence was found** in the Go modules that are compiled into the three binaries, nor in the web packages of `apps/web`. Nothing needs an owner decision.
## Rule
A dependency is acceptable when its licence can be combined with the AGPL-3.0:
| Compatible | Not acceptable |
|---|---|
| MIT, MIT-0, ISC, BSD-2-Clause, BSD-3-Clause, 0BSD, Unlicense, CC0-1.0, Apache-2.0 (one-way compatible with GPLv3/AGPLv3), MPL-2.0 (its §3.3 names the AGPL as a Secondary License), LGPL-3.0, GPL-3.0 and AGPL-3.0, OFL-1.1 for fonts, CC-BY-4.0 for data | Proprietary or "source-available" licences (SSPL, BUSL, Elastic, FSL, Bitwarden SDK, non-commercial or no-derivatives terms), GPL-2.0-only, and anything with no licence |
Run the scan again before approving a new dependency and before each release.
## Go (compiled into `nebulactrl-server`, `nebulactrl-agent`, `nebulactrl`)
Command (the tool must be built once, because `go run` cannot cross-compile it; the agent and server ship for Linux):
```sh
GOBIN=/tmp/gl go install github.com/google/go-licenses/v2@latest
GOOS=linux GOARCH=amd64 /tmp/gl/go-licenses csv ./cmd/... ./internal/... ./pkg/...
```
go-licenses v2.0.1 reported 134 licence entries across 112 third-party modules. Summary by licence, counted per module:
| Licence | Modules |
|---|---|
| MIT | 37 |
| Apache-2.0 | 36 |
| BSD-3-Clause | 17 |
| Apache-2.0 plus BSD-3-Clause files | 8 |
| MPL-2.0 | 3 |
| ISC | 2 |
| BSD-2-Clause | 2 |
| MIT plus MPL-2.0 files | 2 |
| BSD-3-Clause plus MPL-2.0 files | 1 |
| BSD-2-Clause plus ISC files | 1 |
| BSD-2-Clause, BSD-3-Clause and ISC files | 1 |
| Apache-2.0 plus MIT files | 1 |
| MIT-0 | 1 |
No GPL, LGPL, AGPL, SSPL, BUSL, proprietary or non-commercial module is in the build.
Notes:
- **MPL-2.0** applies to `github.com/riverqueue/river` (the job queue and its pgx driver), `github.com/cyphar/filepath-securejoin`. MPL-2.0 is file-level copyleft: NebulaCtrl links them unmodified, so only changes to those files would have to be published, and the AGPL is a permitted Secondary License.
- **`github.com/segmentio/asm`** is reported "no license found" by go-licenses because the licence sits in a file the classifier does not recognise. The file is MIT No Attribution (MIT-0), read directly from the module.
- **Apache-2.0** modules that ship a `NOTICE` file are carried in the top-level [`NOTICE`](): `aws-sdk-go-v2`, `smithy-go`, `go-oidc`, `skeema/knownhosts` and `go.yaml.in/yaml/v3`. Distribution of the binaries must keep that file.
- The scan was also run over `./tools/...` and `./tests/...` (including the `integration` build tag): the only additional module is `github.com/stretchr/testify` (MIT, with BSD-3-Clause and ISC internals). These are not shipped.
- **`github.com/tailscale/hujson`** (BSD-3-Clause, compatible) is added for the optional managed Tailscale ACL push of the [server mesh](https://nebulactrl.com/docs/operations/mesh#the-acl-mode), which edits the user's policy without destroying its comments. It was added after the scan above (2026-10-27) and is not part of the counts; its licence is the upstream project's published BSD-3-Clause and was not re-scanned with go-licenses.
- `templates/` is embedded in the server binary. It carries its own attribution file (`templates/NOTICE`, Apache-2.0 text for Coolify-derived templates and MIT for Dokploy-derived ones); go-licenses therefore reports that package as Apache-2.0. The licences of the **applications** the templates deploy are recorded per template in its `upstream` block.
### Modules in the build
| Module | Version | Licence |
|---|---|---|
| `dario.cat/mergo` | v1.0.0 | BSD-3-Clause |
| `github.com/ProtonMail/go-crypto` | v1.1.6 | BSD-3-Clause |
| `github.com/alexedwards/argon2id` | v1.0.0 | MIT |
| `github.com/alexedwards/scs/pgxstore` | v0.0.0-20251002162104-209de6e426de | MIT |
| `github.com/alexedwards/scs/v2` | v2.9.0 | MIT |
| `github.com/apapsch/go-jsonmerge/v2` | v2.0.0 | MIT |
| `github.com/aws/aws-sdk-go-v2` | v1.47.4 | Apache-2.0 + BSD-3-Clause |
| `github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream` | v1.7.24 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/credentials` | v1.20.10 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager` | v0.4.19 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/internal/configsources` | v1.5.7 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/internal/endpoints/v2` | v2.8.7 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/internal/v4a` | v1.5.7 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding` | v1.13.22 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/service/internal/checksum` | v1.11.8 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/service/internal/presigned-url` | v1.14.7 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/service/internal/s3shared` | v1.20.7 | Apache-2.0 |
| `github.com/aws/aws-sdk-go-v2/service/s3` | v1.114.5 | Apache-2.0 |
| `github.com/aws/smithy-go` | v1.28.6 | Apache-2.0 + BSD-3-Clause |
| `github.com/axiomhq/hyperloglog` | v0.3.0 | MIT |
| `github.com/boyter/go-string` | v1.0.5 | MIT |
| `github.com/cespare/xxhash/v2` | v2.3.0 | MIT |
| `github.com/cloudflare/circl` | v1.6.3 | BSD-3-Clause |
| `github.com/coder/websocket` | v1.8.15 | ISC |
| `github.com/containerd/errdefs` | v1.0.0 | Apache-2.0 |
| `github.com/containerd/errdefs/pkg` | v0.3.0 | Apache-2.0 |
| `github.com/coreos/go-oidc/v3` | v3.21.0 | Apache-2.0 |
| `github.com/cyphar/filepath-securejoin` | v0.6.1 | BSD-3-Clause + MPL-2.0 |
| `github.com/danielgtaylor/huma/v2` | v2.39.1 | MIT |
| `github.com/dgryski/go-metro` | v0.0.0-20250106013310-edb8663e5e33 | MIT |
| `github.com/distribution/reference` | v0.6.0 | Apache-2.0 |
| `github.com/docker/go-connections` | v0.8.2 | Apache-2.0 |
| `github.com/docker/go-units` | v0.5.0 | Apache-2.0 |
| `github.com/emirpasic/gods` | v1.18.1 | BSD-2-Clause + ISC |
| `github.com/felixge/httpsnoop` | v1.1.0 | MIT |
| `github.com/fxamacker/cbor/v2` | v2.9.4 | MIT |
| `github.com/go-chi/chi/v5` | v5.5.0 | MIT |
| `github.com/go-git/gcfg` | v1.5.1-0.20230307220236-3a3c6141e376 | BSD-3-Clause |
| `github.com/go-git/go-billy/v5` | v5.9.2 | Apache-2.0 |
| `github.com/go-git/go-git/v5` | v5.19.3 | Apache-2.0 |
| `github.com/go-jose/go-jose/v4` | v4.1.4 | Apache-2.0 + BSD-3-Clause |
| `github.com/go-logr/logr` | v1.4.4 | Apache-2.0 |
| `github.com/go-logr/stdr` | v1.2.2 | Apache-2.0 |
| `github.com/go-viper/mapstructure/v2` | v2.5.0 | MIT |
| `github.com/go-webauthn/webauthn` | v0.18.2 | BSD-3-Clause |
| `github.com/go-webauthn/x` | v0.3.1 | BSD-2-Clause + BSD-3-Clause + ISC |
| `github.com/golang-jwt/jwt/v5` | v5.3.1 | MIT |
| `github.com/golang/groupcache` | v0.0.0-20241129210726-2c02b8208cf8 | Apache-2.0 |
| `github.com/google/go-tpm` | v0.9.8 | Apache-2.0 |
| `github.com/google/jsonschema-go` | v0.4.3 | MIT |
| `github.com/google/uuid` | v1.6.0 | BSD-3-Clause |
| `github.com/jackc/pgpassfile` | v1.0.0 | MIT |
| `github.com/jackc/pgservicefile` | v0.0.0-20240606120523-5a60cdf6a761 | MIT |
| `github.com/jackc/pgx/v5` | v5.11.0 | MIT |
| `github.com/jackc/puddle/v2` | v2.2.3 | MIT |
| `github.com/jbenet/go-context` | v0.0.0-20150711004518-d14ea06fba99 | MIT |
| `github.com/kamstrup/intmap` | v0.5.2 | BSD-2-Clause |
| `github.com/kevinburke/ssh_config` | v1.2.0 | MIT |
| `github.com/medama-io/go-useragent` | v1.2.5 | MIT |
| `github.com/mfridman/interpolate` | v0.0.2 | MIT |
| `github.com/moby/docker-image-spec` | v1.3.1 | Apache-2.0 |
| `github.com/moby/moby/api` | v1.56.1 | Apache-2.0 |
| `github.com/moby/moby/client` | v0.6.2 | Apache-2.0 |
| `github.com/modelcontextprotocol/go-sdk` | v1.8.0 | Apache-2.0 + MIT |
| `github.com/monperrus/crawler-user-agents` | v1.65.0 | MIT |
| `github.com/oapi-codegen/runtime` | v1.8.0 | Apache-2.0 |
| `github.com/opencontainers/go-digest` | v1.0.0 | Apache-2.0 |
| `github.com/opencontainers/image-spec` | v1.1.1 | Apache-2.0 |
| `github.com/oschwald/maxminddb-golang/v2` | v2.8.0 | ISC |
| `github.com/philhofer/fwd` | v1.2.0 | MIT |
| `github.com/pjbgf/sha1cd` | v0.7.0 | Apache-2.0 |
| `github.com/pressly/goose/v3` | v3.29.0 | MIT |
| `github.com/riverqueue/river` | v0.49.0 | MIT + MPL-2.0 |
| `github.com/riverqueue/river/riverdriver` | v0.49.0 | MPL-2.0 |
| `github.com/riverqueue/river/riverdriver/riverpgxv5` | v0.49.0 | MPL-2.0 |
| `github.com/riverqueue/river/rivershared` | v0.49.0 | MIT + MPL-2.0 |
| `github.com/riverqueue/river/rivertype` | v0.49.0 | MPL-2.0 |
| `github.com/segmentio/asm` | v1.2.1 | MIT-0 |
| `github.com/segmentio/encoding` | v0.5.4 | MIT |
| `github.com/sergi/go-diff` | v1.3.2-0.20230802210424-5b0b94c5c0d3 | MIT |
| `github.com/sethvargo/go-retry` | v0.5.0 | Apache-2.0 |
| `github.com/shirou/gopsutil/v4` | v4.26.9 | BSD-3-Clause |
| `github.com/skeema/knownhosts` | v1.3.1 | Apache-2.0 |
| `github.com/spf13/cobra` | v1.10.2 | Apache-2.0 |
| `github.com/spf13/pflag` | v1.0.10 | BSD-3-Clause |
| `github.com/tidwall/gjson` | v1.19.0 | MIT |
| `github.com/tidwall/match` | v1.2.0 | MIT |
| `github.com/tidwall/pretty` | v1.2.1 | MIT |
| `github.com/tidwall/sjson` | v1.2.5 | MIT |
| `github.com/tinylib/msgp` | v1.6.4 | MIT |
| `github.com/tklauser/go-sysconf` | v0.4.0 | BSD-3-Clause |
| `github.com/tklauser/numcpus` | v0.12.0 | Apache-2.0 |
| `github.com/x448/float16` | v0.8.4 | MIT |
| `github.com/xanzy/ssh-agent` | v0.3.3 | Apache-2.0 |
| `github.com/yosida95/uritemplate/v3` | v3.0.2 | BSD-3-Clause |
| `go.etcd.io/bbolt` | v1.5.0 | MIT |
| `go.opentelemetry.io/auto/sdk` | v1.2.1 | Apache-2.0 |
| `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` | v0.72.0 | Apache-2.0 + BSD-3-Clause |
| `go.opentelemetry.io/otel` | v1.47.0 | Apache-2.0 + BSD-3-Clause |
| `go.opentelemetry.io/otel/log` | v1.47.0 | Apache-2.0 + BSD-3-Clause |
| `go.opentelemetry.io/otel/metric` | v1.47.0 | Apache-2.0 + BSD-3-Clause |
| `go.opentelemetry.io/otel/trace` | v1.47.0 | Apache-2.0 + BSD-3-Clause |
| `go.uber.org/multierr` | v1.11.0 | MIT |
| `go.yaml.in/yaml/v3` | v3.0.5 | MIT |
| `golang.org/x/crypto` | v0.58.0 | BSD-3-Clause |
| `golang.org/x/net` | v0.61.0 | BSD-3-Clause |
| `golang.org/x/oauth2` | v0.37.0 | BSD-3-Clause |
| `golang.org/x/sync` | v0.24.0 | BSD-3-Clause |
| `golang.org/x/sys` | v0.49.0 | BSD-3-Clause |
| `golang.org/x/text` | v0.43.0 | BSD-3-Clause |
| `golang.org/x/time` | v0.16.0 | BSD-3-Clause |
| `gopkg.in/warnings.v0` | v0.1.2 | BSD-2-Clause |
## Web (`apps/web`)
Method: the `license` field of every `package.json` in the resolved `node_modules` tree, starting from the `dependencies` and `devDependencies` of `apps/web/package.json` and following nested dependencies. 260 packages: 91 reachable from `dependencies` (bundled into the shipped app) and 169 only from `devDependencies` (build and test tools, not shipped).
| Licence | Shipped (91) | Dev only (169) |
|---|---|---|
| MIT | 66 | 133 |
| ISC | 17 | 15 |
| OFL-1.1 (the three IBM Plex and Space Grotesk font packages) | 3 | – |
| BSD-3-Clause | 1 | 3 |
| BSD-2-Clause | – | 2 |
| Apache-2.0 | 1 (`class-variance-authority`) | 6 (TypeScript, Playwright, `detect-libc`, `baseline-browser-mapping`) |
| 0BSD, Unlicense, MIT AND ISC | 1 each (`tslib`, `isbot`, `victory-vendor`) | – |
| MIT OR Apache-2.0 | – | 2 (Biome) |
| MPL-2.0 | – | 4 (`lightningcss`, build-time CSS transform only) |
| CC-BY-4.0 | – | 1 (`caniuse-lite` browser data, build time) |
| CC0-1.0 | – | 1 (`simple-icons`, source of the brand glyph sprite) |
| Python-2.0, PSF-2.0 | – | 2 (`argparse`, dev tooling) |
| Unknown or missing | 0 | 0 |
Direct dependencies that ship in the app:
| Package | Version | Licence |
|---|---|---|
| `@fontsource-variable/ibm-plex-sans` | 5.3.0 | OFL-1.1 |
| `@fontsource/ibm-plex-mono` | 5.3.0 | OFL-1.1 |
| `@fontsource/space-grotesk` | 5.3.0 | OFL-1.1 |
| `@tanstack/react-query` | 5.103.1 | MIT |
| `@tanstack/react-router` | 1.170.38 | MIT |
| `@tanstack/react-virtual` | 3.14.13 | MIT |
| `@xterm/addon-fit` | 0.11.0 | MIT |
| `@xterm/xterm` | 6.0.0 | MIT |
| `@xyflow/react` | 12.12.0 | MIT |
| `class-variance-authority` | 0.7.1 | Apache-2.0 |
| `cn` | 0.3.2 | MIT |
| `lucide-react` | 1.61.0 | ISC |
| `motion` | 13.5.1 | MIT |
| `react` | 19.3.0 | MIT |
| `react-dom` | 19.3.0 | MIT |
| `react-is` | 19.3.0 | MIT |
| `recharts` | 3.10.1 | MIT |
| `zod` | 4.6.5 | MIT |
Notes:
- No GPL, LGPL, AGPL, SSPL, BUSL, proprietary or non-commercial package is in the tree. Fonts are under the SIL OFL-1.1, which permits bundling.
- Code adapted into the repository keeps its own notice in the source files: `lucide-animated` icons (MIT, `apps/web/src/components/icons/animated/LICENSE`) and the dithercn image recipes (MIT, Copyright (c) 2026 Leo and Shadcn Labs (startercn); header comments in `packages/dither/src/components/images/look.ts` and `algorithms.ts`, full MIT text in `NOTICE`). Both are in the top-level `NOTICE`. dithercn's other catalogue entries are rebuilt in-house from its public documentation and no dithercn package is installed. The `nebulactrl/nebula` design-system files are the owner's own Apache-2.0 work and are relicensed by the owner under this repository's licence.
- The brand glyphs in `apps/web/src/assets/brand-sprite.svg` come from `simple-icons` (CC0). The logos remain trademarks of their owners and are used to identify the products.
- The marketing site `apps/site` did not exist when this scan was run (no `node_modules`). Run the same scan when it lands, and every manifest in the repository must carry `"license": "AGPL-3.0-only"` (with `"private": true` for unpublished packages).
## Images NebulaCtrl deploys are separate works
NebulaCtrl pulls and runs these container images unmodified, as separate processes that talk to NebulaCtrl over the network or the Docker API. They are not linked into, bundled with or derived from NebulaCtrl, so their licences neither extend to NebulaCtrl nor does the AGPL of NebulaCtrl extend to them. Each keeps its own licence and its own source-offer obligations (which NebulaCtrl does not take on by pulling the image on the user's server).
| Image | Used for | Licence (upstream) |
|---|---|---|
| Traefik v3 | per-server proxy | MIT |
| SeaweedFS | platform S3 storage | Apache-2.0 |
| Loki | platform log store | **AGPL-3.0** |
| Grafana | optional dashboards | **AGPL-3.0** |
| Headscale | tailnet control server | BSD-3-Clause |
| Pocket ID | default managed sign-in provider | BSD-2-Clause |
| Authelia | alternative managed sign-in provider | Apache-2.0 |
| Tailscale | per-service tailnet device, the proxy's managed node, and the [server mesh](https://nebulactrl.com/docs/operations/mesh) server and link nodes | BSD-3-Clause |
| `cloudflared` | Cloudflare Tunnel connector | Apache-2.0 |
| nginx (alpine-slim) | maintenance and error pages | BSD-2-Clause |
| Postgres, MySQL, MariaDB, Valkey, Redis, MongoDB, ClickHouse | managed databases | per engine; the catalog labels Redis (RSALv2 / SSPLv1 / AGPLv3) and MongoDB (SSPL) as source-available and requires `acceptLicense` where relevant |
The licences of Loki, Grafana, SeaweedFS, Headscale, Pocket ID, Authelia and the database engines are the values recorded in `internal/platform/catalog.go` and `internal/databases/engines/catalog.go`; the others are the upstream projects' published licences and were not re-verified against each image on 2026-10-27. The licence class of every one-click template is handled separately by the template catalog (`permissive`, `copyleft`, `source-available`, see [templates](https://nebulactrl.com/docs/operations/templates)).
---
# Navigation transitions
Source: https://nebulactrl.com/docs/architecture/navigation-transitions
Every route change in the web app runs as a [View Transition](https://developer.mozilla.org/docs/Web/API/View_Transition_API). Cards, rows, sidebar labels and page titles morph into their destination; the rest of the page cross-fades with a small lift. This replaced the earlier overlay-based card-to-canvas morph.
Code lives in `apps/web/src/lib/transitions/`; the look lives in the `::view-transition-*` block at the end of `apps/web/src/styles.css`.
The design rules (when to morph, the budgets, what never animates) are in DESIGN.md § 6.3; this page is the mechanics.
## What plays
| Variant | When | Timing |
|---|---|---|
| `page` | A card, row or link press with the pointer; back and forward; navigations started from code | Content area 180 ms cross-fade plus an 8 px lift; shared elements 260 ms, ease-out |
| `fast` | A press inside the sidebar or the command palette; any keyboard activation; in-page tab switches | 150 ms throughout |
| `reduced` | `prefers-reduced-motion: reduce` | Plain 120 ms cross-fade, no lift, nothing morphs |
| none | Browser without `document.startViewTransition`; a change of search or hash only; a held key or an arrow, Home, End or Page key; a navigation within 120 ms of the previous one; the initial load | Instant |
Only the content area (`#main-content`) animates. The sidebar and the top bar are in the unchanged root snapshot and show their new state at once. A navigation with no shell on either side (login, setup) cross-fades the whole document instead.
## How names are assigned
An element opts in with `data-vt="<kind>:<id>"` (helpers `vt(kind, id)` and `vtAnchor(kind, id)` in `#/lib/transitions`). That attribute has no visual effect. The `view-transition-name` itself exists only for the length of one transition:
1. The router calls `router.startViewTransition(update)` for every committed navigation. `installRouteTransitions` (called in `main.tsx`) replaces that member, because neither `defaultViewTransition` nor the per-navigation `viewTransition` option hands back the `ViewTransition` object that tells us when to clean up.
2. `interaction.ts` has recorded the last pointer or key press. If it landed on (or inside a link or row containing) a `data-vt` element, that element is the source. Otherwise the leaving page's `data-vt-anchor` elements are the sources (a canvas shrinks back into its card).
3. The sources and `#main-content` get their names, and the browser captures the old page.
4. After `update()` has rendered the new page, the first element carrying the same `data-vt` key that is not the source takes the name. A persistent source (a sidebar label that becomes a page title) hands it over. With no counterpart the source keeps its name and cross-fades in place.
5. When the transition finishes or is skipped, every name that transition set is removed and the inline style it replaced is restored. A transition that was superseded only releases what it still holds.
Because only the pressed element is named, duplicates (many rows for one service) never collide.
## Where it is wired
| Source | Target |
|---|---|
| Project card (`project:<id>`) | Canvas page root (`project:<id>`, anchor): grows in, and shrinks back on the way out |
| Server card (`server:<id>`) | Server page header (`server:<id>`, anchor), both ways |
| Deployment and service table rows (`service:<id>`) | Inspector header (`service:<id>`) |
| Sidebar label (`nav:<page>`) | Page title (`nav:<page>`) in `PageFrame` |
| Settings nav item (`data-vt-tab="settings"`) | The active item slides, the section body cross-fades (`startTabTransition`) |
To add one: put `{...vt("kind", id)}` on the source and the same on its destination, and add the kind to `CLASS_BY_KIND` in `vt-names.ts` if it needs its own class.
## Tests
`bun test src/lib/transitions` covers the name registry (assignment, hand-over, restore, superseded owners), the navigation classifier (variants, reduced motion, repeating keys, unsupported browsers, rapid navigations) and arming/pairing of shared elements. The DOM glue (`interaction.ts`, `route-transitions.ts`) is not unit-tested.