# Threat model

Scope: NebulaCtrl P0 as built from this repository (control plane, agent, Traefik proxy, web UI, REST API, CLI, optional MCP endpoint). It follows the trust boundaries of spec §11.2 and states what the platform does and, as importantly, what it does not protect against. Report vulnerabilities as described in [`SECURITY.md`]().

## Security boundary in one paragraph

**Teams are an authorization boundary inside one shared trust domain.** An instance hosts one or more teams (workspaces, [teams](https://nebulactrl.com/docs/operations/teams.md)). *Inside a team* everyone who can sign in or holds a token with `deployments:write` is trusted to run arbitrary code in containers on the team's enrolled hosts, and Docker is root, so the platform's own allow-lists are what keep that from being root on the host: **nobody below the instance admin can gain root on a host through the platform unless the instance admin made that server _unrestricted_** (a per-server switch: [Docker options](https://nebulactrl.com/docs/operations/docker-options.md#the-unrestricted-switch-per-server), [Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks.md#unrestricted-per-server)). On an unrestricted server any deployer can take the host over. Builds are a separate, explicitly trusted path (below). *Between teams* the control plane enforces the separation in code: every lookup joins through the team, an inaccessible resource is a 404, and projects, servers, integrations, tokens, notification channels, backup destinations and the audit log are per team. A server is enrolled by exactly one team, so one team's containers never share a Docker daemon with another team's. What teams do **not** get is separate cryptography or separate infrastructure: one control-plane process, one Postgres and **one `NEBULACTRL_SECRET_KEY` serve every team**, so an authorization bug, a database dump together with the key, or a compromised control-plane host exposes all teams at once, and the instance admin (the first user) administers instance-wide settings. Projects and environments within a team organize work and per-environment Docker networks keep their traffic apart on a host ([what is and is not isolated](#4-deployed-apps-and-build-inputs-are-untrusted-workloads)), but they do **not** isolate hostile tenants from each other: they share a kernel, a Docker daemon and root. Do not host mutually untrusting customers on one NebulaCtrl installation; give each its own instance, database and key.

## Assets

| Asset | Why it matters | Where it lives |
|---|---|---|
| Docker daemon on each agent host | Docker access = root on the host | agent host |
| `NEBULACTRL_SECRET_KEY` (and any `NEBULACTRL_SECRET_KEY_PREVIOUS`) | decrypts every stored secret of every team | operator's key file (`NEBULACTRL_SECRET_KEY_FILE`, mode 0400) or `.env` / vault, control-plane process memory; as an environment variable it is also visible to anyone with Docker access on the control-plane host |
| Secret variables, git tokens, SSH deploy keys, registry credentials | access to your apps, repos, registries | encrypted in Postgres (AES-256-GCM); deploy keys are write-only and reach the agent as send-time secrets ([SSH deploy keys](https://nebulactrl.com/docs/operations/ssh-deploy-keys.md)); registry push and pull credentials are write-only and reach agents the same way |
| Container images in a registry | the code that runs on your hosts; a registry write is a code-execution path to every server that pulls from it | the registry you chose (a BYO provider or the managed registry); targets pull by **digest**, never by tag |
| Build server (agent identity, git credentials, build variables, registry push credential) | a privileged host that handles the build inputs of **every project** of the team | the enrolled build server |
| External secret manager credentials (Vault token or AppRole secret id, Infisical client secret) and the secrets read with them | read access to everything the credential's policy allows | the credential is encrypted in Postgres and write-only; the secrets themselves are fetched at deploy time, held in memory and sent to the agent as send-time secrets, never stored ([secret managers](https://nebulactrl.com/docs/operations/secret-managers.md)) |
| Agent identity (Ed25519 private key) | impersonating a host to the control plane | `agent.db` on the host, mode 0600 |
| API tokens, sessions | act as a user | token hashes / session store in Postgres |
| Postgres | the whole desired state | the control plane's Compose stack |
| Proxy certificates and keys (ACME) | TLS for your domains | `nebulactrl-traefik-acme` on each agent host |

## Trust boundaries

```
 Browser / CLI / MCP client ──HTTPS──▶ Control plane ◀──outbound WSS── Agent ──▶ Docker ──▶ your containers
        (untrusted)                 (trusted coordinator)           (privileged on its host)   (untrusted workloads)
                                         │
                                      Postgres
```

### 1. Browsers, CLI and MCP clients are untrusted

- **Authentication:** local admin account created through the one-time setup page (works only while no user exists; afterwards `409`). Passwords are hashed with **argon2id**; login is limited to **10 attempts per minute per IP**. Sessions use `alexedwards/scs` with a Postgres store; the cookie `nebulactrl_session` is **HttpOnly, SameSite=Lax, and Secure when `NEBULACTRL_PUBLIC_URL` is https**. Run behind HTTPS.
- **CSRF:** Go's `http.CrossOriginProtection` rejects cross-origin unsafe requests that rely on cookies. Requests with an `Authorization: Bearer` header are exempt, since a cross-site page cannot attach one.
- **API tokens:** `nct_…`, 32 random bytes; only a SHA-256 hash is stored; shown once with an 8-character prefix for identification. Scopes: `projects:read`, `projects:write`, `deployments:write`, `deployments:rollback`, `servers:read`, `servers:write`, `admin`. Optional project restriction, expiry, revocation. **Bearer header only**; never accepted in query strings, so tokens do not end up in access logs.
- **Authorization:** checked inside every domain service method (`policy.Require(scope, project)`), not in HTTP handlers, so REST, CLI and MCP share it. Every lookup joins through workspace and project, and an inaccessible resource returns **404, not 403** (no IDOR oracle). Session users hold every scope.
- **Audit:** project/environment/service create/update/delete, variable set/delete (**key only, never the value**), domain add/remove, deploy, rollback, restart, stop, server enroll/revoke, token create/revoke, reconciler limit events, in the same transaction as the mutation. Actor type is `user`, `token` or `mcp`.
- **Idempotency:** mutating deploy-like requests honor `Idempotency-Key` (24 h), so retries do not double-deploy.
- **Residual risk:** a stolen session or `admin`/`deployments:write` token can deploy code to your hosts. Use short expiries, project restrictions and the read-only presets for automation that only reads.

### 2. The control plane is a trusted coordinator that assumes nothing about repositories or agents

- It validates input at the boundary: image references, hostnames (RFC 1123, lowercase, ≤ 253 chars), environment variable keys, relative Dockerfile/context paths with no `..`, server placement.
- It never builds a shell string from user input: commands to agents are typed JSON from a **closed set** (`deploy`, `build_image`, `registry_check`, `restart_service`, `stop_service`, `remove_service`, `sync_routes`, `logs_subscribe`, `logs_unsubscribe`). The agent rejects anything else.
- Agent messages are treated as untrusted data: results are matched to known commands; log lines are redacted and bounded (10,000 stored lines per deployment) before storage and fan-out.
- **Secret handling.** Variables are encrypted with **AES-256-GCM**; ciphertext carries a key-version byte and authenticates (tampering fails to decrypt). Secret values are never returned by any read endpoint (`{key, isSecret:true, set:true}`), never written to audit metadata, and never persisted in `agent_commands.payload`: the dispatcher decrypts them at send time and attaches them to the outgoing frame only. Log redaction replaces every secret value of 6+ characters with `••••••` before persisting or streaming. Short secrets (under 6 characters) are **not** redacted: use real secrets. Stored runtime logs live **unmasked** on the server's disk (the agent cannot know which values are secret when it captures); the control plane masks them when it reads history, and the agent masks each service's secrets in every line **before a log drain ships it** off the host ([log history and drains](https://nebulactrl.com/docs/operations/logs.md)). Drain credentials are sealed with the instance key and travel only in the send-time configuration; a drain sends your application's logs to the destination you configure, so treat it as a trust decision.
- **The one deliberate exception: database connection strings.** A managed database's `DATABASE_URL` and friends are masked like any secret; the *reveal* action (`POST /services/{id}/database/reveal`) returns one in plain text, uncached (`Cache-Control: no-store`), only for sessions and tokens with `deployments:write` (or `admin`), and is audited as `secret.reveal` (service and key, never the value). MCP has no reveal tool. Credential rotation sends the new password to the agent in the command frame (never in `agent_commands.payload`), runs it through the exec's stdin/environment (never argv), and redacts it from results; see [managed databases](https://nebulactrl.com/docs/operations/databases.md).
- **The web terminal and file browser are a deliberate exec surface.** Owners, Admins and Developers (never Viewers; tokens need `deployments:write` and a creator who still holds the role) can run a shell and read or write files **inside a replica container**, relayed over the agent's existing connection as ephemeral stream frames (`exec_*`, `fs_*`; nothing is persisted or redelivered). Production needs an explicit confirmation checked server-side; the WebSocket handshake enforces the `Origin` (no cross-site terminal riding a session cookie); at most 3 sessions per person, 15 minutes idle timeout; every session, download and upload is audited (the executable of a command, never its arguments; never the I/O), and a session that cannot be audited is not opened. The agent only ever reaches containers carrying the service's managed labels, through the Docker API: the host filesystem is out of reach. Residual risk: a shell sees the container's environment, including secrets, and can reach whatever the container's network can; treat the roles that hold it like SSH access. File uploads are read-only in production unless an admin allows them per project. See [web terminal and file browser](https://nebulactrl.com/docs/operations/terminal.md).
- **The host shell is a second, much stricter exec surface** ([host-shell.md](https://nebulactrl.com/docs/security/host-shell.md)): a root shell on the server itself, **off by default and opt-in per server**. Only Owners and Admins with a signed-in session (never a token or MCP principal) whose account has **two-factor authentication** can open one; production servers need the guarded confirm and `confirmProduction` checked server-side; at most 2 sessions per person and 15 minutes idle; `server.shell.enable`, `server.shell.disable`, `server.shell.open` and `server.shell.close` are audited, the I/O never. An agent on the host spawns the pty itself with a minimal environment (never the agent's own variables); an agent in a container uses a privileged `--pid=host` helper container that is always removed, with an orphan sweep as backstop.
- **Host hardening is a third surface that changes the host** ([host-hardening.md](https://nebulactrl.com/docs/security/host-hardening.md)): a Lynis audit (read-only) and a curated catalog of reversible fixes, applied only by Owners and Admins, audited with who, which fixes and the index before and after. The agent runs them as root on the host, directly or through a helper container like the host shell. There is no fix for SSH, the firewall or user accounts, none touches Docker's IP forwarding, every fix is verified after it is made and undone by itself if it did not take effect, and rollback only undoes what NebulaCtrl applied.
- **Single point of trust:** the control plane process holds the decrypted key and can deploy to every host. Anyone with its database *and* the key can read all secrets and impersonate the platform to agents (but not to a host that has no enrolled identity). Protect the key, the host and the Postgres volume as one trust domain.
- **Availability:** a single instance; a compromised or stopped control plane cannot stop running workloads, but cannot manage them either.

### 3. The agent is privileged on its host

- The agent holds the Docker socket: **Docker access is root-equivalent on the host**. Anyone who can make the agent run a container can read the host filesystem, other containers' data and the agent's own state. Therefore: **a control-plane administrator is a root administrator of every enrolled host.** A compromised control plane (database + key + code) is a compromise of all agent hosts.
- It only dials **outbound** to `wss://<control-plane>/agent/v1/connect`; no inbound port for management. `ws://` is refused unless `--insecure-dev`.
- **Identity:** at enrollment the agent generates an Ed25519 keypair; only the public key is sent. Each connection runs a challenge: the server sends a fresh nonce, the agent signs `nonce || agentId || serverTime`; stale nonces (> 30 s) and revoked identities are rejected (close 4401). The private key never leaves `agent.db` (mode 0600). Enrollment tokens are 32 random bytes, stored hashed, single use, 1 hour lifetime, revocable.
- **Commands are idempotent and durable**; replays return the stored result instead of re-running.
- **Revocation** is immediate for the control-plane link. It does not remove containers or secrets already on the host: treat a revoked, suspected host as compromised and rotate its secrets.
- **Residual risk:** root on an agent host reveals the agent's private key and the secrets of the services deployed there (they are present in container environments). It does not reveal other hosts' secrets or the control plane's key; one agent cannot read another's secrets.

### 4. Deployed apps and build inputs are untrusted workloads

- **Containers** are created without `--privileged`, without host networking, and without the Docker socket or arbitrary host bind mounts; the platform never adds those, and the API has no field to request them. Volumes are named Docker volumes. Memory and CPU limits are applied from the service config; there are no default limits when none are set.
- **Network:** the Docker network layout is described in full in [networking](https://nebulactrl.com/docs/operations/networking.md). In short:
  - **Isolated now.** Every environment has its own bridge network `nebulactrl-env-<id>`; app, task and tailnet-sidecar containers no longer join the shared `nebulactrl-edge`, so containers of different projects, environments and teams on one host cannot reach each other by name or address. Only Traefik, the error-pages server and `cloudflared` stay on `nebulactrl-edge`; Traefik joins the environment network of each routed service while the route exists and leaves it afterwards. Agent health probes use the container's address on its environment network.
  - **Shared networks are opt-in.** A team **shared network** (`nebulactrl-shared-<id>`) is created by someone with `servers:write`, lives on exactly one server, and only services attached to it (`deployments:write` on the service; staged and deployed like any other setting) can join. Each gets the alias `<service>.<env>.<project>`. A network of another team is a 404 for everyone; attaching a service that runs on another server is `network_wrong_server` (422). Create, delete, attach and detach are audited (`network.*`).
  - **Block outbound internet.** The setting is part of the release (`config.blockEgress`), staged and rolled back with it. A blocked service joins only `nebulactrl-env-<id>-int`, a Docker `internal` network with no gateway, so it has no route to the internet, the LAN or the host; every other service of the environment joins `-int` too, with the same alias, so it still reaches it. It defaults to **on for managed databases** and off for apps. A blocked service cannot join shared networks (a shared bridge would give its internet back).
  - **Not isolated.** Containers of one environment trust each other: they share a network and every port. All containers of a host share its kernel, its Docker daemon and its root: a container escape, or root on the host, crosses every network boundary. Docker networks are not a hostile-tenant sandbox (see the boundary paragraph above). Outbound filtering is all-or-nothing per service: there are no per-destination egress rules, and a service that is allowed out can reach anything the host can route to, including the LAN and a cloud provider's metadata address.
- **By default builds run on the target host** with its Docker daemon (ADR 0004). A Dockerfile executes arbitrary commands during `docker build`, and BuildKit may read build context files. Git repositories are cloned to a per-deployment directory and removed afterwards; the build has a 30 minute timeout; `dockerfilePath` and `buildContext` cannot escape the clone. Without a Dockerfile, Railpack (ADR 0006) runs the project's own install and build commands (`npm install`, `pip install`, …) the same way; on hosts without the containerd image store the agent also runs a privileged BuildKit container (`nebulactrl-buildkit`) for those builds, which adds nothing a Docker-socket holder lacked. Build variables reach builds as BuildKit secrets via the docker CLI's environment, never as arguments. **Only deploy repositories you trust.** Git tokens are passed only inside the command frame and used for the clone as HTTPS BasicAuth (`x-access-token`); they are not passed into the build's environment.
- **Build servers and registries are optional and widen the blast radius** ([ADR 0008](https://nebulactrl.com/docs/decisions/0008-build-servers.md), [build servers](https://nebulactrl.com/docs/operations/build-servers.md)). A build server is an enrolled server with `build_enabled`; it runs the same untrusted build code as a target would (Dockerfiles, Railpack steps), in a **privileged BuildKit container**, but away from your running workloads. What it gains and loses:
  - **Better:** a malicious or compromised build cannot reach production data, the runtime secrets or other running services; **runtime variables never go to a build server** (only the git credential, build variables and the registry push credential do); the target never sees the repository credential.
  - **Worse:** one host now handles the build inputs of **every project** of the team (git tokens and SSH keys of all of them, build variables, and a push credential), so an escape from a build container is a compromise of all of them and of the agent's identity. Treat it as a high-value, **dedicated** host: role *Builds*, no workloads, nothing else on it. Per-project BuildKit builders (`nebulactrl-buildkit-<project8>`, each with its own cache) stop one project's build from reading or poisoning another's **cache** (cache mounts are shared by id inside a builder); they do not stop a container escape. A Developer restricted to project A can still run code on the host that builds project B, which is inside the trusted-team boundary above and outside what NebulaCtrl isolates.
  - **Pull-request previews do not build on a shared build server** unless the owner turns on **Allow previews** for that server; fork code is untrusted, so leave it off or dedicate a build server to previews.
  - **Registry credentials are never ambient.** The push credential is sealed in Postgres, attached to the `build_image` command at send time only (never in `agent_commands.payload`), written by the build agent to a mode-0700 temporary `DOCKER_CONFIG` that exists for one build and is always removed, and redacted from build logs. The target receives a separate, optionally read-only pull credential for each deploy. No `docker login` ever leaves a long-lived credential in a host's `~/.docker/config.json`.
  - **Managed registry:** the control plane is its token server, so credentials are per command, scoped to one repository and the needed actions, and expiring (15 minutes by default, at most 6 hours for a build), exchanged for tokens valid at most 15 minutes; no long-lived registry password exists on any host. The registry host holds every image of the team and is a high-value target: keep it patched and its volume private. Docker only accepts TLS for non-loopback registries, so it needs a real certificate.
  - **BYO registry:** the credential cannot be scoped finer than the provider allows (a GHCR classic token is account-wide), and the provider's staff and its own breaches are outside NebulaCtrl. Image layers and the image configuration are readable by anyone with pull access, and plain `--build-arg` values persist in the image: put secrets in build **secret** variables, which reach BuildKit as secrets.
  - **Digest pinning.** Targets pull `repo@sha256:…`, so a tag overwritten or a repository tampered with after the build cannot change what a release runs: a different image has a different digest and fails to pull. The first trust decision is the push: whoever holds a push credential can push any content to that repository.
  - **Plain-HTTP registries** are accepted for loopback addresses only.
- **Images** come from whatever registry you name. Pin digests where it matters; an image tag can change. No vulnerability scanning is included (recommended in spec §11.3; use external scanners).
- **Container escape** is a host compromise: Docker is not a hostile-tenant sandbox.
- **Proxy:** Traefik is the only component publishing ports. Routes come from the database's domain table only and are pushed by the agent as the full dynamic configuration through Traefik's REST provider on the admin entrypoint. That entrypoint is protected by basic auth with a per-host random secret (kept in the agent's local store; only its bcrypt hash is given to Traefik). It is published only on `127.0.0.1:9000` of the host, or, when the agent itself runs in a container, not published at all and reachable only on the edge network. Containers you deploy join the edge network, but without the secret they get `401` from the admin entrypoint. The probe entrypoint (`:8099`) is not published, but it is reachable from containers on the edge network; it serves only the same routes as the public entrypoints plus liveness. Any local root user can still read the secret and reconfigure the proxy; this is part of the trusted-team boundary.
- **Per-service tailnet sidecar** ([service tailnet](https://nebulactrl.com/docs/operations/service-tailnet.md)): a pinned `tailscale/tailscale` container in userspace mode (`cap_drop: ALL`, `no-new-privileges`, no TUN device, no published ports, no Docker socket) on the service's environment network only; it advertises no routes, so a tailnet peer reaches only the ports named in the serve config, and **your tailnet policy (ACLs, tag ownership) is the access boundary**. A sidecar is as trusted as any container on that environment network and can reach its siblings. The OAuth client secret (or Headscale API key) stays in the control plane, sealed with `NEBULACTRL_SECRET_KEY`; sidecars only ever receive a **single-use, tagged auth key minted at send time** (never persisted, never logged; masked by the agent). An integration's allowed-tag list stops a project member from assigning a more privileged tag. Funnel is public by design: it is opt-in with a typed confirmation, audited (`tailnet.funnel_enabled`) and bypasses NebulaCtrl's proxy. A Headscale *pre-auth key* credential is reusable and remains visible to anyone with Docker access on the hosts.
- **Server mesh nodes and cross-server links** ([mesh](https://nebulactrl.com/docs/operations/mesh.md)): two kinds of agent-created containers, from a fixed spec that no user-supplied Docker option can change and that does not depend on the server's `unrestricted` switch. A **server node** runs in userspace mode (`cap_drop: ALL`, `no-new-privileges`, no TUN device, no host network, no Docker socket) and forwards only the TCP ports of the services you expose. A **link node** adds `NET_ADMIN`, `NET_RAW` and `/dev/net/tun` **inside its own network namespace only** (the same shape as the proxy's managed tailnet container): not privileged, no Docker socket, no host network, and it changes no host firewall rule. Both carry one tag per team; keys are single-use, pre-authorized, non-ephemeral, tagged and valid for an hour, delivered in the command frame and never persisted or logged. **The access policy is the boundary**, and it is generated: one rule per link, with the link's own tailnet IP as the source and the granted server-node ports as the destination; a link belongs to one environment and a grant cannot join it to another environment's service, so environments are separated at the tailnet layer without a tag each. Replication links and service links are different nodes, so a consumer cannot reach a replication port. Other services of the same environment on the consuming server can use the environment's link (the environment is the trust boundary, as everywhere else). On **Headscale** NebulaCtrl owns the policy (database mode) and overwrites hand edits. On **Tailscale** the default is advisory (you paste the rules; with an advisory policy and an allow-all tailnet a link node can reach any exposed port of the remote server node, which the Mesh page warns about). The opt-in **managed push needs the `policy_file` scope: a token with it can rewrite the entire tailnet policy, and the control plane holds it**, so a compromised control plane could open the tailnet; the mode is audited, refused when another team shares the tailnet, and edits only rules whose endpoints are all mesh-node IPs. Managed Headscale falls back to Tailscale's **public DERP relays** when there is no direct path: traffic stays end-to-end encrypted, but relays see ciphertext, endpoint addresses and volume, and the control server needs outbound internet. Mesh traffic is encrypted by WireGuard between nodes, not on the local hop to the provider container, and the mesh adds no authentication beyond the provider's own.
- **Postgres read replicas** ([database replicas](https://nebulactrl.com/docs/operations/database-replicas.md)): the replication role (`nebula_r_<id>`) has the REPLICATION privilege only, a per-replica scram password (generated, sealed, delivered at send time, never in a payload, the audit log, the API or the logs) and a managed `pg_hba.conf` entry; the primary's WAL retention for a dead replica is capped (the smaller of 4 GiB and 10 % of the disk). A replica keeps **Block outbound internet** and reaches its primary only through the environment network or the mesh link. **Promote is manual and authorised like a production deploy**: `confirm` with the replica's slug, the production approval where it applies, and an explicit `acceptDataLoss` for a forced Promote. The old primary is **fenced** (stopped, restart policy `no`, never started again by a deploy) so two writers cannot run through the platform, and a fenced service has no "unfence". Residual risk: replication is asynchronous, so a forced Promote loses what the replica had not received, and if the old primary's host returns and Docker restarts it before the agent reconnects, writes in that window are lost and diverge. A replica is a copy of the data: anyone who can read the replica's service can read the whole database.

### 5. MCP (optional)

Enable with `NEBULACTRL_MCP_ENABLED=true`; otherwise `/mcp` returns 404. When on:

- **Auth:** bearer API token only; **60 requests per minute per token**; every call is audited with actor type `mcp` and the client's name.
- **Authorization in the server, not the prompt:** each tool maps to the scopes of the REST operations it calls (`policy.Require` applies in the domain service, on every call), and a token **lists only the tools it holds every scope for**. Toolsets only reduce what is listed: a client is offered `core` and `observe` by default, the URL (`/mcp?toolsets=…`) opts in to more, and an explicit list also fences `find_operation` / `call_operation` to those toolsets. They never grant anything a token's scopes do not. Reading logs needs its own scope, `logs:read`. Create tokens for agents with the *MCP read-only* preset or *MCP deploy*, and restrict them to specific projects. See the [MCP reference](https://nebulactrl.com/docs/reference/mcp.md#scopes-and-token-presets).
- **Production guard (server-side, fail-closed):** when an MCP principal changes a **production** environment (deploy or apply, rollback, restart, stop, delete, restore, one-off command, deleting a volume, domain, environment or project, and the other operations listed in the [MCP reference](https://nebulactrl.com/docs/reference/mcp.md#production-approval)), the domain service refuses unless a person has approved that exact action; if the approval store is unavailable it refuses too. The client is asked through MCP elicitation, or gets a plan and a link to `/approvals/{id}` in the web app (session users only: Owners, Admins and Developers with deploy rights decide; the approval endpoints are denied over MCP). An approval is single-use and bound to the token, the tool, the arguments and the generations that were reviewed, lives at most 10 minutes, and any change in between voids it. A model-supplied `confirm` is never accepted for production. The only bypass is the per-token setting `allowProductionWithoutApproval` (**Let MCP clients change production without approval**): off by default, settable only by an Owner or Admin, honoured only while the creator still holds that role, and audited. Plain REST and CLI tokens are **not** guarded unless created with `requireProductionApproval`, so CI and deploy hooks keep working. Every outcome (requested, approved, declined, expired, refused, error, used) is audited. Non-production environments need no approval.
- **Staged changes:** MCP edits to services, variables, domains, volumes and links only stage; `apply_changes` is the only path to a running container and it requires the generation the client reviewed.
- **Bounded output:** `tail_logs` returns at most 500 lines, redacted; any tool result is capped at 64 KB with `truncated: true`. Log text is wrapped as untrusted with a per-call marker.
- **Deliberate exec surface: `run_command`.** It is **shell-equivalent** (root-equivalent on that service's data): it runs a command you name in a throwaway container of the service's own image, with the service's environment variables and network, so a token that may call it can read every secret that service holds and do anything the service can. It needs `deployments:write`, is subject to the production guard, waits at most `waitSeconds`, returns bounded redacted output and is audited with the command line (clipped). Do not give it to a token an agent uses on untrusted input.
- **Denied surface (the catalog's deny list):** no sign-in or account management, token, enrollment-token, invite and reset-link minting, deploy hook enabling or rotation, secret or credential reveal, integration credential writes, instance and SSO settings, team and role changes, the `unrestricted` switch, host shell and host hardening, approving an approval; no streaming follow modes. Everything else the API offers is available, behind its scopes, through curated tools or `find_operation` / `call_operation` (denied operations are never found nor callable). There is no raw Docker API or arbitrary SQL, and the web terminal and file upload are not exposed. No tool returns a secret: `service_get` returns configuration with secrets redacted, and a variable set through MCP is never echoed back.
- **Errors** are structured `{code, message}`; an acknowledged command whose agent is offline is `outcome_uncertain`.
- **Prompt injection:** deployment/runtime logs and repository content are attacker-controlled text that an LLM client may read. The server cannot stop a client from acting on injected instructions. The production guard puts a person between an injected instruction and production, but it does not protect other environments: **limit what the token can do** (read-only, project restricted), and keep deploy tokens and `run_command` away from agents that ingest untrusted logs.

## Data in transit and at rest

| Data | Protection |
|---|---|
| Browser ↔ control plane | TLS terminated by the host's own Traefik (the dashboard's system route, Let's Encrypt) with `--domain`, or by your proxy; without a domain the dashboard is plain HTTP and the installer says so |
| Agent ↔ control plane | TLS (`wss://`) plus Ed25519 challenge identity at the application layer (not mTLS) |
| Secrets in Postgres | AES-256-GCM; the key ring has key ids in the ciphertext, every value is bound to its table, column and row (a copied ciphertext does not open), and `nebulactrl-server rekey` rotates keys; releases keep their sealed environment only inside the rollback window. Details: [secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest.md) |
| Everything else in Postgres | **not encrypted by NebulaCtrl**: use disk encryption/storage-level encryption if required |
| Secrets in agent memory/containers | environment variables of the running container (visible to anyone with Docker access on the host) |
| Registry credentials | sealed with the key in Postgres; reach agents only in the command frame; a temporary 0700 `DOCKER_CONFIG` for the length of one build; managed registry: an expiring credential scoped to one repository, exchanged for a JWT valid at most 15 minutes |
| Images between build server, registry and target | TLS to the registry (required for non-loopback hosts); pulls verify the digest |
| Backups | contain ciphertext plus the rest of the state; keep the key elsewhere ([backup](https://nebulactrl.com/docs/operations/backup-restore.md)) |

## Threats and mitigations

| Threat | Mitigation | Residual |
|---|---|---|
| Stolen admin password | argon2id, login rate limit, HTTPS-only cookie, optional passkeys (WebAuthn, phishing-resistant, bound to the public hostname) and TOTP two-step verification with recovery codes, per-team "require 2FA" ([login methods](https://nebulactrl.com/docs/security/login-methods.md)) | a person who never enrols is protected by the password alone; `/mcp` is not subject to the team 2FA rule |
| Stolen API token | hashed storage, scopes, project restriction, expiry, revocation, bearer-only; over MCP the production approval applies even to a stolen token | token is a bearer secret; a token with `allowProductionWithoutApproval`, or `run_command`, is as powerful as the role that created it; REST tokens are not production-guarded by default |
| An AI client steered by injected text (a log line, a README) into a harmful action | scopes and toolsets, project restriction, staged edits that need a reviewed generation, server-side human approval for production, deny list for credentials and shells, untrusted-text wrapping of logs, audit | non-production environments take changes without approval; `run_command` is shell-equivalent; the client can still act on injected text within the token's scopes |
| Cross-site request forgery | `SameSite=Lax`, cross-origin protection | none known |
| IDOR across projects and teams | 404 on policy mismatch, workspace/project joins | teams share one database and one key: a bug here is a cross-team leak, not a contained one |
| Rogue/compromised agent | identity per host, closed command set, revocation, cannot read other hosts' secrets | root on the host |
| Cross-project or cross-team traffic on a host | each environment has its own Docker network and only Traefik, the pages server and cloudflared are on the shared `nebulactrl-edge`; a service reaches only its environment, the shared networks it was explicitly attached to, and (unless outbound is blocked) the internet; shared networks are team-level, one server each, and never cross teams (404 across teams, 422 `network_wrong_server`) | same host, same kernel and same Docker daemon: a container escape or root on the host defeats every network boundary; a service that may reach the internet can reach any address the host can route to (LAN, cloud metadata) |
| A compromised app calls out or exfiltrates data | per-service **Block outbound internet**: the container joins only an `internal` Docker network that has no gateway (on by default for managed databases) | all-or-nothing per service, no per-destination rules; a service that is allowed out is not filtered at all; traffic through the proxy, a tailnet sidecar or the agent (backups) is deliberate |
| Cross-environment or cross-server reach through the mesh | one link node per environment and server pair, with its own source IP; generated deny-by-default rules that list only granted (server-node IP, port) pairs; a grant that joins a link to another environment is refused (`mesh_environment_mismatch`); replication and service links are separate nodes (`mesh_purpose_mismatch`); links and devices are added only by explicit actions (deploy of the consumer, a confirmed canvas edge, a replica or Promote) and removed after a grace period | an advisory Tailscale policy that is looser than the generated rules still allows more; the connectivity check proves intent, not exclusivity; a Tailscale policy token (`policy_file`) can rewrite the whole policy; managed Headscale uses Tailscale's public DERP relays as the fallback |
| A wrong or unwanted Promote | manual only, `confirm` by slug, production approval, a plan that lists the data at risk, the consumers per server and what will be rewritten; a switchover freezes, stops and fences the old primary and waits for the replica to catch up (and rolls back if it cannot); consumer rewrites are staged and can be discarded | forced mode loses up to the replication lag by design; the old primary can restart before it is fenced on a forced Promote; there is no automatic detection of a lost primary |
| Platform services exposed to the internet (S3, Loki, Headscale, sign-in) | each is a normal service behind the server's proxy with a generated hostname and Let's Encrypt certificate (an ACME email is required, or the set-up refuses); images pinned by digest and run **unmodified**; outbound access blocked except Headscale; credentials generated, sealed, delivered as secret variables and never in configuration files; S3 requires a signed request (a wrongly signed one is refused), Loki sits behind basic auth, Authelia's configuration holds no secret (they are read from the environment) and its users have argon2id hashes; all of it is created by an administrator (scope `admin`, and an instance administrator for single sign-on) and audited | the endpoints are reachable from the internet like any domain (restrict by IP in the domain's access rules if you can); the S3 store on the databases' own server is not an off-site backup; an `http://` public URL carries S3 traffic in clear text; Loki and Grafana are AGPL-3.0 ([platform services](https://nebulactrl.com/docs/operations/platform-services.md)) |
| Replay of agent auth | fresh nonce per connection, 30 s window | none known |
| Malicious or stale one-click template | templates ship inside the binary after review (no runtime fetch); images pinned by tag **and** digest, checked against the registry without pulling; strict schema (no shell strings, no host mounts/privileged); file contents are plain text and rejected inside volume paths; generated secrets are stored encrypted and never returned | the app's own licence and security are the upstream project's; source-available and proprietary templates are labelled |
| Database leak (without key) | secrets are ciphertext | other state readable: variable names, secret flags, topology, domains, git URLs, audit log |
| Database write access (SQL injection, a bad restore script) without the key | a sealed value is bound to its table, column and row: copying one secret's ciphertext into another row or column fails to decrypt instead of delivering that secret somewhere else | the attacker can still delete or corrupt values (denial of service) |
| Database leak + key | none | full secret disclosure of every team; rotate everything. The key and the database sit on the same host in the default install, so a compromised control-plane host is this case |
| Old secret values kept in releases | each release stores its sealed environment; an hourly job drops it from releases outside the rollback window (the newest `imageRetention` succeeded ones); a rollback to a pruned release uses the current variables and says so | values inside the window, and in database backups taken before they were pruned, stay until the backup expires |
| Secrets in control-plane memory | no core dumps, process not dumpable (Linux), key bytes not kept after start-up, external secret-manager values expire from memory after 30 minutes | root or a debugger on the control-plane host can read the process; Go strings cannot be wiped |
| Secrets in logs/audit/payloads | redaction (6+ chars), key-only audit, send-time secret attachment, no secret in logs | short secrets, secrets printed in encoded forms |
| Malicious Dockerfile/image | host-level builds are *explicitly trusted* | host compromise possible |
| Compromised or malicious build server | closed command set, dedicated host (role *Builds*) with no workloads and no runtime secrets, per-project builders, revocation of the agent identity, per-build temporary registry credentials; images are deployed by digest, so what a build server pushed later cannot replace a release already recorded | the build host sees the build inputs and push credential of every project; a privileged-builder escape compromises all of them; a malicious build can push a bad image that the **next** deployment will run |
| Untrusted pull-request code on a shared build server | previews do not build on a shared build server unless **Allow previews** is on for it ([previews](https://nebulactrl.com/docs/operations/previews.md#where-previews-build)) | with it on, fork code runs next to other projects' builds and the push credential |
| Registry leak, deletion or outage | credentials sealed, short-lived for the managed registry; images pinned by digest; the target keeps `imageRetention` local images; a rollback fails before touching the running release; a deploy falls back to building on the target when no registry is usable | a BYO lifecycle rule or an out-of-band delete removes rollback targets; a registry compromise is a code-execution path to every server that pulls from it |
| Malicious or over-privileged Compose file (stack) | unless the instance admin made the server **unrestricted**, the agent normalises the file with `docker compose config` and refuses privileged mode, host network/PID/IPC/UTS/user namespaces, devices, unsafe capabilities, security options other than `no-new-privileges`, host bind mounts (including the Docker socket), external or renamed volumes and networks, volume/network `driver_opts` that mount host paths, `file:` secrets and configs and reserved labels; the control plane runs the same check on save; compose runs in a clean environment (no agent secrets), with project, container names and networks fixed by NebulaCtrl's override; ports are never published on the host ([Compose stacks](https://nebulactrl.com/docs/operations/compose-stacks.md), [ADR 0007](https://nebulactrl.com/docs/decisions/0007-compose-stacks.md)) | Docker is still root: the check is a deny list on the normalised model, not a sandbox, and a new Compose key that grants host access is allowed until the list learns it; on a server an instance admin made **unrestricted** any deployer of a stack can take over the host (audited, warned) |
| Over-privileged Docker options on a plain service (devices, GPUs, runtime, privileged, host PID, unconfined seccomp/AppArmor, `NET_ADMIN`, `SYS_ADMIN`) | typed fields only (no free-text flags); these options work **only on a server the instance admin made unrestricted** (instance admin only, name typed to confirm, audited as `server.unrestricted.enable`/`.disable`); refused with 422 `requires_unrestricted_server` on save, on a move to another server, on template install and at every deploy, rollback, reconcile and scale-up; the agent re-validates every option against the server policy carried in the deploy payload; never possible: host networking, `--ip`, restart and log options, host IPC, host binds, the Docker socket; hardening options (`capDrop`, `noNewPrivileges`, `readOnlyRootfs`, `pidsLimit`) and a curated list of namespaced sysctls are available everywhere ([Docker options](https://nebulactrl.com/docs/operations/docker-options.md)) | on an unrestricted server any deployer can take the host over; a device node, GPU driver or toolkit bug is a way out of the container; hardening is opt-in per service; host sysctls such as `vm.max_map_count` are only warned about in the plan |
| SSRF via git URL / registry | URLs are only used by the control plane for `git ls-remote` of the configured repository and by the agent for pulls; a registry host is contacted by the build and target agents (and plain-HTTP only on loopback) | the control plane connects to whatever host the URL names; restrict egress if needed |
| Command injection | typed commands, argument vectors, no shell | none known |
| Hostname hijack/conflict | global unique hostnames, 409 on conflict, DNS checks | DNS correctness is operator-owned |
| Bypassing Cloudflare by connecting to a proxied domain's server IP (skips its WAF, rate limits, bot rules and IP rules) | the **origin lock** ([Cloudflare](https://nebulactrl.com/docs/operations/cloudflare.md#origin-lock)), on by default for every proxied domain: Traefik answers `403` unless the TCP peer is in Cloudflare's published ranges, and, while the zone's Authenticated Origin Pulls is on, a TLS handshake without Cloudflare's client certificate (pinned CA) fails. Applied per route in Traefik, not by host firewall rules (Docker-published ports bypass `ufw`) | both layers prove "comes from Cloudflare", not "from **your** Cloudflare account": anyone who learns the IP can point their own zone at it and pass; hide the IP or use a Cloudflare Tunnel (no open port). Opting a domain out, DNS-only domains, tunnel domains, domains whose DNS NebulaCtrl does not manage, wildcard hosts and TCP/UDP routes are not locked. AOP is IP-only when the token lacks *Zone Settings: Edit* or the zone is Flexible/Off (shown in the panel) |
| A forged visitor address (`X-Forwarded-For`, `CF-Connecting-IP`) to slip past IP allow/deny rules, rate limits or to pollute analytics | the entrypoints trust forwarded headers only from Cloudflare's ranges (Traefik discards them from every other peer); on a locked domain the visitor is the **last** `X-Forwarded-For` entry (the address Cloudflare saw connect), entries to its left are never read, and a user-set `trustedProxyDepth` is ignored; the Cloudflare lock checks the TCP peer, the user's rules check the visitor, as two separate middlewares | on a domain that is **not** locked (DNS-only, opted out) the forwarded header is untrusted: rules see the connecting address, and a depth chosen by the operator is only as safe as the CDN in front |
| A stale, poisoned or unreachable Cloudflare IP list (lock too wide, or visitors blocked) | the list is fetched every 3 hours from Cloudflare's API with the published text files as fallback; an answer is refused unless it has valid CIDRs of the right family, none wider than `/8` (v4) or `/16` (v6), at least 10 v4 and 5 v6 ranges and no more than a 50% shrink; the last good list is kept on failure; a binary-embedded list covers first start and offline installs; every change is audited (`cloudflare.ips.updated`) and announced, and 24 hours of failures raise a notification | a range Cloudflare adds while fetching is down is blocked until the next successful fetch; a hostile answer that passes the plausibility checks would be accepted (the fetch is TLS to Cloudflare's hosts only, no credentials are sent) |
| A tampered or expired pinned origin-pull CA | embedded in the binary with its source URL and SHA-256 fingerprint pinned in code; start-up fails when they differ, and a unit test fails 180 days before the certificate expires (2029-11-01) | the CA is refreshed with a release, not at runtime |
| Over-reaching Cloudflare token (Zone Settings: Edit to turn on AOP) | the permission is optional and requested only for the origin lock's AOP layer; NebulaCtrl changes exactly one zone setting (`tls_client_auth`), only to turn it on and never off, and audits it (`cloudflare.zone.aop.enable`) | the token can do whatever its permissions allow in the Cloudflare account; limit its Zone Resources |
| Lost/changed secret key | a key ring (previous keys keep old values readable), `rekey`, documented backup of the key apart from the database | losing every key that sealed a value loses that value; no recovery |
| Control plane outage | agents keep workloads running, reconcile on reconnect | no HA |

## Hardening checklist

1. Serve the control plane only over HTTPS; keep `NEBULACTRL_PUBLIC_URL` `https://`.
2. Keep Postgres unpublished (the compose file does); encrypt the volume at rest if your threat model needs it.
3. Deliver `NEBULACTRL_SECRET_KEY` as a file (`NEBULACTRL_SECRET_KEY_FILE`, mode 0400) or from a vault rather than an environment variable, and back it up apart from the dumps. Rotate it with `rekey` ([secrets at rest](https://nebulactrl.com/docs/security/secrets-at-rest.md#rotating-the-key)).
4. Create tokens with the narrowest scopes and a project restriction, with an expiry; revoke unused ones.
5. Keep MCP off unless needed; if on, start with the *MCP read-only* preset, leave **Let MCP clients change production without approval** off, and withhold `deployments:write` from agents that read untrusted logs or repositories.
6. Restrict who can reach the control plane (firewall/VPN) when possible; agents only need outbound access to it.
7. Update Docker, the host OS and the agent regularly; use hosts dedicated to NebulaCtrl workloads.
8. Review the audit log (**Settings → Audit log**).
9. Deploy only code and images you trust; pin image digests for critical services.
10. Rotate affected secrets after revoking a suspected host.
11. Turn on two-step verification for your own account and require it for the team (**Settings → Security**); keep the recovery codes offline. Leave open sign-up off and set allowed email domains on every single-sign-on provider ([login methods](https://nebulactrl.com/docs/security/login-methods.md)).
12. Block outbound internet for every service that does not need it (managed databases start with it on) and keep shared networks few and named for a purpose ([networking](https://nebulactrl.com/docs/operations/networking.md)).
13. If you use build servers: dedicate them (role *Builds*), keep them patched and free of other workloads, give the push credential the least scope the registry allows (a bot account, a repository prefix), use a separate read-only pull credential, leave **Allow previews** off, do not enable "delete untagged" lifecycle rules in a BYO registry, and revoke a suspect build server like any host ([build servers](https://nebulactrl.com/docs/operations/build-servers.md)).
14. If you use the [server mesh](https://nebulactrl.com/docs/operations/mesh.md): leave Tailscale on the **advisory** ACL mode unless you accept that the control plane then holds a token that can rewrite your whole policy, tighten an allow-all tailnet, keep a dedicated tag (`tag:nebula-mesh`) and review the generated rules (**Settings → Mesh → Preview rules**); remember that managed Headscale uses Tailscale's public DERP relays. For [read replicas](https://nebulactrl.com/docs/operations/database-replicas.md), keep backups of the primary (a replica is not one) and rehearse a Promote on a non-production environment first.