Log history and log drains

Runtime logs (what your containers write to stdout and stderr) are stored on the server that runs them and can be shipped from that server to Loki, OpenTelemetry, syslog or any HTTP intake. The control plane stores no runtime log line: it asks the server's agent.

Log history#

Every agent captures the output of every managed service container, continuously. A viewer is not needed: closing the Logs tab discards nothing. Build and deploy output are separate: they are stored per deployment (10,000 lines), see deployments.

Limits take effect at once; lowering them deletes the oldest segments within a minute. Turning capture off stops storing new lines; what is stored stays until it expires, and turning capture on again continues from where Docker still has the lines.

A container's output from before the agent first saw it is taken from Docker's own log (at most the retention window), so history starts with what Docker still has. Docker's own log rotation (max-size on the container) is independent of this store.

Reading it#

Service → Logs and Environment → Observability have a Live | History switch and a time range (last 15 minutes … 7 days, or a custom range). History uses the same search box as the live view: the query is sent to the server and evaluated there, so searching 7 days does not move 7 days of log over the wire. The query language is the one documented in the search box: free text, "phrases", level:error, -service:web, @status:[400 TO 499], @path.to.field:value for JSON and logfmt lines, OR/NOT/parentheses, wildcards. The Go implementation on the agent is checked against the web one with a shared fixture (tests/fixtures/logquery/cases.json), so the same query matches the same lines in both.

API: GET /api/v1/services/{serviceId}/logs/history and GET /api/v1/environments/{environmentId}/logs/history (scope projects:read):

The answer has items (ts, line, serviceId, serviceName, replica, deploymentId, stream), nextCursor when more may follow, oldest/newest (what the server still has), capturing, and partial: true when the agent stopped scanning before the page was full (a rare query over a lot of log; keep paging with the cursor). Secret values are masked like in the live view.

History is unavailable while the server is offline. The service endpoint answers 503 server_offline and the UI says "History is stored on <server> and is unavailable while it is offline"; the environment endpoint lists the affected services in unavailable and still returns the others. An agent older than this feature answers 409 agent_outdated.

A service that moved to another server keeps its older history on the old one; the API reads the server the service is placed on now.

Downloading logs#

To attach a log to a bug report or keep a copy, download it as a file. Secret values are masked exactly as in the log views, and every download is audited (service.logs.download, deployment.logs.download); the audit entry records the range and format, never the content.

Query parameters: format (txt, default, one <time> <line> per line, with [build], [deploy] or [debug] added to deployment lines (runtime lines carry none); or ndjson, one {ts, stream, line, …} object per line), from and to (RFC 3339, from inclusive, to exclusive), q (the log search language, service logs only), component (Compose stacks, service logs only), streams (deployment logs only). Service lines come oldest first. A service download holds at most 200,000 lines: a longer range ends with a # download truncated … line (text) or {"note": …} object (NDJSON), so narrow the range. A read error after the first page ends the file with the same kind of note, because the bytes already sent cannot become an error response.

nebulactrl services download-logs acme/production/web --from 24h -o web.log
nebulactrl services download-logs web --format ndjson --grep 'level:error' > errors.ndjson
nebulactrl deployments download-logs 6f1c0d5e-… --streams all -o deploy.log

--from/--to take an RFC 3339 time, a date, or a duration ago (90m, 24h). Without -o the file goes to stdout; with it the file is created with mode 0600, and only once the server answered, so a refusal leaves nothing behind.

Log drains#

A drain ships lines from each agent, from its own host, to a destination. Drains are configured per team (Settings → Log drains, scope admin), optionally narrowed to projects, services and a query.

• Credentials are send-time secrets: sealed in the database, never returned by the API (a mask and the header names only), delivered to the agents with the configuration and never persisted in a command. They are masked from every error message.
• Secrets in lines: before a line leaves the host, the agent masks the secret values of that service (variables, git token, deploy key, registry credentials; 6+ characters, also their URL-escaped and base64 forms) with ••••••. The values come with the configuration and are refreshed at every deploy; a secret you changed and have not deployed yet is not known to the agent. A drain's query is matched against the masked line.
• Start: a new drain ships what is written after it was saved, not the stored history.
• Delivery: batches of up to 500 lines or 1 MB, at least once (a crash can repeat the last two seconds). A failing destination is retried with exponential backoff (1 s to 5 min, honouring Retry-After). The log store is the buffer: the drain keeps a position per service and reads from the disk, so a destination that is down for hours loses nothing as long as the lines are still stored and the backlog fits the buffer bound (default 256 MB per server and drain; the oldest lines are skipped and counted as dropped past it). A destination that rejects a batch for good (HTTP 400, 413, 422, other non-retryable 4xx) has that batch dropped and counted, so one bad record cannot block the drain; 401, 403, 408, 425, 429 and 5xx are retried.
• Health: Settings → Log drains shows, per server, Delivering, Retrying (the last delivery failed), Failing (five failures in a row) or Disabled, the last error (masked), lines sent and dropped, and the backlog on disk. Agents report every 30 seconds.
• Test: Send a test line delivers one line from a connected server (or the one you pick), which is where reachability and credentials count.
• A drain ships from every server of the team that hosts matching services. A server that is offline ships nothing until it reconnects; its backlog waits on its disk.

Quick reference#

# Loki with basic auth, only errors of one project
curl -X POST $URL/api/v1/log-drains -H "Authorization: Bearer $TOKEN" -d '{
  "name":"grafana","kind":"loki",
  "config":{"loki":{"url":"https://logs-prod.grafana.net/loki/api/v1/push","tenantId":"12345"}},
  "secrets":{"username":"12345","password":"glc_…"},
  "allProjects":false,"projectIds":["<project-id>"],"query":"level:error"}'

# Datadog through the generic HTTP drain
curl -X POST $URL/api/v1/log-drains -H "Authorization: Bearer $TOKEN" -d '{
  "name":"datadog","kind":"http",
  "config":{"http":{"url":"https://http-intake.logs.datadoghq.com/api/v2/logs","extraFields":{"ddsource":"nebulactrl"}}},
  "secrets":{"headers":{"DD-API-KEY":"…"}}}'

curl -X POST $URL/api/v1/log-drains/<id>/test -H "Authorization: Bearer $TOKEN" -d '{}'

Changes are audited (log_drain.create|update|delete|test, server.log_settings.update); credentials appear only as "changed".

Limits to know#

• Only stdout/stderr of containers Docker still has: a container that logs to a file inside itself is not captured. Stdout and stderr share one stream (no separate stderr facet).
• Lines are ordered by arrival at the agent; lines of two replicas can be a few milliseconds out of time order. A page is sorted by time, paging is by arrival.
• History and drains need an agent of this version: an older one answers history requests with 409 agent_outdated. Update agents with the install script.
• There is no log search across servers in one scan beyond the environment view (one request per service, merged by time) and no full-text index: a query scans the compressed blocks of its time range (blocks outside the range are skipped by the time index; a scan is bounded to about 256 MB of decompressed log or 4 seconds, then returns a cursor to continue).
• HTTP access lines of the proxy are not stored here: see traffic analytics.