# Tasks: cron jobs, one-off commands and the pre- and post-deploy commands Four ways to run a command next to a service. All of them run in a **throwaway container of the service's own image**, on the service's server, with the service's environment variables, on its environment network (so `postgres:5432` resolves) and the edge network. They publish no ports, never restart, use the service's memory and CPU limits, have a timeout, and the container is removed when the command ends, whatever its outcome. The command is an **argv array without a shell**. Use `["sh", "-c", "…"]` for shell syntax. It replaces the image's entrypoint (like `docker exec`, and unlike `docker run image args`), so an image with `ENTRYPOINT ["/app/server"]` runs `["/app/server", "migrate"]` as written. ## Cron tasks Per service, up to 20 tasks: `name`, `schedule`, `command`, `enabled`, `timeoutSeconds` (default 1800), `concurrency` (`forbid` or `allow`). - **Schedule**: five fields (`minute hour day-of-month month day-of-week`) or `@hourly`, `@daily`, `@weekly`, `@monthly`, `@yearly`. Parsed by the in-house `internal/cronexpr`. The finest resolution is one minute. - **Time zone**: each task has an IANA `timezone` (`Europe/Berlin`, `America/New_York`, …; default `UTC`, and every task created before the field existed stays `UTC`). The schedule is read as wall-clock time in that zone, so "every day at 03:00" stays at 03:00 across daylight-saving changes (the UTC instant moves by an hour). A wall time that does not exist (the spring-forward gap) is skipped for that day; one that happens twice (the autumn hour) runs once, the first time. An unknown zone is `422 invalid_timezone`. Set it with `timezone` on `POST /services/{id}/tasks` and `PATCH /tasks/{id}` (changing it recomputes the next run), `nebulactrl tasks create --timezone` / `tasks update --timezone`, or the **Time zone** field of the task dialog, which previews the next runs in the task's zone, in UTC and in your own zone. The list shows the zone next to the schedule. `POST /task-schedules/preview` takes `timezone` too. The control plane binary embeds the zone database, so zones work on minimal images. - A River periodic job evaluates due tasks every minute. A task that is due runs **once**, however long the control plane was away: missed ticks are not replayed. - `forbid` (default): while the previous run of the task is still active the tick is recorded as **skipped** ("the previous run is still running"). `allow` starts another run. A streak of identical skips is recorded once, so it never pushes real history out. - A tick is also skipped, with the reason, when the service is stopped, has never been deployed or its server is offline. - **History**: the newest 50 finished runs of each task (and 50 one-off runs per service) are kept with their output; older ones are deleted. Output is redacted with the service's secret values and capped at 10,000 lines. - **Run now** starts a run immediately, ignoring the schedule and `concurrency`. - An acked run whose agent never answers is failed after `timeout + 10 minutes` ("the agent never reported a result"). - **Notifications**: a failed or timed-out **cron** run (and runs started by features such as preview seeding) emits `task.failed`; runs a person started from the UI, CLI or MCP do not, you are looking at them. ## One-off commands `POST /api/v1/services/{id}/run {command, timeoutSeconds?}` (scope `deployments:write`, audited as `service.run_command` with the command line). Returns `202` with a run. Follow it with `GET /task-runs/{runId}/logs?follow=true` (server-sent events; `event: end` carries `{status, exitCode}`), or `GET /task-runs/{runId}` for the status and exit code. `409 not_deployed` / `server_offline` when it cannot start. - UI: the dock's **More** menu and ⌘K ("Run command in …"); the Tasks section of the service settings also has the button, plus the run history drawer. - CLI: `nebulactrl services run -- ` streams the output and **exits with the command's exit code**. `--detach` returns the run id. - MCP: `run_command` (needs `deployments:write`; waits up to `waitSeconds`, default 20, returns at most 500 redacted lines, 64 KB) and `get_task_run`. Docker exec level access is root-equivalent on that service's data, so the scope is the same one that can deploy. Output is untrusted application text. ## Pre-deploy command Service settings → Deploy → **Pre-deploy command** (`config.preDeployCommand`, `preDeployTimeoutSeconds`, default 600, at most 3600). Order of a deploy on the agent: pull or build the image → **pre-deploy** → stop the old container (recreate only) → create and start the new one → health check → route → verify. The pre-deploy container uses the **new image** and the deploy's exact environment. Until it succeeds nothing of the previous release has been touched, so for rolling **and** recreate deploys the old release keeps serving while migrations run. A non-zero exit (or a timeout) fails the deployment at the step `predeploy`; the message carries the exit code and the last 20 output lines, and says that the previous release is still serving. The output is in the deployment logs. The LED strip keeps its three LEDs (Build, Release, Verify): `predeploy` lights Release and the card shows a small "pre-deploy" label under it while it runs. It runs for every deployment that creates a **new release** (manual, staged, git push, preview). It does **not** run for rollbacks, reconcile redeploys or server drains: those re-run an existing release and must not touch data. Beware that migrations are not undone by a rollback. ## Post-deploy command Service settings → Deploy → **Post-deploy command** (`config.postDeployCommand`, `postDeployTimeoutSeconds`, default 600, at most 3600). It mirrors the pre-deploy command, but runs **after** the swap, for work that needs the new release to be live: cache warm-up, a search reindex, a smoke call. Order of a deploy on the agent: pull or build → pre-deploy → create and start → health check → route → verify → **retire the previous release** → **post-deploy**. The post-deploy container is a one-shot container of the **new image** with the deploy's exact environment, on the environment network (it reaches the live service by its name, without an alias of its own). It uses the same memory and CPU limits as the service and is removed when it ends. **A failure only warns.** The new release is already serving, so a non-zero exit, a timeout or a command that cannot start never fails or rolls back the deployment. Instead: - the deployment is `succeeded`, and its message ends with `Warning: post-deploy command failed with exit code N`; - the timeline shows the step **Post-deploy** with the status `warning` (a triangle) and the sentence "The new release is live; the deployment was not rolled back." The command's output is in the deployment logs; - a notification goes out as `task.failed` with severity **warning** (title "Post-deploy command failed on "). Channels that subscribe to task failures at the `warning` level or lower receive it; the deployment itself is still announced as `deploy.succeeded`. Like the pre-deploy command it runs for every deployment that creates a **new release** (manual, staged, git push, preview), not for rollbacks, reconcile redeploys or drains, and a service that is itself a one-shot has none. The deployment stays in `verifying` until the command ends (up to its timeout), so a new deploy of the same service is refused as in progress until then. ## Operational notes - The agent finds the image from the service's running container on the host, else from the release (pulling it with the service's registry credentials). - Containers are named `nc-task--` / `nc-predeploy--` / `nc-postdeploy--` and carry `nebulactrl.task-run`, `nebulactrl.task-service`, `nebulactrl.task-kind` labels, but **not** `nebulactrl.service`: they are never replicas. The agent sweeps leftovers (exited ones older than 10 minutes, running ones far past their deadline) on every reconcile. - A redelivered `run_task` command (agent restarted mid-run) **adopts** the existing container instead of running the command a second time. If the agent shuts down while a task runs, the container is left running and adopted after restart; the logs written during the gap are not replayed. - Command payloads persisted in `agent_commands` hold the command line and the variable **keys** only; values are attached at send time. Do not put secrets into the command line itself (they would be stored), use variables. - `GET /services/{id}/task-runs`, `/task-runs/{id}` and the output need `projects:read`; creating, editing and deleting tasks needs `projects:write`; running needs `deployments:write`.