Skip to content

Operations

Open as Markdown

Day to day, the server has only two things to operate: one binary and one Postgres database. The server keeps no state on disk; telemetry lives in the engines, and everything it reads from AgentPod and SuperPipeline stays there. The optional canary does write a tracker to the path passed with --results. That small footprint is useful, but we will still be precise about what to health-check, migrate, back up and restart.

Command Does
superwitness serve runs the service; also what runs with no command
superwitness migrate runs pending migrations once and exits
superwitness rubric-add records a rubric version (Verdicts)
superwitness canary the live acceptance test: canary run, canary record-alert and canary status (The canary)
superwitness skill prints the operations skill for agents; superwitness skill install writes it to ~/.claude/skills/superwitness-ops/SKILL.md, or under --dir <dir> instead
superwitness version prints the version; --version and -version do the same

suwi is the same binary under a short name, so suwi skill install is superwitness skill install.

serve exits 2 when its settings are missing or invalid: missing settings are all named in one message, and any other invalid setting stops it at the first one found (Configuration). It exits 1 when it cannot start for another reason, such as an unreadable secret file. It stops cleanly on SIGINT or SIGTERM, giving requests in flight up to 10 seconds. It logs JSON to standard output.

GET /health needs no token. It answers 200 whenever the process can answer at all, so a monitor can tell “SuperWitness is down” from “a source is down”. Each source’s state is in the body:

{"sources":{"agentpod":"ok","logs":"ok","superpipeline":"ok","traces":"ok","verdicts":"ok"},"status":"ok","version":"v0.0.12"}
  • status is always ok, and version is the running release.
  • Each of agentpod, logs, superpipeline and traces is a GET /health against that product or engine: any 2xx is ok, 401 or 403 unauthorized, anything else unavailable. Each check gets 1 second, after which it reads timeout. The engine checks carry the SW_TRACES_TOKEN_FILE or SW_LOGS_TOKEN_FILE token when one is set.
  • verdicts is a ping of SuperWitness’s own Postgres. It reads unavailable until migrations have succeeded.

To alert on a source that keeps failing during real reads, use the superwitness.source.fetches metric (Sending telemetry).

serve answers at once and runs its migrations in the background, over SW_MIGRATE_DATABASE_URL, else SW_DATABASE_URL. While they fail it retries, waiting 1 second and doubling to at most 30, and logs each failure. Until they succeed:

  • /health shows verdicts as unavailable;
  • run documents show sources.verdicts as unavailable, and everything else in them as usual;
  • POST /v1/verdicts answers 503 store_unavailable, which is retryable, and the record_verdict tool returns the same error.

A Postgres that is down never takes run documents with it.

SuperWitness uses two Postgres roles, set up in Install:

  • superwitness_owner owns the database and the tables, and is used only for migrations, through SW_MIGRATE_DATABASE_URL.
  • superwitness_app is the runtime role, through SW_DATABASE_URL, with SELECT and INSERT on verdicts and rubrics, SELECT, INSERT and UPDATE on runs (the run registry), and SELECT, INSERT, UPDATE and DELETE on sessions (app sign-ins), and nothing else (the grants).

The split keeps the running service from rewriting verdicts through SQL: it does not own the tables, so it cannot disable or drop the triggers that refuse UPDATE, DELETE and TRUNCATE. It does not protect against a compromised process that holds the owner’s password, so if SW_MIGRATE_DATABASE_URL is in the service’s settings file, the service holds that password. To keep it out:

  1. leave SW_MIGRATE_DATABASE_URL out of /etc/superwitness/env, and set it only in your own environment when you run superwitness migrate;

  2. as the owner, grant the runtime role read access to the migration bookkeeping table:

    GRANT SELECT ON goose_db_version TO superwitness_app;

serve then runs its background migration pass as the runtime role. With that grant, the pass succeeds when nothing is pending; without it, the pass always fails. When a migration is pending it fails either way, and verdicts stay unavailable until you run superwitness migrate as the owner.

The owner password is now outside the long-running process, while startup can still prove that the schema is already current. A pending migration fails closed for verdicts rather than quietly running with the wrong role.

  1. Download and verify the new release as in Install.

  2. Run the new binary’s migrations as the owner, before restarting:

    Terminal window
    SW_MIGRATE_DATABASE_URL='postgres://superwitness_owner:<password>@127.0.0.1:5432/superwitness?sslmode=disable' \
    superwitness_<version>_linux_amd64/superwitness migrate

    It exits 0 when the schema is up to date, and non-zero with the error otherwise. It gives up after 60 seconds if Postgres does not answer.

  3. Install the binary and restart: sudo systemctl restart superwitness.

Once the restart succeeds, the new binary is running against its matching schema. Check /health before you call the upgrade finished; it keeps a source problem separate from a process that did not come back.

A migration that adds a table needs a grant of its own for the runtime role: nothing is granted through ALTER DEFAULT PRIVILEGES.

SuperWitness’s only state is plain Postgres: the superwitness database, with its verdicts and rubrics tables, the run registry in runs, and app sign-ins in sessions. Back it up with the usual tools, for example as the owner role or the Postgres superuser:

Terminal window
pg_dump -Fc -f superwitness.dump 'postgres://superwitness_owner@127.0.0.1:5432/superwitness'

That dump contains the server’s database state. If you run the canary, back up its --results tracker separately. The short-lived telemetry engines do not belong in either backup.

Use the owner or the superuser: the runtime role cannot read the migration bookkeeping table unless you granted it above.

The telemetry engines are not SuperWitness’s to back up. Telemetry is lossy by design, sampled and kept for a short time, and SuperWitness never treats a missing span as missing evidence. Evidence that belongs to AgentPod or SuperPipeline stays in those products, not in SuperWitness’s database.

SW_LISTEN is the address serve binds. Left unset it is :8790, every interface. The Install guide sets 127.0.0.1:8790; to reach it from other machines, bind a private address of the host instead, or keep loopback and put a reverse proxy in front.

In fake mode (SW_FAKE_SOURCES=1) SuperWitness accepts development tokens, so it refuses to start unless SW_LISTEN is a loopback address: localhost, a 127.x.x.x address or [::1], with a port. Left unset in fake mode, it is 127.0.0.1:8790.

A client has 5 seconds to send a request’s headers.