Operations
Day to day, the server has only two things to operate: one binary and one Postgres database. The
server keeps no state on disk; telemetry lives in the engines, and everything it reads from
AgentPod and SuperPipeline stays there. The optional canary does write a tracker to the path passed
with --results. That small footprint is useful, but we will still be precise about what to
health-check, migrate, back up and restart.
The commands
Section titled “The commands”| Command | Does |
|---|---|
superwitness serve |
runs the service; also what runs with no command |
superwitness migrate |
runs pending migrations once and exits |
superwitness rubric-add |
records a rubric version (Verdicts) |
superwitness canary |
the live acceptance test: canary run, canary record-alert and canary status (The canary) |
superwitness skill |
prints the operations skill for agents; superwitness skill install writes it to ~/.claude/skills/superwitness-ops/SKILL.md, or under --dir <dir> instead |
superwitness version |
prints the version; --version and -version do the same |
suwi is the same binary under a short name, so suwi skill install is superwitness skill install.
serve exits 2 when its settings are missing or invalid: missing settings are all named in one
message, and any other invalid setting stops it at the first one found
(Configuration). It exits 1 when it cannot start for another reason, such as an unreadable secret file. It stops cleanly on
SIGINT or SIGTERM, giving requests in flight up to 10 seconds. It logs JSON to standard
output.
Health
Section titled “Health”GET /health needs no token. It answers 200 whenever the process can answer at all, so a
monitor can tell “SuperWitness is down” from “a source is down”. Each source’s state is in the
body:
{"sources":{"agentpod":"ok","logs":"ok","superpipeline":"ok","traces":"ok","verdicts":"ok"},"status":"ok","version":"v0.0.12"}statusis alwaysok, andversionis the running release.- Each of
agentpod,logs,superpipelineandtracesis aGET /healthagainst that product or engine: any 2xx isok, 401 or 403unauthorized, anything elseunavailable. Each check gets 1 second, after which it readstimeout. The engine checks carry theSW_TRACES_TOKEN_FILEorSW_LOGS_TOKEN_FILEtoken when one is set. verdictsis a ping of SuperWitness’s own Postgres. It readsunavailableuntil migrations have succeeded.
To alert on a source that keeps failing during real reads, use the
superwitness.source.fetches metric (Sending telemetry).
Migrations at start
Section titled “Migrations at start”serve answers at once and runs its migrations in the background, over
SW_MIGRATE_DATABASE_URL, else SW_DATABASE_URL. While they fail it retries, waiting 1 second
and doubling to at most 30, and logs each failure. Until they succeed:
/healthshowsverdictsasunavailable;- run documents show
sources.verdictsasunavailable, and everything else in them as usual; POST /v1/verdictsanswers 503store_unavailable, which is retryable, and therecord_verdicttool returns the same error.
A Postgres that is down never takes run documents with it.
The two database roles
Section titled “The two database roles”SuperWitness uses two Postgres roles, set up in Install:
superwitness_ownerowns the database and the tables, and is used only for migrations, throughSW_MIGRATE_DATABASE_URL.superwitness_appis the runtime role, throughSW_DATABASE_URL, withSELECTandINSERTonverdictsandrubrics,SELECT,INSERTandUPDATEonruns(the run registry), andSELECT,INSERT,UPDATEandDELETEonsessions(app sign-ins), and nothing else (the grants).
The split keeps the running service from rewriting verdicts through SQL: it does not own the
tables, so it cannot disable or drop the triggers that refuse UPDATE, DELETE and TRUNCATE.
It does not protect against a compromised process that holds the owner’s password, so if
SW_MIGRATE_DATABASE_URL is in the service’s settings file, the service holds that password.
To keep it out:
-
leave
SW_MIGRATE_DATABASE_URLout of/etc/superwitness/env, and set it only in your own environment when you runsuperwitness migrate; -
as the owner, grant the runtime role read access to the migration bookkeeping table:
GRANT SELECT ON goose_db_version TO superwitness_app;
serve then runs its background migration pass as the runtime role. With that grant, the pass
succeeds when nothing is pending; without it, the pass always fails. When a migration is
pending it fails either way, and verdicts stay unavailable until you run superwitness migrate
as the owner.
The owner password is now outside the long-running process, while startup can still prove that the schema is already current. A pending migration fails closed for verdicts rather than quietly running with the wrong role.
Upgrades
Section titled “Upgrades”-
Download and verify the new release as in Install.
-
Run the new binary’s migrations as the owner, before restarting:
Terminal window SW_MIGRATE_DATABASE_URL='postgres://superwitness_owner:<password>@127.0.0.1:5432/superwitness?sslmode=disable' \superwitness_<version>_linux_amd64/superwitness migrateIt exits 0 when the schema is up to date, and non-zero with the error otherwise. It gives up after 60 seconds if Postgres does not answer.
-
Install the binary and restart:
sudo systemctl restart superwitness.
Once the restart succeeds, the new binary is running against its matching schema. Check
/health before you call the upgrade finished; it keeps a source problem separate from
a process that did not come back.
A migration that adds a table needs a grant of its own for the runtime role: nothing is granted
through ALTER DEFAULT PRIVILEGES.
Backups
Section titled “Backups”SuperWitness’s only state is plain Postgres: the superwitness database, with its verdicts
and rubrics tables, the run registry in runs, and app sign-ins in sessions. Back it up with the usual tools, for example as the owner role or the
Postgres superuser:
pg_dump -Fc -f superwitness.dump 'postgres://superwitness_owner@127.0.0.1:5432/superwitness'That dump contains the server’s database state. If you run the canary, back up its --results
tracker separately. The short-lived telemetry engines do not belong in either backup.
Use the owner or the superuser: the runtime role cannot read the migration bookkeeping table unless you granted it above.
The telemetry engines are not SuperWitness’s to back up. Telemetry is lossy by design, sampled and kept for a short time, and SuperWitness never treats a missing span as missing evidence. Evidence that belongs to AgentPod or SuperPipeline stays in those products, not in SuperWitness’s database.
The listen address
Section titled “The listen address”SW_LISTEN is the address serve binds. Left unset it is :8790, every interface. The
Install guide sets 127.0.0.1:8790; to reach it from other machines, bind
a private address of the host instead, or keep loopback and put a reverse proxy in front.
In fake mode (SW_FAKE_SOURCES=1) SuperWitness accepts development tokens, so it refuses to
start unless SW_LISTEN is a loopback address: localhost, a 127.x.x.x address or [::1],
with a port. Left unset in fake mode, it is 127.0.0.1:8790.
A client has 5 seconds to send a request’s headers.