Skip to content

The canary

Open as Markdown

superwitness canary is the live acceptance test we run when “the services are up” is not a convincing ending. It creates one card on a SuperPipeline board, waits for an agent to run it, and then checks, as a black-box client of every product involved, that you can observe the run end to end. The canary is accepted once the first nine flows have been proven on real runs and flow 10 has been proven by a forced failure that reaches the alert channel.

It is meant to run on the SuperWitness host, nightly from a timer and by hand when needed. It keeps its history in a JSON-lines file, the tracker, which it appends one result to per run.

Flow Proves
1 The run document has the run, its card, the work stage and its gates, from SuperPipeline’s run evidence route
2 The AgentPod ledger has the run’s attempts, each with a fingerprint whose harness is SWC_EXPECT_HARNESS
3 The run’s trace is joined, with the hub’s dispatch, attempt and turn spans in it
4 The agent’s node continued the trace the hub started
5 A SuperPipeline API span carries the run’s run.id
6 The gate on SWC_GATE_STAGE is shown pending, and a later run sees it decided by a human who is not the run’s agent
7 record_verdict over MCP records a verdict, and a retry with the same idempotency key returns the same one
8 A synthetic error line, sent to SWC_OTLP_LOGS_URL with the run’s id, shows up in the run’s logs and errors
9 Two markers planted in the card’s prompt reach neither VictoriaLogs nor VictoriaTraces
10 A failure reaches your alert channel (see record-alert)

Flow 6 needs a person to decide the gate, so it is usually proven by a later run than the one that saw the gate pending. The canary looks back at up to seven earlier runs for it.

Terminal window
superwitness canary run --trigger nightly --results /var/lib/superwitness/canary.jsonl
Flag Meaning
--trigger nightly or manual (the default)
--force-fail record a deliberate failure without creating a card or calling any product, to prove flow 10
--results the tracker to append the result to; required
--result-out also write this run’s result, as indented JSON, to this file

It prints one line, canary PASS … or canary FAIL …, and exits 0 on a pass, 1 on a failure, and 2 when its flags are wrong or the tracker cannot be read. A result that cannot be appended to the tracker counts as a failure. The whole run is bounded at 70 minutes.

The line tells you whether this run passed; the tracker preserves which flows it proved. One run need not perform a tiny miracle and prove all ten if earlier runs already supplied valid evidence.

Terminal window
superwitness canary record-alert --results /var/lib/superwitness/canary.jsonl --ntfy-id <id> --query '<how you checked>'

Your alerting wrapper calls this after the alert channel (ntfy) has accepted and stored the message for a forced failure. It records flow 10’s proof. All three flags are required, and the last result in the tracker must be a forced failure; otherwise it records nothing and exits 2.

Once recorded, the next status read can distinguish “we emitted an alert” from “the alert channel actually kept it.” That distinction is the entire reason flow 10 exists.

Terminal window
superwitness canary status --results /var/lib/superwitness/canary.jsonl

Prints which flows have been proven, when, and on which run, with the last run’s result. With --json it prints the same summary as one JSON object.

You now have a compact acceptance ledger: the latest result, each proven flow, and the run that earned it, rather than ten green boxes with no memory.

Flags may be written with one dash or two, as with every Go flag set.

The canary reads its own environment. Names it shares with the service keep the SW_ prefix and mean the same thing (Configuration); canary-only names start with SWC_. deploy/canary/canary.env.example in the repository is a starting file.

Variable Required Default Meaning
SW_SUPERPIPELINE_URL yes none SuperPipeline, where the canary creates its card
SWC_SUPERWITNESS_URL yes none SuperWitness itself, for /v1 and /mcp; usually loopback
SW_TRACES_URL, SW_LOGS_URL yes none VictoriaTraces and VictoriaLogs, for flows 5 and 9
SW_TRACES_TOKEN_FILE, SW_LOGS_TOKEN_FILE no none bearer tokens for the engines
SWC_BOARD_ID yes none the SuperPipeline board the canary’s cards run on
SW_HUB_URL without the plane none the AgentPod hub
SWC_HUB_CLIENT_ID without the plane none the canary’s own hub service credential id, svc_…
SWC_HUB_CLIENT_SECRET_FILE without the plane none a file holding that credential’s secret
SW_ORG_PLANE_URL no none the organization plane; setting it replaces the three hub settings with the two below
SWC_ORG_PLANE_SERVICE_CREDENTIAL_FILE with the plane none a file holding the canary’s plane service credential, svc_<id>:<secret>
SWC_SUPERWITNESS_AUDIENCE with the plane none SuperWitness’s SW_PUBLIC_URL, the audience of its tokens (not SWC_SUPERWITNESS_URL)
SWC_OTLP_LOGS_URL no http://127.0.0.1:4318/v1/logs the collector’s OTLP/HTTP logs endpoint, for flow 8’s synthetic error line
SWC_WORK_STAGE no work the board stage the card is created on and an agent claims
SWC_GATE_STAGE no review the board stage whose gate flow 6 watches
SWC_EXPECT_HARNESS no hermes the harness flow 2 expects in each attempt’s fingerprint
SWC_STANDARD no rubric:superwitness-canary@1 the standard flow 7’s verdict names; must be rubric:<id>@<version>
SWC_CLAIM_TIMEOUT no 10m how long an agent has to claim the card
SWC_RUN_TIMEOUT no 30m how long the run has to end completed
SWC_SETTLE_DELAY no 90s how long to wait after the run ends, for telemetry to arrive
SWC_DOC_TIMEOUT no 5m how long to keep fetching the run document until it is complete
SWC_POLL_INTERVAL no 15s the wait between polls

The five durations are Go durations and must be positive. A missing required setting, or a duration or SWC_STANDARD that does not parse, is recorded as a failed run whose error starts setup:, and canary run exits 1.

Like the service, the canary refuses a secret file that is empty or readable by group or others. Every call it makes has a timeout, so one hung product cannot keep it running past its bound.