The canary
superwitness canary is the live acceptance test we run when “the services are up” is not a
convincing ending. It creates one card on a SuperPipeline board, waits for an agent to run it, and
then checks, as a black-box client of every product involved, that you can observe the run end to
end. The canary is accepted once the first nine flows have been proven on real runs and flow 10
has been proven by a forced failure that reaches the alert channel.
It is meant to run on the SuperWitness host, nightly from a timer and by hand when needed. It keeps its history in a JSON-lines file, the tracker, which it appends one result to per run.
The ten flows
Section titled “The ten flows”| Flow | Proves |
|---|---|
| 1 | The run document has the run, its card, the work stage and its gates, from SuperPipeline’s run evidence route |
| 2 | The AgentPod ledger has the run’s attempts, each with a fingerprint whose harness is SWC_EXPECT_HARNESS |
| 3 | The run’s trace is joined, with the hub’s dispatch, attempt and turn spans in it |
| 4 | The agent’s node continued the trace the hub started |
| 5 | A SuperPipeline API span carries the run’s run.id |
| 6 | The gate on SWC_GATE_STAGE is shown pending, and a later run sees it decided by a human who is not the run’s agent |
| 7 | record_verdict over MCP records a verdict, and a retry with the same idempotency key returns the same one |
| 8 | A synthetic error line, sent to SWC_OTLP_LOGS_URL with the run’s id, shows up in the run’s logs and errors |
| 9 | Two markers planted in the card’s prompt reach neither VictoriaLogs nor VictoriaTraces |
| 10 | A failure reaches your alert channel (see record-alert) |
Flow 6 needs a person to decide the gate, so it is usually proven by a later run than the one that saw the gate pending. The canary looks back at up to seven earlier runs for it.
The commands
Section titled “The commands”canary run
Section titled “canary run”superwitness canary run --trigger nightly --results /var/lib/superwitness/canary.jsonl| Flag | Meaning |
|---|---|
--trigger |
nightly or manual (the default) |
--force-fail |
record a deliberate failure without creating a card or calling any product, to prove flow 10 |
--results |
the tracker to append the result to; required |
--result-out |
also write this run’s result, as indented JSON, to this file |
It prints one line, canary PASS … or canary FAIL …, and exits 0 on a pass, 1 on a
failure, and 2 when its flags are wrong or the tracker cannot be read. A result that cannot be
appended to the tracker counts as a failure. The whole run is bounded at 70 minutes.
The line tells you whether this run passed; the tracker preserves which flows it proved. One run need not perform a tiny miracle and prove all ten if earlier runs already supplied valid evidence.
canary record-alert
Section titled “canary record-alert”superwitness canary record-alert --results /var/lib/superwitness/canary.jsonl --ntfy-id <id> --query '<how you checked>'Your alerting wrapper calls this after the alert channel (ntfy) has accepted and stored the message for a forced failure. It records flow 10’s proof. All three flags are required, and the last result in the tracker must be a forced failure; otherwise it records nothing and exits 2.
Once recorded, the next status read can distinguish “we emitted an alert” from “the alert channel actually kept it.” That distinction is the entire reason flow 10 exists.
canary status
Section titled “canary status”superwitness canary status --results /var/lib/superwitness/canary.jsonlPrints which flows have been proven, when, and on which run, with the last run’s result. With
--json it prints the same summary as one JSON object.
You now have a compact acceptance ledger: the latest result, each proven flow, and the run that earned it, rather than ten green boxes with no memory.
Flags may be written with one dash or two, as with every Go flag set.
Settings
Section titled “Settings”The canary reads its own environment. Names it shares with the service keep the SW_ prefix
and mean the same thing (Configuration); canary-only names start with
SWC_. deploy/canary/canary.env.example in the repository is a starting file.
| Variable | Required | Default | Meaning |
|---|---|---|---|
SW_SUPERPIPELINE_URL |
yes | none | SuperPipeline, where the canary creates its card |
SWC_SUPERWITNESS_URL |
yes | none | SuperWitness itself, for /v1 and /mcp; usually loopback |
SW_TRACES_URL, SW_LOGS_URL |
yes | none | VictoriaTraces and VictoriaLogs, for flows 5 and 9 |
SW_TRACES_TOKEN_FILE, SW_LOGS_TOKEN_FILE |
no | none | bearer tokens for the engines |
SWC_BOARD_ID |
yes | none | the SuperPipeline board the canary’s cards run on |
SW_HUB_URL |
without the plane | none | the AgentPod hub |
SWC_HUB_CLIENT_ID |
without the plane | none | the canary’s own hub service credential id, svc_… |
SWC_HUB_CLIENT_SECRET_FILE |
without the plane | none | a file holding that credential’s secret |
SW_ORG_PLANE_URL |
no | none | the organization plane; setting it replaces the three hub settings with the two below |
SWC_ORG_PLANE_SERVICE_CREDENTIAL_FILE |
with the plane | none | a file holding the canary’s plane service credential, svc_<id>:<secret> |
SWC_SUPERWITNESS_AUDIENCE |
with the plane | none | SuperWitness’s SW_PUBLIC_URL, the audience of its tokens (not SWC_SUPERWITNESS_URL) |
SWC_OTLP_LOGS_URL |
no | http://127.0.0.1:4318/v1/logs |
the collector’s OTLP/HTTP logs endpoint, for flow 8’s synthetic error line |
SWC_WORK_STAGE |
no | work |
the board stage the card is created on and an agent claims |
SWC_GATE_STAGE |
no | review |
the board stage whose gate flow 6 watches |
SWC_EXPECT_HARNESS |
no | hermes |
the harness flow 2 expects in each attempt’s fingerprint |
SWC_STANDARD |
no | rubric:superwitness-canary@1 |
the standard flow 7’s verdict names; must be rubric:<id>@<version> |
SWC_CLAIM_TIMEOUT |
no | 10m |
how long an agent has to claim the card |
SWC_RUN_TIMEOUT |
no | 30m |
how long the run has to end completed |
SWC_SETTLE_DELAY |
no | 90s |
how long to wait after the run ends, for telemetry to arrive |
SWC_DOC_TIMEOUT |
no | 5m |
how long to keep fetching the run document until it is complete |
SWC_POLL_INTERVAL |
no | 15s |
the wait between polls |
The five durations are Go durations and must be positive. A missing required setting, or a
duration or SWC_STANDARD that does not parse, is recorded as a failed run whose error starts
setup:, and canary run exits 1.
Like the service, the canary refuses a secret file that is empty or readable by group or others. Every call it makes has a timeout, so one hung product cannot keep it running past its bound.