Skip to content

The app

Open as Markdown

The API is not the only way in. SuperWitness serves a web app from the same binary and address, so you can browse reported runs, open one, record a verdict against a rubric and revise it without building a client first. People on an allowlist sign in with their AgentPod account; agents and scripts keep using the HTTP API and the MCP tools with bearer tokens.

The app is off until someone is allowed to sign in. Four settings turn it on (Configuration):

Setting Meaning
SW_ALLOWED_PRINCIPALS comma-separated AgentPod principal ids (prn_…) who may sign in. Empty, nobody may, and the app is off.
SW_APP_CLIENT_ID the AgentPod OAuth client browsers sign in through.
SW_SESSION_SECRET_FILE a file holding at least 32 random bytes, readable by SuperWitness alone. It keys the short-lived cookie that carries a sign-in to AgentPod and back.
SW_TRUSTED_PROXIES addresses or CIDR prefixes allowed to say who a client is with CF-Connecting-IP. Unset, it is this host’s own addresses.

SW_PUBLIC_URL must be https when sign-in is on: the app’s cookies are Secure, and a browser drops them over plain http. (Fake mode, which listens on loopback only, is the exception.) Make the secret with, for example:

Terminal window
head -c 48 /dev/urandom | base64 | sudo tee /etc/superwitness/session-secret >/dev/null
sudo chmod 0600 /etc/superwitness/session-secret && sudo chown superwitness /etc/superwitness/session-secret

The cookie key is now private to the service. Once the OAuth client has the callback and audience below, an allowlisted person can complete sign-in instead of meeting a very secure dead end.

At AgentPod, the OAuth client named by SW_APP_CLIENT_ID must list {SW_PUBLIC_URL}/auth/callback as a redirect URI and SW_PUBLIC_URL as an audience. SuperWitness also needs its own service credential to hold evidence:read, as it already does to read attempts: it uses it to look up who signed in.

Sign in sends the browser to AgentPod, which signs the person in if they are not already and sends them back to /auth/callback with a one-time code. SuperWitness trades the code for a token server to server, with PKCE, checks the token, and looks up the AgentPod principal behind it. It then checks that the principal is a person, is not suspended, and is on SW_ALLOWED_PRINCIPALS. Anyone else, including other valid AgentPod accounts, agents and services, sees Not authorised. Each refusal is logged as auth.signin_refused with a reason, for example not_allowlisted, login_missing (the sign-in cookie was absent) or no_code (the callback carried no code). A sign-in that succeeds is logged as auth.signin with the principal. The log never holds a token or a cookie.

With SW_ORG_PLANE_ISSUER set, the app signs people in through the organization plane as the OAuth client named by SW_APP_CLIENT_ID (the plane registers it with the redirect SW_PUBLIC_URL/auth/callback). Both the authorize and the token request carry resource=SW_PUBLIC_URL, and the authorize asks for scope=openid offline_access. The token’s sub is the person’s prn_ id, checked against SW_ALLOWED_PRINCIPALS with no lookup. Three more refusal reasons apply: product_not_enabled (the person’s workspace has not enabled SuperWitness), issuer_unavailable (the plane did not answer) and no_refresh_token (the plane issued no refresh token). The pages then name the plane’s host instead of AgentPod.

Under the plane the session holds the plane’s refresh token, server side: it is encrypted in the database with a key derived from the session cookie, which the database never holds, and it never reaches the browser. That makes SuperWitness one of the person’s connected apps at the plane, and the session lives only as long as that grant:

  • when the plane’s access token is past its five-minute life, the next request refreshes it with the stored refresh token and keeps the rotated one. Requests on one session share a single refresh. If the new refresh token cannot be saved, the request answers 503 and a retry within the plane’s 30-second replay window gets the same answer back;
  • if the plane refuses the refresh (the person signed SuperWitness out at the plane, left the workspace, or the workspace turned SuperWitness off), the session ends at that request and the app shows the sign-in screen (auth.session_ended, reason grant_refused);
  • if the plane cannot be reached, the session carries on, retrying every 30 seconds and logging auth.refresh_unavailable. It ends if the plane is still unreachable 10 minutes after the first failure (reason issuer_unreachable);
  • a session signed in before sessions held a grant ends at its next request (reason no_grant); signing in again is silent while the plane’s own session lasts.

The sign-in itself rides on a signed __Host-sw_login cookie that lives ten minutes and is used once. A signed-in browser then holds a __Host-sw_session cookie and a SuperWitness session, never an AgentPod token. Over plain http (fake mode only) the cookies are sw_login and sw_session, without the __Host- prefix.

  • the session lasts at most 12 hours, and ends after 2 hours unused;
  • every request checks the allowlist again, so taking someone off it (and restarting SuperWitness) ends their access at their next click;
  • Sign out (POST /auth/logout) ends the session at once and clears the cookie. Under the plane it first revokes the session’s refresh token at the plane’s revocation endpoint (found through the plane’s discovery document, and used only on the plane’s own origin); a revocation that fails is logged as auth.revoke_failed and the session ends all the same. If the database cannot be reached the answer is 503 store_unavailable, with the cookie cleared all the same, and the session expires on its own. A sign-out is logged as auth.signout with the principal, and one the database refused as auth.signout_failed. Signing in again from a browser that still holds a session ends the old one first, logged the same way with reason signed_in_again;
  • sessions do not need AgentPod: if AgentPod is down, open sessions keep working and only new sign-ins wait.

Changes made with a session must come from the app’s own page (their Origin must equal SW_PUBLIC_URL); anything else is refused with 403 origin_mismatch. Requests with a bearer token are not affected, and a request that carries a bearer token is judged by the token alone. Run reports (POST /v1/runs) and /mcp never accept a session.

  • Runs: every run sources have reported, newest first, grouped under Today, Yesterday and earlier dates in your own time zone. The sidebar has All, Needs verdict (finished runs no verdict names yet), Failed and Waiting, and a view per board. Each card shows the title, the status, the board, who ran it, how long it took or when it started, and its latest verdict. The filters are in the address, so a view can be bookmarked. With no runs yet, the page links to the Run registry API.
  • A run: its source and reference, title, status and Record verdict; a strip with the agent, attempts, errors, duration and cost; and tabs for the trace (a span waterfall, 500 spans at a time), logs (with a level filter), errors, verdicts (with revisions shown as history: a superseded verdict is struck through and links to the one that replaced it) and attempts (each with its configuration fingerprint). A source that did not answer shows as unavailable, and a value no source supplied reads unknown, never 0. A run missing from the list still opens by its address, /runs/superpipeline/{board}/{run}.
  • Rubrics: every rubric version, and each one’s scale and text. Rubrics are added with superwitness rubric-add.

The sidebar’s dot is green when every source answers /health, amber otherwise; hover for which. The app has four vibes (Daylight, Paper, Studio and Quiet), each in light and dark. Open the account chip, then Appearance, to choose the vibe, to follow the sun, stay light or stay dark, and to pick an accent; the choice is kept for this browser. On a phone the sidebar folds into a menu.

Click a span’s name in the Trace tab for its details, including the request and response where the span maps to the agent’s session; the Transcript tab reads the attempt as a conversation. Either can be cited in a verdict. Transcripts has the details.

Record verdict opens a drawer. Choose the subject (the run, or one of its attempts) and a rubric, then give the value the rubric’s scale asks for (rubric scales): a decision, a score from 0 to 1, a label, or text. Add a comment of up to 10,000 characters if you like. Spans ticked in the Trace tab, up to 100, go with it as evidence. A rubric whose scale the app cannot offer an input for is listed but cannot be chosen.

The drawer makes its idempotency key when it opens, so a double click or a retry records one verdict. Revise appears on your own latest verdicts: it opens the drawer with the same subject and rubric and records a verdict that supersedes the old one. The app explains refusals in plain words, for example “You’ve already revised this verdict” when someone revised it first.

After you save, the verdict appears on the run and in the registry’s latest-verdict view. A revision keeps the old record visible in history, so correcting a judgement does not airbrush the first one.

SuperWitness limits requests in its own process, by client address or by principal:

What Limit
/auth/* 10 a minute per client address, bursts of 20
changes made with a session 30 a minute per person, bursts of 10
POST /v1/runs 600 a minute per reporter, bursts of 100

Over a limit the answer is 429 rate_limited with Retry-After. The client address is the connection’s, unless the connection comes from SW_TRUSTED_PROXIES, in which case it is the CF-Connecting-IP header.

SuperWitness serves the app, the API and /mcp on one address. To publish the app through a tunnel or a reverse proxy, point the proxy at SuperWitness from an address in SW_TRUSTED_PROXIES and have it set CF-Connecting-IP. A request that arrives that way, through the public edge, is answered 404 on /mcp before any authentication, so the MCP tools stay on your private network. Callers on the private network, which carry no CF-Connecting-IP, are unaffected. Set SW_PUBLIC_URL to the public https address, and add that address as an audience to every AgentPod client whose tokens call SuperWitness, before you change it, so that tokens carry both the old and the new audience during the switch.

What What you see
AgentPod, during sign-in “AgentPod sign-in is unavailable”; open sessions keep working
a source that run’s tab says it is unavailable; the sidebar dot turns amber
SuperWitness’s database “Can’t reach the database”
a link to a run no source knows “Run not found”