NFLTR›Docs›Guides

From Alert to Approved Fix

An alert fires. The hub receives it on its hook, sends read-only agents to the machines that may hold part of the answer, and proposes one fix as an exact prompt for one machine. The owner signs that prompt; the fix runs there and nowhere else; the hub's record shows which agent ran what, on which machine.

The incident on this page: checkout fails every request after release r42, which cut its upstream timeout from 2000 ms to 20 ms. Three machines take part, each joined as a node with its own access: checkout-vm, the server running checkout and the only machine that may change; cluster-eu, a read-only view of the cluster where payments, checkout's upstream, runs; and ci-runner, the deploy job's log. The owner's approval key stays on the owner's machine.

Every command below runs as written on one computer, where three directories stand in for the three machines. That is for a first look only: on one computer the nodes run as the same user that holds the approval key, which a real setup never does (see In production).

1. The machines

The stand-ins: checkout's releases and its operator tool checkoutctl (status reads, rollback changes the running release), the cluster's view of payments, the deploy log, and the alert Alertmanager would send.

mkdir -p checkout/releases cluster ci
printf 'release=r41\nupstream_timeout_ms=2000\n' > checkout/releases/r41.env
printf 'release=r42\nupstream_timeout_ms=20\n' > checkout/releases/r42.env
cp checkout/releases/r42.env checkout/current.env
cat > checkout/checkoutctl <<'EOF'
#!/bin/sh
cd "$(dirname "$0")"
case "$1" in
status)   cat current.env ;;
rollback) cp "releases/$2.env" current.env &&
          echo "$(date -u +%FT%TZ) rollback to $2" >> actions.log ;;
*)        echo "usage: checkoutctl status|rollback RELEASE" >&2; exit 2 ;;
esac
EOF
chmod +x checkout/checkoutctl
echo 'payments: 3/3 ready, p99 180 ms' > cluster/payments.txt
echo 'deploy-checkout #142: r42 rolled out; PR #88 set upstream_timeout_ms 2000 -> 20' > ci/deploy-142.log
echo '{"status":"firing","alerts":[{"labels":{"alertname":"CheckoutErrorRate","service":"checkout","instance":"checkout-vm"},"annotations":{"description":"POST /checkout error rate 100% over 2m"}}]}' > alert.json

2. The owner's key

On the owner's machine, make the approval key pair once. The private half stays in the file (mode 0600); the printed public half goes to the node that may change things.

$ nfltr orch hub approval-key --out owner.key
nfltr-approval-pub-v1:3q2+7w...

On a Mac, --signer keychain keeps the private key in the keychain instead, and each signature asks for Touch ID.

3. Join the machines as nodes

The node on checkout-vm gets a tool policy: a command that rolls checkout back is denied unless the owner approved that turn's exact prompt (unless_approved). The node checks the grant against the owner's public key; the prompt has no say.

cat > checkout-policy.json <<'EOF'
{"pre_tool_use":[
  {"tool":"Bash","input_field":"command","deny_regex":"\\bcheckoutctl\\b.*\\brollback\\b",
   "reason":"checkout changes need the owner's approval","unless_approved":true}
]}
EOF

One node per machine, each told what it can reach. checkout-vm also takes work only from the hub incident (--serve-hubs) and only when a spawn names it (--require-pin), so no other hub and no loosely routed task lands there.

$ nfltr node join --name checkout-vm --labels env=linux-vm --harness claude-code --allow-all-tools --max-agents 1 \
    --machine-description "checkout service: checkoutctl status (read), checkoutctl rollback RELEASE (write), in data dir checkout" \
    --data-dir checkout=checkout --tool-policy-file checkout-policy.json --approval-public-key "$PUB" \
    --serve-hubs incident --require-pin
$ nfltr node join --name cluster-eu --labels env=k8s --harness claude-code --allow-all-tools --max-agents 1 \
    --machine-description "read-only view of cluster eu: payments (checkout's upstream), in data dir cluster" \
    --data-dir cluster=cluster
$ nfltr node join --name ci-runner --labels env=ci --harness claude-code --allow-all-tools --max-agents 1 \
    --machine-description "CI: the deploy-checkout job's logs, in data dir ci" \
    --data-dir ci=ci

Each runs until stopped. More on nodes, labels and policies in Join machines as nodes.

4. Start the hub

The hub's standing instructions are yours to write. These say how to handle an alert, not what the cause or the fix is:

cat > runbook.md <<'EOF'
Handle each alert from this hub's Alertmanager hook.
- Investigate through agents on the nodes in list_nodes, one per environment
  that may hold part of the answer. Investigation is read-only.
- Find the change that caused the alert. Propose one fix as an exact prompt
  for one node with request_approval, then wait for the owner's grant. Spawn
  exactly that prompt, pinned to that node, with the grant.
- After the fix ran, confirm the alert's condition is gone with a read-only
  check. Report the cause, what each machine showed and the fix, then revoke
  the hook.
EOF

Then the hub, on the operator's machine. Its output is also kept in incident.log, where the approval request will be.

$ nfltr orch --hub-id incident --hub-guidance-file runbook.md \
    "Create a hook for checkout's Alertmanager alerts and tell me its URL and secret file, then handle the alert it receives." 2>&1 | tee incident.log
nfltr orch: hub → create_hook
Hook ready: POST alerts to https://nfltr.xyz/api/v1/webhooks/HOOK_ID/deliveries with the bearer secret in ~/.config/nfltr/hub/incident.HOOK_ID.secret

The hook's secret never enters the hub's conversation: nfltr writes it once to that file on the hub's machine. The session keeps running while the hook is open.

5. The alert

Point an Alertmanager receiver at the hook, with the secret file copied to where Alertmanager reads it:

receivers:
  - name: nfltr-hub
    webhook_configs:
      - url: https://nfltr.xyz/api/v1/webhooks/HOOK_ID/deliveries
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/nfltr-hook-secret

Or send the alert by hand, from another terminal, with HOOK_URL and SECRET_FILE set to what the hub printed. The secret goes from file to file, never onto a command line:

$ (umask 077; { printf 'Authorization: Bearer '; cat "$SECRET_FILE"; } > alert-auth.txt)
$ curl -sS -o /dev/null -w '%{http_code}\n' -H @alert-auth.txt -H 'Content-Type: application/json' \
    --data-binary @alert.json "$HOOK_URL"
202

A POST without the secret, or with another, gets 401 and never reaches the hub. The alert's body reaches the hub as untrusted content, never as instructions.

6. The hub investigates

In the hub's terminal: one read-only agent per machine, then the request for the owner, with the prompt verbatim between marker lines and its SHA-256. A fix the hub tries without approval is refused by checkout-vm's node and changes nothing:

nfltr orch: hub → spawn_agent machine=checkout-vm: "On checkout-vm: read checkout's status and config (read-only)."
nfltr orch: hub → spawn_agent: "In the cluster: is payments (checkout's upstream) healthy, and how fast does it answer? (read-only)"
nfltr orch: hub → spawn_agent: "In CI: what was the last checkout deploy, from which change? (read-only)"
...
nfltr orch: hub → request_approval: node checkout-vm, prompt sha256 79215c78...a8bf (82 bytes); the owner signs it on their own machine with:
  nfltr orch hub approve --key-file KEY_FILE --node checkout-vm --prompt-file PROMPT_FILE --expect-digest 79215c78...a8bf --ttl 15m
----- prompt to approve (exactly the lines between these markers) -----
On checkout-vm: roll checkout back to release r41 with `checkoutctl rollback r41`.
----- end of prompt -----

7. The owner approves

Save the prompt between the markers, exactly:

$ awk '/^----- end of prompt/{p=0} p; /^----- prompt to approve/{p=1}' incident.log > fix.txt

Read it, decide, and sign it with the digest the request named (SHA below). A file that differs by one byte is refused and nothing is signed. The grant is printed on stdout; it is valid for this prompt on this node, once, for 15 minutes.

$ nfltr orch hub approve --key-file owner.key --node checkout-vm --prompt-file fix.txt --ttl 15m --expect-digest "$SHA"
nfltr-approval-v1.eyJ...

8. The fix runs on checkout-vm

Stop the waiting session with Ctrl+C (the hub keeps its hook, agents and results), then give it the grant: the same --hub-id resumes the same session. The hub spawns the approved prompt pinned to checkout-vm with the grant, checks the result, revokes the hook and ends.

$ nfltr orch --hub-id incident --hub-guidance-file runbook.md "The owner approved the fix: $GRANT"
nfltr orch: hub incident has work outstanding; resuming its claude session ...
nfltr orch: hub → spawn_agent machine=checkout-vm: "On checkout-vm: roll checkout back to release r41 with `checkoutctl rollback r41`."
...
nfltr orch: the hub session ended; hub incident keeps its 5 agent(s) and their results. ...

checkout runs r41 again, changed once:

$ checkout/checkoutctl status
release=r41
upstream_timeout_ms=2000
$ cat checkout/actions.log
2026-10-15T09:41:07Z rollback to r41

9. Who ran what

The hub's record lists every agent: its machine, its state and its prompt. Here the hub tried the fix once without a grant; checkout-vm's node refused that rollback, so the action log holds one change.

$ nfltr orch hub runs --hub-id incident
hub incident stopped (reattach: nfltr orch --hub-id 'incident'), 5 agent(s)
  agent-...-15624d22 completed turns=1 machine=checkout-vm ... "On checkout-vm: read checkout's status and config (read-only)."
  agent-...-17e95a31 completed turns=1 machine=cluster-eu ... "In the cluster: is payments (checkout's upstream) healthy, and how fast does it answer? (read-only)"
  agent-...-afee78c3 completed turns=1 machine=ci-runner ... "In CI: what was the last checkout deploy, from which change? (read-only)"
  agent-...-7c60f4d0 completed turns=1 machine=checkout-vm ... "On checkout-vm: roll checkout back to release r41 with `checkoutctl rollback r41`."
  agent-...-27cf03cc completed turns=1 machine=checkout-vm ... "On checkout-vm: roll checkout back to release r41 with `checkoutctl rollback r41`."

nfltr orch hub show --hub-id incident <agent-id> gives one agent's turns, result and events. Each node also keeps a ledger of the grants it spent.

In production

Next steps