Troubleshooting
Orchestration problems on hosted nfltr.xyz, by symptom: what you see, why, and the fix. Most failures reach the hub as an agent completion it can act on; the rest show when a command starts.
Start with the readiness check; it names the relay the CLI resolved, whether the relay accepts your key, whether claude is installed and logged in, and how many nodes and workers it sees. It counts only nodes that serve this project's hub: a node joined with --serve-hubs naming other hubs is listed apart, and if it is the only one the check fails.
$ nfltr orch doctor"API key rejected by relay (HTTP 403)"
Symptom: nfltr orch "<goal>" stops with the hub cannot start: API key rejected by relay (HTTP 403): invalid API key — save a valid key with `nfltr config add-api-key`, or set NFLTR_API_KEY and exits 1 before any model runs; nfltr orch doctor fails its relay row the same way. In Claude Code the nfltr server stays connected and its server_status tool returns the same reason, so the session can tell you. Fix: create or copy a key in the dashboard and save it with nfltr config add-api-key, or set NFLTR_API_KEY; never put it on a command line.
With no key at all, nfltr orch nodes list says no API key for relay …: save your nfltr.xyz key with `nfltr config add-api-key`, or set NFLTR_API_KEY.
A node is missing
Symptom: nfltr orch nodes list does not show a machine, or the hub says no node matches.
$ nfltr orch nodes list- Is
nfltr node joinstill running on that machine? It is a long-running process; run it under your service manager, or in tmux/screen. - Does it use the same account? The node and the hub must use the same API key (
NFLTR_API_KEYor the saved key). - Did it refuse to start? Read its output: see the credential refusals below.
- Did a second
nfltr node joinrun on the same machine? Both get the idnode-<hostname>. The second is refused while the first is joined (node node-… is already joined to this account (machine …, connected since …): stop that node first, or join this one under another id with --name). If two connect at once, the older stops withnode … stopped: another connection joined with the same node id and replaced this one. Run one node per id, or give each its own--name.
tools_not_allowed
Symptom: an agent completes with tools_not_allowed, naming the node and the denied calls.
Why: the node joined without --allow-all-tools, so its agents may edit files but every command that needs approval (shell, git push, reading outside the task checkout) is refused. Fix: stop the node and join it again with --allow-all-tools. For the zero-setup local hub: nfltr orch nodes stop, then rerun nfltr orch --allow-all-tools (in Claude Code, re-add the server as claude mcp add nfltr -- nfltr orch hub --max-agents 2 --allow-all-tools). There it applies only to the local node the command starts; a node on another machine keeps the flags it joined with.
"no connected worker satisfies constraints …"
Why: nothing eligible for the spawn's machine, labels or repository connected within its bound (the spawn's timeout_ms, else about 30 seconds). The error lists what was connected. Fix: check the machine id and labels in nfltr orch nodes list, start or rejoin the node, and ask the hub to retry. With several harnesses installed on a node, join it with --harness claude-code (or have the spawn name one).
"timeout_ms … elapsed with no connected worker …"
Why: the spawn's own timeout_ms passed while nothing eligible was connected. When it adds N agent process(es) a node launched for it never connected, a node accepted the launch and then stopped or lost its connection before the agent registered. (… before an eligible worker had a free slot means the machines were connected but full.) Fix: as above: check nfltr orch nodes list and rejoin the node.
A repository is refused
Why: the spawn named a repository that matches none of the node's --allow-repo globs, so no node takes it. While the agent waits, its waiting says so for each node, for example none admits it: node on my-laptop advertises nfltr.workspace_repo_allow="", which does not admit repository "https://github.com/your-org/app" (its --allow-repo patterns match none). When its timeout_ms passes it ends with … constraints {workspace.repo=…}; last launch: no connected node admits repository "…": it matches none of their nfltr.workspace_repo_allow patterns (my-laptop@my-laptop allows none (joined without --allow-repo)). Fix: rejoin the node with a matching glob and --allow-all-tools (repository work needs a shell), for example --allow-all-tools --allow-repo 'github.com/your-org/*'. The node must also be able to fetch it for its mirror, with the machine's own git credentials.
node_lost
Why: the node was killed, stopped (Ctrl-C, SIGTERM), or its machine went away, while the agent ran. The node owns its agents, so they end with it. A node that stops (Ctrl-C, SIGTERM) reports its agents at once; one that lost its connection is reported about 30 seconds after it left, unless it comes back sooner. What happens: the hub receives node_lost as a completion and decides; it can re-issue the work. A spawn that names the machine waits for it to rejoin (up to its timeout_ms) and then runs there. Fix: run the same nfltr node join again on that machine.
budget_exceeded
Why: the hub's --hub-max-tokens/--hub-max-usd, or an agent's own budget, was reached. Running turns are stopped and further spawns and messages refused. Token caps count input, output and cache-write tokens; the message shows cache-read tokens apart (not counted) names the caps and what they cover (the hub budget (max_tokens=6000, max_usd=$2 for all agents of this hub together) was passed: they used …) and says cost not reported for tokens of turns that reported no cost (claude reports a turn's cost only when it ends, so a stopped or lost turn never does; nfltr does not estimate it). Fix: start the hub with a higher cap. The stopped turns' uncommitted edits come back as recovery patches, so work is not lost.
Relay restart or deploy window
Symptom: a short connection outage. Nodes and agents reconnect by themselves; a hub started during the window waits up to two minutes for the relay before failing with the relay address. A spawn that could not dispatch stays pending, within its bound, and runs when the nodes are back. A dropped task stream is resumed, not reported as a failure. Fix: usually nothing; if a hub start failed, run it again.
An agent is slow to notice a lost connection
Why: when the relay vanishes without closing the connection (power loss, a NAT or Wi-Fi change), the agent learns of it only because the relay's heartbeats stop arriving. Fix: pick a keepalive preset with --keepalive (or NFLTR_KEEPALIVE, or keepalive: in an nfltr up file). For example, nfltr http 3000 --keepalive fast, or nfltr node join --max-agents 2 --keepalive mobile.
| Preset | Use it for | Notices within | Idle traffic per agent |
|---|---|---|---|
fast | quick failover, interactive use | ~5 s | ~36 B/s |
default | balanced | 15 s | ~11 B/s |
nat | long idle behind NATs and load balancers | 30 s | ~5 B/s |
datacenter | stable networks | 45 s | ~4 B/s |
mobile | laptops and phones on flaky Wi-Fi or metered links | 60 s | ~4 B/s |
nfltr diagnose shows the effective setting.
The hub exited
Agents keep running without their hub. After Ctrl-C the session names them: interrupted; 1 agent(s) of hub … keep running (… on node-…) and their results are kept: run `nfltr orch --hub-id … "<goal>"` … to collect them. Run nfltr orch again in the same project directory (the default hub id is per project), or pass the same --hub-id; the hub reattaches and receives every completion it missed, exactly once.
$ nfltr orch --hub-id release-checkA hub session does the work itself, or asks you to approve commands
Why: a Claude Code session with its own shell and file tools can work locally instead of spawning; under claude -p its commands then need approval it cannot get. nfltr orch "<goal>" starts its session with only the hub tools, so its work runs in agents. Fix: in your own session, say where the work should run ("have an agent on the build server do this"); with nfltr orch, grant the session tools only if you want it to use them itself: --hub-tools Read,Grep.
Exit status of nfltr orch "<goal>"
For scripts: 0 the session ran to its end; 1 the hub could not start (key rejected, relay unreachable, its tools did not come up) or claude failed; 3 an agent of this session ended budget_exceeded. The answer itself is not judged and its text never sets the status: a hub that ends with a question in its answer exits 0; to keep a question open until you answer, the hub uses an owner request, which keeps the session open.
The spend line
nfltr orch ends with spend: this session: hub $X, agents …; this hub id, all sessions: agents …. The first part is this session; the second is every session of the hub id. at least $X or cost unknown means the hub session was stopped (for example by SIGTERM) before Claude reported its last turn's cost; the line after received … at … says which signal and when.
"hub … is already open by pid …"
Why: only one live process may hold a hub id, and a second session in the same project resolves the same id. Fix: stop the other process, or give the second session its own --hub-id (in Claude Code: claude mcp add nfltr-review -- nfltr mcp --toolset hub --hub-id review). A lock left by a dead process is reclaimed automatically.
Credential and clean-config refusals
clean Claude config not established: agents run with a clean Claude config, so a keychain-only login is not read. ExportCLAUDE_CODE_OAUTH_TOKEN(fromclaude setup-token) orANTHROPIC_API_KEYwhere the node runs. To use the machine's own Claude Code setup instead, join with--inherit-host-claude-config.cross-session isolation not verified: agents are isolated from the machine's other Claude sessions and the node checks that before any task. Update theclaudeCLI; join with--allow-cross-session-messagingonly if agents must reach those sessions.- Authentication errors from the relay: check the key with
nfltr config listorNFLTR_API_KEY; the key is never printed.
Recover
- Missed results: reattach the hub (same project or
--hub-id); completions are delivered exactly once. - Partial work: a turn stopped by
stop_agent, a timeout or a budget leaves a recovery patch of its uncommitted edits in the completion; committed work is on the agent's branch. Ask the hub to continue the agent (send_message) or hand the patch to a new one. - Inspect a task:
$ nfltr orch task status --task TASK_ID --events 20
$ nfltr orch task artifact list --task TASK_IDRelated: Join machines as nodes · Hub tools reference · Observe orchestration.