Skip to main content
HITL (human-in-the-loop) means the agent pauses in the middle of execution and hands the decision back to a person — it does not continue until the answer arrives. This is not a malfunction; it is an intentional safety gate or information gate built into the agent product: a human must approve before a sensitive action, supply missing information, or decide between multiple options. You probably already use it every day:
  • Claude Code pauses before running a risky command or editing a file outside its permissions, and asks “Allow this?” — it only proceeds once approved, and takes a different path when denied.
  • Codex approval modes such as suggest and auto-edit decide which actions need confirmation first; it is essentially the same gate.
  • AI SDK apps often surface a tool call in the UI when the tool has no execute, let a human confirm it, then return the result with addToolResult so the model can continue — this is the most common approval-gate pattern in homegrown applications.
For the subject under test, this “pause and wait for a human” branch needs regression coverage like any other behavior: does the right thing happen after approval? does it really not happen after rejection? what if it asks the wrong person, or pauses in the wrong place? But evals run automatically, with no human in the loop. NiceEval compresses the human into three programmable actions: notice that the agent paused (t.parked()), inspect what it is waiting for (t.requireInputRequest()), and answer in place of the human (t.respond()). Approval, rejection, and follow-up information all become paths you can write into an eval.

​
How a pause is represented: waiting + input.requested

The Turn.status returned from send has three values: "completed" (finished), "failed" (errored), and "waiting" — the turn has not finished; the agent is paused waiting on human input. waiting is not a failure, it is a suspension: once the answer arrives, the same turn continues instead of resending the request or opening a new conversation. The paused turn must also say what it is waiting for. Each pending question emits one input.requested event with a stable request id, the action where it paused, and optional options such as approve / deny. Both parts matter: waiting tells evals they should answer, and input.requested tells them what to answer.

​
One complete handshake

t.respond does not use a separate channel — it is just another normal send, and the human decision arrives as structured input: A complete HITL handshake: after t.send the app pauses on an approval, the turn returns as waiting with input.requested; t.parked and t.requireInputRequest inspect the pause; t.respond, as a second normal send, hands back the structured decision, the same turn continues to completion, and assertions proceed as usual. Two easy mistakes to avoid:
  • The response turn is not a new conversation. It happens on the same session line — ctx.session still points to the same conversation, and the adapter uses it to recover the state it paused on in the previous turn.
  • You don’t have to guess the answer from free text. input.responses carries { requestId, optionId } per request (or { requestId, text } for free-text answers); an optionId that hits one of the request’s options has already been validated on the eval side, so a typo throws instead of silently reaching the application. See Inputs for the different answers for what each answer shape looks like once it reaches the adapter.

​
On the eval side: three actions

  • t.parked() asserts that the turn really stopped in waiting; conversely, t.succeeded() fails on a waiting turn — a pause is an explicit state, never mistaken for “it finished”.
  • t.requireInputRequest(filter) picks exactly one pending request, matching fields such as id, prompt, action, or optionIds. It throws when 0 or more than 1 requests match — this is how you disambiguate when several requests are paused at once.
  • t.respond(...) answers it and emits the next turn; when several requests of the same kind should get the same answer (for example, approving a batch of changes one by one), t.respondAll(optionId) handles them all at once.
Approval and rejection are two branches of the same gate — coverage is only complete when you write an eval for each: what should happen after approval happens, and what should happen after rejection does not. See Drive for the full API-by-API usage.

​
On the adapter side: two obligations, one continuation

HITL asks the adapter to do exactly three things — the first two happen on the paused turn, the third happens on the next turn when the answer arrives:
  1. Return status: "waiting" honestly when the turn pauses — do not report a suspension as "completed", or parked() stops working and succeeded() passes when it should not.
  2. Emit one input.requested event for each pending question, with a stable id and as many fields populated as possible — the eval-side checks and matching all depend on them.
  3. On the next send, hand over the decision before continuing: take the verdict from input.responses by requestId (do not guess by order), then continue from where the previous turn paused, instead of resending the request.
The paused execution state (for example, a partially read SSE stream) lives on the session line, in ctx.session: create an adapter-private createSessionSlot<Pending>(), call ctx.session.set(slot, state) on the paused turn, and ctx.session.take(slot) on the response turn to retrieve it. take(slot) clears it once read. See Write Send step 5 for the full skeleton of how this fits together with streaming. Like the rest of NiceEval’s capabilities, HITL is not declared with a boolean flag: doing it is having it. If send has returned "waiting" and emitted input.requested, t.respond works; if it has not, the eval fails explicitly at the first parked() or requireInputRequest(), instead of silently passing. See Adapter for how capabilities follow from construction.

​
Rejection is not a failure

A human veto and a tool error are two different things, and the event stream marks them with different statuses: a call a human rejects gets a tool operation.finished with status: "rejected", not "failed". That way t.noFailedActions() is not falsely tripped by a legitimate human rejection, and t.calledTool("deploy", { status: "rejected" }) can precisely assert “the call was made, and a human blocked it” — which is exactly the assertion the rejection-branch eval needs to write.

​
Which agents do not need HITL

Not every agent has this branch. Skip the whole feature when it does not exist — the adapter does not need to handle waiting, and t.respond never shows up in the eval:
  • Agents that finish each turn in one pass: question answering, retrieval, translation, and other send-and-reply services have no step that waits for a person.
  • Runtime modes with the approval gate turned off: Claude Code and Codex are HITL products, but as subjects under test in a Sandbox they usually run fully automated (permission confirmation skipped) — that is exactly how the built-in claude-code, codex, and bub sandbox agents run, so none of them do HITL.
  • Human review outside the execution loop, such as a ticket system that reviews agent output later. That is another system’s process; it does not go through send, so an eval cannot observe it and should not try to.
The rule is simple: use HITL only when your agent’s execution loop has a branch that stops for a human and then continues from the same point, and you want to write approval and rejection into the eval.
  • Drive — The full eval-side usage: t.parked(), t.requireInputRequest(), t.respond() / t.respondAll().
  • Adapter — The structured input that answers carry to the adapter, four typical shapes.
  • Write Send — The adapter-side practice: typed session slots and the full skeleton for streaming + HITL.
  • Connect Your Agent — The integration overview: minimal integration, parameter channels, and the incremental map.