TOOL | Evaluation design
Should this task be an agent, a workflow, or ordinary software?
Ten questions about the task you are about to automate produce one verdict: build an agent, wire a fixed workflow, or skip the model. See which single answer you would change to flip it.
Describe the task's path, its rules, and what it can act on. Ten questions produce one verdict with the rule that fired. The evidence comes from how reliability compounds along a chain, where agent runs break, and the research on silent failure. Whether automating pays at all is a separate question for the error-rate calculator. Six of the ten questions match the context-of-use profiler's own dimensions; the export records the answers so they can be copied into it.
Showing your last valid result. Update the inputs above to recompute.
Verdict
Agent Variable path
The first matching rule from a list of twelve, with every other matching rule shown in the trace.
Build the agent. The evaluation consequences of the answers just given are what the context-of-use profiler produces. Six of the ten answers match the profiler's own questions word for word, so they can be copied across.
Why
The next step depends on what the previous step found. The task needs a model that chooses its own path.
Shows the fired rule's clauses alongside your own answers to each question the rule read.
- the next step depends on what the previous step found
- Can the steps be written down before the task runs?
- It depends on what each step finds
What would change this verdict
For each question, every other option that produces a different verdict if substituted alone.
The verdict does not hinge on one answer. No single change to one question would produce a different outcome.
| Question | Your answer | Change to | Verdict becomes |
|---|---|---|---|
| Can the steps be written down before the task runs? | It depends on what each step finds | The steps are always the same | Fixed workflow |
| Can the steps be written down before the task runs? | It depends on what each step finds | A few branches, all known in advance | Fixed workflow |
| Could a person write the rule that maps input to output? | No, it needs judgment | Yes, and existing software already does it | No model |
| Do the systems the task acts on expose callable interfaces with scoped permissions? | All actions are callable and individually scoped | No API exists | Not yet |
| How could anyone tell whether a step went right? | Verifiable by execution | Expert judgment only | Fixed workflow |
| How could anyone tell whether a step went right? | Verifiable by execution | Not verifiable | Fixed workflow |
Not yet settled
One entry per optional question left unanswered, naming what filling it in would decide.
Rule trace
| Rule | Fired | The rule needs |
|---|---|---|
| No callable interface | did not apply | the systems it must act on expose no callable interface |
| Manual steps, no one waiting | did not apply | the plan is for it to act without waiting for anyone |
| Existing software solves it | did not apply | software that already exists performs this mapping |
| Rules and a fixed sequence | did not apply | the mapping can be written as rules, however many |
| Steps run in the same order | did not apply | the steps run in the same order every time |
| Branches known in advance | did not apply | the branches are known in advance and can be listed |
| Long run, no automatic check | did not apply | no automatic check can tell a good step from a bad one |
| Irreversible and unattended | did not apply | the plan is for it to act before a person sees the action |
| Third party affected, unattended | did not apply | the plan is for it to act with no one waiting on it |
| Untrusted input, unattended | did not apply | it reads material it did not write and cannot vouch for |
| No oracle and no review | did not apply | no automatic check can tell a good step from a bad one |
| Variable path | fired |
How the verdict is decided
The rule list. The page runs one ordered list of twelve rules against the ten answers. Each rule carries one or more clauses, and a rule fires only when every clause passes. The first rule to fire sets the verdict. Every remaining rule is still evaluated so the trace can report whether it matched but was beaten.
Disqualifiers. Rules 1 and 2 are disqualifiers. A task whose target systems expose no callable interface, or whose plan requires full autonomy over actions that can only be done by hand, returns "not yet" regardless of every other answer. The model is not the first thing to build.
No model. Rules 3 and 4 catch tasks that need no model at all. If software already performs the mapping, or if the mapping can be written as rules and the steps follow a fixed order, the verdict is "no model." Rule 4 requires both conditions: many rules with a genuinely variable path fall through to a workflow or agent verdict, because the path variability is what makes it more than a scripting job.
Fixed workflows. Rules 5 and 6 cover fixed workflows: a sequence that runs the same way every time, or one whose branches are known in advance. Rule 7 catches a variable-path task whose steps cannot be checked automatically and whose depth reaches six or more steps. Across 1,794 annotated coding-agent trajectories, the error that decided a failed run's fate landed at a median of step 7 of a median 27-step run, and the first visible sign arrived around step 16; the median recovery window was one step. And per-step reliability compounds: at 0.95 per step across 20 steps, a pure chain succeeds about 36% of the time. Building it as a pipeline with explicit checkpoints makes each stage independently testable, and the cascade-reliability calculator compounds those per-step rates across the chain.
The gate rules. Rules 8 through 11 sit between the workflow rules and the bare "agent" verdict. They fire when the answers point to an agent but an unguarded consequence makes unattended operation unsafe: irreversible actions with no human in the loop, consequences reaching third parties, untrusted input where the agent acts before a person sees the action, or no automatic check with no review. The verdict is "agent with a gate," and the gate is an approval step or a reversible sandbox. Which actions need the gate is what the agent action risk matrix shows; which cannot be undone is what the irreversible-action inventory lists.
The defaults. Rule 12 fires for any variable-path task that cleared rules 7 through 11. The verdict is "agent." The twelve rules run in order. The trace shows the first rule whose clauses all hold, and the higher rules whose clauses stopped them.
The counterfactual panel. The panel re-runs the rule list once per question, once per option in that question's enum, with every other answer held constant. It records each substitution whose verdict differs from the current one. When no single-answer change flips the verdict, the panel says so explicitly rather than rendering empty.
Formula: The first rule in the list whose conditions all match the answers produces the verdict.
Questions
Why does a long autonomous run with no per-step check get a fixed-workflow verdict?
Because the decisive error in a failed long run lands early and stays invisible. In annotated coding-agent trajectories, the median fatal step was step 7 of a 27-step run, and nothing showed until step 16. A fixed workflow with a checkpoint after each step is the shape that catches the failure where it happens. If you can add an automatic check at each step, re-answer the ground-truth question and the verdict may change.
What is the difference between this page and the context-of-use profiler?
This page asks whether the task should be an agent at all, before anything is built. The profiler starts from a system whose shape is already decided and returns the evaluation consequences: failure families, case classes, slices and evidence expectations. Six of the ten questions here are the profiler's own dimensions. The export records the answers so they can be copied into the profiler.
Why are there five verdicts instead of three?
The three core shapes are agent, fixed workflow and no model. "Agent with a gate" exists because an agent-shaped task whose actions are irreversible, whose consequences reach third parties, or whose steps cannot be checked still needs the agent pattern but cannot run unattended as described. "Not yet" exists because a task whose target systems have no callable interface or whose critical actions are manual-only has a build prerequisite before the shape question matters.
Does this page compute any cost or error rate?
It does not. Whether automating pays at all is what the acceptable-error-rate calculator computes. What a chain's end-to-end success rate looks like at a given per-step rate and depth is what the cascade-reliability calculator computes.
What does the "what would change this verdict" panel show?
For each of the ten questions, every other answer option that would produce a different verdict if that one answer alone were changed. It shows which fact is load-bearing and what revising it would buy. When no single change to one answer flips the verdict, the panel says so.