Free field guide · prompt-free evidence
Make the agent outcome machine-readable—or mark it unknown.
A process exit and a text reply are not enough for downstream automation. Use these eight gates to distinguish valid completion, failure, timeout, max-turn termination, refusal, and invalid output.
Operational information only · No prompts, responses, tool payloads, or production access required
- schema_version
- pin the contract that was requested
- outcome
- use one closed machine-readable status
- termination_reason
- separate timeout, max-turn, and cancellation
- validation_state
- report valid, invalid, or not observed
rule requested → observed → validated → attributed
Eight evidence gates
Reliability begins at the output boundary.
Passing these gates does not prove that an agent is correct. It makes completion, termination, validation, and retry evidence explicit enough to measure without scraping terminal prose.
Version one result envelope
Pin the schema requested for every attempt instead of letting each caller infer a shape.
Keep JSON stdout clean
When structured output is requested, send progress and terminal decoration elsewhere.
Close the outcome vocabulary
Use completed, failed, refused, cancelled, timed_out, max_turns, invalid_output, and unknown.
Separate exit from outcome
Preserve process exit code without treating a zero exit as proof of a valid agent result.
Make termination explicit
Expose timeout, max-turn, cancellation, and external-stop evidence in the envelope.
Validate before accepting
Record schema validation state and a bounded error class; never repair silently.
Attribute every retry
Keep attempt number and prior-attempt relationship so first-pass reliability stays measurable.
Publish evidence coverage
Report the share of attempts with known outcome, termination, validation, latency, and cost evidence.
Minimum result contract
Keep status, termination, and validation separate.
This is a field map, not a universal schema. Keys can remain pseudonymous. Prompts, responses, tool payloads, credentials, direct identifiers, and customer data do not belong in the prompt-free measurement row.
| Field | Purpose | Truth rule |
|---|---|---|
schema_version | Identify the requested result contract. | Pin it per attempt. |
run_key, attempt_no | Attribute one observed execution and its retries. | Pseudonymous and stable. |
outcome | State the closed machine-readable result. | Unknown is not completed. |
termination_reason | Distinguish timeout, max-turn, cancellation, and external stop. | Null only when not applicable. |
validation_state | Report whether the observed envelope matched the requested schema. | Invalid is preserved, not repaired. |
error_class | Group failures without storing sensitive payloads. | Bounded vocabulary; no stack trace required. |
latency_ms, cost | Compare supplied runtime and spend evidence with outcomes. | Missing is null, not zero. |
observed_at | Support bounded windows and contract changes. | Use one documented timezone. |
Evidence boundaries
A valid envelope supports measurement. It does not prove correctness.
It can support
- Structured completion and failure counts
- First-pass and retry rates
- Timeout and max-turn attribution
- Schema-valid output coverage
- Latency and cost by supplied outcome
It cannot prove
- That the agent response is factually correct
- That a zero process exit means completed work
- That missing output or cost was zero
- That a retry caused an improved business result
- Any guaranteed reliability or savings improvement
One workflow, one decision
Need the prompt-free baseline before implementation?
The WrightOps AI Agent Cost & Reliability Snapshot turns up to 50 normalized attempts into deterministic aggregate JSON and Markdown, evidence gaps, and up to five human-reviewed priorities.
Three business days after accepted inputs and provider-confirmed settled payment. No implementation, production access, or outcome guarantee.