Closeout Specification, Draft 0.1
Status: Draft. An evaluator can implement this edition. A later edition that changes behavior will use a new specVersion.
1. Purpose
Closeout is an open specification for declaring what must be verified before an agent's work can be accepted, and recording the evidence used to make that decision.
A skill explains how to perform work. A closeout policy makes a requirement mandatory at a gate. Installing a skill does not add a requirement. An agent that never loads a skill does not remove a requirement.
The interoperability rule is:
Given the same effective policy, candidate identity, and evidence, compliant evaluators reach the same acceptance decision.
Two reviewers do not have to produce the same findings. Evaluators that receive the same findings have to agree on whether those findings satisfy the policy.
2. Files
A repository keeps policy in version control and keeps results out of it.
| Path | Role |
|---|---|
AGENTS.md |
Project instructions. Point at the policy. Do not copy the policy into prose. |
.agents/skills/ |
Procedures for doing or reviewing work. Agent Skills packaging: SKILL.md, plus optional references/, scripts/, and assets/. |
.agents/closeout.yaml |
Public entry point. The acceptance requirements. |
.agents/closeout/*.yaml |
Policy documents imported by the entry point. |
.closeout/ |
Evidence and decisions written by a runner. Not policy. |
Paths inside a policy are relative to the repository root. A path is one or more segments separated by /. A segment is not empty, ., or ... A path does not start with / or ~, and it does not contain a backslash. The resolved path stays inside the repository root.
validate, run, decision, and evidence add read the policy from the commit at refs/heads/main on the remote origin. The runner fetches that ref and checks it against a second answer from the remote. The work tree, the local main branch, and the local origin/main ref are not policy sources. When the remote ref cannot be read, acceptance is blocked. The runner does not substitute ~/.agents/closeout.yaml for an unreadable remote.
When that commit contains .agents/closeout.yaml, that file is the acceptance policy. A file with no items is still a policy. When that commit does not contain that file, those commands read ~/.agents/closeout.yaml. Imports and skills in the global file resolve from the home directory. Commands run in the candidate repository. The decision warning says which global file was used.
closeout try reads ~/.agents/closeout.yaml and evaluates that document against the candidate. Imports and skills in the file resolve from the home directory. The command prints the result and does not write a decision file. A trial does not add, remove, or replace a requirement on origin/main.
When the public entry is absent on that commit and ~/.agents/closeout.yaml is absent, the repository has no requirements. validate succeeds. A decision for any gate is accepted, with message no closeout policy. An invalid global file blocks acceptance.
When a file exists and loading or validation fails, acceptance is blocked. The failure is not treated as zero requirements.
3. Policy document
A public policy file is YAML 1.2. Duplicate keys are an error. The JSON value after parsing matches schema/policy.schema.json.
specVersion: "0.1"
description: Requires Bun and a typecheck script.
imports:
- path: .agents/closeout/bun-quality.yaml
as: quality
items:
- id: adversarial-review
kind: review
gate: beforePR
skill: .agents/skills/adversarial-review/SKILL.md
independence:
differentSession: true
differentModel: true
failOn: P1
specVersion is the string 0.1. Unknown fields are errors. description is optional prose for humans, including prerequisites of a shared collection. It does not change evaluation except through the file hash in the policy identity.
imports and items default to empty arrays. Omit setup when there are no steps. An empty setup array is invalid. Every item that resolves is required at its gate. This edition has no optional item.
An imported file uses the same schema. Only the entry file may contain setup. This edition defines one gate, beforePR.
3.1 Retry limit
The entry policy may contain:
retry:
maxFailedAttemptsPerItem: 5
scope: task
Both fields are required when retry is present. maxFailedAttemptsPerItem is an integer from 1 to 100000. scope is task or candidate. An imported file cannot contain retry. A policy without retry allows further attempts without a failure limit.
The limit counts failed attempts, including the first execution. A record counts when its evaluated state is failed. Passed, missing, stale, invalid, untrusted, and independence records do not consume the limit. Commands and setup steps use their execution results; reviews use findings against failOn.
For candidate scope, failures share itemId, policyDigest, base, and head. For task scope, failures share itemId, policyDigest, and task, across all base and head commits. --task <id> supplies the stable task ID for run, decision, try, and evidence add. Task scope requires a nonblank ID of at most 256 bytes. The orchestrator keeps it fixed throughout the task, including new commits and agent sessions. An operator or orchestrator may authorize a new task ID after escalation.
Once a requirement reaches its limit, its state is exhausted and acceptance is blocked. The runner executes no further attempts for that requirement and refuses further review evidence for it. The agent stops retrying and asks the operator for help. Exhaustion never accepts the work. Requirements skipped by paths stay skipped. An exhausted setup step prevents later setup steps and commands from running.
The runner serializes attempts against the same evidence directory when retry is present. Concurrent attempts cannot spend the same remaining budget. Evidence stays in that directory throughout the task. A new commit, agent session, or invocation does not remove recorded failures.
Item ids and import names match ^[a-z][a-z0-9]*(-[a-z0-9]+)*$.
4. Imports
An import adds that file's requirements under a namespace. There is no override and no last-file-wins rule.
Resolution is depth-first. For an import { path, as } inside a document whose namespace prefix is P:
- The child prefix is
aswhenPis empty, otherwiseP/as. - The child's imports are resolved first.
- Each of the child's items is then added with id
childPrefix/item.id.
Items of the entry file keep their own ids. Import path values are repository-root-relative, including imports nested inside another file.
These are errors:
- An import path that breaks section 2.
- A missing import file.
- An import cycle. The cycle is the chain of policy files.
- Two imports in the same file with the same
as. - Two resolved items with the same qualified id.
- A setup step id that matches another setup step or an item.
setupin an imported file.
This edition imports only files inside the same repository. A shared collection is vendored into the repository and imported by path.
5. Requirements
5.1 command
id: typecheck
kind: command
gate: beforePR
exec: ["bun", "run", "typecheck"]
timeoutSeconds: 120
exec is an argv array. The runner does not invoke a shell. timeoutSeconds is required, from 1 to 86400.
The runner checks out the candidate commit in one detached git worktree. It runs the setup steps from section 5.4, then each in-scope command, then deletes that worktree. A command's exit, timeout, the change in worktree status during that command, and whether it moved HEAD, are the evidence. A statement in a chat transcript is not command evidence.
A shared collection names its prerequisites in description. A command that assumes a package manager does not become valid in a repository that does not provide it.
5.2 review
id: adversarial-review
kind: review
gate: beforePR
skill: .agents/skills/adversarial-review/SKILL.md
independence:
differentSession: true
differentModel: true
failOn: P1
skill is a repository-root-relative path ending in /SKILL.md. The file has to exist. The skill directory is part of the policy identity, including references/ and any other regular files under that directory. Symlinks inside the skill directory are an error.
failOn is P0, P1, P2, or P3. Severity order from most severe to least is P0, P1, P2, P3. A finding fails the requirement when its severity is as severe as failOn or more severe. failOn: P1 fails on P0 and P1.
independence.differentSession and independence.differentModel are required booleans. true means the reviewer's value and the candidate's value are both non-empty and different. An empty or unknown value does not satisfy true.
The candidate session, model, and provider are inputs supplied by the orchestrator. The reviewer session, model, and provider are fields on the review record. The YAML flags are requirements. The runner records the identities it compared. It does not infer them from a model name it cannot see.
A review record carries findings. It does not carry an outcome. The evaluator computes the outcome from the findings, failOn, and independence.
Each finding has:
| Field | Meaning |
|---|---|
severity |
P0, P1, P2, or P3. |
location |
Where the reviewer looked. |
explanation |
What is wrong, or what the reviewer checked. |
evidence |
The observation that supports the finding. |
5.3 paths
paths is optional on a command, a review, and a setup step. Omit it and the requirement always applies. An empty list is invalid.
id: engine-tests
kind: command
gate: beforePR
exec: ["bun", "test"]
timeoutSeconds: 120
paths:
- apps/engine/**
- apps/engine/
Each entry is a repository-relative pattern, from 1 to 256 characters. An item has at most 100 patterns. A pattern has no leading slash, no leading !, no backslash, no . or .. segment, and no Windows drive prefix.
The runner reads the changed paths with git diff --name-only --no-renames -z <base> <head>. It uses the resolved base and head. A rename contributes both the old path and the new path. A deleted path counts.
A pattern that ends with / matches a path that starts with that prefix. apps/engine/ matches apps/engine/main.ts. It does not match apps/engine or apps/engine-extra. Any other pattern is a glob. * and ? match inside one path segment. A segment that is ** matches across segments. apps/engine/** matches apps/engine/main.ts.
A command, a review, or a setup step whose patterns match none of the changed paths is skipped. Its message is no changed path matches. A skipped requirement is not executed. It does not block or reject the decision. Requirements without paths still apply.
If any setup step or item at the gate has paths and that diff cannot be read, the decision is blocked. The message starts with changed paths are unavailable. The runner does not run the setup steps or the commands.
5.4 setup
setup is an optional array on the entry document. Each step prepares the worktree for the commands. An imported file that contains setup does not load. The message is setup is only allowed on the entry policy.
specVersion: "0.1"
setup:
- id: js
exec: ["bun", "install", "--frozen-lockfile"]
timeoutSeconds: 600
- id: worker
paths: ["apps/worker/**"]
exec: ["make", "-C", "apps/worker", "install"]
timeoutSeconds: 1800
items:
- id: markdown
kind: command
gate: beforePR
exec: ["make", "check-markdown"]
timeoutSeconds: 120
Each step requires id, exec, and timeoutSeconds. exec is an argv array. The runner does not invoke a shell. timeoutSeconds is from 1 to 86400. paths follows section 5.3. The array has from 1 to 32 steps.
A step has no kind and no gate in the file. The resolved kind is setup and the gate is beforePR. Ids are unique across setup steps and items. A repeated id fails to load with duplicate requirement id and that id.
The runner uses the same worktree as the commands. It runs each in-scope setup step in the order written, then the in-scope commands. A setup step may leave files behind. The runner does not fail that step for dirt. It fails the step when the process exits non-zero, times out, cannot start, or moves HEAD.
When a setup step fails, the runner runs no later setup step and no command. A later setup step or command that has no evidence is skipped, with message setup failed. That skip does not block. The decision is rejected because the setup step failed. A setup step that is only missing, because nothing has run, does not skip later requirements. Those requirements stay missing and the decision is blocked. A setup step skipped because no path matches is not a failure, and later steps still run. The runner does not execute a review. A missing review stays missing.
Dirt left by an earlier setup step does not fail a later command. A command fails for dirt when the worktree status after that command is non-empty and differs from the status before it.
6. Effective policy identity
The identity is sha256: plus 64 lowercase hex characters. It is the SHA-256 of the UTF-8 bytes of a canonical JSON document:
{
"specVersion": "0.1",
"files": [{ "path": "repo-relative path", "sha256": "64 hex characters of the file bytes" }],
"items": []
}
files is sorted by path. It contains each policy file that was read, and every regular file under each referenced skill directory, once each. setup is present only when the entry has setup steps, in the order written. items stays in execution order.
When the entry has retry, the canonical document also contains that object. Changing its limit or scope changes the policy digest.
Canonical JSON for this edition:
- Objects have keys sorted by UTF-16 code unit, ascending.
- Arrays keep their order.
- Strings use JSON string encoding.
- The only numbers are safe integers, written in base 10 without an exponent.
- There is no insignificant whitespace.
A resolved command item is { id, kind, gate, exec, timeoutSeconds }. A resolved review item is { id, kind, gate, skill, independence, failOn }. independence contains differentSession and differentModel. An unsupported item is { id, kind, gate, supported: false }. When an item has paths, the object also contains that array in the order written. An item without paths omits the field.
A resolved setup step is { id, kind, gate, exec, timeoutSeconds } and its kind is setup. When the entry has setup steps, the document contains a setup array in that order. A document with no setup steps omits setup. When a setup step has paths, that object also contains the array. A step without paths omits the field.
Changing a policy file, an imported file, or a file in a referenced skill directory changes the digest. Evidence bound to the previous digest does not satisfy the new one.
7. Evidence
A record matches a requirement only when itemId, base, head, and policyDigest are equal to the evaluation inputs. base and head are lowercase hex git commit ids, 40 or 64 characters, resolved by the runner before execution. attempt is a positive integer. Two records with the same identity and the same attempt are an error for that item.
The evaluator uses the matching record with the greatest attempt.
Records produced with --task also carry task. Under task retry scope, matching evidence has the same task ID as the candidate. The reference runner allocates attempt numbers across all bases and tasks for the same policy, head, and requirement so records retain distinct storage paths.
Command records are accepted only from a producer the evaluator has been configured to trust. The reference runner trusts producer name closeout-reference at its own version, and only when exec.argv equals the policy argv. A content hash binds the record to its bytes. It does not prove who wrote the file.
Review records that match the evidence schema are intake. The evaluator recomputes pass or fail. Any schema-invalid file in the evidence directory blocks the decision.
The reference runner stores records under .closeout/evidence/ and command logs under .closeout/logs/. Those directories are outside the committed policy.
7.1 Verification artifacts in clients
Command and review records carry artifacts. Each reference names a path and the SHA-256 of the file bytes. The array can be empty and holds at most 16 references. A client can display an actual terminal transcript, screenshot, video, Markdown report, PDF, HTML document or archive through this field. The file format does not determine whether a requirement passes.
The reference runner writes log paths relative to its evidence directory, .closeout/ by default. For example, logs/<policy-digest>/<head>/<item-id>/1.log resolves inside that directory. A portable copy keeps the evidence records and their referenced files together with the same relative paths and hashes. A client checks those bytes before opening or saving its copy. Private authenticated download URLs belong to the client's transport, while the evidence record retains its portable path.
An orchestrator may publish files while the agent is still working. When it associates a file with a requirement, it retains the record's itemId, base, head, policyDigest, attempt and producer identity. A later attachment does not rewrite an earlier attempt. The evaluator still applies sections 7 and 8 to the matching records.
The agent's chat response remains its answer. A client can link that response to the supporting evidence without repeating the answer in an evidence panel. A clarification, progress message or ordinary answer does not itself create a verification requirement. Requirements come from the effective policy at the requested gate.
A screenshot can show the observed interface, a video can show an interaction, and a runner transcript can show actual command output. Agent-authored prose describes the evidence. Command outcomes come from the recorded execution, and review outcomes come from the findings and independence checks.
8. Evaluation
For each resolved setup step, then each resolved item, at the requested gate:
| Condition | Item state | Decision class |
|---|---|---|
Requirement has paths, and no changed path matches |
skipped |
not required |
Failed attempts in the configured scope reach maxFailedAttemptsPerItem |
exhausted |
blocked |
Kind is not command, review, or setup |
unsupported |
blocked |
| No records for the requirement | missing |
blocked |
| Records exist, none match this base, head, and digest | stale |
blocked |
| Duplicate attempt among matching records | invalid |
blocked |
| Command producer is not trusted, or argv differs | untrusted or invalid |
blocked |
Command timed out, could not start, exited non-zero, moved HEAD, or left new dirt |
failed |
rejected |
Command exited 0, did not move HEAD, and left no new dirt |
passed |
satisfied |
Setup timed out, could not start, exited non-zero, or moved HEAD |
failed |
rejected |
Setup exited 0 and did not move HEAD |
passed |
satisfied |
| A later setup step or command has no evidence because an earlier setup step failed | skipped |
not required |
A finding meets failOn |
failed |
rejected |
| Independence is not satisfied | independence |
blocked |
Review findings do not meet failOn, and independence holds |
passed |
satisfied |
The decision is blocked when any requirement is in the blocked class, or when policy loading failed, or when the evidence directory is unreadable, or when a setup step or an item at the gate has paths and the runner cannot read the changed paths. Otherwise the decision is rejected when any requirement failed. Otherwise the decision is accepted.
Dirt left by an earlier setup step does not fail a later command. New dirt means the worktree status after that command is non-empty and differs from the status before it.
When the changed paths cannot be read, the message starts with changed paths are unavailable. The runner does not execute the setup steps or the commands, and it does not mark them skipped.
A candidate commit that changed cannot reuse evidence recorded for the previous commit. That evidence is stale.
An unsupported kind blocks acceptance when the item applies. An item with paths that match no changed path is skipped, including an unsupported kind. A matching unsupported kind stays unsupported.
No reviewer identity that satisfies independence blocks acceptance. The runner does not substitute the candidate's own session or model.
9. Result and exit codes
The decision document matches schema/decision.schema.json. The reference runner prints it with --json. Without --json, it prints the decision, the commits, the policy digest, and one line per item.
| Exit | Meaning |
|---|---|
| 0 | accepted. Also validate when the policy is valid or absent. |
| 1 | rejected. A supported requirement failed. |
| 2 | The invocation is invalid. |
| 3 | blocked. Policy invalid, unsupported kind, retry limit exhausted, missing or stale evidence, independence unmet, or the commits do not resolve. |
Reference commands:
closeout validate
closeout run \
--gate beforePR \
--base <base-commit> \
--head <candidate-commit> \
--json
closeout decision \
--gate beforePR \
--base <base-commit> \
--head <candidate-commit> \
--candidate-session <id> \
--candidate-model <id> \
--json
closeout try \
--gate beforePR \
--base <base-commit> \
--head <candidate-commit>
closeout evidence add \
--gate beforePR \
--item <qualified-id> \
--base <base-commit> \
--head <candidate-commit> \
--session <reviewer-session> \
--model <reviewer-model> \
--findings findings.json
findings.json is an array of finding objects. evidence add writes the next attempt. It does not decide acceptance.
try evaluates the draft at ~/.agents/closeout.yaml. It uses the same exit statuses as a decision and does not write a decision file.
--candidate-session, --candidate-model, and --candidate-provider supply the candidate identity used for independence. Omitting them leaves the value empty, which does not satisfy a true independence flag.
A hook may consult closeout decision and hand the result back to an agent. The message names the decision and each requirement whose state is not skipped. A hook that allows the session to end does not write an accepted decision and does not grant permission to open or merge a pull request. The orchestrator checks the current decision at its own boundary.
Human review of a pull request is a separate decision from this document.
The reference runner can also seal a markdown copy of the decision for an OpenPGP public key. closeout run --format markdown --pgp-key <public-key> encrypts that copy. closeout verify --pgp-key <private-key> --head <commit> decrypts it. The runner holds the public key. The verifier holds the private key. closeout keygen --pgp --out <dir> writes closeout-public.asc and closeout-private.asc.
The plaintext is a deterministic markdown rendering of the decision, including the canonical decision JSON. verify checks the decision schema, renders the markdown again, and requires those bytes to match the decrypted body. --head must equal the head in the decision. With --require-accepted, the exit status is the decision's own status. Without that flag, a decrypted blocked or rejected decision exits 0.
A public key can encrypt any report. Decryption shows the body that was encrypted. Evaluators still agree on acceptance from the policy, the candidate, and the evidence. The sealed report is a copy of a decision addressed to the private key. It is not command evidence and is not review evidence. A hook that consults closeout decision does not decrypt a sealed report and does not treat one found in the workspace as acceptance.
examples/github/verify-sealed.yml is a GitHub Actions workflow for that check. It reads closeout-report.asc from the pull request head, runs closeout verify --require-accepted with the private key, and requires the sealed decision's policy digest to match origin/main. It comments on the pull request when the proof check is valid and the decision is accepted. The workflow is the one on the default branch. It does not execute the pull request's code.
10. Conformance
conformance/<name>/repo/ is a sample repository. conformance/<name>/expect.json is the outcome a compliant evaluator reproduces.
check is validate, decision, or digest. A decision fixture with git: true is committed by the harness, and that commit is both base and head. Evidence templates may use $base, $head, $digest, and $version for those commits, the effective policy digest, and the reference runner version. A digest fixture names touch. The digest taken before and after appending a newline to that file must differ.
The cases cover malformed policy, duplicate ids, import cycles, import conflicts, path escape, exhausted retries, stale evidence, a missing reviewer, independence, findings that meet failOn, and a referenced criteria file changing the digest.
The reference runner in this repository is the Rust binary closeout. cargo test runs these fixtures against it.