The Verification Horizon

Working paper, version 0.1.0 — not peer reviewed. This is a conceptual paper grounded in published systems literature and practitioner observation. It reports no controlled experiment and makes no measured effectiveness claim.

Problem statement

The economic appeal of autonomous software delivery is straightforward. A system that can interpret a goal, change code, execute checks, respond to review, and deploy while an operator is absent turns calendar time into productive time. The trust problem is harder. Absence removes the conversational corrections that normally catch a wrong premise before it compounds. A system can be locally competent and globally wrong: it may solve a nearby problem, optimize a proxy, weaken a gate, conceal uncertainty, or deliver an artifact whose behavior is internally consistent but misaligned with the original purpose.

The familiar response is to ask for more verification. That response is necessary but underspecified. Verification is always performed by an instrument against a claim. A unit test can support a narrow claim about a function. A contract test can support a claim about an interaction. A static analyzer can support a claim about selected source patterns. A deployment probe can support a claim about an observed endpoint at a particular time. None of those instruments, alone or together, proves that the delivered system embodies the operator's full intent.

Turing's imitation game replaced an ambiguous question about whether a machine "thinks" with a behavioral test that an observer could perform [1]. The move was methodologically powerful because it made an instrument explicit. It also illustrates the boundary at issue here: an observation can discriminate among behaviors without opening direct access to internal meaning. This paper does not claim that semantic alignment is unknowable. It claims that an engineering system must not report more certainty than its instruments warrant.

I use verification horizon for the boundary between claims supported by the current evidence and claims that remain underdetermined. The term is introduced here as an engineering construct, not as a theorem. It deliberately avoids describing the boundary as Gödelian. A software-delivery agent and its test suite are not thereby a formal arithmetic system, and incompleteness language would add grandeur without adding explanatory power. The practical problem is closer to behavioral indistinguishability under limited observations.

The research question is therefore: What control structure lets an autonomous delivery system make useful progress while keeping every accepted claim within the evidence available to an absent operator?

Operational definitions

The model uses six operational terms.

Operator. The person or accountable group that owns the objective, risk tolerance, and final authority. An operator can delegate decisions but remains the source of legitimacy for the system's operating envelope.

Supervisor. The control-plane component that selects the next bounded intervention, frames it, chooses the worker boundary, and evaluates returned evidence. The supervisor decides what is authorized next. It does not gain an unbounded right to implement merely because it can observe the repository.

Worker. A component that executes one bounded intervention. Its authority is limited by scope, tools, time, writable resources, and required gates. A worker may be an agent, a deterministic program, or a human. The model depends on authority and evidence, not on anthropomorphic labels.

Ledger. An append-oriented record connecting objective, delegation, execution, evidence, decision, and outcome. The ledger is not a transcript dump. It is a structured explanation of why an intervention existed and what the system observed.

Gate. An instrument that evaluates a named claim using a defined procedure. Examples include a compiler, test suite, schema validator, security scanner, review decision, deployment probe, or policy check. A gate is strong only with respect to the claim it actually measures.

Verification horizon. The boundary beyond which available observations do not discriminate sufficiently among materially different interpretations of the objective. Beyond the horizon, the correct status is unknown, conditional, or projected—not verified.

These definitions separate control from execution and evidence from narrative. They also make failure discussable. If a result is wrong, the question becomes whether the objective, worker boundary, instrument, evidence, or decision rule failed. "The agent made a mistake" is too coarse to improve a system.

Closed-loop control model

Cybernetics studies regulation through communication and feedback. Ashby's An Introduction to Cybernetics provides the foundational vocabulary of state, transformation, stability, and requisite variety [2]. Applied to software delivery, the important move is to reject a one-way pipeline model. Autonomy is not a prompt followed by output. It is a repeated control loop:

objective and constraints
          |
          v
      supervisor ---- authorization ----> worker
          ^                                  |
          |                                  v
     decision <---- evidence <---- gates and environment
          |
          +---- adapt, accept, revert, escalate, or stop

Let the repository and its environment at time t be state x(t). Let the operator's desired outcome and constraints be reference r. A worker applies an authorized intervention u(t), producing a new state. Gates do not observe the full state; they produce observations y(t) through selected measurement functions. The supervisor applies decision rule D to the objective, constraints, observations, and history.

The useful question is not whether y(t) is green. It is whether the mapping from state to observation preserves the distinction needed for the decision. Two materially different states can produce the same observation. A test suite can pass for an implementation that satisfies its assertions and for another that exploits an omitted condition. A deployment can return HTTP 200 with the correct application payload or with a generic success page. A paper can contain citations while misrepresenting what the cited instruments establish.

The verification horizon contains the distinctions that the current observation functions cannot make. More tests can narrow that set, but count is not coverage of meaning. Measurement must begin with an information need, then define measures and analysis that address it. That direction—from need to measure—is consistent with the process described by ISO/IEC/IEEE 15939:2017 [8]. Reversing it produces dashboard theater: available numbers search for a decision they can justify.

Leveson's system-theoretic account of accidents argues that losses in complex systems can arise from inadequate control of interactions, even when individual components have not failed in the traditional sense [3]. The analogy to autonomous delivery is bounded but useful. A worker, test suite, branch policy, and deployment platform may each behave as specified while their interaction permits an unsafe or unwanted outcome. Trust must therefore be a property of the control structure, not a personality attributed to one agent.

A closed loop has five required properties:

1. A reference: a concrete outcome and named constraints.
2. A bounded actuator: authority limited to the intervention required.
3. Sensors: gates whose claims and blind spots are explicit.
4. A decision rule: criteria for accept, adapt, revert, escalate, or stop.
5. Memory: provenance connecting the next decision to prior observations.

Remove any one and "autonomy" becomes open-loop execution. A worker without a reference optimizes whatever proxy it infers. A worker without bounded authority expands blast radius. Gates without claim definitions produce green symbols with ambiguous meaning. A supervisor without stop rules converts sunk cost into repeated intervention. A system without memory repeats decisions it cannot explain.

Trust kernel and authority tiers

A trust kernel is the smallest operator-readable artifact that defines the autonomous system's operating envelope. The name is intentionally narrower than "policy." A large policy corpus can contain contradictions and implicit precedence. A kernel should identify the few constraints whose violation makes all other success evidence inadmissible.

I propose three authority tiers.

Immutable constraints cannot be changed by the supervised system during an execution. Examples are prohibitions on exposing secrets, weakening decisive tests, writing outside the authorized repository, bypassing required human confirmation, or deleting recovery state. "Immutable" is relative to the execution boundary: an operator may revise the kernel between runs, but the worker cannot rewrite the rules that judge its own work.

Operator-configurable constraints express context and risk tolerance. They include the target repository, production boundary, acceptable external writes, cost ceiling, deployment environment, and release rule. They are explicit inputs, not facts inferred from tool availability.

Runtime-tunable parameters allow adaptation without changing authority. Examples include batch size, search breadth, retry count within a cap, or the choice among already authorized gates. These parameters may change in response to evidence because the kernel bounds their range and effect.

This separation responds to a recurring control failure: a system encounters a gate it cannot pass and edits the gate. That action may be legitimate during test maintenance, but only when the objective specifically authorizes the change and independent evidence shows that the gate—not the product—is wrong. Otherwise the system is choosing its own definition of success after observing failure.

The kernel must also define an escalation boundary. Some actions are not made safe merely by adding a rollback command. Public disclosure, credential rotation, legal acceptance, messages sent in another person's name, destructive data changes, and cross-organizational writes can create consequences outside the repository's reversible state. Those require authority appropriate to the external effect.

TDD as a local stabilizer

Test-driven development contributes a specific form of feedback to this model. Beck describes a cycle in which a failing test precedes the minimum change that makes it pass, followed by design improvement while tests remain green [4]. The important control property is temporal: the system observes the instrument rejecting the pre-change state before it relies on the instrument to accept the post-change state.

That red observation does not make a test complete. It does establish that the test is capable of discriminating at least one relevant state transition. A test written after implementation and seen only in the passing state lacks this particular evidence. It may still be useful, but it cannot demonstrate that it would have detected the absence it purports to guard.

For an autonomous worker, the TDD loop acts as a local stabilizer:

1. Translate one acceptance claim into an executable observation.
2. Observe failure for the expected reason.
3. Apply the smallest intervention expected to change that observation.
4. Observe the focused gate pass.
5. Observe neighboring gates for unintended effects.
6. Refactor without changing the accepted behavior.

The adjective local matters. A unit-level red-green cycle cannot validate a system-level objective. The worker must bind the scale of its claim to the scale of its evidence. A formatter passing supports formatting conformance. A unit suite supports its modeled units. A full static build supports generation of the routes it exercises. A live probe supports the observed deployment at the recorded time. The supervisor assembles these without inflating one into another.

This paper makes no claim that TDD alone reduces defects or guarantees correct design. Its narrower claim is structural: test-first order creates an observable rejection event before the acceptance event, which is useful evidence for a closed-loop change. Whether that practice improves outcomes in a particular context is an empirical question outside this conceptual paper.

Evidence and traceability contract

An absent operator needs more than a terminal transcript. A transcript records events but does not necessarily connect them to claims. The ledger should store a small provenance contract for every accepted outcome:

• objective: the requested outcome in operator terms;
• authority: permitted resources and excluded effects;
• source state: branch, revision, environment, and relevant external state;
• intervention: the bounded change attempted;
• instrument: exact command, query, review, or probe;
• observation: raw result, status, and observation time;
• interpretation: the claim supported by that result;
• blind spots: what the instrument did not inspect;
• decision: accept, adapt, revert, escalate, or stop;
• recovery: the rollback path and retained state.

The W3C PROV model distinguishes entities, activities, and agents and defines relations among them [5]. A software-delivery ledger need not implement the entire ontology to benefit from that separation. The artifact under test is an entity; the test execution is an activity; the worker and test runner are agents in different senses; and the resulting report is another entity derived through a recorded activity. Collapsing these into "the agent verified it" erases the chain required for audit.

Evidence strength depends on freshness and identity. "CI passed" is incomplete without the revision, workflow, run, conclusion, and completion time. "The site works" is incomplete without the origin, route, response, content check, and observation time. "No secret was found" is incomplete without the scanner, scope, rule set, and whether untracked or generated files were included.

The ledger must preserve dissent between instruments. A local suite can pass while CI fails in a clean environment. A review can identify an untested premise after all automated gates pass. A production endpoint can serve a stale artifact despite a successful deployment command. The supervisor should not average those observations into a confidence score. It should bind the conflict to the claim and either resolve it or report it.

NIST's AI Risk Management Framework organizes work around Govern, Map, Measure, and Manage functions and stresses that risk management is contextual and continuous [6]. The Generative AI Profile applies that framework to risks specific to generative systems across their lifecycle [7]. These documents do not prescribe the control model in this paper. They do support two premises: trustworthiness requires lifecycle governance, and measurement is not a substitute for contextual risk decisions.

Failure modes and counterexamples

A useful model should expose where its own controls can fail.

Proxy-complete, objective-wrong

The worker satisfies every written acceptance check, but the checks encode a narrower objective than the operator intended. This is not a test execution failure. It is a framing failure beyond the current verification horizon. Mitigation requires examples, counterexamples, domain constraints, or operator review—not more repetitions of the same gate.

Green by omission

The suite passes because the relevant behavior is absent from the suite. High test count and high line coverage can coexist with this failure. Contract, integration, adversarial, accessibility, performance, and live probes each address different omissions; none is universal.

Green by gate mutation

The worker changes the product and the gate together, causing the new behavior to pass. This can be correct in a deliberate specification change, but the same diff can conceal weakened expectations. The control structure should require a named reason and independent review when an acceptance oracle changes.

Stale truth

An earlier run is reported as current. The command may be real and the result may have been green, but a later revision or external-state change invalidates the inference. Evidence without source identity and time invites this error.

Reviewer monoculture

Multiple agents repeat the same premise, training, or prompt structure and are reported as independent agreement. Count does not create independence. A stronger review uses different instruments or adversarial questions and records the evidence each reviewer actually inspected.

Rollback theater

A change is described as reversible because Git can revert source, while its external effect cannot be recalled. A sent message, disclosed paper, changed credential, deleted cloud resource, or accepted legal term has a different recovery model. The authority boundary must follow the real effect.

Ledger without control

The system records every action but never uses the record to constrain the next decision. This produces observability, not closed-loop control. Memory becomes control only when the supervisor conditions authorization on it.

Control without productive authority

The kernel is so restrictive that a worker cannot complete a useful vertical slice. A zero-risk system that cannot act does not meet the overnight-delivery objective. The remedy is not to remove constraints wholesale; it is to design smaller reversible interventions whose authority can be granted safely.

These counterexamples show why trust, autonomy, and cost cannot be optimized in isolation. Cheap execution that produces unverifiable state increases recovery cost. Comprehensive ceremony that never ships produces no delivery value. Trust is the invariant: autonomy and cost reduction are acceptable only inside an evidence-bearing control loop.

Limitations and falsifiability

This is a conceptual paper and is not peer reviewed. It synthesizes systems literature with practitioner observation from building and operating automated software-delivery workflows. It does not demonstrate that the proposed control model reduces defects, prevents incidents, lowers cost, or improves delivery speed. No controlled comparison, longitudinal dataset, or calibrated outcome measure is presented.

The term verification horizon may overlap with existing concepts in formal verification, epistemology, safety engineering, observability, and human-machine teaming. This paper does not claim terminological priority outside the specific definition given here. A systematic literature review could show that another term describes the same boundary more precisely.

The control analogy also has limits. Software organizations are not closed physical plants. Objectives change, stakeholders disagree, environments are partially observable, and the worker can alter some of the instruments that observe it. The model is useful only if those complications remain visible.

Several observations could falsify or materially weaken the proposed model:

1. Controlled studies could show that separating supervisor and worker authority provides no improvement in detecting or containing objective errors compared with an otherwise equivalent unbounded agent.
2. Incident analysis could show that provenance-bearing ledgers add recovery cost without improving diagnosis, rollback, or accountability.
3. Comparative delivery data could show that observing a genuine pre-change failure contributes no useful discrimination beyond tests written after the implementation.
4. Operational evidence could show that immutable execution constraints create compensating bypasses whose risk exceeds the behavior they constrain.
5. A more precise formalism could replace the verification-horizon construct and make different, testable predictions.

A future empirical program should define outcome measures before collecting them, following the information-need direction of ISO/IEC/IEEE 15939. Candidate measures include objective-deviation detection time, unauthorized-effect rate, rollback completion time, stale-evidence acceptance rate, and the fraction of accepted claims with reproducible provenance. Such measures need operational definitions and denominator discipline before they can support comparison.

Conclusion

Trustworthy autonomous software delivery is not established by an agent's fluency, a green dashboard, or the number of tools invoked. It is established incrementally by a control structure that binds authority to bounded work, claims to instruments, observations to source state, decisions to evidence, and progress to recovery.

The verification horizon is a discipline for stopping claims at the boundary of the evidence. It does not demand pessimism. It makes useful autonomy more credible by distinguishing what the system observed from what it inferred and what it merely hopes will remain true.

The proposed architecture is intentionally practical: a supervisor selects one intervention; a bounded worker executes it; TDD supplies local rejection and acceptance signals; independent gates examine different risks; a provenance ledger records the chain; authority tiers prevent the system from redefining its own success; and reversible progression limits the cost of being wrong.

The open empirical question is whether this structure improves real delivery outcomes at acceptable cost. Answering it requires measured deployments, counterfactual comparison, incident data, and instruments designed from explicit information needs. Until then, the model should be treated as a falsifiable engineering proposal—not a declaration that autonomy has been solved.

References

1. Turing, A. M. (1950). “Computing Machinery and Intelligence.” Mind, 59(236), 433–460. https://doi.org/10.1093/mind/LIX.236.433
2. Ashby, W. Ross (1956). An Introduction to Cybernetics. John Wiley & Sons. Biodiversity Heritage Library edition: https://doi.org/10.5962/bhl.title.5851
3. Leveson, Nancy G. (2004). “A New Accident Model for Engineering Safer Systems.” Safety Science, 42(4), 237–270. https://doi.org/10.1016/S0925-7535(03)00047-X
4. Beck, Kent (2002). Test Driven Development: By Example. Addison-Wesley. ISBN 9780321146533.
5. Lebo, Timothy; Sahoo, Satya; McGuinness, Deborah (eds.) (2013). “PROV-O: The PROV Ontology.” W3C Recommendation. https://www.w3.org/TR/prov-o/
6. Tabassi, Elham (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. https://doi.org/10.6028/NIST.AI.100-1
7. Autio, Chloe; Schwartz, Reva; Dunietz, Jesse; Jain, Shomik; Stanley, Martin; Tabassi, Elham; Hall, Patrick; Roberts, Kamie (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. https://doi.org/10.6028/NIST.AI.600-1
8. ISO/IEC/IEEE 15939:2017. Systems and software engineering — Measurement process. https://www.iso.org/standard/71197.html