§03-iv Minerva — Governed Production — Governed benchmark generation
Mars® Spec › §03-iv Minerva — Governed Production › Governed benchmark generation
← Governed data generation · Section index · Outputs →
3. Governed benchmark generation
3.1 What it is
Governed benchmark generation is the derivation of a formal evaluation surface from the governing specification and domain model. A benchmark instantiates the orders and considerations of the governing specification into concrete, evaluable test cases: each test case is a governed evaluation act whose expected outcome is derivable from the governing specification.
A benchmark is not a dataset. It is a specification-derived artifact: its test cases are generated from the governing specification’s clauses and the domain model’s aspects, not from empirical observations. Its function is to make model behavior evaluable against the governing specification without requiring live inference on proprietary or sensitive data.
3.2 What a benchmark enables
A benchmark addresses a gap that neither data acquisition nor governed data generation can fill: the evaluability gap — where the governing specification defines requirements that can be formally expressed as test cases, but no empirical data exists or can be generated to cover them. Benchmarks are also the operative artifact for model certification against a domain: a licensee who wants to certify a model’s behavior profile against their governing specification requires a benchmark as the evaluation surface.
A benchmark also serves as the evaluation surface for the model-gap signal in the reinforcement loop (§03-iii §2.8): where a model-gap is identified and the amendment action is model re-training or model replacement, the benchmark governs what the updated model must satisfy before the gap is declared closed.
3.3 Governing conditions
A governed benchmark generation act requires:
-
A governing specification. Each test case is derived from a clause, provision, or consideration of the governing specification. The derivation is recorded in the benchmark’s recomputation witness.
-
A domain model — one or more. Test cases are typed against the model’s dimensions: each test case is bound to one or more domain aspects and tagged with the order type(s), perspective tag(s), and variant identifier(s) of those aspects. Where multiple domain models are active, each test case records the domain model version encoded in its bound aspect identifier(s).
-
A declared benchmark specification. A registered artifact declaring: the test-case generation method, the governing specification version under which test cases were derived, the aspect-coverage target, and the expected-outcome derivation procedure (how expected outputs are determined from the governing specification).
3.4 Test case structure
Each test case in a governed benchmark carries:
| Field | Content |
|---|---|
| Case identity | A stable identifier unique within the benchmark version |
| Content hash | A content hash computed over all remaining test-case fields (excluding the case identity); the case identity together with the content hash constitute the test case’s version identity |
| Aspect binding | The domain aspect identifier(s) from the order-decomposition tree |
| Order type | The order type(s) of the bound aspects |
| Input | The governed input — a prompt or non-linguistic input consistent with the declared invocation class |
| Expected outcome | The outcome derivable from the governing specification for this input; drawn from the conformance verdict enumeration or a declared extension |
| Derivation record | The specification clause(s) and domain model aspect(s) from which the expected outcome is derived |
| Falsifier annotation | The condition under which the expected outcome would change (specification amendment, aspect scope change) |
The benchmark as a whole carries a content hash computed over the ordered collection of test-case content hashes, where test cases are ordered lexicographically by their content hash for the purpose of computing the aggregate. On benchmark staleness (§3.8), the stale test cases are identified by comparing the new benchmark’s per-case hashes against the prior version’s; only cases whose content hash has changed are stale. This scopes consumer re-evaluation to the changed cases rather than requiring full benchmark re-evaluation.
3.5 Recomputation witness for governed benchmark generation
The inventive matter is the co-required set of three structural invariants, not the count. A recomputation witness for a governed benchmark generation act is defined by the co-presence of three structural invariants — each independently required, none waivable. The canonical realization records each invariant as a distinct component. A witness carrying all three invariants is conforming; a witness omitting any invariant is structurally deficient.
| Invariant | Canonical component | What it requires |
|---|---|---|
| Derivation trace | (i) | For each test case: the case identity and content hash (from §3.4), the specification clause(s) and domain model aspect(s) from which it was derived, the expected-outcome derivation procedure applied, and the benchmark specification version. The case identity and content hash are the binding between the witness entry and the specific test case in the delivered benchmark; a consumer verifying the witness matches the case identity against the received benchmark and recomputes the content hash from the received test-case fields to confirm no tampering. |
| Coverage record | (ii) | For each aspect in the declared aspect-coverage target: the number of test cases bound to that aspect and any gap-on-insufficiency where coverage targets were not met |
| Falsifier record | (iii) | For each test case: the conditions under which the expected outcome would be falsified by a governing specification amendment, with the delta-attestation classification (§02b §5.2) that would apply to each such amendment |
An independent inspector can re-derive from this witness: what test cases were generated, from which specification clauses and domain model aspects, under what derivation procedure, whether coverage targets were met, and under what specification amendments each test case’s expected outcome would change — without access to the generation infrastructure.
3.6 Model evaluation scenarios
A governed benchmark serves as the evaluation surface for model fitness assessment across three scenarios. The benchmark mechanism is identical across all three; what varies is the certificate path and what the resulting model-behavior certificate asserts.
New model. A model with no prior conformance record under the current governing specification. The benchmark evaluation establishes a behavior baseline: the first model-behavior certificate for this model / governing-specification pair. The certificate’s certification-path record (Field 3) carries the benchmark identity and version. No prior certificate is referenced; delta comparison is not applicable. The baseline certificate becomes the reference for all subsequent evaluations of this model under this governing specification version.
Modified model. A model whose weights, parameters, or generation behavior have been altered from a prior version by any mechanism — including but not limited to fine-tuning, reinforcement learning from human or automated feedback, adapter insertion, prefix-tuning, prompt-tuning, quantization, distillation, or any automated post-training optimization procedure. The benchmark evaluation produces a model-behavior certificate that is structurally comparable to the prior version’s certificate. The certificate path must carry a reference to the prior model version’s model-behavior certificate and a declared modification record identifying the modification mechanism (the licensee’s declared label for the post-training procedure applied). A model version whose modification mechanism is undeclared or whose prior model-behavior certificate is absent does not qualify for delta comparison and is treated as a new model under the governing specification.
The requirement for a declared modification record means that automated post-training procedures — including opaque or vendor-managed optimization pipelines — must be declared as a registered modification artifact before the resulting model version may be evaluated under the delta path. A model version that carries no declared modification record and no modification artifact identity is not evaluable as a modified model; it is evaluated as a new model and receives no delta classification. This prevents undeclared optimization from bypassing the delta-attestation discipline.
The delta between the two certificates is classifiable under the same four-valued lifecycle used for specification amendments:
| Delta class | Meaning |
|---|---|
| Scope-preserving | All aspects conforming under the prior version remain conforming; no regressions |
| Scope-extending | Prior conforming aspects remain conforming; additional aspects now conforming |
| Scope-restricting | One or more aspects conforming under the prior version are now conditionally-conforming or non-conforming |
| Scope-breaking | One or more aspects that were conforming under the prior version are now non-conforming at a severity that invalidates the prior baseline |
A scope-restricting or scope-breaking modification delta triggers the delta-attestation cascade (§02b §5.5) for any downstream governed artifact whose certificate path references the prior model version. This makes model modification — regardless of the modification mechanism — evaluable under the same delta-attestation discipline as governing specification amendments.
External model. A model whose development history, training procedure, and prior evaluations are outside the licensee’s governed context. No prior certificate under this governing specification exists. The benchmark evaluation establishes whether the external model is operable within the licensee’s governed deployment. The certificate’s certification-path record (Field 3) carries the benchmark identity and version rather than a specification amendment chain — there being no prior certificate or amendment chain to reference for an external model. External model documentation — vendor safety cards, published eval results, model cards — is not a governing specification artifact under this architecture and does not substitute for or supplement the benchmark-derived model-behavior certificate.
Model-behavior certificate. Regardless of scenario, the model evaluation produces a model-behavior certificate: a conformance certificate (§02b §2) whose target identity is the model artifact (identified by model identifier, version, and provenance hash), whose grounded verdict is drawn from a registered verdict vocabulary whose canonical values are {conforming, non-conforming, conditionally-conforming} against the benchmark’s aspect-coverage surface (additional verdict values may be declared in a registered extension to the vocabulary, consistent with §03-iii §2.6), and whose recomputation witness (Certificate Field 7) is the model-evaluation witness defined below. The model-behavior certificate is the operative artifact for model-gap closure in the reinforcement loop (§03-iii §2.10).
Model-evaluation witness — structural invariants. The inventive matter is the co-required set of three structural invariants, not the count. A model-evaluation witness is defined by the co-presence of three structural invariants — each independently required, none waivable.
| Invariant | Canonical component | What it requires |
|---|---|---|
| Evaluation trace | (i) | For each test case evaluated: the test case identity and content hash; the model’s output; the per-case outcome (drawn from the registered outcome vocabulary defined in §3.7, whose canonical values are {match, mismatch, partial-match, tolerance-boundary, indeterminate}); and the aspect binding and order type of the test case |
| Verdict derivation record | (ii) | The aggregation procedure applied to per-case outcomes to produce the aggregate verdict; the aspect-level conformance fractions; and, for a modified model, the delta classification (scope-preserving / scope-extending / scope-restricting / scope-breaking) against the prior model version’s certificate, together with the modification record identity |
| Benchmark version binding | (iii) | The benchmark identity, version (content hash), and governing specification version under which the evaluation was conducted; and the consumer identity and version if a registered consumer conducted the evaluation |
An independent inspector can re-derive from this witness: which test cases were evaluated, what the model produced, whether each case matched its expected outcome, how the aggregate verdict was computed, and whether any modification-induced regression occurred — without access to the model deployment or the consumer’s internal state.
3.7 Benchmark consumer interface
A benchmark is a published governed artifact. Any registered governance component that declares a benchmark evaluation interface may consume a governed benchmark, evaluate a model against it, and return a typed verification result that feeds into the certificate path.
Consumer registration. A benchmark consumer is registered as a typed artifact in the specification registry. Its registration record declares:
| Field | Content |
|---|---|
| Consumer identity | A stable identifier for the consumer (governance component name, version, provenance hash) |
| Evaluation interface | The protocol by which the consumer receives the benchmark — pull (registry-fetch), push (direct delivery), or other declared delivery protocol |
| Input contract | The benchmark fields the consumer requires; consumers must declare which test-case fields they consume |
| Output contract | The typed verification result the consumer returns; must include at minimum: a per-test-case outcome record, an aggregate verdict, and a recomputation witness for the evaluation act |
| Scope declaration | The aspects and order types over which the consumer’s evaluation is authoritative |
Delivery topology. This Mars® specification is agnostic on delivery topology. Two admitted embodiments, non-exclusively:
-
Registry-pull. The benchmark is published to the specification registry. Registered consumers fetch it on a declared schedule or on benchmark version change notification. This pattern suits consumers operating autonomously — for example, a pre-invocation governance layer (AIGP) running its own model verification procedure against the published benchmark independent of the licensee’s pipeline.
-
Direct push. The licensee routes the benchmark directly to declared consumers as part of the benchmark generation act or the model evaluation pipeline. The push event is recorded in the benchmark’s recomputation witness as a delivery record naming the consumer, the benchmark version delivered, and the delivery timestamp.
Both embodiments may be used simultaneously for different consumers. The licensee’s pipeline architecture determines which pattern applies to each registered consumer.
Consumer verification result. Each consumer returns a typed verification result carrying: the consumer identity and version, the benchmark identity and version evaluated, a per-test-case outcome record (the outcome observed, the expected outcome from the benchmark, and a per-case outcome classification — see below), an aggregate verdict, and a recomputation witness for the consumer’s evaluation act. The recomputation witness must satisfy the structural requirements of §03-iii §3.4: it must be sufficient for an independent inspector to re-derive the consumer’s verdict from the benchmark test cases and the model’s outputs, without access to the consumer’s internal state.
Per-case outcome classification. The per-case outcome classification is drawn from a registered outcome vocabulary whose canonical values are:
| Value | Meaning |
|---|---|
| match | The observed outcome satisfies the expected outcome as specified in the test case, within any declared tolerance |
| mismatch | The observed outcome does not satisfy the expected outcome and does not fall within any declared tolerance or extension |
| partial-match | The observed outcome satisfies the expected outcome on a declared subset of aspects or conditions; the specific satisfied and unsatisfied aspects are recorded in the per-case record |
| tolerance-boundary | The observed outcome satisfies the expected outcome only when a declared tolerance is applied; the tolerance value and the raw distance are recorded in the per-case record |
| indeterminate | The consumer’s evaluation procedure cannot classify the outcome for this test case (e.g., the test case’s aspect is outside the consumer’s declared scope, or the model’s output is malformed); the reason is recorded in the per-case record |
Additional outcome values may be declared in a registered extension to the outcome vocabulary, consistent with the same extensibility rule as the verdict vocabulary (§03-iii §2.6). A consumer whose outcome vocabulary omits partial-match or indeterminate — treating all outcomes as binary match/mismatch — produces a structurally narrower evaluation record; this does not violate the specification but must be noted in the consumer’s scope declaration so that downstream pipeline components receiving the verification result can interpret the binary classification correctly.
Multi-consumer certificate path. Where multiple consumers evaluate the same benchmark against the same model, their verification results are collected into a multi-consumer evaluation record: a single record identifying the benchmark version, the model version, and the set of consumer verification results. The model-behavior certificate (§3.6) carries or references this record in Certificate Field 3 (certification-path record). Each consumer’s result is independently recomputable; the multi-consumer record does not merge them into a single verdict.
The licensee determines which consumer verdicts are required for model-gap closure and which are informational. The governing specification declares the admissibility conditions: a model-behavior certificate may require unanimous conformance across all registered consumers, majority conformance, or conformance from a declared subset. The admissibility condition is a registered parameter of the governing specification, not a hard-coded rule.
3.8 Benchmark lifecycle
Versioning. A benchmark is a versioned registered artifact. Its version is bound to the governing specification version and domain model version under which it was generated. A benchmark generated under specification version N is not valid for evaluating against specification version N+1 unless a delta-attestation analysis confirms the benchmark’s test cases are unaffected by the amendment (scope-preserving or scope-extending without test-case invalidation).
Staleness. A specification amendment classified as scope-restricting or scope-breaking triggers benchmark staleness for any test case whose derivation record references the amended clause(s). Stale test cases are flagged and excluded from the evaluation until regenerated under the updated specification version.
Consumer staleness. Where a registered consumer’s scope declaration references aspects affected by a scope-restricting or scope-breaking specification amendment, the consumer’s prior verification results for those aspects are marked stale. The consumer must re-evaluate the updated benchmark version before its results may be used for model-gap closure on the affected aspects.
Benchmark-anchored gap closure. As stated in §03-iii §2.11, a benchmark may resolve an evaluability gap. The benchmark identity, version, and the gap-closure correspondence record are carried in the gap closure witness.
3.9 Conformance certificate for generated benchmarks
A governed benchmark generation act may receive a governing-benchmark certificate: a conformance certificate (§02b §2) whose target identity is the benchmark artifact, whose recomputation witness is the three-invariant benchmark generation witness of §3.5, and whose grounded verdict reflects the coverage and derivation-correctness outcomes.
3.10 Train/evaluation disjointness
A model evaluation against a benchmark is sound only if the model was not trained on the benchmark’s cases. This section makes that soundness condition a structural, checkable invariant rather than an assumed discipline — using the per-item content hashes that generated corpora (§2.3) and benchmark test cases (§3.4) already carry.
Disjointness invariant. A model-behavior certificate (§3.6) issued from a benchmark evaluation is conforming only if the benchmark’s test cases and the model’s registered training corpus are disjoint under a declared similarity bound. Disjointness is evaluated over content: exact-match is detected by content-hash intersection of benchmark case hashes against training-item hashes; near-duplication is detected by a declared similarity measure and threshold over the two hash-addressed sets. The declared similarity measure and threshold are registered parameters of the governing specification. A benchmark case whose content matches or falls within the similarity threshold of a training item is a contaminated case; a model-behavior certificate whose evaluation included contaminated cases is structurally deficient and is narrowed in declared scope to exclude the contaminated cases, or refused where contamination is pervasive.
Benchmark-exposure record. Every model-behavior certificate carries a benchmark-exposure record: the identity and version of the model’s registered training corpus, the benchmark identity and version, the declared similarity measure and threshold, and the disjointness result (clean / narrowed-by-N-contaminated-cases / refused). A model whose training corpus is undeclared cannot be evaluated for disjointness and is treated for this purpose as potentially-contaminated: its model-behavior certificate carries an exposure-undeclared annotation and may not claim clean-disjointness. An external model (§3.6) whose training history is outside the governed context carries exposure-external and is evaluated with the disjointness result explicitly undeterminable — the annotation is the honest disclosure, not a pass.
Closed-loop separation. The closed generative loop (§3.6; §02b §5.5 generative-loop trajectory witness) co-generates a benchmark and a dataset and evaluates a candidate against the co-generated benchmark. Co-generation makes contamination the default rather than the exception: without a separation invariant, the loop evaluates a model against cases drawn from the same generative act that produced its training data. The loop is therefore conforming only when the generated training partition and the generated benchmark partition are disjoint under the declared similarity bound at each iteration, and the trajectory witness records the per-iteration disjointness result. A generative-loop trajectory witness that does not carry per-iteration train/benchmark disjointness results is structurally incomplete, and a loop that evaluates against a benchmark not disjoint from its own generated training partition is producing overfitting-by-construction, not a governed evaluation.