Evidence
Rigor that can be reproduced: every claim comes with its source and the command that checks it.
Each row gives the claim, the number as published in the repository, how it was measured, the command that reproduces it and the source files. Commands run from the root of the Exemplar repository; they are the ones in the README’s tests and verification section, with Python run through uv.
Verification table
Each row: the claim, the number, how it was measured, the command and the file.
The Z3 encoding agrees with two other ways of computing the answer
3,850 randomized cases
Published in docs/DEVPOST.md, line 93
- Over 6,000 randomized cases ran during development with 0 mismatches (README.md, line 215)
- 2,000 randomly generated sketches compare the Z3 encoding with a NumPy evaluator and 1,850 more compare the solver with an exhaustive search (README.md, line 215)
Randomly generated rule sketches and traces give the same answer from the Z3 encoding, from a separate evaluator and from a brute-force search.
How it was measured
Hypothesis generates sketches, traces and labelled examples. One comparison checks the Z3 encoding against the NumPy evaluator; another checks the solver against an exhaustive search over the same instance.
A third, literal reference implementation of the semantics checks the vectorised evaluator bit for bit.
The command below runs the deep configuration, ten times the default number of cases (about 5 to 7 minutes).
Reproduce it (about 5 to 7 minutes)
SYNTH_PROP_SCALE=10 uv run --project engine pytest engine/tests/synth/test_synth_encoding.py engine/tests/synth/test_synth_solve.pySource files
- engine/tests/synth/test_synth_encoding.py — Z3 encoding against the NumPy evaluator
- engine/tests/synth/test_synth_solve.py — solver against exhaustive search
- engine/tests/synth/synth_support.py — instance generators and the brute-force reference
- engine/tests/dsl/dsl_reference.py — the literal reference evaluator
Deliberately planted solver bugs are caught
7 of 7 deliberately planted solver bugs caught
Published in docs/DEVPOST.md, line 94
The agreement test above has power: it fails when the encoder is wrong.
How it was measured
Each case monkeypatches one realistic bug into the encoder, such as an off-by-one on a window edge, a dropped GE/LE swap or a lost midnight wrap, and checks that the same comparison reports a mismatch.
A control case runs the unmodified encoder through the identical harness and must pass. The streams are derandomised, so the result is the same on every run.
Reproduce it (about 15 seconds)
uv run --project engine pytest engine/tests/synth/test_synth_mutations.pySource files
- engine/tests/synth/test_synth_mutations.py — the planted bugs and the control
The false-alarm bound holds in simulation
A 100,000-draw simulation of the false-alarm bound
Published in docs/DEVPOST.md, line 95
For exchangeable negatives, the chance that a calibrated rule fires on one is at most the chosen level, including when scores tie.
How it was measured
Two independent checks. First, an exact enumeration of every calibration outcome of a small discrete distribution with ties everywhere, which must come out at or below the level with no tolerance.
Second, a seeded Monte Carlo simulation: the count of false alarms must not exceed the upper quantile of the matching binomial distribution at 1e-6, so a failure means the bound is violated, not bad luck. A deliberately wrong quantile (k minus 2) fails the same check, which shows the test can fail.
The conformal rank uses exact rational arithmetic, because floating-point rounding of the rank is off by one for some levels.
Reproduce it
uv run --project engine pytest engine/tests/calibrate/test_calibrate_validity.pySource files
- engine/tests/calibrate/test_calibrate_validity.py — exact enumeration and the simulation
- engine/tests/calibrate/test_calibrate_conformal.py — rank and threshold unit tests
- engine/exemplar/calibrate/conformal.py — the implementation under test
The rule logic obeys its stated laws
Property tests
Properties of the rule language hold on generated rules and traces, not just on hand-picked examples.
How it was measured
Property tests (Hypothesis) state that the sign of robustness agrees with the boolean result, that negation is an exact duality, and that widening an eventually or then window never decreases robustness.
The semantics are written down operator by operator in docs/DSL.md, and the tests follow that document.
Reproduce it
uv run --project engine pytest engine/tests/dslSource files
- engine/tests/dsl/test_dsl_properties.py — the laws
- engine/tests/dsl/dsl_strategies.py — generators for rules and traces
- docs/DSL.md — the specified semantics
Test suites
860+ engine tests, 205 evaluation tests and 22 webhook tests
Published in docs/DEVPOST.md, line 96
The engine, the evaluation harness and the webhook code each have their own suite.
How it was measured
Engine tests cover the rule language, the solver, calibration, the Ring client, the API and the store. Evaluation tests cover the harness and the dataset tooling. Tests that call Bedrock or Ring skip themselves unless credentials are present.
Reproduce it
uv run --project engine pytest engine/tests uv run --project engine pytest eval/tests cd apps/web && node --test lib/ring/ring.test.mjsSource files
- engine/tests — engine suite
- eval/tests — evaluation suite
- apps/web/lib/ring/ring.test.mjs — webhook and WHEP allow-list tests
Signed webhooks are verified like production
22 tests
Published in README.md, line 188
Signed, tampered, re-serialised, unsigned and duplicate payloads are accepted or refused as they should be, in both Python and Node, against the same fixtures.
How it was measured
The signature is HMAC-SHA256 over the raw request body, compared in constant time, and deliveries are deduplicated on the request id. The Playground sends no webhooks, so these paths are exercised with signed fixtures and a replay tool.
Reproduce it
cd apps/web && node --test lib/ring/ring.test.mjs uv run --project engine pytest engine/tests/ringSource files
- apps/web/lib/ring/ring.test.mjs — Node tests
- engine/tests/ring/test_webhook.py — Python tests
- engine/tests/ring/fixtures — the shared signed fixtures
- scripts/send_signed_webhook.py — replay tool
The API contract is enforced against a mock
33 of 33 operations
Published in docs/DEVPOST.md, line 97
Every operation of the engine API contract answers with a schema-valid response.
How it was measured
The validator boots the mock engine and exercises each operation in the contract, failing on any response or event-stream frame that breaks it. The engine’s own test suite checks that every path and method in the contract is served and schema-valid.
Reproduce it
cd apps/web && npm run mock:validateSource files
- apps/web/mock/validate.mjs — the validator
- engine/exemplar/contracts/api.yaml — the contract
- engine/tests/test_contracts.py — contract checks on the engine side
This site’s own check
The browser code on this site is a TypeScript port of the Python engine. This report compares the two on golden files the engine generated.
Engine commit dc911769ad, generated 2026-10-18.
| Family | Cases | Identical | Tie-verified | Engine-supplied | Excluded (operational) | Mismatches |
|---|---|---|---|---|---|---|
| dsl.random | 7,886 | 7,886 | 0 | 0 | 0 | 0 |
| dsl.real | 30,806 | 30,806 | 0 | 0 | 0 | 0 |
| dsl.sweep | 5,351 | 5,351 | 0 | 0 | 0 | 0 |
| dsl.edge | 72 | 72 | 0 | 0 | 0 | 0 |
| dsl.text | 2,110 | 2,110 | 0 | 0 | 0 | 0 |
| synth.recorded | 49 | 49 | 0 | 0 | 0 | 0 |
| synth.random | 1,298 | 1,260 | 0 | 38 | 0 | 0 |
| synth.random.sat | 576 | 459 | 0 | 117 | 0 | 0 |
| lab.grid | 223,850 | 202,270 | 6,120 | 15,460 | 0 | 0 |
| cegis.presets | 133 | 133 | 0 | 0 | 0 | 0 |
| cal | 20,571 | 20,571 | 0 | 0 | 0 | 0 |
| cal.real | 467 | 467 | 0 | 0 | 0 | 0 |
| mdl | 3,510 | 3,510 | 0 | 0 | 0 | 0 |
Identical: the browser computed the same result as the engine, without reading it. Tie-verified: equally optimal, the engine chose another of the tied alternatives, verified here. Engine-supplied: shown from the engine, not re-solved here.
| Preset | Solves | Identical | Engine-supplied |
|---|---|---|---|
| p1-playground | 12 | 12 | 0 |
Cost
What one watched event costs, and how that changes as rules are added.
Published figures
$0.0014–$0.0015 per watched event in our live runs
Published in docs/DEVPOST.md, line 68
$0.0018 on average over 1,005 clips
Published in docs/DEVPOST.md, line 68
about a tenth of a millisecond
Published in docs/DEVPOST.md, line 68
$0.0133
Published in docs/eval/cost_vs_rules.md, line 23
Model cost per event as rules are added
Exemplar has Claude read an event once and scores every rule on the cached facts, so the cost stays at $0.00179 per event. The comparison line asks a strong vision model one question per rule, at $0.0133 per rule per event. One-time teaching cost is not included.
- Exemplar (perceive once, score every rule)
- Zero-shot prompting (one call per rule)
10 rules: Exemplar $0.00179 per event; zero-shot prompting $0.1330 per event (74x).
The same numbers as a table
Copied from docs/eval/cost_vs_rules.md in the repository, exactly as published.
| Rules deployed | Exemplar | Zero-shot prompting | Ratio |
|---|---|---|---|
| 1 | $0.00179 | $0.0133 | 7x |
| 2 | $0.00179 | $0.0266 | 15x |
| 3 | $0.00179 | $0.0399 | 22x |
| 4 | $0.00179 | $0.0532 | 30x |
| 5 | $0.00179 | $0.0665 | 37x |
| 6 | $0.00179 | $0.0798 | 45x |
| 7 | $0.00179 | $0.0931 | 52x |
| 8 | $0.00179 | $0.1064 | 59x |
| 9 | $0.00179 | $0.1197 | 67x |
| 10 | $0.00179 | $0.1330 | 74x |
How the numbers are obtained
- Exemplar. The mean cost of the fast model perceiving one clip, over 1,005 clips of 16 frames each. The trace is cached, so adding a rule adds no model call; scoring a rule on a cached trace takes about 0.05 to 0.1 ms in the NumPy evaluator.
- Prompting per rule. The median cost of one strong-model call on the same 16 frames (527 calls). That baseline asks one concept per call, so n rules cost n calls per event.
- Both prices come from the token counts the model returned and the per-token prices in eval/harness/budget.py. A strong-model re-check of events near a rule’s decision boundary is billed on top; the live Playground run did not need it.
Method, ledgers and the original chart: docs/eval/cost_vs_rules.md and docs/eval/cost_vs_rules.svg. Regenerate with:
uv run --project engine python eval/plots/cost_vs_rules.pyAccessibility
Worded as an automated-check result.
Accessibility: automated checks
0 axe violations
Published in README.md, line 220
An axe-core scan of every screen of the Exemplar web app, run against the mock engine, reports no violations. Contrast tokens are checked separately.
How it was measured
This is the result of automated checks. Automated scans cover only part of what accessibility needs, so it is not a conformance claim.
Reproduce it (one-time setup, then a few minutes)
cd apps/web npm i --prefix mock/.cache/pw playwright-core npm i --prefix mock/.cache/axe axe-core npm run dev:mock # in one terminal; then, in another: node mock/a11y.mjs npm run mock:contrastSource files
- apps/web/mock/a11y.mjs — the axe scan of every screen
- apps/web/mock/contrast.mjs — contrast checks on the colour tokens
- apps/web/mock/keyboard.mjs — keyboard walkthrough
Developer feedback
Entries logged while building on Ring and AWS, with links.
51 friction-log entries
Published in docs/DEVPOST.md, line 113
Each entry gives the task, the steps, what we expected and what happened, a severity, the workaround we used and a concrete suggestion. Entries marked live were reproduced against the Ring Developer Playground. The product feedback covers every tool we used: what worked, onboarding, and whether we would build with it again.