What proves each requirement
A model says what the system promises: its requirements. archec status
answers the next question. What proves each promise today?
For each requirement it shows what satisfies it, where that is built in the generated code, and every piece of evidence: a test case, a property the model checker proved, timing measured on a board, a log of the real system. When something the evidence depended on has changed since, it says so. When nothing proves a requirement, it says that too.
How it works#
The commands that prove things record what they found:
| Command | What it records |
|---|---|
archec test |
each case that ran, passed or failed, on the simulator |
archec verify |
each formal property NuSMV or nuXmv proved or refuted |
archec timing |
each requirement a chain, a thread or a monitor of the traces speaks for |
archec replay |
each requirement checked on the real system's logs |
archec live --replay |
the final state of each requirement |
archec test --against <ref> |
each case that fails on an older build and passes on this one: a fix |
archec cad |
each limit and clearance a requirement puts on a part's geometry (CAD as code) |
archec synth |
each limit a requirement puts on what synthesis measures of an FPGA component (limit Pwm.luts <= 500), and each FPGA component placed and routed for its device (FPGA components) |
Each record goes in .arche/evidence/, beside your main file, one file per
command. You do nothing more: run your tests as usual.
Then ask:
$ archec status
REQ-FAN Fan on when hot
The fan shall start when the air is above 28.
satisfied by Thermostat (ctl)
built into gen/c/thermostat.c:11-19,97-128, gen/c/thermostat.h:16-20
✓ test TP/TC-HOT web sim 2 h ago
✓ verify REQ-FAN NuSMV 2 h ago ctl in Idle leads to ctl in Cooling
✓ timing chain react board nucleo-f446re 3 h ago chain react max 212ms ≤ budget 1s (40 run(s))
REQ-STOP Fan stops when cool (out of date)
! test TP/TC-COOL web sim out of date: Thermostat: Cooling -> Idle on READING changed
REQ-SAFE Never both (nothing proves it)
✗ nothing proves it add a test case that verifies it, or a formal property for archec verify
3 requirement(s): 1 proved, 1 out of date, 1 unproved, 0 failing
A requirement is in one of four states:
- proved: it has evidence, all of it current and passing.
- out of date: something a piece of evidence depended on changed since it ran. Run it again.
- failing: a current piece of evidence failed.
- nothing proves it: no evidence at all. A case that verifies it but never
ran is listed, with
never ran.
What "out of date" means#
Each record keeps a fingerprint (a short hash) of every model element the run
depended on. When you run archec status, it hashes the model again and
compares.
What a run depends on:
- A test case: the case itself, the requirements it verifies, the parts it names, the parts that satisfy those requirements, and every part upstream of them (a part that sends them messages).
- A formal property: the requirement, the parts it names, and the parts upstream of them.
- A timing run or a replay: the requirement, the deployment, the parts measured, and the code that ran (see below).
A part is hashed piece by piece: each function, port, variable, state and
transition apart, then the rest. So the message names the piece that changed:
Thermostat.update changed, or Thermostat: Idle -> Cooling on READING changed
for a guard. A part with no link to what a case checks is not in its
fingerprints: editing it changes nothing. Comments and layout do not count. A
requirement is hashed by its wording only, so changing its status or owner
keeps its evidence current.
Runs on a board or an emulator#
archec timing, replay and live take three options to say where they ran:
--board <id>: on that board (board nucleo-f446rein the report).--place <where>: anywhere else, such asrenodeorhost.--firmware <file>: the firmware that ran. Its hash is kept. When the file changes, the evidence is out of date.
The environment can say the same: ARCHE_BOARD, ARCHE_EVIDENCE_PLACE and
ARCHE_FIRMWARE. A script that flashes a board can set them once. Without
them, the place is trace or log.
The generated files that hold the code of the measured parts are hashed too.
archec status finds them through the genmaps archec gen writes
(<out>/.archec/genmap.json), up to three folders down. The same genmaps give
the built into lines.
The build a board runs#
Firmware generated with --trace (--target embassy or c) says which build
it is, in the first line of its trace:
arche-trace 0 build 79bbe5250c9e 3f2a1c9b0d
The first word is the model's build fingerprint: one short hash of everything
code is generated from (components, messages, systems, deployments, types,
the .py files of bodies). Editing a requirement's wording or a test case does
not change it. The second is the git commit the code was generated at:
3f2a1c9b0d-dirty when tracked files had changes, nogit outside a
repository (ARCHE_BUILD_COMMIT says it instead, in CI).
archec replay, timing and live --replay read that line before anything
else:
- The trace is of this model's build. One grey line says so, and each
record keeps the build (
"build": {"fingerprint", "commit"}).--firmwareis no longer needed to know what ran. When the model changes later, the record is out of date:the model changed since build 79bbe5250c9e ran (rebuild and flash). - The trace is of another build. The first line of the report is
✗ the board runs an older build: rebuild and flash, before any verdict. Nothing is recorded, and the command exits 1.--any-buildjudges the trace anyway (an old log you want to read); it still records nothing. - The trace has no build line (a CAN log, a firmware generated before): nothing is said, and everything works as before.
To ask only that question:
$ archec status --build # the model's build now
build 79bbe5250c9e 3f2a1c9b0d
$ archec status --build build/board/firmware.trace
✓ build 79bbe5250c9e, commit 3f2a1c9b0d: the firmware is this model's
It exits 1 when the trace is of another build, 2 when the trace does not say.
A script that flashes a board asks it first, as examples/greenhouse/board.sh
does two seconds after the reset.
The test ladder#
archec test main.arche --ladder runs the same proof on three rungs (four
for a model with FPGA parts: ghdl, between renode and board), lowest
first, and stops at the first rung that fails:
$ archec test main.arche --ladder --board 0483:374b
✓ sim 4 case(s) passed on the browser simulator
✓ renode board.sh --renode held (212 s)
✓ board board.sh held (95 s)
✓ ladder: held up to board
| Rung | What runs | It runs when |
|---|---|---|
sim |
the selected cases (--plan, --case) on the simulator |
always |
renode |
the project's hook with --renode |
the deployment has a binary on a board, renode is installed, and there is a hook |
ghdl |
archec test --target vhdl: the cases on the generated VHDL, in GHDL (fpga.md) |
the model has FPGA parts, and GHDL is here; the rung is named only for such a model |
board |
the hook with no argument | the same, a flash tool of the chip (probe-rs…) is installed, and you asked with --board <probe> |
For an FPGA there is one rung more, above ghdl: synth, the design
synthesised, placed and routed for its device. The ladder does not run it:
archec synth does (fpga.md), and archec status
shows it. It says a component is built for its device and meets its clock.
Its requirements are still proved by its tests.
The board rung flashes, so it never runs unless --board asks for it. A rung
whose tool is missing is skipped, with the reason, not failed:
– renode skipped: renode not installed. Nothing is downloaded: Renode and
probe-rs are yours to install.
The hook is a script beside the model that builds the firmware with its
trace, runs it, and hands the trace to archec timing and archec replay:
board.sh, else --hook <script> or ARCHE_LADDER_HOOK.
examples/greenhouse/board.sh is one to copy. Its contract:
hook --renoderuns the firmware in Renode;hookflashes the board and records its trace.- It exits 0 when everything held, 2 when it could not run (a tool or the
board is missing: the rung is skipped, with the hook's last
✗line as the reason), anything else when something did not hold. - It is given
ARCHEC(this archec),ARCHE_LADDER(renodeorboard),ARCHE_EVIDENCE_PLACE=renodeon the Renode rung, andARCHE_PROBEandARCHE_BOARD(the value of--board) on the board rung. So what itsarchec timingandarchec replayrecord says which rung it came from.
Each rung's output is kept in <out>/.archec/ladder/<rung>.log (--verbose
prints it as it runs), and the ladder in <out>/.archec/ladder.json (--json
prints it).
Every record has a place, and a place is on a rung: sim and web sim are
the simulator, renode is Renode, board <id> is the board. archec status
shows the highest rung where everything current passed, once a rung above the
simulator has evidence:
REQ-SAFE No short cycling
highest rung renode (sim ✓, renode ✓, board –)
✓ test TP-GREENHOUSE/TC-SAFE web sim 2 h ago
✓ replay REQ-SAFE renode 2 h ago 1 checked
– is a rung with no evidence, ✗ one where something current failed, !
one that is out of date. --json gives rung and rungs per requirement.
A test that proves a fix#
A regression test is worth something when it fails on the build that has the
bug and passes on the one that fixes it. archec test --against <git-ref>
runs both:
$ archec test main.arche --against HEAD
archec test --against HEAD 1 case(s) on 3f2a1c9b0d (the model as it was, the cases as they are), then here
✓ TP/TC-WARM proves the fix: it fails on HEAD (3f2a1c9b0d) and passes here
✓ 1 case(s) prove the fix
The older build is a checkout of the ref in a temporary folder (a git
worktree, removed afterwards). The cases run there as they are written now,
added to the model as it was, then on the working tree. Both runs are on the
simulator. With no --plan or --case, the cases are the ones that are new
or changed since the ref.
| On the ref | Here | What it says |
|---|---|---|
| fails | passes | proves the fix: a fix record |
| passes | passes | this test does not catch the bug: recorded too, in yellow |
| any | fails | fails here: the fix is not in |
The command exits 0 only when every case proves the fix. archec status
shows a fix line under the requirements the case verifies:
✓ test TP/TC-WARM web sim just now
✓ fix TP/TC-WARM web sim just now proves the fix: it fails on HEAD (3f2a1c9b0d) and passes here
A fix record says what a test is worth, not whether the requirement holds:
it never changes a requirement's state. It depends on the case alone, so it
goes out of date only when the case is edited.
In the editor and in the documents#
In VS Code, a line above each requirement says the same in short: proved by 3 tests · timing on board nucleo-f446re ✓ · 2 h ago, out of date: … or
nothing proves it. A requirement proved above the simulator, or by a test
that proves a fix, says so: … · proves a fix · up to renode. Click it for
the full status in the Arche Status output.
Three commands of the palette run the checks above with the archec the extension ships, and show their report in the same output:
- Arche: Run Test Ladder…: it asks how far up, and for the probe when the board is included.
- Arche: Prove a Fix Against a Git Ref…: it asks for the ref and the case.
- Arche: Check Board's Build from Trace…: it asks for the trace file.
They work with no internet. Renode and probe-rs are not part of the extension: a rung whose tool is not installed is reported skipped. See the VS Code extension.
archec doc adds a chip to each requirement (proved, out of date, failing,
nothing proves it) and a "Proved by" line with each piece of evidence, its
place and its date. It does so only when the project has evidence.
In CI#
archec status exits 1 when a requirement fails, is out of date, or has
nothing that proves it. That makes a gate:
archec test main.arche
archec verify main.arche
archec status main.arche
--lenient exits 1 only when a requirement fails. --req REQ-FAN looks at one
requirement. --json prints the whole report for another tool.
Commit the evidence, or not#
The files in .arche/evidence/ are small JSON lines, sorted, one line per
subject and place. A new run replaces its own lines. Commit them, and the
repository shows what proved the model at each commit, CI's runs included. To
keep them on your machine only, add .arche/evidence/ to .gitignore.
Any of the commands takes --no-record to record nothing (or set
ARCHE_NO_RECORD=1).
Limits#
- The dependencies are found from the model's text and links, not from what a run really did. A case that names no part depends on the whole system. A part reached only through a function call, not a link, is not seen as upstream.
- Types and constants a part uses are not fingerprinted, only the messages of
its ports. A body in a
.pyfile is hashed as a whole file; a TypeScript handler file is not hashed. - A firmware is compared only where it is: on a machine without it, its record stays current.
archec liverecords its final state; a live session under podman is recorded when it stops, not as it runs.- The cases that run on arche-cli (
archec testwithout--sim web) are recorded assim; the tests of the generated C (gen --target c-tests --run) are not recorded yet. - One record is kept per subject, place and requirement: the history is git's.
- The build fingerprint is of the model's text. A change in a TypeScript handler file, a driver's source or the toolchain gives the same fingerprint: the commit beside it tells those apart.
- The ladder's rungs above the simulator are what the project's hook does. archec does not build or flash the firmware itself, and does not check that the hook ran the selected cases.
--againstcarries a case to the older build as text. A case made from acasestemplate, or written in another file than its plan, is not carried. A case that names a part the older model does not have cannot run there: the command says so and proves nothing.- Nothing here has run on a real board yet: the board rung is tested with a stand-in hook.