Benchmark · TRACE (working name)

TRACE scorecard — run-1

Run window 2026-10-06 · 5 products · scorecard.json · results · what we are fixing · plan · re-run and challenge

What TRACE measures, and why

TRACE is the public, re-runnable part of the method we use internally to steer our own work. That method is called ARRF. It places each product between the two reference points above and asks, for each part of an answer (accuracy, currency, authority, application to the facts and so on), how much of the distance from the floor to the ceiling we have closed. There is no single overall score: each dimension is reported on its own.

Before we may say anything about a product, it has to earn a step on a ladder. The first step (R1) is "every claim traceable to a source we read, with its status computed as at a stated date". Later steps are about being more rigorous and repeatable than AI research (R2), applying the law to a reader's facts (R3) and structuring answers to the standard of specialist advice (R4). A product never states a step it has not earned. We never claim to be a law firm, to give legal advice, or to make anyone fully compliant.

How confident each answer should be is the job of our risk engine, RRIE. Calibration (whether a stated confidence matches how often the answer is right) cannot be measured yet, because our published records do not yet carry a stated confidence for each claim. We say so rather than estimate it.

Half of the internal question bank is held back and never published, so that our own monthly re-scoring stays honest. Only the other half will ever appear here.

Where each product stands

Track B checks each product's published records: 170 Advennt, 170 Crypto, 170 Data Protection, 170 Financial Integrity, 170 World Payments Monitor. Products are listed alphabetically. This is not a ranking. B1 and B2 count every failing row the fleet provenance gate finds, split into fatal rows (the gate fails them) and warnings. FATAL on B1 or B2 means new fatal rows exist against that product's debt baseline. Failing rows are listed in the repository's per-product failing files.

What we are fixing because of this run

Each finding below is an internal fix with its own reference, re-measured in the next run.

• FR-1335 Crypto, Data Protection. Most of these products' sources store their official-source grade in a different field from the one our provenance check reads, so those sources are never tested as official sources. The check will read both fields.
• FR-1336 Advennt. Advennt's own provenance check skips its regional records (Africa, Asia-Pacific, EEA, Latin America and others), so failures in those records do not show up there. TRACE found them; the check will cover them.
• FR-1337 World Payments, Crypto, Data Protection, Financial Integrity. These four products have no recorded baseline of existing provenance debt, so every fatal row counts as new and nothing stops the debt from growing. Each will record its existing debt, fail only on new debt, and show the remaining debt every run.
• FR-1338 All five. Our rule for which websites count as official sources has defects: a United States congressional site is graded official for Gibraltar, and Dubai's virtual-assets regulator is not graded official for the UAE. The rule will be corrected and re-measured.
• FR-1339 All five. Date coherence (B4) and gap honesty (B5) cannot be measured yet, because no record carries separate checked, published and changed dates for each claim, or a register of known gaps. The shared record format will carry them.

Action plan

Every month:

1. Before each run, we record in advance what will be measured, which models judge it and which questions are used.
2. We run the checks on every product's published records, and the answer tests once the question set exists.
3. We publish the results here as found. Each run is kept; past runs are never overwritten. Bad results get the same space as good ones.
4. Every failure is tied to the internal fix that owns it.
5. The next run measures it again. A fix is not done until its measure moves.

What comes next, in order:

Before results can stop being a self-assessment (the open-alpha gates):

Re-run and challenge

The repository is not yet public: it stays private until the work is more advanced. The commit and re-run command below are shown anyway, so that this run is pinned to exact code and inputs. They become usable by anyone when the repository opens.

Challenges are open now through our contact page: email [email protected] naming the product, the figure and why you think it is wrong, or write to Asymmetric Intelligence Limited at the registered office (Unit G02, Eurocity, Europort Avenue, Gibraltar GX11 1AA). Challenges and our rulings will be published with the next run. Open challenges: 0.

Changelog

• run-0: page scaffold; no results yet.
• run-1: first Track B run on recorded provenance of all five union consumers (published JID records). Results as found.

Data: CC BY 4.0. Information, not legal advice. Machine-readable: runs/run-1/scorecard.json.