Find out whether a policychange is actually real.

A robot lab runs fifty trials, gets forty successes, and writes down eighty percent. That number is doing far more work than it can carry. The interval around it is roughly twenty points wide, so the next checkpoint can look better without being better. Change the camera angle and the same policy moves further than most model upgrades do. Ship the same weights with the wrong normalization file and a working policy stops working, with nothing in the logs but a warning.

None of this is a secret. It is in the literature, and it keeps happening anyway, because there is no standard layer that checks it.

Six modules, each answering one question you can act on. They work on their own or together, and everything they produce is a Markdown and HTML report.

Three rules run through all of it, and they are the reason the numbers are worth anything.

Fail closed, not open

If the contract check cannot prove the deployed policy matches the evaluated one, it does not pass you with a warning. It returns a non zero exit code and names the field it could not verify. Some things are not written to disk anywhere, like which of several shipped normalizer statistics is actually live, so those are listed as unverified and you supply them once.

No rate without an interval

The report layer will not print a bare percentage. Every rate carries a Wilson interval, every comparison carries a paired difference, and when a result is still inside the noise it says so and tells you how many more trials would settle it. This is a small rule that changes what a report can be used for.

Abstain instead of guessing

The judge is conformally calibrated, which means it is allowed to answer only when it can meet a stated error rate, and it hands the rest to a human. Coverage is reported next to accuracy, because a judge that answers everything at sixty percent is worse than useless to a team that has to trust it.

Everything below was produced by the tool itself on public data and on a live policy, on rented GPUs. The bundles and the scripts that made them are in the repository, so none of it has to be taken on trust.

Module by module

We checked whether this is just our tool

A limitation in our own monitor is one thing. A limitation in the approach is another, and it is the more useful finding, so we tested it. We took nine other detectors covering the families used in the recent literature, including the cheap half of VLA-FAIL, the distance and density signals from FAIL-Detect, and standard anomaly detectors, then fitted each one on successful episodes only and scored it two ways on exactly the same held-out episodes.

The measure is AUROC, which is the chance that a failed episode scores higher than a successful one. It needs no threshold, so a poor score cannot be blamed on bad tuning. 0.5 is a coin flip.

Every one of them is at chance on the failures the robots actually had, and every one of them catches injected faults. Methods that share almost nothing in how they work, a distance, a density, a tree ensemble, a kernel method, a learned novelty score and a sequence model, do not agree this closely by accident. The signal they all read does not carry the answer. Our own monitor is the best of the group on execution faults, scoring a perfect 1.00 on all three, and just as blind as the rest on real failures.

Worse than chance, and why

On DROID seven of the ten score below 0.5, some of them clearly so. That is not the same as having no signal. It means the score runs backwards: failed episodes look calmer than successful ones, so a monitor tuned on this data would raise its alarms on the successes first.

The cause turned out to be simple once we measured it. On DROID a failed episode runs 188 steps against 239 for a successful one, and covers 5.2 units of total motion against 7.6. A policy that fails often does less. It gives up early, or never reaches the part of the task where the interesting motion happens. Any score built on magnitude or movement therefore reads failure as quiet and quiet as normal. Anyone deploying an action-stream monitor should check the direction of this effect on their own data before trusting it.

Policy CI, the first two runs

Two rented A10 boxes, 2,280 episodes, one ACT checkpoint from the LeRobot hub with its weights left alone. The battery for the second run varies where the cube sits and how far it is turned. The turning matters: the simulator never rotates the cube itself, in training or in evaluation, so that axis is outside anything the policy has seen.

A tool about honest measurement has to be honest about itself, so this section exists and stays current.

• No robot has been gated by it. Everything measured on physical robots is replay. The guard watched recordings and never intervened, so nothing here tells you what happens when it actually stops an arm.
• The live numbers come from one policy on one task. Three attempts at a second combination failed, for three different and documented reasons. Public simulation checkpoints that work well enough to calibrate against are scarcer than expected.
• The judge is only as good as its vision channel. The open weight model reaches 0.55 balanced accuracy on its own. Fusing and calibrating is what gets it to 0.92, and that is on the quarter of episodes it is willing to answer.
• Policy CI has only run in simulation. Nothing yet connects what it measures to a real robot, and the deploy record it writes says so in its own text. It also varies three scene factors on one task, so the only regions it can describe are the ones that battery samples.

This is one person's work so far, and the gaps in section 05 are not going to close on their own. If you run robots, study them, or just like breaking things carefully, there is real work here. Every item below is something we cannot do alone, ordered by how much it would change the answer to whether this is ready to trust.

The data we have been working from

The benchmark currently runs over 54 public datasets, all of them open. RoboMIND and AgiBotWorld task collections, the Open X-Embodiment sets converted to LeRobot format, DROID, BotFails, several HIL-SERL and DAgger intervention logs, SO-100 and SO-101 pick and place sets, and the RoboArena pairwise sessions. Between them they cover Franka, UR5, SO-100, SO-101 and bimanual platforms.

That mix is a starting point, not a standard. It is weighted towards tabletop manipulation and towards whatever happened to be public and well formed, which is its own kind of bias. If you have logs from a different robot, a different task family, or a setup where you expect this to fall over, those are the most useful ones to run. You do not have to share the data to share the result: the reports are text and the raw logs never need to leave your machine.

Python 3.11 or newer. The core has no GPU requirement, and the vision channel is optional.

pip install robotruth
robotruth --version

Or from source, if you want to run the tests or change something:

git clone https://github.com/mulkakhileshmj/robotruth.git
cd robotruth
uv venv .venv && uv pip install -e ".[dev]"
.venv/bin/python -m pytest -q

Check a deployment before it goes out:

robotruth contract extract runs/ckpt_0400/pretrained_model -o evaluated.json --role evaluated
robotruth contract extract /robot/current_policy -o deployed.json --role deployed
robotruth contract check evaluated.json deployed.json

Settle whether a new checkpoint is better:

robotruth stats plan --p-a 0.80 --p-b 0.90
robotruth stats compare results.csv --a base --b cand