Open source, Apache 2.0 / v0.1.5 / updated 20 Sep 2026
robotruth is a Python library and command line tool for testing learned robot policies. It checks that the policy you deployed is the one you evaluated, puts a confidence interval on every number it prints, tells you when the robot itself has drifted, and watches a running policy for faults. It sits next to your stack and reads files. It is not a benchmark, a leaderboard or a model.
A robot lab runs fifty trials, gets forty successes, and writes down eighty percent. That number is doing far more work than it can carry. The interval around it is roughly twenty points wide, so the next checkpoint can look better without being better. Change the camera angle and the same policy moves further than most model upgrades do. Ship the same weights with the wrong normalization file and a working policy stops working, with nothing in the logs but a warning.
None of this is a secret. It is in the literature, and it keeps happening anyway, because there is no standard layer that checks it.
| Finding | Source |
|---|---|
| Of 13 audited real robot VLA papers, 0 report a confidence interval | PhAIL, arXiv 2605.29710 |
| A 50 trial success rate carries a 20 to 30 point wide 95% interval | Toyota Research Institute, LBM study |
| Moving a camera or a tote shifts task completion by 22 points | PhAIL |
| Same weights, different normalization metadata: 28/28 drops to 2/28 | arXiv 2606.03724 |
| Same policy, second identical arm: 98% drops to 18% | SPACE, arXiv 2606.24049 |
| Video only success judges cap at 0.77 balanced accuracy | FailBench, arXiv 2609.03611 |
Six modules, each answering one question you can act on. They work on their own or together, and everything they produce is a Markdown and HTML report.
Is the policy you evaluated the policy you deployed? Weights, normalizer statistics, action semantics, control rate, cameras and embodiment go into a manifest that is hashed and compared. The comparison fails closed, so anything it cannot prove is reported as unverified rather than assumed fine.
Is B really better than A? Intervals on every rate, paired designs, sequential tests you are allowed to peek at, censored time to success, Bradley Terry rankings, and an audit that takes success counts someone else published and tells you which comparisons their own numbers can support.
What actually happened in each rollout? One record per episode with outcome, interventions, failure class and provenance, plus the fleet numbers almost nobody reports: interventions per hour, mean time between interventions, autonomous fraction. Reads LeRobot datasets, writes MCAP for Foxglove.
Did the cell or the arm change under you? Lighting and camera geometry from a single frame, and per joint lag, backlash, offset and gain from one excitation run. When you have a pile of fingerprints from a working cell instead of a known tolerance, the drift threshold is calibrated from them rather than guessed.
Did the episode succeed? Motion features fused with a vision language model, then calibrated so that it abstains instead of guessing when it is not confident. It reports how often it answers, not only how often it is right. The open weight backend runs on your own GPU, so no API key is needed.
Has the robot stopped behaving like itself right now? Per step scores with time uniform thresholds calibrated on your own successful rollouts, emitting ok, slow, handover or stop. It catches execution faults such as a stalled arm or erratic control. It does not catch task failures, and section 04 says so with numbers.
Three rules run through all of it, and they are the reason the numbers are worth anything.
If the contract check cannot prove the deployed policy matches the evaluated one, it does not pass you with a warning. It returns a non zero exit code and names the field it could not verify. Some things are not written to disk anywhere, like which of several shipped normalizer statistics is actually live, so those are listed as unverified and you supply them once.
The report layer will not print a bare percentage. Every rate carries a Wilson interval, every comparison carries a paired difference, and when a result is still inside the noise it says so and tells you how many more trials would settle it. This is a small rule that changes what a report can be used for.
The judge is conformally calibrated, which means it is allowed to answer only when it can meet a stated error rate, and it hands the rest to a human. Coverage is reported next to accuracy, because a judge that answers everything at sixty percent is worse than useless to a team that has to trust it.
Everything below was produced by the tool itself on public data and on a live policy, on rented GPUs. The bundles and the scripts that made them are in the repository, so none of it has to be taken on trust.
| Module | What was measured | Result |
|---|---|---|
| Contract | Five public checkpoints, ACT through GR00T | Caught a loader silently dropping normalizer statistics, which took a working policy from 83% to 0% with only a log warning |
| Statistics | Audit of published success counts | Of 5 comparisons, 2 hold up and 3 are inside the noise |
| Episodes | A real assisted autonomy deployment log | 129.6 interventions per hour [120.9, 138.7] |
| Fingerprint | Five physical SO-100 and SO-101 arms | Told apart from their logs alone, backlash 0.107 to 0.478, lag 101 to 135 ms |
| Fingerprint | Calibrated drift threshold, 270 clean episodes | 3 false alarms, against 23 for the rule of thumb it replaced, with no loss of sensitivity |
| Judge | 323 labelled real episodes, 20 tasks | 0.921 balanced accuracy [0.617, 0.972] on the quarter it chooses to answer |
| Guard | 338 held out successful episodes | 0 false alarms [0.000, 0.011] |
| Guard | Stalls and erratic control on three real robot corpora | 361 of 362 caught, within 0.1 to 0.2 s |
| Guard | Faults that arrive gradually over up to 4 s | Still caught every time, with the delay tracking how long the fault takes to grow |
The guard used to be described as catching a failing policy. Testing it against real robot logs showed it catching 5 of 145 real failures on one corpus and 0 of 58 on another, and a threshold free check confirmed there was no signal to recover rather than a setting to tune. A robot that fails a task usually keeps moving normally, so the motion stream does not give it away. What the guard reliably catches is the robot no longer behaving like itself, and the wording was changed to match. The task failure question belongs to the judge, and miscalibration belongs to the fingerprint, which spots an offset eight times smaller than the guard needs.
A tool about honest measurement has to be honest about itself, so this section exists and stays current.
This is one person's work so far, and the gaps in section 05 are not going to close on their own. If you run robots, study them, or just like breaking things carefully, there is real work here. Every item below is something we cannot do alone, ordered by how much it would change the answer to whether this is ready to trust.
Every real robot result we have is replay. The guard watched recordings and never once stopped an arm. If you have a working cell and are willing to run the guard in the loop, even in a corner of the lab on a task you do not care about, that single experiment is worth more than anything else on this list. We want to know what happens when it fires on a robot that is actually moving.
All the live numbers come from one ACT policy on one task. We tried three other public checkpoints and all three failed, for three different reasons we have written down. If you have a policy that reliably does its task, in simulation or otherwise, running the same battery against it would tell us whether any of the envelope generalises or whether we got lucky.
The guard is at chance on real task failures, and we showed the signal is not in the action stream to begin with. This is a research problem, not a tuning problem. If you have a view on what a cheap per step signal for "the robot is moving fine but the task is going wrong" looks like, we would like to hear it, including the answer that there is not one.
Nobody outside has installed this. Take it, point it at your own logs, and tell us where it confused you, what it got wrong about your data, and which error message wasted your time. A blunt issue about a bad assumption is more useful to us than a pull request.
Every run is published with the scripts that produced it, including the ones that went badly. If you think a result is wrong, or that an interval is doing more work than it should, say so. We have already had to publish one correction against our own tool and we would rather do that again than be quietly wrong.
The benchmark currently runs over 54 public datasets, all of them open. RoboMIND and AgiBotWorld task collections, the Open X-Embodiment sets converted to LeRobot format, DROID, BotFails, several HIL-SERL and DAgger intervention logs, SO-100 and SO-101 pick and place sets, and the RoboArena pairwise sessions. Between them they cover Franka, UR5, SO-100, SO-101 and bimanual platforms.
That mix is a starting point, not a standard. It is weighted towards tabletop manipulation and towards whatever happened to be public and well formed, which is its own kind of bias. If you have logs from a different robot, a different task family, or a setup where you expect this to fall over, those are the most useful ones to run. You do not have to share the data to share the result: the reports are text and the raw logs never need to leave your machine.
Open an issue on GitHub describing what you have and what you want to try, or just send a report the tool produced with your read on whether it told you the truth. Independent researchers, students and lab engineers are all equally welcome. There is no contributor agreement to sign and no roadmap to fit into.
Python 3.11 or newer. The core has no GPU requirement, and the vision channel is optional.
git clone https://github.com/mulkakhileshmj/robotruth.git cd robotruth uv venv .venv && uv pip install -e ".[dev]"
Check a deployment before it goes out:
robotruth contract extract runs/ckpt_0400/pretrained_model -o evaluated.json --role evaluated robotruth contract extract /robot/current_policy -o deployed.json --role deployed robotruth contract check evaluated.json deployed.json
Settle whether a new checkpoint is better:
robotruth stats plan --p-a 0.80 --p-b 0.90 robotruth stats compare results.csv --a base --b cand
The repository README covers every command. The validation bundles under examples/validation hold the raw output behind every number on this page, including the runs that went badly.