77% vs 53%: the consistency gap

A model can look reliable per run and still be unreliable in aggregate. One study (arXiv 2609.08832) reports 77% success per run, versus 53% when all five runs must succeed.

For an agent that moves money or messages customers, the gap between 'usually works' and 'works every time' is the whole game. Keeprun flags when the same input gives divergent outcomes.