[P] Built confidence scoring for autoresearch because keeps that don't reproduce are worse than discards

Been running autoresearch for about a week. ~100 experiments per night on an H100. The keep rate is around 15%, which matches what Karpathy posted in his own discussion threads (#32 and #43).

The problem isn't the keep/discard loop. That works. The problem is that some of those keeps don't hold up. Karpathy's session #43 shows that 5% warmup (a keep in session #32) actually hurt performance when run again. A 0.02% improvement in val_bpb could be a real win or GPU nondeterminism. After extended runs it gets worse: 68 experiments for a single keep.

If you build on a false keep (change architecture based on it, stack more experiments on top), you're compounding noise. That's worse than a clean discard.

So I built three CLIs:

autojudge estimates noise floor from your recent experiments, checks if the result sits on the Pareto front (val_bpb vs memory), and returns a confidence scored verdict: STRONG_KEEP, KEEP, MARGINAL, RETEST, DISCARD, or CRASH. MARGINAL means "this might be noise, retest before building on it." Exit codes are scripting friendly.

autosteer analyzes which categories of experiments (architecture, hyperparams, optimizer) historically produced real improvements and suggests what to try next. Exploit mode when you're on a streak, explore when you're stuck. Stops the random walk.

autoevolve is more experimental. It puts multiple agents on separate git worktrees with different strategies competing on the same problem. Winning ideas get cross pollinated.

The difference in practice: instead of waking up to a TSV and guessing which keeps are real, you wake up to ranked results with confidence scores and a clear next step.

Caveats: noise floor estimation needs ~5 experiments to stabilize. autosteer's suggestions are category level, not causal. autoevolve is the newest and least polished.

pip install autojudge autosteer autoevolve

https://preview.redd.it/ekm1db5lfmpg1.png?width=800&format=png&auto=webp&s=68265f92001c7582d049a74969e8bf0993e021d9