ProgramBench: Can we really rebuild huge binaries from scratch? (doesn't look like it)

There's been quite a few case studies recently on agents building whole programs from scratch, but most of them test a single or just a few projects with hand-tuned setups.

We've spent the last couple of months formalizing this setting and building a benchmark of 200 tasks while doubling down on testing, cheat prevention, and task diversity.

Our agent ONLY gets a target executable and some readme/usage files. The agent must choose a language, design abstraction layers, and architect the entire program. No internet access or any other way of cheating. No decompilation.

We've also spent some 50k to generate 6M lines of behavioral tests and then filtered them down to keep the best ones. Because they are just testing executables as a black box, we do not make any assumptions on even the language that the LM uses to implement the program.

All of the results are at programbench.com . There's also a big FAQ at the bottom.

We've just open-sourced our github, huggingface and docker images.

Essentially you can just start evaluating with pip install programbench && programbench eval <your submission>

Github is at https://github.com/facebookresearch/programbench

Sorry that it's just closed source models right now, we have a few open-source models in the pipeline, but so far we've had an even harder time at getting them to behave well with these tasks (open source models tend to be somewhat more overfitted to things like SWE-bench, so they often have a harder time with new benchmarks).

We're also planning to open the benchmark for submissions quite soon, similar to what we did on SWE-bench and its variants.