Science Sandboxes

Two sealed environments for measuring whether an agent can actually do experimental science — not recall it. In each, the agent gets one action, a budget of experiments, and no explanation of the system it is probing.

• CodonBox — an invented genetic system. The agent submits a sequence, gets back a single number, and has to work out how the world converts sequence into fitness.
• MPRAbox — regulatory-sequence library design, scored by a pretrained model acting as the experimental oracle.

Both are scored against ground truth the agent never sees. This skill covers getting them running and reading what comes out.

What this skill deliberately withholds

A benchmark's scores are only worth something while its task material stays unseen, and a skill is loaded straight into an agent's context — which is precisely the contamination path. So the split here is deliberate:

Documented: installation, building the substrate, the oracle's interface contract, how to launch a scored run, and how to read the grading output.

Not documented, on purpose: task prompts, per-world structural parameters, the hidden codon tables, held-out evaluation set identities, and reference trajectories. All of it is in the repository, where a human setting up a run can read it deliberately.

One case is worth stating plainly, because it is easy to leak by accident. CodonBox tells the agent only the alphabet and the sequence length. That sequences contain codons at all, how long a codon is, which positions carry information, whether they act additively — every bit of that is what the agent is being scored on inferring. Publishing a per-world parameter table into a document agents read would hand over the answer. If you extend this skill, keep that line.

Requirements

Before anything below runs:

• git, a C compiler, and Python 3.11+. The folding substrate and the CodonBox oracle need nothing else and run fully offline.
• For the CodonBox agent loop only — an Anthropic API key in ANTHROPIC_API_KEY, and pip install anthropic. A run is one model call per experiment, 500 experiments per world by default, against a conversation that grows all run. Price a single world before launching all eight.
• For MPRAbox only — a ~700 MB Malinois checkpoint download, PyTorch, and in practice a GPU. The checkpoint is covered by the boda2 licence (AGPL-3.0), which is separate from this repository's MIT and applies to the weights themselves.

Get the code

git clone https://github.com/asr2210/science-sandbox
cd science-sandbox

MPRAbox's boda2 dependency is a git submodule, fetched later by its own setup step — you do not need --recurse-submodules for CodonBox.

CodonBox

The scoring substrate

Underneath the invented genetics is one fixed piece of physics, shared by every world. Each 16-residue chain of hydrophobic (H) and polar (P) residues is folded exhaustively on a two-dimensional square lattice and scored by the classic HP contact potential (Dill): its fitness is the count of favourable non-consecutive H–H contacts in the best fold it can reach. Because that space is small enough to enumerate completely, every world inherits the same known ceiling — which is what makes scores comparable from one world to the next.

Build it:

cd codonbox
gcc -O3 -o hpfold hpfold.c
./hpfold build 16 contacts_rebuilt.bin

That enumerates the self-avoiding walks once and caches the contact lists — 802,075 folds, well under a second. The repository already ships contacts_16.bin and table_16.bin, so this is a rebuild rather than a prerequisite.

Build to a fresh filename rather than over the shipped one, because that makes the artefacts checkable — both reproduce byte-for-byte from source:

cmp contacts_rebuilt.bin contacts_16.bin && echo "contacts reproduce byte-for-byte"

The full fitness lookup over all 2^16 chains is optional and much slower — about eight minutes single-threaded, producing exactly 327,680 bytes (65,536 entries of one signed byte of energy plus a uint32 degeneracy). The oracle folds on demand when it is absent, so skip it unless you are running many worlds:

./hpfold enumerate contacts_16.bin 16 table_rebuilt.bin
cmp table_rebuilt.bin table_16.bin && echo "fitness table reproduces byte-for-byte"

Score a chain directly

score takes an H/P chain and prints two numbers — the energy and the degeneracy (how many distinct optimal folds achieve it):

./hpfold score contacts_16.bin HHHHHHHHHHHHHHHH
./hpfold score contacts_16.bin PPPPPPPPPPPPPPPP

Fitness is the negated energy, so -9 69 is a fitness of 9 reached 69 different ways. Nine is the ceiling for a 16-residue chain — an invariant of the lattice, not of any particular world, and the denominator every run is graded against. The all-polar chain prints 0 802076: with no hydrophobic residues there are no contacts to make, so every fold ties for optimal.

The oracle contract

This is the part to understand before you interpret anybody's run. The oracle is deliberately silent: every string is accepted and scored. A sequence that uses characters outside the alphabet, or that does not divide into exactly the right number of codons, simply returns fitness 0 — no error, no hint about what was wrong.

That design has a consequence when you read results. A long opening run of zeros is normally the agent failing to find the encoding at all, not a broken harness — and the two look identical from the outside unless you know this.

You can exercise the whole stack offline against a world you build yourself, which keeps the benchmark's own worlds out of it:

python3 - <<'PY'
from world import make_world
from oracle import Oracle

w = make_world("demo", alphabet_size=4, codon_length=3, seed="demo-seed")
print("alphabet:", w.alphabet, "codons:", len(w.code))
print("H codons:", sum(1 for v in w.code.values() if v == "H"), "of", len(w.code))

o = Oracle(w, n_residues=16, contacts_path="contacts_16.bin", folder_bin="./hpfold")
print("outside alphabet ->", o.query("Z" * 48))
print("wrong length     ->", o.query("A" * 47))
print("well-formed      ->", o.query("ABCD" * 12))
PY

Codon tables come out of a deterministic construction seeded from the world's own parameters, with the assignment balanced so that half of them map to H — 32 of 64 above. Because that seed is pinned, a world rebuilt today grades identically to the run that originally used it.

To see which worlds ship, ask the catalogue rather than hardcoding a list:

python3 - <<'PY'
from world import build_catalogue
print(sorted(build_catalogue()))
PY

Drive a scored run

Each run is one experiment per turn — the harness enforces a single query per model turn, so the agent has to think between experiments instead of sweeping in a loop. It also keeps an append-only notebook, which is the artefact a human grades the reasoning from.

pip install anthropic
export ANTHROPIC_API_KEY="your-key"
export MODEL_ID="the-model-identifier-your-account-serves"
python harness.py earthlike --budget 500 --model "$MODEL_ID"

earthlike is the control world; --budget is the experiment count. Set MODEL_ID to whatever identifier your account serves — the harness defaults to one that may not match your access.

Read the results

Each run writes to runs/<world>/: the notebook, a JSONL of every query, a transcript, and summary.json with the grading fields. Read the score from the summary:

python3 - <<'PY'
import json, pathlib

for p in sorted(pathlib.Path("runs").glob("*/summary.json")):
    d = json.load(open(p))
    # Deliberately not printing best_sequence — it is solution material for that world.
    print(f"{d['world']:>20}  {d['queries_used']:>4}/{d['budget']} queries  "
          f"fitness {d['best_fitness']}/{d['true_optimum_fitness']}  "
          f"({d['fraction_of_optimum']:.0%} of optimum)")
PY

fraction_of_optimum is the headline number. Note what it does not capture: an agent can stumble onto a high-fitness sequence without ever understanding the world, which is why the notebook is graded separately by a human. There is no automated grader for the reasoning, by design.

runs/<world>/GROUND_TRUTH.json holds the world's true structure. It exists for post-hoc grading — keep it out of any context an evaluated agent can see, and out of anything you publish.

MPRAbox

MPRAbox scores library design rather than rule discovery. In each design round the agent produces a library of 50,000 200 bp sequences; the Malinois model labels them as a sealed in-silico assay; those labels then train a surrogate model from scratch, which is scored on held-out test sets. The agent is given only the per-set Pearson correlations and their mean — never sequence-level predictions, set identities, or anything about the oracle's internals.

Setup is the heavy part. It wants its own environment, and setup.sh then clones boda2, pulls the ~700 MB checkpoint, builds the evaluation index, and ends with a correlation sanity check against the oracle:

cd mprabox
python3.12 -m venv venv && source venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install numpy scipy pandas matplotlib lightning
bash setup.sh

The CUDA wheel index above is the project's own choice; substitute the build matching your hardware.

Scoring is the agent-facing half. prepare.py takes a directory holding three sequence files — sequences_0.txt, sequences_1.txt, sequences_2.txt — and returns measurements, so the designing agent treats it the way it would a collaborator running the assay:

python prepare.py path/to/your_library/

The instructions/ directory holds one task file per condition; a run starts by copying the variant you are testing to instructions.md and pointing a coding agent at it. Which variant you chose is what makes two runs comparable, so record it alongside the score. Completed runs ship under results/, laid out by condition — the quickest way to see the expected output shape before generating your own.

Both MPRAbox blocks above are unexecuted here — see verified.unverified_reason.

Try it

Data — none fetched. The substrate is enumerated locally by the block itself, and the world is one you construct, so nothing external is queried and nothing can rot. The only network access is cloning the repository (MIT).

Run — in a fresh empty directory:

git clone --depth 1 https://github.com/asr2210/science-sandbox
cd science-sandbox/codonbox
gcc -O3 -o hpfold hpfold.c
./hpfold build 16 contacts_rebuilt.bin
cmp contacts_rebuilt.bin contacts_16.bin && echo "contacts reproduce byte-for-byte"
echo "--- ceiling ---"
./hpfold score contacts_16.bin HHHHHHHHHHHHHHHH
python3 - <<'PY'
from world import make_world
from oracle import Oracle
w = make_world("demo", alphabet_size=4, codon_length=3, seed="demo-seed")
o = Oracle(w, n_residues=16, contacts_path="contacts_16.bin", folder_bin="./hpfold")
print("H codons:", sum(1 for v in w.code.values() if v == "H"), "of", len(w.code))
print("outside alphabet:", o.query("Z" * 48)["fitness"])
print("well-formed:", o.query("ABCD" * 12)["fitness"])
PY

Expect —

Invariants (a failure here means the skill or the tool is wrong):

• build 16 reports 802,075 folds.
• The all-hydrophobic chain scores energy -9, i.e. fitness 9 — the ceiling for 16 residues. No chain scores better.
• The codon table is balanced: 32 H of 64 codons for a 4-letter alphabet with 3-character codons.
• A sequence using characters outside the alphabet scores 0, silently.

Observed, 2026-09-07, science-sandbox @ 2812348 (a mismatch is drift to investigate, not a bug):

• score on the all-H chain prints -9 69 — 69 distinct optimal folds.
• The well-formed query in the demo world returns fitness 7.
• contacts_16.bin is 6,719,114 bytes and reproduces byte-identically against the copy committed to the repository.

Licence and citation

The repository is MIT (© 2025 Ananth Rao). The Malinois weights MPRAbox uses are governed separately by the boda2 licence (AGPL-3.0) — it restricts redistribution of the model, not your use of the benchmark.

The sandboxes accompany Science sandboxes measure the scientific capability of AI agents (arXiv:2608.30165). Cite that, and say which commit you ran — the worlds are seed-pinned, so a commit identifies the exact benchmark you scored against.