CLIO Coder / Research
Open questions, and how to work on them.
Each page states a question, what is known and measured so far, and a first experiment you can run with CLIO Coder, labelled “proposed, not yet run” until someone has. The team, students, and outside researchers are welcome: open a GitHub issue to join, and read the contributor guide. What has shipped is in Docs and Tutorials.
Decision models / 12 min
Can a small decision model improve an agent's work?
Open questions on System One: evaluating on trajectories, asking better questions, and bringing a decision model, with a first experiment.
Open question. System One is off by default.
Read the articleReproducible agent science / 10 min
Can an agent run be a scientific record?
What a sealed run record proves and what it cannot, how far repeated runs differ, and a first experiment to measure that spread with CLIO Coder.
Open question. The records have shipped; no study exists yet.
Read the articleMemory / 8 min
Does agent memory change outcomes?
Whether lessons change what an agent does, whether self-approval is safe, and a first experiment on one repeated mistake.
Open question. Memory has shipped; its effect is unmeasured.
Read the articleContext / 8 min
What does an agent lose when its context is reduced?
Fidelity of summaries, the cost of reduction, and search by meaning, with a first experiment that counts how many planted facts survive.
Open question. Cost is measured; fidelity is not.
Read the articleFleets and local models / 7 min
When do several agents or small models pay off?
Compete, council, heterogeneous fleets and local models, with no comparative numbers yet and a first experiment on one seeded bug.
Open question. The controls have shipped; no comparison exists.
Read the articleScientific and HPC work / 8 min
What would make a scientist trust an agent's change?
Pre-registered checks, validated modernization, a project store and scheduler use on scientific code, with a first experiment on catching a numerical error.
Open question. The skills have shipped; no study exists.
Read the articleMethod
How each page is organized
Every research page follows the same seven parts, so you can compare questions and see at a glance what is still unknown.
- The question, stated so that an answer could turn out wrong.
- What is known and measured so far, with sample sizes and the build that was measured. A page says when a sample is small, when a result came from a design that no longer exists, and when nothing has been measured yet.
- Why the answer matters to people who use coding agents on real projects.
- Directions to work on: studies someone could take up, each with what to vary and measure.
- How to work on it today with CLIO Coder, using only commands that exist in the stated release.
- A first experiment, labelled “proposed, not yet run” until someone has run it and recorded the result on the page.
- How to join the work.
Join
Work on a question
Pick a question, run its first experiment or design a better one, and open a GitHub issue describing what you plan to measure. The contributor guide covers setting up a checkout and sending changes. Results, including null and negative ones, are published on the page with their numbers, so the next person starts from what you found.