In mid-October, Sakana AI published a repository called Agentic Scientist. The usage is simple: point it at a directory with a research task written as task.md, and coding agents propose ideas, write programs, run experiments, compare against baselines, perform ablations, pass a sealed final evaluation, and draft a paper. The interesting part is that no new model was built. What was designed is the ordering and the control procedure that turns existing coding agents into "AI scientists."

That distinction matters in practice. The lesson for developers is less about flashy automated discovery and more about how the repository forces reproducibility and sealed evaluation.

What it is, and what it is not

Agentic Scientist is published as the SakanaAI/AI-Scientist-v3 repository under Apache 2.0, with the paper at arXiv:2610.18933. The authors are Robert T. Lange, Luca Grillotti, Yutaro Yamada, Rujikorn Charakorn, and David Ha.

The central line from the official description is this: "Starting from a task.md, the agents propose ideas, run experiments, evaluate outcomes against the task rubric, and iterate toward stronger programs. The pipeline then runs baselines and ablations, a sealed final evaluation, post-hoc analyses, and writes the paper."

Three things are worth pinning down.

[What it is]
- An orchestration layer that drives coding agents as experimenters
- Runs claude / codex / cursor / opencode / gemini / pi via Headless CLI
- Enforces reproducibility by writing artifacts to one run directory per run

[What it is not]
- Not a new foundation model or training algorithm
- Does not guarantee that "one paper" is verified new knowledge
- Does not mean it is safe to run unattended without human review

So the repository is less a claim that "AI does science" and more a reference implementation of how to build and seal a graph where an agent stacks results across many stages.

The six-phase pipeline

The pipeline runs in sequence, with optional stages after it.

[Agentic Scientist pipeline]

1. Task drafting      task.md -> sealed evaluator and rubric
                      (only when a task scaffold is absent; experimental)
2. Investigation      idea -> experiment -> evaluation loop (the core agent work)
3. Ablations          winning program vs baselines and ablated variants
4. Final evaluation   re-scored in a sealed environment
5. Post-hoc analysis  mines the artifacts for extra evidence
6. Paper              writes, compiles, and audits the manuscript
(+) Review / Revision  optional: send the manuscript to an external reviewer

Each phase runs in its own container and reads only the run-local snapshot.

Per-phase output, summarized:

Phase What it does What it leaves
Task drafting Turns task.md into a sealed evaluator and rubric Drafting logs, validation audit
Investigation Repeats idea, experiment, evaluation Ideas, attempts, winning program, optimizer memory
Ablations Compares baselines and ablated variants Comparison results, replications
Final evaluation Re-scores in a sealed environment Final scores
Post-hoc analysis Additional analysis over artifacts Selected analyses
Paper Writes, compiles, audits the manuscript Manuscript, PDF, manifest, optional AI reviews

One design choice stands out: "Investigation plus the task snapshot is enough to write a paper." The other phases only add evidence, and when they fail they are reported as degraded. A single phase dying does not collapse the whole run, and the missing evidence is not hidden.

The run directory is the unit of reproducibility

Every run leaves results in one predictable directory.

results/<task>/<agent>/<run_id>/
|- task_drafting/    drafting logs and validation audit
|- task/             immutable run-local task snapshot
|- investigation/    ideas, attempts, winner, optimizer memory
|- ablations/        baselines and ablations
|- final_evaluation/ sealed final evaluation
|- posthoc/          post-hoc analyses
`- paper/            manuscript, PDF, manifest, optional AI reviews

Two things to notice. First, each phase reads only the "run-local snapshot" and never modifies examples/, so an experiment cannot contaminate the example code and the run is easy to repeat under identical conditions. Second, the directory structure is a contract: which phase must produce what is fixed by path, which makes partial reruns possible.

The actual execution flow

The commands below are the official examples from the repository README. They were not executed for this article; they are here to show how the system enforces order and control.

# 1) Prepare the agent image and the Python environment
docker build -t agentic-scientist/agent-cli:latest -f src/docker/agent-cli/Dockerfile .
uv sync && source .venv/bin/activate

# 2) Check what is ready and preview the exact commands
agsci doctor examples/circpack          # host, image, credentials, scaffold
agsci run examples/circpack --dry-run   # print phase commands, run nothing

# 3) Shrink budgets for a fast first end-to-end run
agsci run examples/circpack \
  investigation.execution.max_idea_cycles=2 \
  ablations.execution.num_replications=1 \
  ablations.execution.max_evals_per_variant=2 \
  posthoc.execution.max_selected_analyses=1

# 4) Resume only some phases (pass the original agent and model again)
agsci run examples/circpack execution.phases=ablations,posthoc,paper \
  paths.run_dir=results/circpack/claude/20260331_094917

# 5) Browse a run in the browser
agsci inspect start        # http://127.0.0.1:8765

The default budgets are 3 idea cycles, 5 ablation replications, and 3 post-hoc analyses. Tasks that need GPUs declare them through presets. The GRPO example, for instance, needs two GPUs, one for vLLM sampling and one for training, and downloads about 3 GB of model weights once. Each evaluation trains for 1-2 hours, and the harness evaluation timeout defaults to 7200 seconds. If the GPUs agsci doctor sees do not match the preset, it refuses to start the compute phases.

Agents are swappable via agent.name=<agent> agent.model=<model>, so the same experiment can be repeated across agents and models. The intent reads as comparing which agent handles which phase well.

Permission is risk: --allow yolo

The security warning the repository puts in bold at the top is worth carrying over almost verbatim.

[Official security warning, summarized]

Agentic Scientist launches coding agents with full tool permissions
(Headless --allow yolo). That means they can:
- run arbitrary commands
- inherit the API keys you give them
- be steered by the task files and papers they read

Recommendations:
- run inside the Docker image, not on your workstation
- only run tasks and sources you trust
- review run directories before sharing (agent logs can echo document
  contents and paths)

This points at the same problem covered in the previous article. The moment you give an agent shell, file, and network permissions, that isolation boundary becomes your product's safety boundary. Automated research pipelines are especially exposed, because the documents an agent reads can become instructions.

Limits and open problems

The official docs are clear about the limits too.

[Known limits]

- Drafting a task from a bare research brief is experimental
  -> simple tasks like circle packing work, but larger tasks such as the
     world-model or tabular distribution-shift examples may fail to produce
     a scaffold that matches the full pipeline contract
- Tasks without a shipped scaffold can have drafts rejected; check the audit
- API key usage and GPU costs are separate
- Agent logs can retain document contents (security)

So the repository has more to teach about how a discovery process is controlled and recorded than about what it discovered.

Moving it into a small project

You can steal the structure without running the whole pipeline.

[task.md -> baseline -> experiment -> held-out evaluation]

1. State the task in task.md
   - goal, success criteria, allowed tools, and failure conditions

2. Seal the evaluator first
   - fix the scoring code separately from the experiment
   - if the agent can edit the evaluator, the experiment is meaningless

3. Run experiment and baseline under the same conditions
   - change exactly one variable for at least one comparison

4. Do the held-out evaluation once, at the end
   - do not score the final result on data you looked at during the run

5. Keep artifacts in one directory
   - record ideas, attempts, winner, and results by path

The core is "seal" and "one directory." If the experimenter can touch the evaluator, scores cannot be trusted; if artifacts scatter, the run cannot be repeated. Agentic Scientist is a case where those two principles are enforced by containers and a directory contract.

Conclusion: the bottleneck of automated research is sealing, not ideas

How good an agent is at generating ideas is not the visible metric. What actually builds trust is the order of execution, the sealed evaluation, and reporting that does not hide failures. That is what makes Agentic Scientist interesting: instead of bragging about a new model, it shows as design what to fix and what to retain in an unattended, multi-stage job.

Move the sequence onto one small task and the feel of it comes through. Write a task.md, seal the evaluator first, run the baseline and the experiment under the same conditions, and run the sealed evaluation once at the end. For more on controlling this loop, see What Is Loop Engineering, Model vs Harness in AI Coding, and AI Agent KPIs: From Tokens to Outcomes.

Further reading

References

Go deeper with a course

Making an agent run, evaluate, and verify work across multiple stages is harness, loop, and graph design. Sketching where to seal and where a human steps in is how the feel of an automated research pipeline comes together.