AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentation

The Challenge of Sustained Scientific Discovery

Scientific research is rarely a linear path from hypothesis to discovery. Instead, it is a complex, iterative cycle involving the generation of multiple competing hypotheses, the design of experiments to test them, and the subsequent revision of plans based on empirical evidence. While Artificial Intelligence (AI) has made significant strides in automating specific parts of this cycle—such as literature review or code generation—most existing AI agent systems struggle with the "long-running" nature of science.

Current AI agents typically operate under two constraints. First, many are single-agent systems that follow a single trajectory, effectively "putting all their eggs in one basket." If the initial research direction is a dead end, the agent may struggle to pivot. Second, existing multi-agent systems often rely on a centralized "manager" or "planner." While this structure works for short-term tasks, it creates bottlenecks in complex research where the most promising directions only emerge after dozens of experiments. Centralized systems often assume that the search space can be partitioned into fixed tasks at the start, which is rarely true in open-ended scientific exploration.

Figure 1: The AUTOSCIENTISTS framework enables a decentralized team of agents to conduct parallel experiments, self-organize into teams, and coordinate via a shared state.

AUTOSCIENTISTS is a framework designed to overcome these limitations. It introduces a decentralized, self-organizing multi-agent team that mimics the collaborative nature of human research departments. Rather than following a fixed plan, agents in this system dynamically form teams around emerging ideas, critique each other's proposals, and share knowledge about what failed just as much as what succeeded.

A Decentralized Framework for Research

At the core of the framework is a team of $n$ agents, $A = \{a_1, \dots, a_n\}$, working together to find an optimal program $p^*$ that maximizes a specific evaluation metric $\ell_{\text{eval}}(p; D)$ on a dataset $D$. The agents might start with an initial seed program $p_0$ and work to refine it over days or weeks of continuous experimentation.

The key distinction of this approach is its lack of a central orchestrator. Instead, coordination is achieved through a "Shared State" ($S$). This shared state acts as the laboratory's collective memory and is accessible to all agents through a standardized protocol. The shared state is composed of four distinct layers:

1. The Champion ($p^*$): The current best-performing model or code, along with instructions on how to reproduce it.
2. The Experiment Log ($L$): A complete history of every experiment run by the team, including the code used, the results, and training diagnostics.
3. The Shared Forum ($F$): A space where agents post structured debate, hypothesize about why certain changes worked, and critique the proposals of others.
4. Team-Local State: This includes specific experiment queues ($Q_k$) and dead-end registries ($D_k$) for individual sub-teams.

By using this shared state, the system ensures that no two agents are redundantly performing the same work and that insights discovered by one team can be immediately leveraged by another.

The Architecture of Collaborative Agents

The system divides labor between two primary roles: Analyst Agents and Experiment Agents. These agents operate in a "heartbeat cycle," where they repeatedly read the shared state, decide on an action, and update the state with their findings.

Analyst Agents

Analysts are the "strategists" of the team. They do not run the experiments themselves; instead, they monitor the experiment log $L$ and the shared forum $F$. Their job is to look for patterns in the data. If an analyst notices that a certain type of model architecture consistently fails, they will register that direction in the "Dead-End Registry." Conversely, if they see a small but consistent improvement from a specific hyperparameter change, they will propose new experiments to explore that direction further. They are also responsible for maintaining the team's long-term research hypotheses.

Experiment Agents

Experiment agents are the "tacticians." They claim the highest-priority proposals from the experiment queue $Q_k$ and execute them. This involves modifying the codebase, running the training on a GPU, and reporting the results back to the log. Importantly, experiment agents do more than just report a single number; they provide a "mechanistic analysis" of why they think the experiment yielded a certain result, which then feeds back into the analyst's strategic planning.

Dynamic Self-Organization and Peer Review

One of the most characteristic features of this framework is how it handles the organization of these agents. Scientific priorities shift as results come in. To reflect this, the system alternates between a Discussion Phase and an Execution Phase.

When experimental progress stagnates, or at the very beginning of a run, the agents enter a structured discussion. They propose candidates for research directions and critique them. Unlike a voting system that might settle on a single "winner," AUTOSCIENTISTS allows for the formation of a team roster $R = \{(T_k, \text{axis}_k, \text{members}_k)\}$, where $T_k$ represents a team focused on a specific "axis" of research (e.g., one team might focus on the optimizer while another focuses on data augmentation).

The discussion phase is decentralized; there is no manager to decide when it ends. Instead, agents vote to continue or finish. Once a consensus is reached, the last agent (alphabetically) consolidates the proposals into a roster. This allows for emergent coordination—teams can split, merge, or dissolve naturally as research axes are exhausted or shown to be highly promising.

Ensuring Robustness: Noise-Aware Gating

In scientific experimentation, "luck" (such as a particularly favorable random seed) can often look like a genuine improvement. This is a significant problem for AI agents, which might otherwise "promote" a noisy result as the new champion $p^*$, leading the entire team down a suboptimal path.

To prevent this, the framework implements a noise-aware champion validation mechanism. When a new candidate program $p'$ shows an improvement $\Delta = \ell(p') - \ell(p^*)$, it is not immediately accepted. Instead:

• If the improvement is large ($\Delta > M\sigma$, where $\sigma$ is the measured noise floor and $M=2$), it is accepted.
• If the improvement is small ($0 < \Delta \leq M\sigma$), the system triggers a mandatory re-run with a different random seed. The improvement is only accepted if both runs show a strict gain.

This rigorous gating ensures that the team's "champion" is a robust foundation for future iterations.

Empirical Evaluation Across Scientific Domains

The researchers evaluated AUTOSCIENTISTS across several challenging benchmarks to see if this decentralized approach actually translates to better scientific outcomes.

Biomedical Machine Learning (BioML-Bench)

On the BioML-Bench, which covers 24 tasks in areas like drug discovery and protein engineering, AUTOSCIENTISTS achieved a mean leaderboard percentile of 74.40%. This was a significant improvement over the 66.07% achieved by Autoresearch, a leading single-agent baseline. Notably, in the drug discovery tasks, the system improved the mean percentile from 46.16% to 64.52%.

Figure 2: Performance of AUTOSCIENTISTS across 24 biomedical tasks compared to other AI agent baselines.

Language Model Optimization

When applied to optimizing a small language model (GPT nanochat), the framework demonstrated a 1.9x acceleration. It reached the same performance target in 34 experiments that required 65 experiments for the single-agent baseline. Furthermore, when both systems started from a highly optimized "champion" model, the single agent plateaued, while the decentralized team found seven additional accepted improvements by exploring diverse axes like matrix initialization and normalization order.

Protein Fitness Prediction

In the ProteinGym benchmark, the system was tasked with improving an existing Gaussian-process model (Kermut) for predicting how mutations affect protein function. The agents successfully discovered an ensemble of three models incorporating diverse features and "quantile-warped" targets. This discovered recipe improved the Spearman’s correlation ($\rho$) from 0.657 to 0.700 across 217 different assays.

Figure 3: Progress comparison between AUTOSCIENTISTS and single-agent baselines in language model optimization, showing faster convergence and sustained improvement.

Insights from Ablation Studies

To understand which parts of the framework were most critical, the authors conducted "ablation" studies—removing one piece of the system at a time:

• Removing Analysts: This led to a significant drop in proposal quality, as experiment agents lacked the strategic oversight to avoid redundant or poorly justified trials.
• Removing Cross-Agent Feedback: Disabling peer critique led to a higher rate of "false starts," where the system pursued hypotheses that other agents could have easily identified as flawed.
• Removing Self-Organization: Fixing the teams at the start of the run prevented the system from pivoting when a specific research direction (like a certain optimization schedule) reached its limit.

These results suggest that the system's strength lies in the interplay between its components: the shared memory prevents duplication, the peer critique filters bad ideas, and the dynamic teams ensure that resources are always directed toward the most promising evidence.

Conclusion

AUTOSCIENTISTS moves AI-driven research away from single, linear trajectories toward a more robust, decentralized team model. By allowing agents to self-organize, critique each other, and maintain a shared memory of both successes and failures, the framework achieves more sustained and efficient scientific discovery.

While the system does consume more LLM tokens than simpler models due to its discussion phases, its ability to find better models within a fixed budget of physical experiments (like GPU training time or lab assays) makes it a more practical tool for the types of long-running, high-stakes problems found in modern science. Future work will likely look at scaling these teams even further and allowing for more automated interaction with physical laboratory hardware.

Figure 4: A sample model card generated by the system, detailing the architecture, metrics, and training data of a discovered model for drug safety screening.