Mixture-of-Agents: Layered Intelligence Across Open-Weight Models

The LLM Scaling Ceiling: Beyond Monolithic Transformers

For years, the artificial intelligence industry operated under a singular scaling dogma: to achieve higher reasoning benchmarks, labs had to train a larger, monolithic dense transformer on more compute clusters.

However, scaling monolithic models introduces steep economic diminishing returns: - Serving a 405-billion parameter model requires massive multi-node GPU clusters (as explored in our analysis of the xAI Colossus Cluster). - A single monolithic model, no matter how capable, exhibits blind spots, hallucination tendencies, and style biases.

In June 2024, Together AI published Mixture-of-Agents (MoA) (*Mixture-of-Agents Enhances Large Language Model Capabilities*, Wang et al.), introducing a radical architectural paradigm: a layered feedforward network composed of diverse, heterogeneous language models that collaboratively critique, synthesize, and refine solutions.

By combining open-source models (such as Qwen, Llama 3, and Mistral) in structured layers, MoA achieved a score of 65.1% on AlpacaEval 2.0, outperforming GPT-4o (57.5%) using exclusively open-weight building blocks.

User Query (x) │ ▼ ┌────────────────────────────────────────────────────────┐ │ Layer 1: Proposer Models (Heterogeneous) │ │ - Model A (Qwen 2.5 72B): Technical Draft │ │ - Model B (Llama 3.3 70B): Logical Outline │ │ - Model C (Mistral Large 2): Creative Analysis │ └──────────────────────────┬─────────────────────────────┘ │ Candidate Responses {y_1, y_2, y_3} ▼ ┌────────────────────────────────────────────────────────┐ │ Layer 2: Aggregator / Refinement Layer │ │ - Model A': Reviews A, B, C -> Generates Refined A' │ │ - Model B': Reviews A, B, C -> Generates Refined B' │ └──────────────────────────┬─────────────────────────────┘ │ Layer 2 Refinements {y'_1, y'_2} ▼ ┌────────────────────────────────────────────────────────┐ │ Layer 3: Final Aggregator (Qwen / Llama) │ │ - Synthesizes all perspectives into canonical response│ └──────────────────────────┬─────────────────────────────┘ │ ▼ Final Output Response (y*)

---

The Mathematical Architecture of Mixture-of-Agents

The MoA architecture is structured analogously to a multi-layer neural network, but where each individual "neuron" is an entire Large Language Model:

#### 1. Layered Formulation Let $L$ denote the number of sequential layers, and let $N_l$ denote the number of LLM agents in layer $l$. - For an input prompt $x$, the first layer ($l = 1$) generates initial candidate completions independently:

$$y_i^{(1)} = \mathcal{M}_i^{(1)}(x), \quad i = 1, \dots, N_1$$

• For subsequent layers $l > 1$, each agent $\mathcal{M}_i^{(l)}$ receives the original prompt $x$ concatenated with all outputs generated by the previous layer:

$$y_i^{(l)} = \mathcal{M}_i^{(l)}\left( x, \left\{ y_1^{(l-1)}, y_2^{(l-1)}, \dots, y_{N_{l-1}}^{(l-1)} \right\} \right)$$

• The final output $y^*$ is emitted by the designated aggregator agent in the terminal layer $L$.

#### 2. The 'Collaborativeness' Phenomenon The theoretical breakthrough behind MoA is an empirical discovery known as the collaborativeness phenomenon: > *An LLM generates a significantly higher-quality response when presented with candidate outputs from other models—even if those other models possess lower individual capability scores than itself.*

When an agent is presented with multiple independent viewpoints, its role shifts from cold generation to critical evaluation and error correction. It cross-verifies facts, synthesizes diverse insights, and eliminates individual hallucinations.

---

Heterogeneous vs Homogeneous Ensembles

MoA architectures can be constructed using either identical models or diverse model families:

Homogeneous MoA: Heterogeneous MoA: ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Llama 3 │ │ Llama 3 │ │ Qwen 72B│ │ Llama70B│ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └──────┬─────┘ └──────┬─────┘ ▼ ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ Aggregator: Llama 3 │ │ Aggregator: Mistral │ └──────────────────────┘ └──────────────────────┘

• Homogeneous Ensembles (Self-MoA): Deploying the same model (e.g. Llama 3 70B) across all proposer and aggregator slots with high sampling temperatures ($ au = 0.7$). While it improves scores, it suffers from correlated failure modes.
• Heterogeneous Ensembles: Combining distinct model architectures trained on different pre-training corpora (e.g. Qwen for mathematics/coding, Llama for conversational nuance, Mistral for multilingual fluency). Heterogeneous ensembles yield the highest benchmark uplifts because model blind spots rarely overlap.

---

Benchmark Disruption on AlpacaEval 2.0 & Arena-Hard

In standardized conversational evaluations, MoA dramatically shifted the cost-performance frontier:

| Model / Architecture | Base Parameters | AlpacaEval 2.0 (Win-Rate) | LC Win-Rate | | :--- | :---: | :---: | :---: | | GPT-4o (Closed Frontier) | Undisclosed | 57.5% | 52.6% | | Claude 3.5 Sonnet (Closed)| Undisclosed | 58.7% | 55.4% | | Qwen 2.5 72B (Single Dense)| 72B | 48.2% | 46.1% | | Llama 3 70B (Single Dense)| 70B | 50.3% | 48.0% | | MoA (Qwen + Llama + Mistral)| Ensemble | 65.1% | 62.8% |

MoA's 65.1% win-rate made it the first open-weights system to decisively surpass GPT-4o on complex instruction-following benchmarks.

---

Latency, Cost, and Token Economics

While MoA delivers frontier reasoning, it introduces significant computational trade-offs:

1. Token Ingestion Explosion: Because each layer ingests the outputs of all preceding models, input token consumption scales quadratically with layer depth:

$$\text{Total Tokens} \approx N \cdot |x| + \sum_{l=2}^L N_l \cdot \left( |x| + N_{l-1} \cdot |y| \right)$$

2. Inference Latency: Running sequential layers introduces multiple round-trip inference cycles, increasing time-to-first-token (TTFT) and total latency to several seconds. 3. Optimized Serving: By deploying high-throughput inference engines like Together AI or local runners like Ollama, and executing proposer models in parallel within each layer, engineering teams can minimize wall-clock latency while slashing API expenditures compared to proprietary closed models.

MoA represents the future of agentic architectures: shifting value from proprietary mega-checkpoints to modular, collaborative orchestration pipelines.