End-to-End RL Training for Emerging Agentic Capabilities
June 20, 2025 • 10 min read
Meet Kimi-Researcher, an autonomous agent that excels at multi-turn search and reasoning. It performs an average of 23 reasoning steps and explores over 200 URLs per task. Built on an internal version of the Kimi k-series model and trained entirely through end-to-end agentic reinforcement learning (RL), it achieved a Pass@1 score of 26.9% — a state-of-the-art result — on Humanity's Last Exam, and Pass@4 accuracy of 40.17%. Starting from an initial HLE score of 8.6%, Kimi-Researcher reached 26.9% almost entirely through end-to-end RL training, providing compelling evidence that end-to-end agentic RL can significantly advance agent intelligence.
Kimi-Researcher has also achieved strong performance across several complex and challenging real-world benchmarks. On xbench, a new, dynamic, professionally-aligned suite designed to bridge AI capabilities with real-world productivity, Kimi-Researcher achieved 69% pass@1 (averaged on 4 runs) on xbench-DeepSearch, outperforming models such as o3 with search tools. On benchmark tests for multi-turn search reasoning (FRAMES, Seal-0) and factual information (SimpleQA), Kimi-Researcher also achieved strong performance.
Figure 1: Potential fluctuations in tools, such as search engines, may affect performance. The results are tested on: HLE on June 17, 2025; and xbench-DeepSearch, Seal-0, Frames, and SimpleQA on June 18, 2025.
All Kimi-Researcher results were evaluated using o3-mini. Scores of other models are referenced from the relevant papers or leaderboards. For benchmarks with fewer than 200 test samples (xbench, Seal-0), we performed four runs and reported the average result (avg@4). We do not compare multi-agent workflows based on multiple frontier models here, as our focus is on evaluating model capabilities.
End-to-end agentic RL is promising but challenging
Kimi-Researcher is an autonomous agentic and thinking model designed to solve complex problems through multi-step planning, reasoning, and tool use. It leverages three main tools: a parallel, real-time internal search tool; a text-based browser tool for interactive web tasks; and a coding tool for automated code execution.
Formally, given the state observation \(s_t\) (for instance, \(s_0\) includes system prompt, tool declarations, and user query), Kimi-Researcher generates \(think_t\) and \(action_t\). An action can either be a tool call or an indication to terminate the trajectory.
Traditional agent development has key limitations:
- Workflow-Based Systems: Multi-agent workflows assign roles to specialized agents and coordinate using prompt-based workflows. While effective, they are tied to specific LLM versions and need frequent manual updates.
- Imitation Learning with Supervised Finetuning (SFT): Aligns models well with human demonstrations but struggles with data labeling — especially for long-horizon, agentic tasks in dynamic environments.
End-to-end agentic reinforcement learning trains a single model to solve problems holistically: given a query, the agent explores a large number of possible strategies, receives rewards for correct solutions, and learns from the full trajectory. However, it introduces new challenges:
- Dynamic Environments: Agents must adapt to constantly changing conditions.
- Long-Horizon Tasks: Kimi-Researcher can run 70+ search queries per trajectory, with context windows reaching hundreds of thousands of tokens.
- Data Scarcity: High-quality RL datasets for agentic QA are rare.
- Rollout Efficiency: Multi-turn reasoning and heavy tool use can slow training.
Approach
Kimi-Researcher is trained via end-to-end reinforcement learning. We observe a consistent improvement in agent performance across different domains.
Training data
To address the scarcity of high-quality agentic datasets, we engineered our training corpus with two complementary objectives.
First, we developed a suite of challenging, tool-centric tasks designed to promote robust tool-use learning. These prompts are deliberately constructed such that solving the task requires invoking specific tools — making naive approaches either infeasible or substantially less efficient.
Second, we curated and synthesized reasoning-intensive tasks to reinforce the agent's core cognitive abilities:
- Math and Code Reasoning: Tasks that target logical inference, algorithmic problem-solving, and sequential computation.
- Hard Search: Scenarios where the agent must iteratively search, synthesize, and reason within context constraints.
RL training
The model is primarily trained using the REINFORCE algorithm. Key factors contributing to stable training:
- On-policy Training: Critical to generate strict on-policy data. During training, we disable LLM engine mechanisms like toolcall format enforcers.
- Negative Sample Control: Strategically discarding some negative samples to prevent entropy collapse during RL training.
Kimi-Researcher uses outcome rewards for training:
- Format Reward: Penalized for trajectories with invalid tool calls or exceeding maximum context/iteration limits.
- Correctness Reward: Rewards based on comparison between model's answer and ground truth.
A gamma-decay factor is applied to correct trajectories, encouraging the model to discover shorter, more efficient exploration.
Context management
A long-horizon research trajectory may involve massive observation contexts. We designed a context-management mechanism that allows the model to retain important information while discarding unnecessary documents, extending a single rollout trajectory to over 50 iterations. An early ablation study shows that a model trained with context management uses 30% more iterations.
Large-scale agent RL infrastructure
To address efficiency and stability challenges, we developed infrastructure with:
- Fully asynchronous rollout: Efficiently orchestrates actor rollouts, environmental interactions, and reward calculations in parallel.
- Turn-level partial rollout: Tasks exceeding time budget are saved to replay buffer, with remaining turns executed with updated model weights (1.5x acceleration).
- Robust sandbox environment: Unified sandbox architecture with Kubernetes-based hybrid cloud for dynamic resource allocation.
Emerging agentic capacities
During end-to-end reinforcement learning, we observed several notable emergent abilities:
Resolving conflicting information: Kimi-Researcher resolves inconsistencies through iterative hypothesis refinement and self-correction.
Caution and rigor: Even for seemingly straightforward questions, it deliberately performs additional searches and cross-validates information before answering.
Use cases
- Academic research
- Legal & regulatory insights
- Obscure information retrieval
- Clinical evidence review
- Corporate financial analysis