Reasoning language models (RLMs) or large reasoning models (LRMs) are large language models that are trained further to solve tasks that take several steps of reasoning. They tend to do better on logic, math, and programming tasks than standard LLMs, can revisit and revise earlier steps, and make use of extra computation while answering as another way to scale performance, alongside the number of training examples, parameters, and training compute.
History
2024 In September 2024, OpenAI released o1-preview, an LLM with enhanced reasoning. The full version, o1, followed in December 2024. OpenAI also began sharing results on its successor, o3. The development of reasoning LLMs has illustrated what Rich Sutton called the "bitter lesson": that scaling compute often outperforms methods that rely on specific human insights. For example, the Generative AI Research Lab (GAIR) explored complex methods such as tree search and reinforcement learning to replicate o1's capabilities. In their "o1 Replication Journey" papers they reported that knowledge distillation (training a smaller model to imitate o1's outputs) worked surprisingly well. This highlighted the effectiveness of distillation in this context. Alibaba released reasoning versions of its Qwen LLMs in November 2024. In December 2024, the team introduced QvQ-72B-Preview, an experimental visual reasoning model. In December 2024, Google introduced Deep Research in Gemini, a feature that runs multi-step research tasks. On December 16, 2024, an experiment with a Llama 3B model showed that by scaling test-time compute, a relatively small model could outperform a much larger Llama 70B model on challenging reasoning tasks. This suggested that better inference strategies can unlock useful reasoning capabilities even in small models.
2025 In January 2025, DeepSeek released R1, a model with comparable performance to o1 at lower cost. The release demonstrated the effectiveness of Group Relative Policy Optimization (GRPO). On January 25, 2025, DeepSeek added a feature to DeepSeek R1 that lets the model search the web while it reasons, making it easier to combine retrieval with reasoning. The effectiveness of distillation for reasoning models was shown in works such as s1-32B, which achieved strong performance through budget forcing and scaling methods. On February 2, 2025, OpenAI released Deep Research based on their o3 model, allowing users to initiate complex research tasks and generate comprehensive reports which incorporate various sources from the web. OpenAI called GPT-4.5 its "last non-chain-of-thought model", and implemented with GPT-5 a router that selects a model based on the difficulty of the task.
2026 In January 2026, Moonshot AI released Kimi K2.5, an open-source 1 trillion parameter MoE model with 32 billion active parameters which was followed by release of Kimi K2.6 in April 2026 with both Kimi models use an “Agent Swarm” system that dynamically decomposes tasks into sub-agents for reasoning and execution, enabling more scalable multi-step problem solving than a single sequential reasoning chain, However both Kimi models despite being identical in parameters have differences in Agent Swarm with K2.5 utilizing a 100 sub-agent format while K2.6 utilized 300 sub-agent which enhanced the latter's ability for task coordination.
Supervised finetuning A large language model (LLM) can be fine-tuned on a dataset of reasoning tasks paired with example solutions and step-by-step (reasoning) traces. The fine-tuned model can then produce its own reasoning traces for new problems. Because human-written traces are costly to collect, researchers have proposed ways to build such datasets automatically. In rejection sampling finetuning (RFT), new reasoning traces are gathered in a loop:
Sample a task prompt. Generate many reasoning traces for the prompt. Use a verifier to remove reasoning traces with a wrong final answer, and optionally remove duplicates
Reinforcement learning A pretrained language model can be further trained with RL. In the RL formalism, a generative language model is a policy π {\displaystyle \pi } . A task prompt is an environmental state x {\displaystyle x} , and the model's response is an action y {\displaystyle y} . The probability that the model responds x {\displaystyle x} with y {\displaystyle y} is π ( y | x ) {\displaystyle \pi (y|x)} . Training a reasoning language model with RL means constructing a reward model r ( x , y ) {\displaystyle r(x,y)} to guide the RL process. Intuitively, the reward says how good a response is for a prompt. For a reasoning task, the reward is high if the response solves the task and low if it does not. A response y {\displaystyle y} may be broken-down into multiple steps, written y 1 , y 2 , … , y n {\displaystyle y_{1},y_{2},\dots ,y_{n}} . Most recent systems use policy-gradient methods such as Proximal Policy Optimization (PPO) because PPO constrains each policy update with a clipped objective, which stabilises training for very large policies.
Outcome reward model
… excerpt ends here. Continue reading the full article.
