Thinking & Reasoning Models

Nikita Namjoshi's video "How do thinking and reasoning models work?" from the Google for Developers channel explains the concepts behind "thinking models" or "reasoning models," such as Gemini, and how they use more computation at inference time to achieve better results in complex tasks.

Video Summary

The video focuses on how Large Language Models (LLMs) can be improved to handle complex tasks like coding, advanced mathematics, and data analysis by utilizing more compute power during the generation phase (inference or test time).

The Problem: LLMs generate responses by predicting one token at a time. When solving complex problems, the model has to figure out the entire solution in a single pass to generate the correct final answer, which is difficult.

The Solution: Chain of Thought (CoT): CoT prompting is a technique where the model is prompted to generate a series of intermediate steps (a "chain of thought") that lead to the final answer. This process forces the model to generate more tokens, meaning more forward passes through the model's weights, which is equivalent to "reasoning" or spending more compute before giving the answer.

Test Time Compute: The concept is about deliberately giving models more compute power when generating a response to improve accuracy, going beyond just increasing compute during the initial training phase (scaling laws).

Strategies for Test Time Compute:

Best-of-N: The model generates multiple candidate responses (e.g., N=100) for a single prompt, and the most frequent response is returned as the answer.

Reward/Verifier Models: A second model, known as a verifier or reward model, is used to assign a quality score to each candidate answer, and the one with the highest score is selected.

Training Thinking Models (Reinforcement Learning): Reinforcement Learning (RL) is used during model post-training to teach LLMs to produce long chains of thought. By training on problems with objective, verifiable answers (like math or code), the model is rewarded for correct final answers, which leads to the surprising emergence of longer, more effective reasoning chains.

Key Terms

Thinking/Reasoning Models - LLMs that use more tokens when generating an answer to achieve better results in complex tasks like coding, advanced mathematics, and data analysis.

Thinking Trace/Thought Summary - A short summary of the internal contents of the model's reasoning path, intended to help users follow how the model arrived at the answer.

Scaling Laws - Describes how model performance improves as you increase the amount of training data and compute (more compute + more data + more parameters = better models).

Inference/Test Time - The phase when a user interacts with the model to generate a response.

Test Time Compute - The concept of making models better by giving them more compute power specifically when generating a response.

Chain of Thought Prompting (CoT) - A way to improve an LLM's ability to provide correct responses to complex reasoning tasks by prompting it to generate a series of intermediate steps (the chain of thought) that lead to the final answer.

Reward Models - A type of model used to evaluate and improve the performance of other AI models, often against specific criteria like correctness, fluency, or relevance.

Reinforcement Learning (RL) - A framework where an agent (the LLM) learns to solve a task by interacting with an environment, performing actions (generating tokens), and receiving a reward (e.g., a verifiable reward for a correct answer) to maximize future rewards.