Post-training of large language models is a term used in recent technical literature for training applied to a large language model (LLM) after its initial large-scale pretraining. The scope varies among surveys. Supervised fine-tuning or instruction tuning and preference-based alignment are common elements, while broader treatments differ on whether reasoning-focused training, parameter-efficient adaptation, distillation, tool integration, or inference-time scaling are included. In training-centered usage, a pretrained or base model is further optimized with structured data such as instruction-response pairs, comparisons between candidate responses, step-level judgments, or outcomes that can be checked automatically. Practical systems may combine several stages, for example supervised training followed by preference optimization or reinforcement learning.
Scope and terminology Recent surveys use post-training as an umbrella label for several lines of research, including instruction tuning, preference alignment, reasoning-oriented training, efficiency methods, and distillation. Their boundaries differ, so the label does not imply a single objective or a universally agreed sequence of stages. Some surveys include increased computation at inference time in a broad account of post-training, while others center the term on additional training after pretraining. Post-training in this sense is also distinct from post-training quantization, which lowers the numerical precision of an already trained network rather than teaching instruction following, preferences, or reasoning behavior.
Development The methods now described as post-training emerged through several lines of work. Multitask instruction tuning trained models on collections of tasks written as natural-language prompts. T0 converted supervised datasets into prompted tasks and evaluated generalization to tasks excluded from its training mixture. Later FLAN experiments studied instruction tuning across larger task mixtures, model families, model sizes, and chain-of-thought data. Preference-based training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and optimized a language model against that reward. InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback. Reviews of reinforcement learning from human feedback describe the approach as a combination of feedback collection, reward learning, and policy optimization, with limitations at each stage. Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and avoided the separately trained reward model and online policy-gradient stage used in a common reinforcement learning from human feedback pipeline. Kahneman-Tversky optimization (KTO) was subsequently proposed for data labeled desirable or undesirable without requiring matched response pairs.
Methods
Supervised fine-tuning and instruction tuning In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, model scale, response construction, data quality, and generalization to unseen tasks. Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning. Results from individual recipes do not establish a fixed amount or composition of data that works for every model or task; outcomes vary with the base model, task distribution, and training procedure. SFT often appears at the beginning of a multi-stage pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a policy that generates outputs for later comparison or verification.
Preference learning and alignment Preference learning uses judgments about candidate outputs rather than only a single target response. Feedback may be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The model is then trained to increase the probability of preferred behavior, commonly subject to a constraint that limits divergence from a reference policy.
Reward modeling and reinforcement learning from human feedback A common reinforcement learning from human feedback (RLHF) pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward. The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The resulting behavior depends on the annotators, instructions, sampling process, and aggregation method used to construct the feedback data.
Direct preference optimization DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss. DPO still depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength. Related objectives use different feedback assumptions. KTO, for example, was designed for examples labeled desirable or undesirable without a matched alternative response. These methods are alternatives within preference-based training, not evidence that one objective is uniformly preferable across datasets and applications.
AI-generated feedback Feedback can also be generated or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences produced by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines. AI-generated feedback reduces dependence on human labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.
… excerpt ends here. Continue reading the full article.
