Generative AI Engineering
Learn fine-tuning, continued pre-training, instruction tuning, SFT, PEFT, LoRA, QLoRA, preference data, RLHF, DPO, GRPO, regression evaluation, data rights, privacy, versioning, and rollback.
54 matching terms
Fine-tuning overview
Fine-tuning
Continuing training from a pre-trained model to adapt its behavior to a narrower task, domain, format, or preference.
Use it when prompting and retrieval alone cannot provide sufficiently consistent learned behavior.
Fine-tune the approved base model on reviewed support responses, then compare it with the unchanged baseline.Fine-tuning overview
Downstream task
A specific application task performed using a general pre-trained model.
Define it precisely before collecting fine-tuning data and selecting metrics.
Downstream task: classify Korean support requests into the approved routing taxonomy.Fine-tuning overview
Domain adaptation
Adapting a model to language, patterns, or distributions found in a particular field or environment.
Use it when the target domain differs materially from the model's general training distribution.
Adapt the model to authorized semiconductor maintenance documents and evaluate on unseen manuals.Fine-tuning overview
Continued pre-training
Further pre-training a base model on additional unlabeled or self-supervised domain data before task-specific tuning.
Use it for broad domain language adaptation when a small set of instruction examples is insufficient.
Continue pre-training on the licensed domain corpus, then run supervised instruction tuning.Fine-tuning overview
Instruction tuning
Fine-tuning on instructions and desired responses so a model follows varied natural-language tasks more reliably.
Use diverse, reviewed instructions that reflect target users, tasks, languages, and safety boundaries.
Train on instruction-response examples covering summarization, extraction, refusal, and clarification.Fine-tuning overview
Post-training
Training performed after broad pre-training to improve task following, preferences, safety, or deployment behavior.
Use it as an umbrella term for supervised tuning, preference optimization, reward modeling, and related alignment work.
Document each post-training stage, its data, objective, checkpoint, and evaluation gate.Fine-tuning overview
Supervised fine-tuning (SFT)
Fine-tuning that learns from inputs paired with reviewed target responses or labels.
Use it to teach response format, task procedure, tone, and examples of desired behavior.
SFT example: user request, approved assistant response, metadata, and data-rights record.Fine-tuning overview
Model alignment
The effort to make model behavior better follow defined human intentions, policies, preferences, and safety requirements.
Use it as the broad goal behind instruction tuning, preference optimization, safety training, and behavioral evaluation.
Define the alignment objectives, unacceptable behaviors, evaluation cases, and human approval gates.Fine-tuning overview
Behavior adaptation
Changing how a model responds, formats output, follows procedures, or handles uncertainty without assuming that it gains current factual knowledge.
Use it when the desired change concerns stable response patterns rather than frequently changing information.
Adapt the model to request clarification when required account evidence is missing.Methods and adapters
Full fine-tuning
Fine-tuning that updates all or most parameters of the base model.
Use it only when the expected benefit justifies substantial memory, compute, storage, and evaluation cost.
Compare full fine-tuning against a PEFT baseline under the same data and evaluation set.Methods and adapters
Parameter-efficient fine-tuning (PEFT)
A family of methods that adapts a model while training a relatively small portion of parameters.
Use it to reduce training memory, storage, and per-task checkpoint size.
Train a separate PEFT adapter for each approved domain while keeping the base weights fixed.Methods and adapters
Adapter
A small set of additional or modified parameters attached to a base model for a particular adaptation.
Use adapters to separate task-specific changes from shared base weights.
Load the finance adapter only for authorized finance workflows.Methods and adapters
Low-rank adaptation (LoRA)
A PEFT method that learns low-rank update matrices for selected model modules while keeping base weights frozen.
Use it when smaller trainable states and reusable adapters fit the deployment design.
Apply LoRA to selected attention projections and record rank, alpha, dropout, and target modules.Methods and adapters
QLoRA
A fine-tuning approach that trains LoRA adapters while the base model is loaded in a quantized representation.
Use it to reduce base-model memory while validating numerical stability and quality against an appropriate baseline.
Load the approved base model in the supported quantization format and train only the LoRA adapter.Methods and adapters
Quantization
Representing model values with lower-precision formats to reduce memory, storage, or compute requirements.
Use it when the supported method and hardware can preserve acceptable quality and numerical behavior.
Evaluate the quantized base model before QLoRA and record format, bit width, library, and compute dtype.Methods and adapters
LoRA rank
The dimensionality of the low-rank update matrices used by a LoRA adapter.
Tune it as a capacity and resource tradeoff rather than assuming a larger rank is always better.
Compare rank 8 and rank 16 with the same seed, data, steps, and evaluation suite.Methods and adapters
LoRA alpha
A scaling configuration that influences the contribution of learned LoRA updates.
Record it together with rank and implementation details because interpretation can depend on the library.
Store rank, alpha, dropout, target modules, and library version in the adapter manifest.Methods and adapters
Target modules
The model layers or projections selected to receive adapter updates.
Choose them according to model architecture, method support, and measured quality.
Target the supported query and value projections, then verify the number of trainable parameters.Methods and adapters
Trainable parameters
The subset of model parameters that optimization is allowed to update.
Report both the count and percentage so full and parameter-efficient runs can be compared.
Log trainable parameters, total parameters, percentage, and adapter checkpoint size.Methods and adapters
Adapter merge
Combining learned adapter updates into base weights or another adapter representation for deployment.
Use it only after compatibility, license, precision, quality, and rollback checks.
Evaluate the merged artifact against the unmerged base-plus-adapter configuration before release.Training formats
Prompt-completion pair
A training example containing an input prompt and the desired completion.
Use a consistent schema and ensure completions demonstrate the exact target behavior.
{"prompt":"Classify the request","completion":"billing_refund"}Training formats
Instruction-response pair
A supervised example pairing a natural-language instruction or conversation with the reviewed response the model should learn to produce.
Use it for SFT and retain source, rights, review, and quality metadata alongside each example.
{"messages":[{"role":"user","content":"배송 취소 방법을 알려줘"},{"role":"assistant","content":"주문 내역에서 취소할 주문을 선택하세요."}]}Training formats
Chat template
A formatting rule that converts system, user, assistant, and tool messages into the token sequence expected by a chat model.
Use the tokenizer and template compatible with the selected base model during both training and inference.
Render the same approved chat template in dataset preprocessing and production inference.Training formats
System, user, and assistant roles
Message-role labels that distinguish durable instructions, user inputs, and model responses in conversational training data.
Use only roles and ordering supported by the selected model's chat template.
System: policy and role; user: request; assistant: reviewed target response.Training formats
Response masking
Excluding selected input tokens from loss calculation so training focuses on intended response tokens.
Use it when the objective should learn assistant outputs without treating every prompt token as a target.
Mask system and user tokens from the loss and verify assistant boundaries after tokenization.Training formats
Sequence packing
Combining multiple shorter examples into longer training sequences to reduce padding and improve utilization.
Use it only when example boundaries, attention behavior, labels, and end tokens remain correct.
Pack short dialogs up to the context limit and test that one example cannot leak into the next.Training formats
Fine-tuning context length
The maximum token length used for each training sequence after formatting and tokenization.
Set it according to the base model, target task, memory budget, truncation risk, and production request pattern.
Measure token-length distribution and choose a limit that preserves critical content without excessive padding.Training formats
Preference dataset
A dataset that records comparative judgments or rewards for alternative model outputs.
Use it for reward modeling or preference optimization with documented labeling rules and reviewer quality checks.
Store prompt, candidate responses, preference result, rationale policy, reviewer group, and consent status.Training formats
Chosen and rejected response pair
Two candidate responses labeled to indicate which one better satisfies the defined preference criteria.
Use pairs that differ meaningfully and apply a consistent rubric instead of personal taste alone.
Chosen: cites the supplied policy and states uncertainty. Rejected: invents an unsupported deadline.Preference alignment
Human feedback
Structured judgments, labels, demonstrations, or corrections supplied by people to guide model behavior.
Define reviewer qualification, instructions, disagreement handling, privacy, compensation, and quality assurance.
Use calibrated reviewers and adjudicate high-impact disagreements before training.Preference alignment
Reinforcement learning from human feedback (RLHF)
A family of post-training approaches that uses human preference signals and reinforcement-learning techniques to shape model behavior.
Use it only with a clear reward design, stable training setup, safety evaluation, and strong data governance.
Train and validate a reward signal, optimize the policy conservatively, and compare against the SFT baseline.Preference alignment
Reward model
A model that scores candidate outputs according to learned preference or quality signals.
Use it when a training or selection process needs a scalable approximation of documented preferences.
Evaluate reward-model accuracy, calibration, subgroup behavior, and exploitability on held-out comparisons.Preference alignment
Preference optimization
Post-training that updates model behavior using comparative preferences, rewards, or related feedback signals.
Use it as a broad category that includes direct-preference and reinforcement-learning approaches.
Define the preference rubric, reference behavior, optimization method, and retention evaluations.Preference alignment
Reinforcement learning from AI feedback (RLAIF)
A family of reinforcement-learning approaches that uses feedback produced or assisted by AI systems instead of relying only on direct human judgments.
Use it only with validated feedback criteria, human oversight, bias checks, and safeguards against self-reinforcing errors.
Calibrate AI feedback against a held-out set of expert human judgments before policy optimization.Preference alignment
Reinforcement fine-tuning
Fine-tuning that optimizes model behavior from reward signals produced by graders, environments, rules, or feedback models.
Use it when outcomes can be scored reliably and the training system can monitor reward hacking, stability, and capability retention.
Score verifiable task outcomes, optimize conservatively, and compare against the unchanged and SFT baselines.Preference alignment
Direct preference optimization (DPO)
A preference-training method that directly uses chosen and rejected responses to update a policy relative to a reference behavior.
Use it when suitable preference pairs and a compatible training implementation are available.
Train DPO from the approved SFT checkpoint and compare helpfulness, safety, and capability retention.Preference alignment
Proximal policy optimization (PPO)
A policy-optimization algorithm that constrains the size of updates while optimizing a reward signal.
Use it in reinforcement-learning pipelines that can support reward, value, rollout, and stability monitoring.
Track reward, divergence from the reference model, response length, and safety metrics during PPO.Preference alignment
Group relative policy optimization (GRPO)
A reinforcement-learning method that compares rewards among multiple sampled outputs for the same prompt to estimate relative advantage.
Use it only when group sampling, reward quality, training stability, and compute cost fit the task.
Sample several solutions per problem, score them with verified checks, and optimize from group-relative rewards.Evaluation and regression
Fine-tuning validation loss
Loss measured on held-out fine-tuning examples without updating the model from those examples.
Use it with task, safety, and behavior metrics because lower loss alone does not prove better production behavior.
Select candidate checkpoints using validation loss and the fixed production-style evaluation suite.Evaluation and regression
Holdout evaluation
Evaluation on examples deliberately excluded from training and model-selection feedback.
Use it for a less-biased estimate of performance on unseen cases and protect the set from repeated tuning.
Run the locked holdout suite only after selecting the candidate checkpoint.Evaluation and regression
Checkpoint selection
Choosing which saved training state should advance based on predefined quality, safety, cost, and regression criteria.
Use a fixed evaluation suite and avoid selecting from training loss alone.
Promote the checkpoint that passes task, safety, multilingual, latency, and memorization gates.Evaluation and regression
Base-model comparison
A controlled evaluation comparing the fine-tuned model with the unchanged base model.
Use identical prompts, retrieval, decoding, and datasets to attribute changes to fine-tuning.
Report wins, ties, regressions, latency, and cost for base and tuned candidates.Evaluation and regression
Catastrophic forgetting
A loss of previously learned capability after training strongly on new data or objectives.
Test retained general capabilities, languages, safety behavior, and out-of-domain tasks after tuning.
Run the baseline capability suite before and after fine-tuning and block material regressions.Evaluation and regression
Capability regression
A measurable decline in an existing model capability after fine-tuning or deployment changes.
Define regression thresholds for important tasks before training and release.
Fail the release if multilingual extraction accuracy drops beyond the approved threshold.Evaluation and regression
Safety regression
A decline in refusal, privacy, fairness, or harmful-content behavior after adaptation.
Run targeted adversarial and policy evaluations for every candidate checkpoint.
Compare jailbreak, PII disclosure, bias, and unsafe-completion rates with the approved baseline.Evaluation and regression
Training-data memorization
A model's retention and possible reproduction of specific training examples beyond desired generalization.
Test for unusual phrase reproduction, secrets, personal data, and near-verbatim outputs before release.
Probe held-out prefixes and block release if the model reproduces sensitive continuations.Evaluation and regression
Fine-tuning versus RAG
A design decision between changing model behavior through training and supplying external knowledge at request time.
Prefer retrieval for current or frequently changing facts and fine-tuning for stable behavior, format, or task patterns, then evaluate a combined approach if needed.
Fine-tuning does not automatically make frequently changing facts current or provide source citations.
Use RAG for current policies and SFT for the approved response structure.Rights and operations
Fine-tuning data rights
The licenses, permissions, contractual terms, and other authority required to use data for model adaptation.
Verify rights at source and example level and retain provenance, restrictions, and removal procedures.
Online availability, API access, or possession of a file does not by itself grant fine-tuning rights.
Record source, rights holder, license or permission, allowed training use, territory, term, and deletion contact.Rights and operations
Personal data in fine-tuning
Information in adaptation data that identifies or can be linked to an individual.
Minimize, redact, obtain proper authority, restrict access, and define retention and deletion before training.
Replace names and account identifiers with controlled placeholders and verify the transformed dataset.Rights and operations
Model registry
A controlled catalog of model, adapter, checkpoint, evaluation, approval, and deployment metadata.
Use it to track which artifact is approved, deployed, deprecated, or eligible for rollback.
Register model digest, base version, adapter, dataset, evaluations, owner, approval, and deployment stage.Rights and operations
Adapter serving
Serving one base model with selectable task- or domain-specific adapters at inference time.
Use it when the runtime can isolate adapters, authorize selection, manage memory, and preserve version compatibility.
Route approved finance requests to the finance adapter and log the exact base-plus-adapter version.Rights and operations
Fine-tuning provider data policy
A service provider's terms and controls for uploaded training data, retention, deletion, model access, geographic processing, and secondary use.
Review the current contract and product documentation before uploading any dataset, and record the approved configuration.
Provider policies can change; recheck them before each material training program and never assume API access grants data-use rights.
Verify retention period, deletion process, training-use setting, access roles, region, encryption, and incident procedure.Rights and operations
Fine-tuned model versioning
Tracking the exact base model, adapter or checkpoint, data, code, and configuration that produced a tuned model.
Use immutable identifiers so evaluations, incidents, rollbacks, and audits refer to the same artifact.
Version = base model digest + adapter digest + dataset hash + code commit + training config.Rights and operations
Fine-tuning rollback
Returning production traffic to a previously approved model or adapter after a problem is detected.
Prepare it before release with artifact retention, compatibility checks, routing controls, and decision authority.
Keep the last approved base-plus-adapter combination deployable and rehearse the rollback path.Start with the least complex approach that reliably meets the requirement, then evaluate combinations when necessary.
| What you need | Approach to consider first |
|---|---|
| Improve response format or tone | Prompt design or SFT |
| Answer from current or private documents | RAG with source-grounded retrieval |
| Learn repeatable task behavior | Fine-tuning with representative examples |
| Adapt broad industry language and patterns | Continued pre-training or domain fine-tuning |
| Learn from preferred and rejected responses | DPO or another validated preference-optimization method |
The methods require different data and optimize different signals; they are not interchangeable labels for fine-tuning.
| Method | Typical data and objective |
|---|---|
| SFT | Instruction-response or labeled examples; learn to reproduce reviewed target behavior. |
| DPO | Chosen/rejected response pairs; directly increase relative preference against reference behavior. |
| GRPO | Multiple sampled outputs and verified rewards; optimize from group-relative advantages. |
Pre-training teaches broad patterns at large scale and produces a base model. Fine-tuning continues from that model with narrower data and objectives to adapt a task, format, domain, preference, or safety behavior.
Frequently changing or source-sensitive knowledge usually fits RAG because documents can be updated and cited at request time. Fine-tuning is generally better suited to stable behavior, format, or task patterns. Some systems combine both.
LoRA trains low-rank adapter updates while keeping base weights frozen. QLoRA additionally loads the base model in a supported quantized representation to reduce memory. Quality and compatibility must be evaluated for the exact stack.
No. A tuned model can still become outdated and does not inherently expose the source of a generated claim. Use retrieval or another verified data connection when freshness and citations matter.
Use SFT for reviewed target responses, DPO for comparative preference pairs, and GRPO for tasks with reliable rewards across multiple sampled outputs. Choose only after confirming the data, objective, evaluation, safety, and compute requirements.
Verify data rights, consent, confidentiality, personal-data handling, provider terms, retention and deletion, geographic processing, model-access controls, secondary training use, encryption, incident response, and the current contract.
Official guide to continuing training from a pre-trained model with datasets, tokenization, training arguments, evaluation, and checkpoints.
Official conceptual guide to LoRA configuration, rank, alpha, target modules, and parameter-efficient adaptation.
Official overview and examples for post-training methods including SFT, reward modeling, DPO, and GRPO.
Review datasets, batches, epochs, loss, optimization, generalization, hardware, and reproducibility first.
Check whether instructions, examples, schemas, or prompt chaining can solve the requirement before training.
Use retrieval concepts when answers require current, private, source-grounded knowledge.
Connect tuned-model evaluation to fixed datasets, benchmarks, human review, regression tests, latency, and cost.