Generative AI Engineering
Understand training data, objectives, batches, epochs, loss, gradients, optimizers, learning rates, validation, overfitting, checkpoints, distributed training, and reproducibility.
61 matching terms
Training foundations
Model training
The process of adjusting model parameters from data so the model can perform a target behavior or task.
Use the term for the overall optimization process, including data preparation, training runs, validation, and checkpoint selection.
Define the training objective, approved dataset, validation plan, compute budget, and stopping criteria before the run.Training foundations
Pre-training
Large-scale training that teaches a model broad patterns before it is adapted to a narrower task or domain.
Use it when distinguishing the creation of a base model from later fine-tuning or alignment.
Pre-train the base model on an authorized multilingual corpus, then evaluate language coverage.Training foundations
Foundation model
A broadly pre-trained model designed to support many downstream tasks and adaptations.
Use it when identifying the general model that later prompting, retrieval, adapters, or fine-tuning build upon.
Record the approved foundation-model family, version, license, and intended adaptation path.Training foundations
Base model
The specific pre-trained checkpoint used as the starting point for inference or additional adaptation.
Use the exact model and revision identifier so training and deployment can be reproduced.
Base model: approved model ID and immutable revision digest.Training foundations
Model parameter
A numerical value inside a model that may be learned during training or held fixed by the chosen method.
Use it when comparing total, frozen, and trainable portions of a model.
Report total parameters and the subset updated by the training method.Training foundations
Transfer learning
Reusing knowledge learned for one setting as the starting point for another related task or domain.
Use it when adapting a pre-trained representation instead of training a new model from random initialization.
Start from the approved language model and adapt it to intent classification.Training foundations
Training objective
The mathematical or behavioral goal that a training run attempts to optimize.
Define it before choosing data, labels, loss functions, and evaluation metrics.
Objective: predict the next token while minimizing validation loss on the approved domain corpus.Training foundations
Supervised learning
Training from examples that pair inputs with expected labels or outputs.
Use it when reliable target answers, classes, or values are available.
Train on support questions paired with reviewed intent labels.Training foundations
Self-supervised learning
Training that derives prediction targets from the structure of unlabeled data itself.
Use it for large-scale representation or language-model learning where manual labeling is impractical.
Hide or shift part of each sequence and train the model to predict the missing or next content.Training data
Training dataset
The examples used to calculate gradients and update model parameters.
Build it from relevant, licensed, representative, and quality-controlled data.
Do not assume that publicly accessible data is automatically authorized for model training.
Create a versioned training dataset with source, license, consent, language, and quality metadata.Training data
Validation dataset
A held-out dataset used during development to compare settings and monitor generalization.
Use it for checkpoint selection, early stopping, and hyperparameter decisions without updating weights from its examples.
Evaluate validation loss at each epoch and keep the best approved checkpoint.Training data
Test dataset
A separately held-out dataset used for a final, less-biased estimate after development choices are complete.
Use it only after model and hyperparameter selection to avoid tuning to the final test.
Lock the test set before experimentation and report its version with the final result.Training data
Training corpus
A collected body of text, code, images, audio, or other material prepared for model learning.
Use the term when discussing source coverage, provenance, filtering, and rights at collection scale.
Document corpus sources, collection dates, exclusions, rights basis, and removal procedures.Training data
Training sample or example
One unit presented to the training pipeline, such as a labeled row, sequence, image, conversation, or preference pair.
Use it when defining schema, sampling weight, quality checks, and what counts toward dataset size.
One example contains input, target, source ID, rights metadata, and quality status.Training data
Label
A target class, value, span, response, or judgment associated with a supervised training example.
Define labels with a documented rubric and measure reviewer agreement for ambiguous tasks.
Label each ticket with one approved intent and record uncertain cases for adjudication.Training data
Data augmentation
Creating additional training examples by applying controlled transformations that preserve the intended label or behavior.
Use it to improve coverage while verifying that transformations remain realistic and do not change the target meaning.
Create paraphrase variants, then review that each variant preserves the original intent label.Training data
Data contamination
The presence of evaluation examples or close variants in training data, causing evaluation to overstate generalization.
Check exact and semantic overlap before training and quarantine protected evaluation sets.
Compare hashes and near-duplicate embeddings across training and evaluation splits.Training data
Training tokenization
The conversion of raw content into model input units and identifiers used during training.
Keep tokenizer choice, vocabulary, normalization, and special tokens compatible with the model and serving pipeline.
Record tokenizer version, vocabulary size, special tokens, truncation, and maximum sequence length.Training data
Data split
The separation of examples into training, validation, and test subsets.
Split by user, time, document, or group when random rows could leak near-duplicates across subsets.
Split by customer account so conversations from one customer do not appear in both training and test data.Training data
Data deduplication
The detection and removal or grouping of identical and near-duplicate training examples.
Use it to reduce memorization, source imbalance, wasted compute, and train-test contamination.
Deduplicate exact hashes, normalized text, and high-similarity document segments before splitting.Training data
Synthetic training data
Examples generated or transformed by software or models rather than directly collected from real events.
Use it to cover rare cases, formats, or privacy-preserving simulations, then validate realism and bias.
Synthetic data can reproduce source bias, errors, or protected expression and still requires provenance and quality checks.
Generate edge-case support dialogs, label them as synthetic, and review them before training.Training data
Confidential training data
Customer records, trade secrets, internal documents, credentials, or other restricted information considered for training.
Exclude it by default unless there is explicit authority, a documented purpose, strict access control, and an approved retention and deletion plan.
Never place credentials, customer secrets, or trade secrets in a training service without approved authority and provider-policy review.
Block credentials and trade secrets during ingestion and route uncertain records to an authorized reviewer.Training data
Data leakage
The unintended use of information during training that would not be available in the intended real-world prediction setting.
Check features, labels, timestamps, and preprocessing to prevent unrealistically high evaluation scores.
Remove post-resolution fields before training a model that predicts ticket escalation at intake.Training loop
Batch
A group of training examples processed together in one forward and backward computation.
Use batches to balance computational efficiency, memory use, and gradient stability.
Assemble each batch with padded sequences and an attention mask.Training loop
Mini-batch
A smaller subset of the training data used for one gradient calculation rather than processing the full dataset at once.
Use it as the standard practical unit for stochastic optimization on large datasets.
Shuffle the training set and construct mini-batches of similar sequence length.Training loop
Batch size
The number of examples or sequences included in a batch.
Tune it according to sequence length, memory, optimization behavior, and effective batch size.
Start with per-device batch size 4 and use gradient accumulation only if required by memory.Training loop
Training step / iteration
One optimizer-update cycle, which may consume one or several micro-batches.
Use it for logging, scheduling, saving checkpoints, and comparing run duration.
Log training loss and learning rate every 20 optimizer steps.Training loop
Epoch
One complete pass through the training dataset under the configured sampling process.
Use epoch counts with validation curves rather than treating more epochs as automatically better.
For 1,000 examples and batch size 50, one epoch contains about 20 batches.Training loop
Forward pass
The computation that sends model inputs through the network to produce predictions and a loss value.
Use it when explaining the first half of each training iteration before gradients are calculated.
Run the forward pass on token IDs and compare predicted tokens with the target labels.Training loop
Loss function
A function that measures how far model predictions are from the training objective.
Choose it to match the task, label format, class balance, and desired behavior.
Track token-level training loss and sequence-level task metrics separately.Training loop
Backpropagation
The process of propagating loss derivatives backward through the model to compute gradients.
Use it to explain how an error signal reaches trainable parameters.
Backpropagate the scaled loss, clip gradients, and then update the optimizer.Training loop
Gradient
A set of derivatives indicating how parameter changes would affect the loss.
Monitor gradient norms to detect instability, vanishing updates, or exploding gradients.
Alert when the gradient norm becomes non-finite before an optimizer step.Training loop
Optimizer
An algorithm that updates trainable parameters using gradients and configured state.
Select and configure it together with learning rate, weight decay, and precision settings.
Use AdamW with explicitly recorded learning rate, weight decay, and optimizer version.Training loop
Stochastic gradient descent (SGD)
An optimization method that updates parameters from gradients estimated on individual examples or mini-batches.
Use it as a foundational optimizer concept and compare momentum or scheduling choices for the task.
Run SGD with a recorded learning rate, momentum, scheduler, and batch configuration.Training loop
AdamW
An adaptive optimizer that decouples weight decay from the gradient-based parameter update.
Use it when supported by the training recipe and record all optimizer hyperparameters for reproducibility.
Record AdamW learning rate, betas, epsilon, weight decay, and implementation version.Training loop
Learning rate
A hyperparameter controlling the scale of parameter updates during optimization.
Tune it carefully because values that are too high can destabilize training and values that are too low can stall progress.
There is no universally correct learning rate; appropriate values depend on model, method, data, batch size, and optimizer.
Compare candidate learning rates using the same data split, seed, and evaluation schedule.Training loop
Learning-rate scheduler
A rule that changes the learning rate over training steps.
Use it to warm up training, decay updates, or react to a plateau according to the experiment plan.
Warm up for the planned fraction of steps, then use cosine decay to the final rate.Training loop
Warmup
An initial period that gradually raises the learning rate from a small value.
Use it to reduce unstable updates near the start of training, especially in large-model optimization.
Record warmup steps as part of the reproducible training configuration.Generalization and quality
Training loss
Loss calculated on batches used to update model parameters.
Use it to monitor optimization progress, but do not treat a low value as proof of generalization.
Plot smoothed training loss by optimizer step and investigate sudden spikes or non-finite values.Generalization and quality
Validation loss
Loss measured on held-out validation examples without updating parameters from those examples.
Compare it with training loss to detect overfitting and select candidate checkpoints.
Keep the checkpoint with the best approved validation behavior, not necessarily the final epoch.Generalization and quality
Convergence
The state in which training updates produce progressively smaller or more stable improvement.
Judge it from training and validation curves together with task metrics, not from training loss alone.
Treat the run as converged only after validation metrics stabilize across checkpoints.Generalization and quality
Generalization
The ability of a trained model to perform well on relevant examples it did not train on.
Measure it on representative held-out and real-world cases.
Compare performance across unseen users, time periods, languages, and difficult edge cases.Generalization and quality
Overfitting
A condition where a model fits training data closely but performs worse on unseen data.
Look for a widening gap between training and validation behavior and respond with data, regularization, or earlier stopping.
Stop when validation loss worsens while training loss continues to decrease.Generalization and quality
Underfitting
A condition where a model fails to learn enough structure to perform well even on training data.
Check model capacity, feature or data quality, optimization settings, and training duration.
Investigate underfitting when both training and validation metrics remain poor.Generalization and quality
Regularization
Techniques that limit or shape learning to reduce overfitting and improve behavior on unseen data.
Choose regularization together with model capacity, data volume, augmentation, and early stopping.
Compare validation behavior with documented weight decay and dropout settings.Generalization and quality
Weight decay
A regularization setting that discourages parameter values from growing excessively during optimization.
Tune it with the optimizer and verify which parameters are excluded from decay.
Record weight decay and the parameter groups to which it applies.Generalization and quality
Dropout
A regularization technique that randomly disables selected activations during training.
Use only where the architecture and training method support it, and disable training behavior during evaluation.
Set and record the supported dropout rate, then compare validation performance.Generalization and quality
Gradient clipping
Limiting gradient magnitude or norm before an optimizer update to reduce unstable steps.
Use it when gradient spikes are possible and monitor how often clipping occurs.
Clip the global gradient norm at the documented threshold and log clipped steps.Generalization and quality
Early stopping
Stopping a training run when a monitored validation metric no longer improves under defined rules.
Use patience and minimum-improvement thresholds to avoid reacting to one noisy evaluation.
Stop after three validation checks without meaningful improvement and restore the best checkpoint.Generalization and quality
Training checkpoint
A saved training state containing model parameters and often optimizer, scheduler, and progress information.
Use checkpoints for recovery, comparison, evaluation, and controlled promotion.
Save versioned checkpoints with dataset, code, tokenizer, and configuration identifiers.Compute and reproducibility
AI accelerator (GPU, TPU, NPU)
Specialized compute hardware used to accelerate tensor operations for model training and inference.
Choose hardware according to framework support, numerical formats, memory, interconnect, availability, and cost.
Record accelerator type, count, memory, runtime, driver, and estimated training cost.Compute and reproducibility
Video memory (VRAM)
Memory on an accelerator used for model weights, optimizer states, activations, gradients, and batches.
Estimate it before training and adjust batch size, precision, sharding, or checkpointing when necessary.
Measure peak allocated VRAM for the longest supported sequence and effective batch configuration.Compute and reproducibility
FP16 and BF16
Reduced-precision numerical formats commonly used to lower training memory and increase accelerator throughput.
Use only formats supported by the hardware and framework, with finite-value and quality checks.
Compare BF16 or FP16 training stability and output quality with the approved baseline.Compute and reproducibility
Gradient accumulation
Combining gradients from several micro-batches before one optimizer update.
Use it to reach a larger effective batch size when device memory is limited.
Effective batch size = per-device batch × accumulation steps × number of data-parallel devices.Compute and reproducibility
Mixed-precision training
Training that uses more than one numerical precision to improve speed or memory efficiency while preserving necessary stability.
Use supported FP16 or BF16 workflows with loss and finite-value monitoring.
Enable BF16 on compatible hardware and verify loss parity against the baseline run.Compute and reproducibility
Gradient checkpointing
A memory-saving technique that recomputes selected activations during backpropagation instead of storing all of them.
Use it when memory is the constraint and additional computation is acceptable.
Enable gradient checkpointing to fit longer sequences, then measure the training slowdown.Compute and reproducibility
Distributed training
Training that coordinates computation or model state across multiple accelerators or machines.
Use it when one device cannot meet model-size or training-time requirements.
Record data-parallel strategy, world size, sharding policy, network setup, and recovery procedure.Compute and reproducibility
Data parallelism
Replicating a model across workers that process different data batches and synchronize updates.
Use it when the model fits on each worker and more batch throughput is needed.
Record worker count, global batch size, gradient synchronization, and failure recovery behavior.Compute and reproducibility
Model parallelism
Splitting model computation or parameters across multiple devices because one device cannot hold or process the entire model efficiently.
Use tensor, pipeline, or sharded approaches according to architecture and framework support.
Document the partition strategy, communication cost, checkpoint format, and serving compatibility.Compute and reproducibility
Random seed
A value used to initialize pseudo-random operations such as shuffling, sampling, dropout, and parameter initialization.
Record it for repeatability while recognizing that distributed hardware and some kernels may remain nondeterministic.
Compare important results across several recorded seeds instead of relying on one favorable run.Compute and reproducibility
Training reproducibility
The ability to repeat a run closely enough to explain and compare its results.
Version data, code, model, tokenizer, dependencies, seeds, configuration, and hardware details.
Store a run manifest containing commit ID, dataset hash, seed, package lock, and checkpoint URI.These units describe different parts of a training run and should not be used interchangeably.
| Term | Meaning |
|---|---|
| Batch | A group processed together in one forward/backward computation. |
| Mini-batch | A practical smaller subset used instead of the entire dataset. |
| Training step | One optimizer update, possibly after several accumulated micro-batches. |
| Epoch | One pass through the training set; 1,000 examples with batch size 50 is about 20 batches per epoch. |
Training data updates parameters, validation data guides development choices, and test data estimates final performance. Mixing the same or near-duplicate examples across them can inflate results and hide poor generalization.
An epoch processes the training set once. Without gradient accumulation, the approximate batches per epoch are dataset size divided by batch size. With accumulation and multiple workers, distinguish per-device, effective, and global batch size.
It is a common overfitting signal. Review data overlap and quality, then consider earlier stopping, regularization, augmentation, model capacity, and multiple task metrics rather than one loss curve alone.
No. Small models and experiments can run on CPUs, while larger neural models commonly benefit from GPUs, TPUs, or NPUs. The choice depends on framework support, model size, sequence length, time, and budget.
Public accessibility alone does not establish training rights. Check copyright, license, contract, privacy, consent, confidentiality, collection rules, and provider terms, and retain provenance and removal procedures.
Official glossary for training, validation, test sets, batches, epochs, loss, overfitting, and other machine-learning fundamentals.
Official guide showing tokenization, training arguments, datasets, epochs, batch size, learning rate, evaluation, and checkpointing.
Official description of a training loop covering batching, forward pass, loss, backpropagation, gradients, and parameter updates.
Review model weights, foundation models, inference, tokens, and context windows.
Connect training work to evaluation datasets, benchmarks, rubrics, regression tests, and operational metrics.
Continue from general training concepts to SFT, LoRA, preference optimization, evaluation, and deployment.