Generative AI Engineering
Understand datasets, criteria, human and model-based judging, regression tests, and efficiency metrics for verifying AI quality.
12 matching terms
Evaluation data
Ground truth
A trusted target answer or label used as a reference for comparison.
Use it for tasks where correct outputs can be defined reliably.
Compare extracted invoice totals with human-verified ground truth values.Evaluation data
Evaluation dataset
A curated set of cases used to measure model or application behavior.
Include normal, difficult, multilingual, and safety-critical cases from real usage.
Create a fixed evaluation dataset of 200 anonymized support questions.Evaluation data
Benchmark
A standardized task or dataset used to compare systems under shared conditions.
Use it for directional comparison, then validate with your own domain cases.
Compare candidate models on a public benchmark and the internal evaluation set.Criteria
Metric
A numerical measure of a selected quality, safety, speed, or cost property.
Choose metrics that map directly to user and business requirements.
Track answer correctness, citation precision, refusal quality, latency, and cost.Criteria
Rubric
A set of written criteria and scoring levels used to judge an output.
Use it when quality requires nuanced judgment rather than exact matching.
Score factual support from 1 to 5, with evidence requirements for each level.Methods
Pointwise evaluation
Scoring one output independently against criteria or a reference.
Use it for pass/fail checks or absolute quality scores.
Rate this answer for correctness and completeness without seeing alternatives.Methods
Pairwise evaluation
Comparing two outputs for the same input and selecting the better one.
Use it when relative preference is easier to judge than an absolute score.
Given responses A and B, choose which better follows the source evidence.Methods
Human evaluation
Assessment performed by people, often using guidelines and labeled examples.
Use trained reviewers for subjective, high-impact, or context-sensitive qualities.
Two domain experts review medical summary faithfulness and resolve disagreements.Methods
LLM-as-a-judge
Using a language model to grade, rank, or critique other model outputs.
Use it for scalable evaluation after calibrating against human judgments.
Judge models can be biased or inconsistent; audit them with representative human-labeled cases.
Ask a judge model to score citation support using a fixed rubric.Operations
Pass rate
The share of evaluation cases that meet a defined acceptance threshold.
Use it to track release gates and quality changes over time.
Require at least a 95% pass rate on critical policy questions before release.Operations
Regression test
A repeated test that checks whether a change breaks behavior that previously worked.
Run it after model, prompt, retrieval, tool, or policy changes.
Rerun the fixed evaluation suite after updating the system instruction.Operations
Efficiency metrics
Measures such as response latency, token use, throughput, and monetary cost.
Evaluate them alongside quality so improvements remain practical at scale.
Compare p95 latency and cost per successful task, not cost per request alone.