Generative AI Engineering
Learn how to build prompt test cases, compare controlled variants, analyze failures, prevent regressions, and balance quality, latency, and cost.
16 matching terms
Test data
Prompt test case
A defined input, context, expected behavior, and evaluation rule used to check a prompt.
Create cases for normal requests, edge conditions, ambiguous inputs, and expected refusals.
Input: missing account ID; expected behavior: request the ID without inventing one.Test data
Golden dataset
A reviewed collection of representative test inputs and trusted expected outcomes or scoring guidance.
Use it to compare prompt changes consistently across releases.
Refresh the dataset as real usage changes, while preserving a stable regression subset.
Keep 200 reviewed support cases with expected category, required facts, and prohibited claims.Experiments
Baseline prompt
The current or simplest prompt used as the reference point for comparison.
Record it before optimization so claimed improvements are measurable.
Baseline v12 uses the production model, default parameters, and unchanged retrieval settings.Experiments
Prompt variant
An alternative prompt wording, structure, example set, model, or inference configuration tested against a baseline.
Name and version variants so results can be reproduced and promoted safely.
Variant B moves constraints after the source and adds one negative example.Experiments
Controlled experiment
A comparison that changes one intended factor while holding relevant inputs and settings constant.
Use it to attribute an observed difference to the prompt change rather than the model, data, or sampling.
Change only the instruction wording; keep the dataset, model version, temperature, and evaluator fixed.Experiments
A/B test
An experiment comparing two prompt or configuration versions on comparable traffic or test samples.
Use it after offline checks when production behavior and user outcomes must be compared.
Protect users with eligibility rules, monitoring, and stop conditions; do not expose high-risk workflows without review.
Randomly route eligible requests to A or B and compare task success, corrections, latency, and cost.Experiments
Prompt ablation
Removing or replacing one prompt component to measure its contribution.
Use it to identify unnecessary instructions, examples, or context that add cost without improving quality.
Remove the third example while keeping every other component unchanged.Evaluation
Batch evaluation
Running a prompt over many prepared cases and aggregating per-case and overall results.
Use it before manual spot checks are mistaken for representative performance.
Run all prompt variants on the fixed evaluation set and export per-case failures.Evaluation
Prompt regression
A previously working behavior that becomes worse after a prompt, model, retrieval, tool, or policy change.
Track it with a stable suite and compare results before every release.
The new concise format improved length but stopped including required source IDs.Reliability
Robustness test
A test of whether acceptable behavior holds across realistic variation, noise, languages, and edge cases.
Use paraphrases, reordered context, typos, missing fields, and adversarially confusing inputs.
Test the same intent with formal, conversational, misspelled, and multilingual wording.Reliability
Sensitivity test
A test measuring how strongly outputs change when a small input or configuration detail changes.
Use it for wording, example order, temperature, context placement, and model-version changes.
Move the same evidence from the beginning to the middle and compare citation accuracy.Reliability
Failure taxonomy
A consistent set of categories used to label and analyze prompt failures.
Define it before large evaluations so teams can prioritize causes rather than only count failures.
Labels: missing fact, unsupported claim, schema error, wrong tool, unsafe disclosure, excessive latency.Evaluation
Acceptance threshold
A predeclared score or rule that a prompt version must meet before release.
Set thresholds by risk and include must-not-fail cases rather than relying only on averages.
Release only if required-field accuracy is at least 98% and no critical privacy case fails.Evaluation
Model and parameter matrix
A planned grid comparing prompts across model versions and inference settings.
Use it when a prompt may behave differently after model migration or parameter changes.
Evaluate prompt v8 on model A and B at two supported temperature settings.Efficiency
Quality, latency, and cost comparison
A joint view of task quality, response time, and resource use for each candidate.
Use it to avoid selecting a slightly better prompt that is too slow or expensive for its workload.
Compare pass rate, p95 latency, input tokens, output tokens, and estimated cost per successful task.Efficiency
Prompt optimization loop
An iterative cycle of defining goals, testing a baseline, analyzing failures, changing the prompt, and retesting.
Use small evidence-based changes and keep versioned results at every iteration.
Define metric -> run baseline -> label failures -> change one component -> rerun -> review regressions.| Step | Required evidence |
|---|---|
| Define the target | User outcome, quality criteria, risk, latency, and cost limits |
| Establish the baseline | Versioned prompt, model, settings, dataset, and per-case results |
| Test a change | Controlled variant, repeated runs where needed, and failure labels |
| Release or reject | Acceptance thresholds, regression review, human approval, and rollback plan |
A few examples may miss common failures, edge cases, variation, and regressions. Use representative cases, stable scoring rules, per-case review, and batch results before drawing conclusions.
No. Select metrics that reflect the task, including required facts, groundedness, format validity, safety, robustness, latency, token use, cost, and human correction effort.
For causal diagnosis, change one main factor at a time. Broader candidate comparisons can change several components, but they show which complete candidate wins rather than which component caused the difference.
Official guidance describing prompt engineering as a test-driven, iterative workflow with defined objectives and systematic evaluation.
Official guide to testing controlled prompt variants on representative data and comparing measurable results.
Start with explicit instructions, examples, constraints, output schemas, and decomposed workflows.
Connect prompt-specific experiments to broader system quality, human review, benchmarks, and operational metrics.
Version tested candidates and promote only the approved immutable revision.
Include prompt injection, privacy, harmful content, fairness, and red-team cases in the test strategy.