Generative AI Engineering
Learn how to structure, ground, compare, and verify AI requests that combine text with images, documents, audio, or video.
17 matching terms
Media inputs
Multimodal prompt
A request that combines text with one or more media types such as images, documents, audio, or video.
Use it when the task depends on information expressed across multiple modalities.
Compare the chart image with the written forecast and identify supported and conflicting claims.Media inputs
Image input
A photograph, screenshot, scan, diagram, or other image supplied for model analysis.
Specify the exact visible property or region the model should inspect.
In the attached screenshot, read only the error banner and list its code and message.Media inputs
Document input
A file such as a PDF, slide deck, or report supplied with both textual and visual structure.
State whether the task must preserve page, section, table, or footnote references.
Extract payment terms from the contract and cite the page and section for each item.Media inputs
Audio input
Speech, music, ambience, or another sound recording supplied for transcription or analysis.
Define the language, speaker-label, timestamp, and uncertainty requirements before transcription.
Transcribe the Korean meeting, label speakers when distinguishable, and timestamp each decision.Media inputs
Video input
A time-based visual input that may also contain speech, text, motion, and scene changes.
Ask for temporal evidence and distinguish what is seen, heard, or inferred.
At each scene change, provide the timestamp, visible action, and spoken claim.Prompt structure
Interleaved input
A sequence where text and media items alternate so instructions or labels stay close to the relevant item.
Use it for comparisons, stepwise demonstrations, or requests involving several media items.
Image A -> its question -> Image B -> its question -> final comparison criteriaPrompt structure
Modality ordering
The order in which text, images, documents, audio, and video parts are presented.
Keep related instructions near their media and test alternative orders for important workflows.
Provide the image, then the extraction instruction, then the output schema.Prompt structure
Image detail and resolution
The amount of visual detail made available for reading small text, objects, or spatial relationships.
Use adequate resolution and crop the relevant area when fine details matter.
Higher resolution can increase processing cost and still cannot recover detail absent from the source image.
Attach a close crop of the serial label instead of a distant photo of the entire device.Evidence grounding
Visual grounding
Linking a statement or extracted result to observable content in an image or video.
Require it when users must verify where a conclusion came from.
For every defect, describe the visible evidence and identify its approximate location.Evidence grounding
Region reference and bounding box
A textual, coordinate-based, or boxed reference to a specific area of an image.
Use regions to disambiguate repeated objects and support downstream review or annotation.
Coordinate formats and accuracy differ across models and APIs; validate them before automation.
Return each detected label with a normalized [x_min, y_min, x_max, y_max] box.Evidence grounding
Temporal reference and timestamp
A time position or interval identifying where an event occurs in audio or video.
Use it for scene analysis, evidence review, editing, and transcript alignment.
List safety incidents as start time, end time, observed action, and confidence.Analysis tasks
OCR instruction
A request to detect and transcribe visible text from an image or scanned document.
Specify languages, layout preservation, illegible-text handling, and whether normalization is allowed.
Transcribe the receipt line by line; use [illegible] instead of guessing missing characters.Analysis tasks
Chart and table interpretation
Reading values, labels, trends, and relationships from visualized or tabular data.
Ask the model to separate directly read values from calculated or inferred conclusions.
Extract the legend and axis units first, then report the largest year-over-year change.Analysis tasks
Multi-image comparison
Analyzing similarities, differences, or changes across two or more images.
Label each image and define stable comparison criteria before requesting conclusions.
Compare Image A and Image B only for layout, visible text, missing controls, and color changes.Analysis tasks
Cross-modal reasoning
Combining evidence from different modalities to answer a question or reach a conclusion.
Use it when no single modality contains the complete answer.
Use the spoken amount, the invoice image, and the policy text to identify any mismatch.Evidence grounding
Multimodal evidence reference
A citation-like pointer to the page, region, frame, or timestamp supporting an answer.
Require references in reviewable outputs and preserve them through later workflow steps.
Claim: total is 48.20; evidence: page 1, lower-right total row; confidence: high.Reliability
Unsupported visual inference
A conclusion that goes beyond what the supplied media visibly or audibly supports.
Instruct the model to mark uncertainty and say when a property cannot be determined.
Do not use appearance alone to infer sensitive traits, identity, intent, or medical conditions.
Report visible damage only; do not infer the cause unless the media directly shows it.| Multimodal prompt engineering | AI media generation prompting |
|---|---|
| Supplies images, documents, audio, or video for understanding and analysis | Describes the image, video, speech, music, or effect to generate |
| Focuses on grounding, regions, timestamps, extraction, comparison, and uncertainty | Focuses on subject, composition, camera, motion, style, sound, and negative constraints |
A text-only request to generate an image is usually a media-generation prompt. It becomes multimodal when the request also supplies an image or another media input for analysis, comparison, editing, or guidance.
Ask the model to distinguish directly observed evidence from calculations or inferences, cite a page, region, frame, or timestamp, and state when the media does not support a conclusion.
Low-risk assistive use may allow automated checks, but consequential decisions should verify source quality, extraction accuracy, unsupported inference, sensitive-trait handling, and model-specific limitations.
Official guidance for combining text with image and video inputs, ordering prompt parts, troubleshooting, and controlling output.
Official overview of prompt content, structure, examples, output formats, iteration, and evaluation.
Review roles, instructions, examples, constraints, and output schemas before adding media inputs.
Budget and assemble large documents, audio, video, history, and supporting evidence.
Use this separate guide when the goal is to create an image rather than analyze one.
Use this separate guide for camera, motion, timing, continuity, and video-generation constraints.