Multimodal

AI models that can process and generate content across multiple data modalities — such as text, images, audio, and video — within a single architecture. A multimodal AI model can accept inputs and produce outputs across more than one data modality (text, images, audio, video, code). Multimodal models align representations across modalities so they can be jointly reasoned over.

Examples include GPT-4V (text and images), Gemini 1.5 Pro (text, images, audio, video), and CLIP (connecting text and image representations for zero-shot visual classification).

Multimodal AI is a foundational concept in Model Architecture. AI models that can process and generate content across multiple data modalities — such as text, images, audio, and video — within a single architecture.