Cross-Modal Retrieval

You have 50,000 product images and 50,000 descriptions. A customer types "red running shoes with white sole" and expects the right image to surface. Or they upload a photo and want visually similar products returned. Cross-modal retrieval makes both searches work off the same index, using a shared embedding space where text and images speak the same geometric language.

By the end of this module you will be able to

Explain how CLIP-style training aligns text and image embeddings into a shared semantic space
Describe the nearest-neighbor retrieval loop for text-to-image and image-to-text queries
Compare late fusion and early fusion strategies for combining modalities at query time
Build a working text-to-image retrieval pipeline using CLIP embeddings and Qdrant
Identify the use cases where cross-modal retrieval outperforms text-only RAG

The Problem: Two Data Types, One Search Box

Traditional retrieval systems pick one modality and stick to it. A keyword search finds documents by matching tokens. A reverse image search finds images by matching pixel signatures. Neither can cross the boundary between a text query and an image result, or between an uploaded photo and a text description.

Cross-modal retrieval collapses that boundary. The insight is simple: if you train a model to map images and text into the same vector space, a text query and a matching image will land near each other regardless of which modality the user started from. The retrieval math stays the same. Only the encoding step changes depending on what the user gave you.

The analogy that makes it click Think of a city where every location is labeled in both English and French. A tourist who asks for "the coffee shop on the corner" and a tourist who says "le cafe au coin" both get pointed to the same building. The shared embedding space is that city. CLIP training teaches the model to put the English sign and the French sign on the same building.

How the Shared Embedding Space Works

CLIP (Contrastive Language-Image Pre-training, Radford et al. 2021) trains two encoders simultaneously: one for text and one for images. The training signal comes from pairs: for a given image, the paired caption should have a high cosine similarity to the image embedding, while captions from other images in the batch should have low similarity.

After training on hundreds of millions of image-caption pairs, the model learns to embed semantically related content near each other regardless of modality. A photo of a golden retriever and the phrase "a golden retriever playing in the grass" end up close in the vector space. A photo of a financial chart and "quarterly revenue trend" end up close. The geometry of meaning becomes modality-independent.

The practical consequence: once you have precomputed image embeddings indexed in a vector store, a text query is just another point in that same space. You encode the query, run a nearest-neighbor search, and return the top-k images whose embeddings are closest to the query embedding. No special multimodal retrieval logic is needed. The embedding space does the work.

The CLIP Retrieval Loop

A production cross-modal retrieval system has two offline steps and one online step.

Offline: index your images. For each image in your catalog, pass it through the CLIP image encoder and store the resulting embedding vector in a vector database, keyed to the image ID or URL. This is a one-time batch job. For a catalog of 50,000 images with a 512-dimensional embedding and CLIP ViT-B/32, this takes minutes on a single GPU.

Offline: optionally index text too. If you also want image-to-text retrieval (a user uploads a photo and gets matching product descriptions), encode your text corpus with the CLIP text encoder and store those embeddings. The same vector store holds both modalities because the embedding dimensions are identical.

Online: encode the query and search. When a user submits a query, encode it with the appropriate encoder (text query: text encoder; image query: image encoder) and run a cosine similarity or approximate nearest-neighbor search against the precomputed index. Return the top-k results. The entire online step is typically under 50ms for million-scale indexes with HNSW indexing.

Multimodal Vector Stores

The good news: cross-modal retrieval does not require a specialized database. Any vector store that supports approximate nearest-neighbor search over floating-point vectors works, because the CLIP embedding space makes both modalities compatible. Qdrant, Weaviate, and Milvus all support storing image and text embeddings in the same collection, with optional payload metadata linking back to source files.

The one practical difference from text-only RAG: your embeddings may come from different encoder sizes across modalities, so pay attention to dimensionality consistency. Both the text and image encoders in a CLIP model family produce vectors of the same dimension (e.g., 512 for ViT-B/32, 768 for ViT-L/14), so as long as you use the same CLIP model for both index-time and query-time encoding, the cosine similarity math is directly comparable.

Late Fusion vs. Early Fusion

When a user submits a query that contains both text and an image ("find products like this photo, but in blue"), you need to decide how to combine the two input signals. Two architectures handle this differently.

Late fusion retrieves separately per modality and then re-ranks the combined result lists. The text query returns a ranked list of candidates. The image query returns a separate ranked list. A re-ranking step (simple score averaging or a learned re-ranker) merges the two lists and returns a unified top-k. Late fusion is easy to implement and modular: you can tune each retrieval branch independently and swap out the re-ranker without touching the encoders.

Early fusion encodes all input modalities together before retrieval. Instead of two separate queries, you produce a single fused embedding that represents the combined intent. Some models, such as ImageBind and more recent multimodal retrieval models, natively support multi-input encoding. Early fusion can produce more coherent results when the two inputs are tightly related, but it requires a model architecture that explicitly supports multi-modal input encoding, not just two separate CLIP encoders.

When to choose each Use late fusion when you are building on CLIP or any pair of separate encoders. Use early fusion only if you have access to a model that natively encodes multiple input modalities simultaneously. For most teams building today, late fusion is the practical choice.

When Cross-Modal Retrieval Beats Text-Only RAG

Cross-modal retrieval is worth the additional complexity in three scenarios where text-only search systematically fails.

Product search with visual attributes. Textual product descriptions are often incomplete or inconsistent. A customer searching for "minimalist white sneaker with a gum sole" will get poor results from keyword matching if the catalog descriptions say "casual lifestyle shoe" or omit the sole color entirely. Cross-modal retrieval finds visually matching items regardless of how the catalog was described, because the image embeddings capture visual attributes that descriptions miss.

Medical imaging combined with clinical notes. Radiologists need to find prior cases whose imaging findings match a current scan. Text-only search retrieves by diagnosis code, which misses similar presentations with different codes. Cross-modal retrieval indexes the images directly, so a query with an example scan returns cases with visually similar findings regardless of how they were labeled in the report. This supports analogical reasoning at scale (Monajatipoor et al. 2022 survey, medRxiv).

E-discovery combining documents and screenshots. Legal review increasingly involves screenshots, PDFs with embedded images, and photos of physical documents alongside text memos. Cross-modal retrieval allows investigators to find document images visually similar to a reference exhibit and text documents semantically related to the same subject in a single unified search, rather than running two disconnected queries.

Interactive 4: Embedding Space Explorer Try it

A 2D projection of a shared embedding space. Blue dots are text queries. Clay dots are image embeddings. Click a text query to animate a nearest-neighbor search across the image embeddings.

Click a blue dot or press the button to select a query

Code: Text-to-Image Retrieval with CLIP and Qdrant

The following Python example indexes a directory of images with CLIP embeddings and queries them with a text string. It assumes Qdrant running locally and transformers installed.

# text_to_image_retrieval.py
# Requirements: transformers, torch, pillow, qdrant-client

from transformers import CLIPProcessor, CLIPModel
import torch
from PIL import Image
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
import pathlib, uuid

# Load CLIP model (ViT-B/32, 512-dim embeddings)
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
model.eval()

def embed_image(img_path: str) -> list[float]:
    img = Image.open(img_path).convert("RGB")
    inputs = processor(images=img, return_tensors="pt")
    with torch.no_grad():
        feats = model.get_image_features(**inputs)
    feats = feats / feats.norm(dim=-1, keepdim=True)
    return feats[0].tolist()

def embed_text(text: str) -> list[float]:
    inputs = processor(text=[text], return_tensors="pt", truncation=True)
    with torch.no_grad():
        feats = model.get_text_features(**inputs)
    feats = feats / feats.norm(dim=-1, keepdim=True)
    return feats[0].tolist()

# Create Qdrant collection (512 dims, cosine similarity)
client = QdrantClient(host="localhost", port=6333)
client.recreate_collection(
    collection_name="product_images",
    vectors_config=VectorParams(size=512, distance=Distance.COSINE),
)

# Index a directory of images
img_dir = pathlib.Path("./product_images")
points = []
for path in img_dir.glob("*.jpg"):
    vec = embed_image(str(path))
    points.append(PointStruct(
        id=str(uuid.uuid4()),
        vector=vec,
        payload={"filename": path.name},
    ))
client.upsert(collection_name="product_images", points=points)
print(f"Indexed {len(points)} images")

# Query with text
query = "red running shoes with white sole"
query_vec = embed_text(query)
results = client.search(
    collection_name="product_images",
    query_vector=query_vec,
    limit=5,
    with_payload=True,
)

for r in results:
    print(f"Score: {r.score:.4f}  File: {r.payload['filename']}")

Think about it first: why do we normalize embeddings before storing them? +

Cosine similarity between two vectors equals their dot product when both are unit vectors (length 1). Normalizing all embeddings to unit length before indexing means that maximum inner product search and cosine similarity search become equivalent, which lets vector databases use highly optimized MIPS (maximum inner product search) indexes. It also removes magnitude as a confounding factor: two vectors pointing in the same direction are considered equally similar regardless of their original magnitudes.

Knowledge Check

What makes it possible to run a text query against an index of image embeddings using cosine similarity?

Late fusion combines query modalities by:

Cross-modal retrieval is most likely to outperform text-only RAG when:

Before you go

CLIP-style training produces a shared embedding space where semantically related text and images land near each other. That shared geometry is what makes cross-modal retrieval possible with standard cosine similarity search.
Late fusion (retrieve per modality, then re-rank) is the practical default for teams building on CLIP or separate encoders. Early fusion requires a model that natively encodes multiple inputs together.
Cross-modal retrieval earns its complexity in product search with visual attributes, medical imaging retrieval, and e-discovery scenarios where text descriptions are incomplete or inconsistent.

Reflection: think of a search or retrieval problem in your organization that currently returns poor results because the relevant information lives in images rather than text. What would it take to build a cross-modal index for that corpus?

Next: Module 5

Multimodal Agents

Agents that can see, read, and act. Module 5 covers how to give an agent tool access to vision, audio, and document understanding so it can reason across modalities and take actions based on what it observes.

Course 13: Multimodal AI

01 Foundations 02 Vision Models 03 Audio and Speech 04 Cross-Modal Retrieval 05 Multimodal Agents 06 Evaluation Capstone

Was this helpful?

← Module 3: Audio and Speech Module 5: Multimodal Agents →