Multimodal AI Capstone: Document Intelligence System

Build a Multimodal Document Intelligence System

A mid-market manufacturing company receives supplier invoices as scanned PDFs and records supplier calls as MP3 files. Your goal: build a system that extracts invoice line items, transcribes the call, verifies the supplier logo with a vision model, and feeds all three outputs to a reasoning model that flags discrepancies and drafts an approval or rejection message.

This is a real enterprise scenario. The four components you will wire together (document AI, ASR, VLM, and reasoning LLM) are the same stack that invoice automation, procurement intelligence, and accounts payable verification systems use. By the end you will have a working end-to-end pipeline you can extend and adapt to your own context.

Work through each step, run the code in your own environment, and substitute your own test files. The code uses realistic API call patterns rather than toy stubs, so each step is executable with a real API key and a sample document, audio file, or image.

Architecture Overview

Step-by-Step Implementation

Step 1

PDF Extraction with Document AI API

Send the scanned PDF to a document AI extraction API. The goal is structured JSON containing line items: description, quantity, unit price, and total. This step handles both OCR and layout understanding in a single call. The response will be used as context in Step 4.

Python · document AI extraction

import anthropic
import base64
from pathlib import Path

def extract_invoice_line_items(pdf_path: str) -> dict:
    """
    Extract structured line items from a scanned invoice PDF.
    Returns a dict with keys: supplier_name, invoice_number,
    invoice_date, line_items (list of dicts), total_amount.
    """
    client = anthropic.Anthropic()

    # Read and encode the PDF
    pdf_bytes = Path(pdf_path).read_bytes()
    pdf_b64 = base64.standard_b64encode(pdf_bytes).decode("utf-8")

    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=2048,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "document",
                        "source": {
                            "type": "base64",
                            "media_type": "application/pdf",
                            "data": pdf_b64,
                        },
                    },
                    {
                        "type": "text",
                        "text": (
                            "Extract all invoice line items from this document. "
                            "Return a JSON object with these keys: "
                            "supplier_name, invoice_number, invoice_date, "
                            "line_items (array of {description, quantity, "
                            "unit_price, total}), total_amount. "
                            "If a field is not present, use null. "
                            "Return only the JSON object, no commentary."
                        ),
                    },
                ],
            }
        ],
    )

    import json
    text = response.content[0].text.strip()
    # Strip markdown code fences if present
    if text.startswith("```"):
        text = text.split("```")[1]
        if text.startswith("json"):
            text = text[4:]
    return json.loads(text.strip())


# Usage
invoice_data = extract_invoice_line_items("supplier_invoice.pdf")
print(f"Supplier: {invoice_data['supplier_name']}")
print(f"Line items: {len(invoice_data['line_items'])}")

Step 2

Whisper Transcription of Supplier Call

Transcribe the supplier call MP3 using the Whisper API. For files under 25 MB, a single API call is sufficient. For longer recordings, chunk the audio into segments before transcribing. The transcript will be passed to the reasoning LLM in Step 4 to check for verbal commitments that differ from the written invoice.

Python · Whisper transcription

from openai import OpenAI
from pathlib import Path

def transcribe_supplier_call(audio_path: str) -> str:
    """
    Transcribe a supplier call MP3 using Whisper.
    Returns the full transcript as a plain string.
    For files over 25 MB, chunk first (not shown here).
    """
    client = OpenAI()

    with open(audio_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file,
            response_format="text",
            language="en",  # set if known; omit to auto-detect
        )

    return transcript


# Usage
transcript = transcribe_supplier_call("supplier_call.mp3")
print(f"Transcript length: {len(transcript.split())} words")
print(transcript[:300] + "...")  # preview first 300 chars

Step 3

VLM Logo Verification

Crop the invoice header region (or pass the full first page image) to a VLM and ask it to compare the visible logo against a reference description of the known supplier's branding. This step catches forged or substituted invoices where the document content looks plausible but the logo does not match. Pass a text description of the known supplier logo rather than a base64 reference image, which simplifies the prompt while still catching obvious mismatches.

Python · VLM logo verification

import anthropic
import base64
from pathlib import Path

KNOWN_SUPPLIERS = {
    "Acme Components Ltd": (
        "Logo is a bold sans-serif wordmark 'ACME' in dark navy blue "
        "with a small gear icon to the left. No other colors present."
    ),
    "Meridian Fasteners": (
        "Logo is an italic red 'M' monogram with 'Meridian Fasteners' "
        "in grey below it. Font is a rounded sans-serif."
    ),
}

def verify_invoice_logo(
    header_image_path: str,
    expected_supplier: str,
) -> dict:
    """
    Compare the logo visible in the invoice header image against
    the known description for the expected supplier.
    Returns: {match: bool, confidence: str, notes: str}
    """
    client = anthropic.Anthropic()

    image_bytes = Path(header_image_path).read_bytes()
    image_b64 = base64.standard_b64encode(image_bytes).decode("utf-8")
    # Detect media type from extension
    ext = Path(header_image_path).suffix.lower()
    media_type = {"jpg": "image/jpeg", "jpeg": "image/jpeg",
                  "png": "image/png"}.get(ext.lstrip("."), "image/jpeg")

    known_desc = KNOWN_SUPPLIERS.get(expected_supplier, "No known description on file.")

    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=512,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": media_type,
                            "data": image_b64,
                        },
                    },
                    {
                        "type": "text",
                        "text": (
                            f"Expected supplier: {expected_supplier}\n"
                            f"Known logo description: {known_desc}\n\n"
                            "Does the logo visible in this invoice header match "
                            "the known description? Reply with a JSON object: "
                            "{\"match\": true/false, \"confidence\": \"high/medium/low\", "
                            "\"notes\": \"brief explanation\"}. "
                            "Return only the JSON object."
                        ),
                    },
                ],
            }
        ],
    )

    import json
    return json.loads(response.content[0].text.strip())


# Usage
logo_result = verify_invoice_logo(
    "invoice_header.png",
    expected_supplier="Acme Components Ltd",
)
print(logo_result)
# {"match": true, "confidence": "high", "notes": "Navy ACME wordmark with gear icon visible."}

Step 4

Reasoning LLM: Discrepancy Detection and Draft Response

Pass all three outputs (structured invoice data, call transcript, and logo verification result) to a reasoning-capable LLM. Ask it to compare the invoice against the transcript for verbal commitments, flag any discrepancies, assess the logo result, and draft either an approval confirmation or a rejection message for the procurement team to review and send.

Python · reasoning LLM with all three modality outputs

import anthropic
import json

def analyze_and_draft_response(
    invoice_data: dict,
    transcript: str,
    logo_result: dict,
) -> dict:
    """
    Combine document AI extraction, call transcript, and VLM logo
    verification into a final analysis and draft approval or rejection.
    Returns: {recommendation: str, discrepancies: list, draft_message: str}
    """
    client = anthropic.Anthropic()

    prompt = f"""You are a procurement verification assistant. Review the following
three inputs from a supplier invoice processing pipeline and produce a
structured analysis.

--- EXTRACTED INVOICE DATA ---
{json.dumps(invoice_data, indent=2)}

--- SUPPLIER CALL TRANSCRIPT ---
{transcript}

--- LOGO VERIFICATION RESULT ---
{json.dumps(logo_result, indent=2)}

Tasks:
1. Compare the invoice line items and totals against any pricing or
   quantity commitments mentioned in the call transcript.
2. Flag any discrepancies between the written invoice and verbal commitments.
3. Note whether the logo verification passed or failed and at what confidence.
4. Draft either an approval confirmation or a rejection message to the supplier.
   The message should be professional, specific about any issues, and suitable
   for a procurement manager to review and send without rewriting.

Return a JSON object with these keys:
- recommendation: "approve" or "reject"
- discrepancies: array of strings describing each discrepancy found (empty if none)
- logo_status: "verified" or "mismatch" or "low_confidence"
- draft_message: full text of the message to send to the supplier

Return only the JSON object."""

    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}],
    )

    text = response.content[0].text.strip()
    if text.startswith("```"):
        text = text.split("```")[1]
        if text.startswith("json"):
            text = text[4:]
    return json.loads(text.strip())


# Wire all four steps together
def run_invoice_intelligence_pipeline(
    pdf_path: str,
    audio_path: str,
    header_image_path: str,
    expected_supplier: str,
) -> dict:
    print("[1/4] Extracting invoice line items...")
    invoice_data = extract_invoice_line_items(pdf_path)

    print("[2/4] Transcribing supplier call...")
    transcript = transcribe_supplier_call(audio_path)

    print("[3/4] Verifying supplier logo...")
    logo_result = verify_invoice_logo(header_image_path, expected_supplier)

    print("[4/4] Running discrepancy analysis...")
    result = analyze_and_draft_response(invoice_data, transcript, logo_result)

    return {
        "invoice_data": invoice_data,
        "transcript_word_count": len(transcript.split()),
        "logo_result": logo_result,
        "analysis": result,
    }


# Run the full pipeline
output = run_invoice_intelligence_pipeline(
    pdf_path="supplier_invoice.pdf",
    audio_path="supplier_call.mp3",
    header_image_path="invoice_header.png",
    expected_supplier="Acme Components Ltd",
)
print(json.dumps(output["analysis"], indent=2))

Extension Challenges

Once the baseline pipeline is working, these three extensions increase its value and production-readiness. Each is self-contained and can be added independently.

Extension 1

Confidence Scoring

Add a confidence field to each step's output. Propagate a weighted aggregate confidence score to the reasoning step. Route items below a threshold to human review automatically rather than generating a draft message.

Extension 2

Human-in-the-Loop Review

Build a simple review queue: items where the aggregate confidence is below your threshold, or where any discrepancy is flagged, go into the queue. A reviewer sees the draft message and can approve, edit, or reject before it is sent. Log all reviewer decisions for model improvement.

Extension 3

Invoice History Lookup

Embed each processed invoice into a vector store keyed by supplier. Before the reasoning step, retrieve the three most recent invoices from the same supplier. Pass them as context so the LLM can flag price increases, changed line items, or unusual patterns relative to historical invoices from the same supplier.

What you built

A four-component multimodal pipeline covering document AI, ASR, vision, and reasoning LLM
Structured JSON extraction from a scanned PDF invoice using a document AI API
Audio transcription of a supplier call with Whisper, returning plain text suitable for downstream reasoning
Logo verification against a known supplier description using a VLM, with a match/confidence/notes output
A reasoning step that combines all three outputs, detects discrepancies between written and verbal commitments, and drafts an approval or rejection message
A full pipeline runner that sequences all four steps and returns a structured result ready for a review queue or direct use

Claim your Certificate of Completion →