Why OCR Alone Fails
Optical character recognition solves one problem well: it converts pixels to characters. It has no model of what those characters mean or how they relate to each other spatially. Feed a scanned invoice through a standard OCR engine and you get a flat string. The line items that were arranged in columns become a single run of text. A table with six columns and forty rows becomes a string that requires extensive parsing to recover any structure at all.
Three failure modes appear consistently across document types:
Tables collapse to flat strings. OCR reads left to right, row by row, with no understanding that cells belong to a column or that a row is a logical unit. A financial statement with subtotals and spanning headers becomes meaningless when linearized this way.
Multi-column forms become gibberish. A two-column form field layout is read across both columns simultaneously, interleaving labels with values and destroying the field-value relationship that makes the form useful.
Regions have no semantic identity. OCR cannot distinguish a document header from a footer, a table from a paragraph, or a signature block from a line item. Downstream systems receive undifferentiated text and must re-infer structure that was present in the original document.
The Four Pipeline Stages
Modern document AI pipelines add three stages before OCR and one stage after it, each addressing a different structural gap.
Three Model Families and When to Use Each
LayoutLM family (Microsoft). LayoutLM models are transformer architectures pretrained on document images with the addition of 2D position embeddings. Unlike standard language models that process text in sequence, LayoutLM conditions on both the token content and its spatial position on the page. This lets the model learn that a number appearing directly below a column header labeled "Unit Price" is a price, not just a number. The family has evolved through several versions, each incorporating more visual signal from the document image. LayoutLM-class models are the standard approach for structured forms and invoices where the field layout is consistent across documents.
Donut (Document understanding transformer). Donut takes a different architectural stance: it treats document understanding as an end-to-end vision task without an explicit OCR step. The model reads the document image directly and produces structured output through a sequence decoder, learning to extract fields without an intermediate text representation. This makes Donut faster and less dependent on OCR quality, which matters when input documents are low-resolution or degraded. The tradeoff is that end-to-end models require more training data to match the precision of pipeline approaches on complex layouts.
Vision-language models as document readers. General-purpose vision-language models trained on broad web data have developed substantial ability to read and reason about documents. Given a high-resolution document image, a capable VLM can extract specific fields, describe table contents, and answer questions about what it sees. The advantage is flexibility: the same model handles invoices, contracts, slide decks, and forms without task-specific training. The limitation is that VLMs are less precise than fine-tuned document models on consistent structured layouts, and their output requires more careful post-processing to enforce schema compliance.
Choosing the Right Approach
The choice of approach depends on three variables: document structure consistency, input quality, and required output precision.
Simple single-column text documents where layout is minimal and the content is flowing prose: standard OCR with post-processing is sufficient. Legal agreements with no tables, plain memos, and single-column reports fall here. Adding a full document AI pipeline adds complexity without adding value.
Structured forms and invoices where the same fields appear at consistent positions across a document class: LayoutLM-class models are the appropriate choice. The spatial pretraining directly addresses the problem, and fine-tuning on a document-type-specific dataset produces high precision. This is the largest category of enterprise document AI use cases: invoices, purchase orders, insurance claims forms, tax documents.
Complex mixed-content documents where layout varies across instances, content mixes tables, figures, and prose, and the extraction task requires reasoning rather than field extraction: a VLM-as-reader approach with careful prompting and output validation is the right architecture. Research papers, annual reports, and regulatory filings with heterogeneous structure fall here.