Skip to main content

olmOCR

olmOCR is a specialized AI tool that converts PDF documents into clean, structured text while preserving important formatting and layout information. What makes olmOCR particularly valuable for developers is its ability to handle challenging PDFs that traditional OCR tools struggle with—including complex layouts, poor-quality scans, handwritten text, and documents with mixed content types. Built on a fine-tuned 7B vision-language model, olmOCR provides enterprise-grade PDF processing at a fraction of the cost of proprietary solutions.

Possible use cases​

olmOCR's advanced PDF processing capabilities make it ideal for applications that need to extract structured data from documents:

  • Document Digitization: Convert legacy paper documents, scanned files, and image-based PDFs into searchable, editable text
  • Data Pipeline Integration: Process large volumes of PDFs for data analysis, machine learning training, or content management systems
  • Legal and Compliance: Extract text from contracts, legal documents, and regulatory filings for analysis and review
  • Academic Research: Process research papers, journals, and academic documents for literature reviews and citation analysis
  • Enterprise Content Management: Digitize business documents, reports, and archives for searchable knowledge bases
  • Financial Services: Extract data from financial statements, invoices, and regulatory documents
  • Healthcare: Process medical records, research papers, and clinical documentation
  • Publishing and Media: Convert printed materials and legacy documents into digital formats

Performance comparison: olmOCR v0.1.68 scores 77.4% on olmOCR-bench, a challenging benchmark that requires models to OCR difficult math equations, tables, handwriting, and other scanned documents.

Getting started​

Requirements:

  • Recent NVIDIA GPU (tested on RTX 4090, L40S, A100, H100) with at least 20 GB of GPU RAM
  • 30GB of free disk space

Install dependencies (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install poppler-utils ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools

Setting up a conda environment and installing packages from source:

conda create -n olmocr python=3.11
conda activate olmocr

git clone https://github.com/allenai/olmocr.git
cd olmocr

pip install -e .[gpu] --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/

Install sglang with flashinfer if you want to run inference on GPU:

pip install sgl-kernel==0.0.3.post1 --force-reinstall --no-deps
pip install "sglang[all]==0.4.2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/

To run locally, a GPU is required, as inference is powered by sglang:

python -m olmocr.pipeline ./localworkspace --pdfs tests/gnarly_pdfs/horribleocr.pdf
python -m olmocr.pipeline ./localworkspace --pdfs tests/gnarly_pdfs/*.pdf # For multiple pdf's.

Prompting strategy​

The prompt below is used to prompt the fine-tuned model, using the same rule to replace {base_text} with the output of document-anchoring. Refer to the anchor script for more details.

Below is the image of one page of a document, as well as some raw textual content that was
previously extracted for it.
Just return the plain text representation of this document as if you were reading it
naturally .
Do not hallucinate.
RAW_TEXT_START
{base_text}
RAW_TEXT_END

JSON schema​

The following JSON schema is used to define the structure of the outputs.

{
    "type": "json_schema",
    "json_schema": {
        "name": "page_response",
        "schema": {
            "type": "object",
            "properties": {
                "primary_language": {
                    "type": ["string", "null"],
                    "description": "The primary language of the text using two-letter codes or null if there is no text at all that you think you should read.",
                },
                "is_rotation_valid": {
                    "type": "boolean",
                    "description": "Is this page oriented correctly for reading? Answer only considering the textual content, do not factor in the rotation of any charts, tables, drawings, or figures.",
                },
                "rotation_correction": {
                    "type": "integer",
                    "description": "Indicates the degree of clockwise rotation needed if the page is not oriented correctly.",
                    "enum": [0, 90, 180, 270],
                    "default": 0,
                },
                "is_table": {
                    "type": "boolean",
                    "description": "Indicates if the majority of the page content is in tabular format.",
                },
                "is_diagram": {
                    "type": "boolean",
                    "description": "Indicates if the majority of the page content is a visual diagram.",
                },
                "natural_text": {
                    "type": ["string", "null"],
                    "description": "The natural text content extracted from the page.",
                },
            },
            "additionalProperties": False,
            "required": [
                "primary_language",
                "is_rotation_valid",
                "rotation_correction",
                "is_table",
                "is_diagram",
                "natural_text",
            ],
        },
        "strict": True,
    },
}