Skip to main content

Molmo2

Molmo2 is a family of vision-language models that understand and analyze images and videos. These models can respond to natural language questions about images and videos, generate detailed captions for them, and ground natural language queries in them via pointing and tracking. What makes Molmo particularly valuable for developers and researchers is the fully open pipeline in terms of data, model weights, and training and evaluation code. Unlike many vision language models that rely on synthetic or proprietary or distilled training data from closed source models, Molmo was built using only real-world, openly available datasets that you can access and verify. To get familiar with Molmo's capabilities, try it out on the Ai2 Playground.

Possible use cases​

Molmo's vision-language capabilities make it ideal for applications that need to understand images and videos:

  • Content Moderation: Automatically analyze images and videos for policy violations or inappropriate content
  • E-commerce: Generate product descriptions, categorize items, and answer customer questions about product images and videos
  • Accessibility Tools: Create alt-text for images and videos, describe visual content for visually impaired users
  • Document Processing: Extract information from charts, graphs, tables, and other visual documents
  • Pointing: Molmo2 can be used for grounding in videos via pointing to actions and objects
  • Tracking: Track objects and object parts in a video useful for various scientific use-cases such as tracking wild-life
  • Quality Control: Inspect products, identify defects, and ensure manufacturing standards
  • Education: Build interactive learning tools that can explain diagrams, charts, and visual concepts
  • Robotics: Enable robots to understand and navigate visual environments via the pointing capability
  • Detecting AI generated artifacts: Molmo2 can point to artifacts in AI generated videos. This could help detect AI generated content or possibly be used as a reward model to improve video generation models

Model variants​

Molmo2 family offers three different models:

  • Molmo2-4B: Our most efficient, performant for most vision applications, especially tasks that require processing long videos with efficient resource usage
  • Molmo2-8B: Strong performance for most vision applications, especially short video tasks, video captioning, and tasks that require processing images
  • Molmo2-O-7B: Our model trained with the Olmo3 7B backbone with open data.

Inference code example​

You can use Molmo with Hugging Face transformers library:

Quick Start​

Setup with Conda​

conda create --name transformers4571 python=3.11
conda activate transformers4571
pip install transformers==4.57.1
pip install torch pillow einops torchvision accelerate decord2 molmo_utils

General Video QA​

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

# load the processor
processor = AutoProcessor.from_pretrained(
    "allenai/Molmo2-8B",
    trust_remote_code=True,
    dtype="auto",
    device_map="auto"
)

# load the model
model = AutoModelForImageTextToText.from_pretrained(
     "allenai/Molmo2-8B",
    trust_remote_code=True,
    dtype="auto",
    device_map="auto"
)

# process the video and text
messages = [
    {
        "role": "user",
        "content": [
            dict(type="text", text="Which animal appears in the video?"),
            dict(type="video", video="https://storage.googleapis.com/oe-training-public/demo_videos/many_penguins.mp4"),
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)

inputs = {k: v.to(model.device) for k, v in inputs.items()}

# generate output
with torch.inference_mode():
    generated_ids = model.generate(**inputs, max_new_tokens=2048)

# only get generated tokens; decode them to text
generated_tokens = generated_ids[0, inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)

# print the generated text
print(generated_text)

# >>>  Penguins appear in the video.

Training Pipeline​

Our training pipeline consists of three stages:

  1. Pretraining: We pretrain on dense captions with length conditioning and transcript prediction using PixMo-Cap following Molmo1. We add NLP data using the supervised fine-tuning data from Tulu, filtered to remove non-English content and code, to better preserve language capabilities.
  2. Supervised fine-tuning: Our data mixture combines PixMo, the Molmo2 datasets, Tulu, and other open-source video and image datasets. We divide these datasets into categories and manually assign each category a sampling rate based on empirical tests. We train for 25k steps with a batch size of 128 and a max sequence length of 16,384.
  3. Long-context supervised fine-tuning: Finally we do a third stage of training with a longer context length. uring this stage we increase the sequence length to 36,864, set F = 384, train for 2k steps, and use context parallelism (CP) on the LLM so each example is processed by a group of 8 GPUs. We also distribute video frame processing by the vision encoder and the attentional pooling after that across each context parallel group and find it effective in reducing the memory footprint of the model.

Training data​

Molmo2 is fine-tuned on both Pixmo(https://huggingface.co/collections/allenai/pixmo) and the Molmo2 datasets (https://huggingface.co/collections/allenai/molmo2-data), which built from a mix of real-world and synthetic openly available sources. These datasets cover diverse visual tasks including image/video captioning, visual question answering, image/video pointing, document understanding, and specialized domains like clock reading and video tracking.

vLLM​

Here is an example shell script for installing packages and serving Molmo2 with vLLM:

# Install vLLM nightly (Molmo2 support is not yet included in the latest stable release)
pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
pip install -U "huggingface_hub[hf_transfer]"

# Use Molmo2 video loader backend
export VLLM_VIDEO_LOADER_BACKEND=molmo2

# vLLM's default `max-num-batched-tokens` is smaller than Molmo2's context length.
# We set it explicitly to avoid runtime errors.
# To enable multi-image inputs in a single request, pass `limit-mm-per-prompt`.
vllm serve allenai/Molmo2-8B --dtype bfloat16 --max-num-batched-tokens 36864 --trust-remote-code --limit-mm-per-prompt '{"image": 6, "video": 1}' --media-io=kwargs '{"video": {"num_frames": 384, "frame_sample_mode": "uniform_last_frame"}}'

For video tracking, we recommend increasing max_fps to 8 (default is 2) to capture fast-moving objects and temporal details more accurately: --media-io=kwargs '{"video": {"num_frames": 384, "frame_sample_mode": "uniform_last_frame", "max_fps": 8}}.