MolmoAct

MolmoAct is a fully open-source action reasoning model for robotic manipulation developed by the Allen Institute for AI. It has state-of-the-art performance among vision-language-action models on multiple benchmarks while being fully open-source. You can find all models in the MolmoAct family here. Learn more about MolmoAct in our announcement blog post or the paper.

Possible Use Cases

MolmoAct’s vision-language-action capabilities make it ideal for applications where systems must see, reason, and act:

General-purpose Manipulation: Execute multi-step tasks using camera observations and natural-language instructions.
Home & Service Robotics: Power assistive robots that can fetch objects, open drawers, or tidy spaces while grounding instructions in 3D scenes.
Industrial & Warehouse Automation: Control arms and grippers for picking, packing, sorting, and simple assembly, adapting to clutter, novel object poses, and changing layouts.
Embodied Evaluation of VLMs: Use MolmoAct as a testbed to study how vision-language understanding translates into concrete behavior, closing the loop between perception, reasoning, and control.
Robotics Research & Benchmarking: Serve as a unified VLA backbone for comparing datasets, training recipes, and task suites, and for studying generalization across robots, scenes, and tasks.
Safety-critical Robotics: Use MolmoAct’s visual reasoning traces and bounded action space to audit planned behavior, reduce unintended motions, and prototype safer deployment strategies for real-world robots.

Model Variants

MolmoAct offers all checkpoints needed to replicate everything:

MolmoAct-7B-D: Our best and demo version of MolmoAct that can be adapted to real world by fine-tuning on your own datasets
MolmoAct-7B-O: Our most open version of MolmoAct that can be adapt to real world by fine-tuneing on your own datasets
MolmoAct-7B-D-Pretrain: MolmoAct checkpoint used to replicate zero-shot results on SimplerEnv (Google Robot)
MolmoAct-7B-D-Pretrain-RT-1: MolmoAct checkpoint used to replicate fine-tuned results on SimplerEnv (Google Robot)
MolmoAct-7B-D-Captioner: MolmoAct checkpoint initialized from Molmo Captioner used to train MolmoAct from scratch
MolmoAct-7B-D-LIBERO-Spatial: MolmoAct checkpoint used to replicate results on LIBERO-Spatial
MolmoAct-7B-D-LIBERO-Object: MolmoAct checkpoint used to replicate results on LIBERO-Object
MolmoAct-7B-D-LIBERO-Goal: MolmoAct checkpoint used to replicate results on LIBERO-Goal
MolmoAct-7B-D-LIBERO-Long: MolmoAct checkpoint used to replicate results on LIBERO-Long

Inference Code Example

You can use MolmoAct (MolmoAct-7B-D as an example) with Hugging Face transformers library (v4.52):

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch
from PIL import Image
import requests
from io import BytesIO

ckpt = "allenai/MolmoAct-7B-D-0812"

# load the processor
processor = AutoProcessor.from_pretrained(
    ckpt,
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
    padding_side="left",
)

# load the model
model = AutoModelForImageTextToText.from_pretrained(
    ckpt,
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)

# task instruction
instruction = "close the box"

# strictly follow this reasoning prompt
prompt = (
    f"The task is {instruction}. "
    "What is the action that the robot should take. "
    f"To figure out the action that the robot should take to {instruction}, "
    "let's think through it step by step. "
    "First, what is the depth map for the first image? "
    "Second, what is the trajectory of the end effector in the first image? "
    "Based on the depth map of the first image and the trajectory of the end effector in the first image, "
    "along with other images from different camera views as additional information, "
    "what is the action that the robot should take?"
)

# apply chat template
text = processor.apply_chat_template(
    [
        {
            "role": "user",
            "content": [dict(type="text", text=prompt)]
        }
    ], 
    tokenize=False, 
    add_generation_prompt=True,
)

# image observation (side + wrist)
url1 = "https://huggingface.co/allenai/MolmoAct-7B-D-0812/resolve/main/example_1.png"
url2 = "https://huggingface.co/allenai/MolmoAct-7B-D-0812/resolve/main/example_2.png"
r1 = requests.get(url1, headers={"User-Agent": "python-requests"}, timeout=30)
r1.raise_for_status()
r2 = requests.get(url2, headers={"User-Agent": "python-requests"}, timeout=30)
r2.raise_for_status()
img1 = Image.open(BytesIO(r1.content)).convert("RGB")
img2 = Image.open(BytesIO(r2.content)).convert("RGB")
imgs = [img1, img2]

# process the image and text
inputs = processor(
    images=[imgs],
    text=text,
    padding=True,
    return_tensors="pt",
)

# move inputs to the correct device
inputs = {k: v.to(model.device) for k, v in inputs.items()}

# generate output
with torch.inference_mode():
    with torch.autocast("cuda", enabled=True, dtype=torch.bfloat16):
        generated_ids = model.generate(**inputs, max_new_tokens=256)

# only get generated tokens; decode them to text
generated_tokens = generated_ids[:, inputs['input_ids'].size(1):]
generated_text = processor.batch_decode(generated_tokens, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

# print the generated text
print(f"generated text: {generated_text}")

# >>>  The depth map of the first image is ... The trajectory of the end effector in the first image is ...
#      Based on these information, along with other images from different camera views as additional information,
#      the action that the robot should take is ...

# parse out all depth perception tokens
depth = model.parse_depth(generated_text)
print(f"generated depth perception tokens: {depth}")

# >>>  [ "<DEPTH_START><DEPTH_1><DEPTH_2>...<DEPTH_END>" ]

# parse out all visual reasoning traces
trace = model.parse_trace(generated_text)
print(f"generated visual reasoning trace: {trace}")

# >>>  [ [[242, 115], [140, 77], [94, 58], [140, 44], [153, 26]]] ]

# parse out all actions, unnormalizing with key of "molmoact"
action = model.parse_action(generated_text, unnorm_key="molmoact")
print(f"generated action: {action}")

# >>>  [ [0.0732076061122558, 0.08228153779226191, -0.027760173818644346, 
#         0.15932856272248652, -0.09686601126895233, 0.043916773912953344, 
#         0.996078431372549] ]

Training Data

MolmoAct is trained on a subset of OXE and MolmoAct Dataset, a dataset with 10k high-quality trajectories of a single-arm Franka robot performing 93 unique manipulation tasks in both home and tabletop environments.

vLLM

MolmoAct is supported by vLLM. Please use vLLM version 0.8.5 to serve MolmoAct.

Possible Use Cases​

Model Variants​

Inference Code Example​

Training Data​

vLLM​

Possible Use Cases

Model Variants

Inference Code Example

Training Data

vLLM