Skip to main content

OlmoE models

OlmoE is a family of mixture-of-experts language models that delivers the performance of much larger models while using significantly fewer computational resources during inference. What makes OlmoE particularly valuable for developers is its intelligent architecture that activates only the most relevant "expert" networks for each input, providing excellent performance-to-cost ratios. This makes it ideal for applications where you need strong AI capabilities but want to minimize infrastructure costs.

Possible use cases​

OlmoE's efficient architecture makes it perfect for cost-conscious applications that still need strong performance:

  • Real-Time Systems: Build responsive applications where inference speed and efficiency are critical
  • Edge Deployment: Deploy powerful language models on resource-constrained devices
  • Startup Applications: Get strong AI capabilities without the infrastructure costs of larger models
  • Development and Prototyping: Experiment with advanced AI features without expensive infrastructure

Performance comparison: OlmoE-1B-7B surpasses the performance of GPT-3.5 Turbo and LLaMA-2-13B, but is faster and incurs only a faction of the inference costs.

Model evolution​

The OlmoE 1B-7B model has shown significant improvements from its September 2024 release to the January 2025 version:

  • Performance Improvements: Notable gains on challenging tasks like ARC-Challenge (62.1% → 67.5%) and MMLU (54.1% → 56.3%)
  • Instruction Tuning: Enhanced instruction-following capabilities through improved RLVR training
  • Benchmark Performance: Consistent improvements across all evaluation metrics
  • Inference Efficiency: Maintained efficient inference while improving quality

Common features​

OlmoE shares many architectural and training features with the Olmo family. For common features like SwiGLU activation functions, rotary positional embeddings, training methodology, data sources, licensing, and openness, please refer to the Olmo documentation.

OlmoE-specific features​

Architecture​

  • Mixture-of-Experts (MoE) transformer architecture
    • Dynamic expert routing with 64 total experts
    • Only 8 experts active at a time
    • Efficient parameter utilization (1.3B active parameters out of 6.9B total)
  • Consistent context window of 2048 tokens

Training methodology​

  • Expert balancing and load balancing
    • Ensures even utilization of expert networks
    • Prevents expert collapse
    • Optimizes routing decisions
  • Two-stage pretraining process
  • Instruction tuning with RLVR
  • Consistent training stability across all model sizes

Performance characteristics​

  • More efficient inference than dense models of similar total parameter count
  • Specialized expert networks for different types of tasks
  • Dynamic adaptation to input characteristics

Instruct models​

OlmoE-Instruct models are specialized, fine-tuned versions of the OlmoE 1B-7B base model, specifically optimized for instruction-following tasks. They are trained on permissive datasets and utilize a multi-stage reinforcement learning approach with verifiable rewards (RLVR). These models offer improved alignment, usability, and performance across a broad range of user requests. Additionally, we've made available the SFT and DPO models. The core models released in this batch include the following:

Inference code example​

You can use OlmoE with the Hugging Face transformers library. Here's an example using the latest model:

from transformers import OlmoeForCausalLM, AutoTokenizer
import torch

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

model = OlmoeForCausalLM.from_pretrained("allenai/OLMoE-1B-7B-0125").to(DEVICE)
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMoE-1B-7B-0125")

inputs = tokenizer("Bitcoin is", return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}

out = model.generate(**inputs, max_length=64)
print(tokenizer.decode(out[0]))
# >> 'Bitcoin is a digital currency that is created and held electronically. No one controls it. Bitcoins aren't printed, like dollars or euros – they're produced by people and businesses running computers all around the world, using software that solves mathematical'

For the September 2024 model, you can use the same code with the model ID allenai/OLMoE-1B-7B-0924. You can also load different checkpoints via the revision parameter, e.g., revision=step10000-tokens41B.

Visit our Olmo suite to find all model variants on Hugging Face.