Skip to main content

Olmo Hybrid

Olmo Hybrid is a 7B base model whose architecture closely matches the Olmo 3 7B base model, except that it is a hybrid model that mixes attention with linear RNN layers. Olmo Hybrid achieves impressive pretraining performance, generally outperforming Olmo 3 across pretraining and mid-training evaluations. In particular, it is exceptional on long-context extrapolation, as measured by performance on the RULER benchmark.

Possible use cases​

Due to its close comparability to Olmo 3, Olmo Hybrid is primarily meant as a strong proof of concept for hybrid models vs. full transformers. We anticipate the Olmo Hybrid recipe will guide our future pretraining recipes, and the artifact could serve as a useful starting point for post-training research.

Performance: At the time of writing, it is our strongest 7B base model and is particular on RULER, a benchmark measuring performance over long input contexts.

Architecture​

Olmo Hybrid follows a similar architecture Olmo 3 except that we replace the sliding-window attention layers from Olmo 3 with linear RNN layers. In these layers, all heads are computed by Gated DeltaNet (GDN) with negative eigenvalues, a particular type of linear RNN, rather than attention. The hybridization is 3:1, i.e., we alternate 3 GDN layers with 1 layer of attention.

Artifacts​

We release the following base model artifacts for Olmo Hybrid:

  1. Pretrained checkpoint
  2. Mid-trained checkpoint
  3. Checkpoint after LC extension

Additionally, we release the following post-trained checkpoints:

  1. SFT checkpoint
  2. DPO checkpoint

All intermediate checkpoints are also available for each stage of training listed above.

Inference code example​

You can use Olmo Hybrid with the Hugging Face transformers library. Here's an example using the latest model:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

olmo = AutoModelForCausalLM.from_pretrained("allenai/Olmo-Hybrid-7B", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("allenai/Olmo-Hybrid-7B")

message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}

response = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.7, temperature=1.0)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
>> 'Language modeling is  a key component of any text-based application, but its effectiveness...'

Visit our Olmo suite to find all model variants on Hugging Face.