Running locally
This guide provides step-by-step instructions for running Ai2 models locally. It is aimed at software developers familiar with Python, but not necessarily experienced with machine learning or large language models.
Prerequisites
Before running any models, ensure your system meets the following requirements:
Minimum Hardware Requirements
While most models will run well on an 80G GPU (e.g. NVIDIA A100 or H100), there's a wide range of requirements across the various models as demonstrated by the following minimum requirements.
- Small, quantized models: Raspberry Pi 4 (8GB RAM) to run Olmo-2-0425-1B-Instruct-GGUF
- Large models: 16 80 GB GPUs (e.g., NVIDIA A100 or H100) to run Llama-3.1-Tülu-3-405B-DPO
Software Requirements
- macOS
- Windows
- Linux
macOS Requirements
- Python 3.8 or newer (install via python.org or Homebrew)
pip(included with Python)- Virtual environment tool:
python -m venv(built-in)
For detailed Python installation instructions, see the macOS-specific Python installation guide.
Windows Requirements
- Python 3.8 or newer (install via python.org or Microsoft Store)
pip(included with Python)- Virtual environment tool:
python -m venv(built-in)
For detailed Python installation instructions, see the windows-specific Python installation guide.
Linux Requirements
- Python 3.8 or newer (install via package manager)
pip(install viasudo apt install python3-pipor equivalent)- Virtual environment tool:
python -m venv(install viasudo apt install python3-venvor equivalent)
For detailed Python installation instructions, see the Linux/Unix-specific Python installation guide.
Hello, world
Below is a minimal example that loads a model and generates a short text completion.
- Olmo 2
- Tülu 3
- Molmo
- OlmoE
- Tülu
Install packages
pip install torch transformers
Code example
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_id = "allenai/OLMo-2-0425-1B"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Install packages
pip install torch transformers
Code example
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("allenai/Llama-3.1-Tulu-3-8B")
model = AutoModelForCausalLM.from_pretrained("allenai/Llama-3.1-Tulu-3-8B")
input_text = """<|user|>
How are you doing?
<|assistant|>
"""
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=50,
temperature=0.7, top_p=0.9, do_sample=True)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
# >> As an AI, I don't have feelings, but I'm functioning normally. How can I assist you today?
Install packages
pip install torch transformers pillow requests einops tensorflow torchvision accelerate
Code example
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from PIL import Image
import requests
processor = AutoProcessor.from_pretrained(
'allenai/Molmo-7B-D-0924',
trust_remote_code=True,
torch_dtype='auto',
device_map='auto'
)
model = AutoModelForCausalLM.from_pretrained(
'allenai/Molmo-7B-D-0924',
trust_remote_code=True,
torch_dtype='auto',
device_map='auto'
)
inputs = processor.process(
images=[Image.open(requests.get("https://picsum.photos/id/237/536/354", stream=True).raw)],
text="Describe this image."
)
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}
output = model.generate_from_batch(
inputs,
GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
tokenizer=processor.tokenizer
)
generated_tokens = output[0,inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
print(generated_text)
# >>> This image features an adorable black Labrador puppy, captured from a top-down perspective. The puppy is sitting on a wooden deck, which is composed ...
Install packages
pip install torch transformers
Code example
from transformers import OlmoeForCausalLM, AutoTokenizer
import torch
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model = OlmoeForCausalLM.from_pretrained("allenai/OLMoE-1B-7B-0125").to(DEVICE)
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMoE-1B-7B-0125")
inputs = tokenizer("Bitcoin is", return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
out = model.generate(**inputs, max_length=64)
print(tokenizer.decode(out[0]))
# >> 'Bitcoin is a digital currency that is created and held electronically. No one controls it. Bitcoins aren't printed, like dollars or euros – they're produced by people and businesses running computers all around the world, using software that solves mathematical'
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("allenai/Llama-3.1-Tulu-3-8B")
model = AutoModelForCausalLM.from_pretrained("allenai/Llama-3.1-Tulu-3-8B")
input_text = "<|user|>\nHow are you doing?\n<|assistant|>\n"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(inputs["input_ids"], max_new_tokens=50,
temperature=0.7, top_p=0.9, do_sample=True)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
# >> As an AI, I don't have feelings, but I'm functioning normally. How can I assist you today?
GPU acceleration with vLLM
vLLM is a high-performance inference engine that significantly improves the runtime performance of Hugging Face models through several key optimizations:
- Continuous Batching: Batches requests dynamically, maximizing GPU utilization
- PagedAttention: Manages GPU memory via paging like virtual memory
- Kernel Fusion: Combines operations into single kernels to reduce bandwidth
- Quantization Support: Enables 4-bit and 8-bit quantization for memory savings
- Tensor Parallelism: Distributes layers across GPUs for faster inference
Follow these three steps to get it running with any Ai2 model (e.g., Olmo):
-
Install
pip install vllm -
Launch an OpenAI‑compatible server
vllm serve allenai/OLMo-2-0425-1B --served-model-name olmo
# Server now listens on http://localhost:8000/v1 -
Query the model
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \