Skip to main content

Training models

Training, and why it matters​

Large Language Models (LLMs) are developed in two main training stages:

  • Pretraining is the foundational phase, where the model learns general language patterns by predicting the next token in a sentence. This is done on massive, diverse datasets that include books, websites, code, and other text sources. The model doesn't learn specific tasks, it learns to model language itself.

  • Fine-tuning is an optional second phase where the model is adapted to specific tasks, domains, or behavioral expectations. This is typically done on much smaller, curated datasets, such as examples of customer support interactions, legal document summaries, or math problem solutions.

Why this matters to software developers​

Understanding the training process helps you reason about:

  • What the model "knows" and how it behaves by default
  • Why it might struggle with your specific use case
  • When it's appropriate to fine-tune, and when it's not

LLMs are not trained to know everything or do every task. They are trained to model statistical patterns in language, and later fine-tuned (if needed) to act in specific ways.

Start with a model that's already close to your goal​

As an application developer, you almost certainly do not need to pretrain a model from scratch, as training state-of-the-art models requires significant computational resources (even the smallest Olmo 1B model requires 5,000 H100 hours per 1 trillion tokens). Instead:

  • Choose a pretrained model that already performs well on your type of task (e.g., chat, summarization, instruction-following, domain-specific language).
  • Use prompt engineering or retrieval-augmented generation (RAG) to get solid results without modifying the model at all.
  • If needed, apply fine-tuning as a targeted refinement, for example, to adapt tone, correct specific mistakes, or teach new behavior.

Key principle: Pretraining is foundational, but not something most developers need to do. Start with a strong base model, and use the lightest technique that meets your needs.

Training your own models​

While most software developers will not need to pretrain their own models, Ai2 makes the process fully transparent and reproducible! This has obvious benefits for research, but also gives software developers a way to understand and evaluate models on a deeper level. For more information about adapting models to your specific needs, see our Fine-Tuning guide.

The following sections provide code examples for training different Ai2 models. Remember that training large language models requires significant computational resources and expertise. Make sure you have access to appropriate hardware (typically multiple high-end GPUs or TPUs) and understand the training process before proceeding. You may also need job management software that will help your training job resume after hardware failures, which are expected when running at scale.

  1. Create a new Python environment: This step will create an isolated Python environment using venv, which helps manage dependencies and avoid conflicts between different projects. You can create a new Python environment using venv:

    python -m venv olmo_core
    source olmo_core/bin/activate  # On Unix/macOS
    # or
    .\olmo_core\Scripts\activate  # On Windows

  2. Install OLMo-core: This step will install the OLMo-core package, which provides the necessary components for training Olmo 3 models. We recommend installing from source:

    git clone https://github.com/allenai/OLMo-core.git
    cd OLMo-core
    pip install -e ."[all]"

    Alternatively, you can install from PyPI with pip install ai2-olmo-core. You will also need to install flash-attn.

  3. Train the model: This step will start the training process for the Olmo 3 7B model. The training script uses distributed training across multiple GPUs for efficient processing:

    torchrun --nproc-per-node=gpu src/scripts/official/OLMo3/OLMo-3-1025-7B-pretrain-1.py \
      --save-folder=models/tutorial-run-01 \
      --train_module.rank_microbatch_size=8192

    For the 32B model, use the corresponding script:

    torchrun --nproc-per-node=gpu src/scripts/official/OLMo3/OLMo-3-1025-32B-pretrain.py \
      --save-folder=models/tutorial-run-01

Customize training parameters (optional): The key components and hyperparameters are defined through the ExperimentConfig class in src/olmo_core/script_utils.py. To override any fields at runtime, pass them as command-line options:

torchrun --nproc-per-node=gpu src/scripts/official/OLMo3/OLMo-3-1025-7B-pretrain-1.py \
  --save-folder=models/tutorial-run-01 \
  --train_module.rank_microbatch_size=8192 \
  --trainer.callbacks.lm_evaluator.enabled=false \
  --trainer.callbacks.downstream_evaluator.enabled=false \
  --trainer.hard_stop='{value: 100, unit: steps}'

Use --dry-run to validate your config without launching training.

Convert to HuggingFace format (optional): To convert your trained model to HuggingFace format (requires transformers):

python src/examples/huggingface/convert_checkpoint_to_hf.py \
  --checkpoint-input-path models/tutorial-run-01/step100 \
  --huggingface-output-dir hf_models/tutorial-run-01/step100