Skip to main content

Deployment options

Because large language models (LLMs) often require expensive GPUs, deploying them to a suitable environment is often preferrable to running locally. When it comes to deploying LLMs in production, there are several approaches you can take. In this guide, we'll cover two main deployment strategies:

  • Google Vertex AI: Deploy your model on Google Cloud's managed ML platform, Vertex AI. This option provides a fully managed service with built-in scaling, monitoring, and API endpoints.

  • Modal.com: Deploy your model on Modal's serverless platform. This option offers a simple deployment process with automatic scaling and competitive pricing.

Hosting our models on Google Vertex AI​

Vertex AI is Google Cloud's managed machine learning platform. It provides tools to build, deploy, and manage machine learning models at scale. With Vertex AI, you can leverage tools for data preparation, model training, hyperparameter tuning, model evaluation, deployment, and monitoring. This guide walks you through deploying the Olmo-2-1124-7B model on Google Cloud’s Vertex AI and serving it via a REST API — putting a 7B model into production with ease.

Getting started​

Make sure you have:

  • A Google Cloud project with the Vertex AI API enabled.
  • A Cloud Storage bucket to host your model files (e.g., gs://ai2-llm/).

The right IAM roles:

  • Vertex AI Admin
  • Storage Admin

Install Google Cloud SDK

pip install google-cloud-aiplatform

Download model​

Download the model and tokenizer from Hugging Face and save them to disk:

from transformers import AutoModelForCausalLM, AutoTokenizer

olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-1124-7B")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-2-1124-7B")

olmo.save_pretrained("./olmo_model")
tokenizer.save_pretrained("./olmo_model")

Upload the saved model to your GCS bucket:

gsutil cp -r ./olmo_model gs://ai2-llm/olmo2-7b-1124

Deploying model​

Use the Vertex AI Python SDK to upload and deploy the model:

from google.cloud import aiplatform

aiplatform.init(project="olmo-2-7b-1124", location="us-central1")

model = aiplatform.Model.upload(
    display_name="olmo-7b-model",
    artifact_uri="gs://ai2-llm/olmo2-7b-1124",
    serving_container_image_uri="europe-docker.pkg.dev/vertex-ai/prediction/pytorch-x.y:latest",
) # Replace pytorch-x.y:latest with the correct PyTorch version tag.

endpoint = model.deploy(
    machine_type="n1-standard-8",
    accelerator_type="NVIDIA_TESLA_T4",
    accelerator_count=1,
)

print(f"Model is live at: {endpoint.resource_name}")

POST request​

Use a POST request to get predictions:

import requests
import json

endpoint_url = "https://us-central1-aiplatform.googleapis.com/v1/projects/ai2-llm/locations/us-central1/endpoints/234234:predict"
message = ["Language modeling is"]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)

payload = {
    "instances": inputs["input_ids"].tolist()
}

headers = {
    "Authorization": "Bearer XXX"  # Replace with your access token
}

response = requests.post(endpoint_url, json=payload, headers=headers)
print(response.json())

Similarly, you can host all our models on Vertex AI. For more details you can visit Google's Documentation.

One click deployment on Model Garden​

You can also go to the Open models on Hugging Face section in the Model Garden, search for our models, and click on an included model. Then, choose the required settings such as region, endpoint name, and machine specifications, and deploy them without writing any code. The following is the equivalent code to achieve the same.

Deploying via Python

# Install the Vertex AI SDK
pip install --upgrade google-cloud-aiplatform
gcloud auth application-default login

# Use the following code in your application to deploy the model

import vertexai
from vertexai.preview import model_garden

vertexai.init(project="ai2-allennlp", location="us-east4")

model = model_garden.OpenModel("allenai/olmo-2-1124-7b")
endpoint = model.deploy(
  accept_eula=True,
  machine_type="a2-ultragpu-1g",
  accelerator_type="NVIDIA_A100_80GB",
  accelerator_count=1,
  endpoint_display_name="olmo-2-1124-7b-mg-one-click-deploy",
  model_display_name="olmo-2-1124-7b-1743620092901",
)

Hosting on Modal.com​

You can also host Ai2 models on modal.com using the OpenAI API and vLLM. This implementation is based on modal.com's reference example for OpenAI with vLLM, with two key modifications:

  1. Using an Olmo 2 model instead of their default model
  2. Downloading model weights dynamically from Hugging Face using huggingface_hub.snapshot_download(), rather than manually uploading them to a Modal's storage volume

Our setup includes:

  • A FastAPI web server
  • Completions and chat endpoints for Olmo
  • Dynamic scaling with keep_warm=0, meaning the app shuts down when idle to save costs

Getting started​

  1. Follow the instructions under Getting Started in the modal.com guide to install the modal library and command-line tools.
  2. Follow the instructions under Secrets in the modal.com Guide to create a modal secret named example-secret-token that defines a value for the variable MODAL_TOKEN for your server.
  3. Run the code below. Modify it as needed to use different GPUs.
import os
import modal

MODEL_NAME = "allenai/OLMo-2-1124-13B-Instruct"
MODEL_REVISION = "3a5c85baefbb1896a54d56fe2e76c0395627ddf4"
MODEL_DIR = "/root/models/{MODEL_NAME}"
N_GPU = 1
GPU_CONFIG = modal.gpu.A100(size="80GB", count=N_GPU)
APP_NAME = "OLMo-2-1124-13B-Instruct-openai"
APP_LABEL = APP_NAME.lower()
ONE_MINUTE = 60
ONE_HOUR = 60 * ONE_MINUTE

def download_model_to_image(model_dir, model_name, model_revision):
    from huggingface_hub import snapshot_download
    from transformers.utils import move_cache
    os.makedirs(model_dir, exist_ok=True)
    snapshot_download(
        model_name,
        revision=model_revision,
        local_dir=model_dir,
    )
    move_cache()

vllm_image = (
    modal.Image.debian_slim(python_version="3.10")
    .pip_install(
        "vllm==0.6.6.post1",
        "torch==2.5.1",
        "transformers==4.47.1",
        "ray==2.10.0",
        "huggingface_hub==0.24.0",
        "hf-transfer==0.1.6",
    )
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
    .run_function(
        download_model_to_image,
        timeout=60 * ONE_MINUTE,
        kwargs={
            "model_dir": MODEL_DIR,
            "model_name": MODEL_NAME,
            "model_revision": MODEL_REVISION,
        },
    )
)
app = modal.App(APP_NAME)

@app.function(
    image=vllm_image,
    gpu=GPU_CONFIG,
    keep_warm=0,
    container_idle_timeout=5 * ONE_MINUTE,
    timeout=24 * ONE_HOUR,
    allow_concurrent_inputs=1000,
    secrets=[modal.Secret.from_name("example-secret-token")],
)
@modal.asgi_app()
def serve():
    import fastapi
    import vllm.entrypoints.openai.api_server as api_server
    from vllm.engine.arg_utils import AsyncEngineArgs
    from vllm.engine.async_llm_engine import AsyncLLMEngine
    from vllm.entrypoints.logger import RequestLogger
    from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
    from vllm.entrypoints.openai.serving_completion import OpenAIServingCompletion
    from vllm.entrypoints.openai.serving_engine import BaseModelPath
    from vllm.usage.usage_lib import UsageContext

    web_app = fastapi.FastAPI(
        title=f"OpenAI-compatible {MODEL_NAME} server",
        description="Run an OpenAI-compatible LLM server with vLLM on modal.com 🚀",
        version="0.0.1",
        docs_url="/docs",
    )
    http_bearer = fastapi.security.HTTPBearer(
        scheme_name="Bearer Token",
        description="See code for authentication details.",
    )
    web_app.add_middleware(
        fastapi.middleware.cors.CORSMiddleware,
        allow_origins=["*"],
        allow_credentials=True,
        allow_methods=["*"],
        allow_headers=["*"],
    )
    # This example uses a token defined in the Modal secret linked above,
    # as described here: https://modal.com/docs/guide/secrets
    async def is_authenticated(api_key=fastapi.Security(http_bearer)):
        if api_key.credentials != os.getenv("MODAL_TOKEN"):
            raise fastapi.HTTPException(
                status_code=fastapi.status.HTTP_401_UNAUTHORIZED,
                detail="Invalid authentication credentials",
            )
        return {"username": "authenticated_user"}
    router = fastapi.APIRouter(dependencies=[fastapi.Depends(is_authenticated)])
    router.include_router(api_server.router)
    web_app.include_router(router)

    engine_args = AsyncEngineArgs(
        model=MODEL_DIR,
        tensor_parallel_size=N_GPU,
        gpu_memory_utilization=0.90,
        max_model_len=4096,
        enforce_eager=False,
    )

    engine = AsyncLLMEngine.from_engine_args(engine_args, usage_context=UsageContext.OPENAI_API_SERVER)
    model_config = get_model_config(engine)
    request_logger = RequestLogger(max_log_len=2048)
    base_model_paths = [BaseModelPath(name=MODEL_NAME.split("/")[1], model_path=MODEL_NAME)]
    api_server.chat = lambda s: OpenAIServingChat(
        engine,
        model_config=model_config,
        base_model_paths=base_model_paths,
        chat_template=None,
        chat_template_content_format=None,
        response_role="assistant",
        lora_modules=[],
        prompt_adapters=[],
        request_logger=request_logger,
    )
    api_server.completion = lambda s: OpenAIServingCompletion(
        engine,
        model_config=model_config,
        base_model_paths=base_model_paths,
        lora_modules=[],
        prompt_adapters=[],
        request_logger=request_logger,
    )
    return web_app


def get_model_config(engine):
    import asyncio
    try:  # adapted from vLLM source -- https://github.com/vllm-project/vllm/blob/507ef787d85dec24490069ffceacbd6b161f4f72/vllm/entrypoints/openai/api_server.py#L235C1-L247C1
        event_loop = asyncio.get_running_loop()
    except RuntimeError:
        event_loop = None
    if event_loop is not None and event_loop.is_running():
        model_config = event_loop.run_until_complete(engine.get_model_config())
    else:
        model_config = asyncio.run(engine.get_model_config())
    return model_config

Deploying the server​

To deploy the API on Modal, run the following command:

modal deploy olmo2_modal_openai.py

This command:

  • Creates a new app on Modal
  • Builds the container image
  • Deploys the Olmo 2 model

Interacting with the server​

Once deployed, a unique URL will appear in your terminal, such as: https://your-workspace-name--olmo-2-1124-instruct-openai-serve.modal.run

Testing the endpoint​

Once deployed, you can test the endpoint using cURL:

curl -X POST \
  -H "Authorization: Bearer [your secret token]" \
  -H "Content-Type: application/json" \
  -d @body.json \
  https://[your-modal-endpoint]/v1/chat/completions

Replace [your-modal-endpoint] with the actual endpoint created by Modal. You can explore the API using Swagger UI by navigating to: https://your-workspace-name--olmo-2-1124-instruct-openai-serve.modal.run/docs

Example request body​

{
  "model": "Olmo-2-1124-13B-Instruct",
  "messages": [
    {
      "role": "user",
      "content": "Who was Alan Turing?"
    }
  ],
  "max_tokens": 100,
  "temperature": 0.9,
  "stream": true
}