RunPod: GPU Cloud Deployment Mental Model

You can run large language models on GPU cloud services such as RunPod. First, you must understand how RunPod works. This guide describes the architecture, the APIs, and two problems that we found. We found them when we deployed a model with 27B parameters.

Architecture: Template, Pod, and Endpoint

RunPod has three layers. A Template is a reusable configuration. It defines the Docker image, the environment variables, the ports, and the disk sizes. A Pod is a running GPU instance. You create it from a Template. An Endpoint is a proxy URL. You use it to reach your service from outside.

Template (reusable config blueprint)
   └── Pod (running GPU instance from template)
        └── Endpoint (proxy URL to access your service)

The two APIs

RunPod has two interfaces. The first is a CLI (runpodctl). It wraps GraphQL. The second is a REST API at https://rest.runpod.io/v1/pods. It gives full control.

The CLI has known bugs. For example, it does not pass disk sizes from templates. For production work, use the REST API. It is more reliable.

The REST API pod create call

This request creates a pod. The list after it explains the important fields.

POST https://rest.runpod.io/v1/pods
Authorization: Bearer <API_KEY>
Content-Type: application/json

{
  "name": "model-deployment",
  "imageName": "vllm/vllm-openai:latest",
  "gpuTypeIds": ["NVIDIA B200"],
  "gpuCount": 1,
  "containerDiskInGb": 100,
  "volumeInGb": 100,
  "volumeMountPath": "/root/.cache/huggingface",
  "ports": ["8000/http", "22/tcp"],
  "cloudType": "SECURE"
}

• containerDiskInGb: the temporary disk. The system erases it at restart. It must hold the Docker image and the model weights during download. The default is only 20-50GB. This was the first critical bug.
• volumeInGb: the persistent disk. It stays after a restart. The system mounts it at volumeMountPath.
• volumeMountPath: set it to /root/.cache/huggingface. Then the model downloads stay after a restart.
• ports: http ports are available through the RunPod proxy URL. tcp ports give direct IP access (SSH).
• gpuTypeIds: an array of GPU type names. You can add fallback GPU types.

Docker ENTRYPOINT and CMD

When you use a custom configuration, you must know how ENTRYPOINT and CMD work together. The container runs ENTRYPOINT + CMD as its final command.

# Normal vLLM image:
ENTRYPOINT: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
CMD: (args from dockerStartCmd)
Final: python3 -m vllm.entrypoints.openai.api_server --model ...

# Override pattern (for injecting env fixes):
ENTRYPOINT: ["bash", "-c"]           ← override
CMD: ["export LD_LIBRARY_PATH=... && python3 -m vllm..."]
Final: bash -c "export LD_LIBRARY_PATH=... && python3 -m vllm..."

The two bugs we found

Disk too small

The command runpodctl pod create --template-id=X ignores containerDiskInGb in the template. It uses a default of 20GB. A model with 27B parameters needs about 50GB for the weights alone. The container disk became full. The container then crashed and restarted again and again.

Fix: Use the REST API. Set containerDiskInGb: 100 in the request.

CUDA-compat library conflict

The CUDA 13.0 container image includes cuda-compat libraries. These libraries hide the libcuda.so of the host driver. The host driver (580) and CUDA 13.0 are compatible. But the linker finds /usr/local/cuda/compat/libcuda.so first, in the container. This file causes the failure.

Fix: Set this environment variable:

export LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

This setting makes the dynamic linker search the real driver libraries of the host first. It uses the compat libraries of the container only after that.

Endpoint URL pattern

When a pod runs, its proxy URL has this form:

https://<POD_ID>-<PORT>.proxy.runpod.net

This URL is compatible with the OpenAI API. Set it as base_url in any OpenAI SDK. Then you can connect your existing LLM pipelines to the pod.

Useful commands

# List available GPU types and stock
runpodctl gpu list

# List your running pods
runpodctl pod list

# Delete a pod
runpodctl pod remove <ID>

# SSH connection details
runpodctl ssh info <ID>

Monitoring with GraphQL

The RunPod GraphQL API gives the GPU utilization and the uptime. Use it to monitor a pod with a program.

curl -H "Authorization: Bearer $KEY" \
  https://api.runpod.io/graphql \
  -d '{"query":"{ pod(input:{podId:\"<ID>\"}) { 
    runtime { 
      uptimeInSeconds 
      gpus { memoryUtilPercent } 
    } 
  } }"}'

No log API

RunPod has no API for container logs. You can only read logs in the web console. This limits automated debugging and monitoring. Send your logs to an external service, for example a logging endpoint in your application. Do not depend on container stdout.

Summary

• Use the REST API for production deployments. The CLI has known bugs with parameter passing.
• Set containerDiskInGb in the request. Do not use the default for large models.
• Mount the HuggingFace cache on a persistent volume. Then the model does not download again after a restart.
• Look for cuda-compat library conflicts. If they occur, set LD_LIBRARY_PATH.
• The endpoint is compatible with the OpenAI API. You can use a RunPod pod in place of any OpenAI SDK service.