Deploying NVIDIA NIM
NVIDIA NIM microservices are self-contained containers that bundle a model-optimized inference engine and download model weights automatically at startup, exposing an OpenAI-compatible API. NIM is not a pre-configured recipe on the General Inference page — enter the image manually with the settings below.
Find a NIM image
NIM images are hosted on nvcr.io. Browse the catalog at build.nvidia.com and copy the image path from the model's Deploy tab. The catalog shows the image and tag joined with a colon (e.g. nvcr.io/nim/meta/llama-3.1-8b-instruct:latest) — on CCluster, split them into the Add or Select Image and Container Tag fields:
| Image | Tag |
|---|---|
nvcr.io/nim/deepseek-ai/deepseek-r1-distill-qwen-32b | latest |
nvcr.io/nim/meta/llama-3.1-8b-instruct | latest |
You also need an NGC API Key — it is used both to pull the image and to download model weights.
Configure the deployment
From the General Inference deployment page, every NIM uses the same configuration:
| Field | Value |
|---|---|
| Add or Select Image | The nvcr.io/nim/... image path (entered manually, without the tag) |
| Container Tag | The image tag (e.g. latest) |
| Container Port | 8000 |
| Protocol | HTTP |
| Health check path | /v1/health/ready |
| Image Registry Username | $oauthtoken (literal string — not a variable) |
| Image Registry Password | Your NGC API Key |
Leave Add command empty — the image's default entrypoint starts the server. Under Optional Details → Environment Variables, also add:
| Key | Value |
|---|---|
NGC_API_KEY | Your NGC API Key |
The NGC API Key is needed twice: the registry credentials let CCluster pull the image from nvcr.io, and the NGC_API_KEY environment variable lets the container download model weights on first boot. Without the environment variable, the container starts but fails to load the model.
NIM downloads model weights at startup, so the first boot can take several minutes for large models. The deployment stays in a pending state until the /v1/health/ready probe passes.
From here the deployment follows the standard General Inference flow — select the cluster and hardware, click Deploy, and monitor the deployment until it is ready. See General Inference for details. Check the model's page on build.nvidia.com for the GPU and memory each NIM supports.
Once ready, query the endpoint like any OpenAI-compatible endpoint. List the served model identifier first, then send requests:
# Find the model identifier served by this NIM
curl https://<endpoint_url>/v1/models \
-H "Authorization: Bearer <your_bearer_token>"
# Chat completion
curl https://<endpoint_url>/v1/chat/completions \
-H "Authorization: Bearer <your_bearer_token>" \
-H "Content-Type: application/json" \
-d '{
"model": "meta/llama-3.1-8b-instruct",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 256
}'
Every endpoint is protected — there are no public endpoints. Secure access with a Bearer token, an mTLS client certificate, or both. For mTLS, select Make it a private endpoint? and CCluster generates a TLS certificate that clients must present. See Securing Endpoints for details.