Deploying TensorRT-LLM
NVIDIA TensorRT-LLM images run through General Inference and expose an OpenAI-compatible API with trtllm-serve. This example walks through deploying TinyLlama/TinyLlama-1.1B-Chat-v1.0 with a validated release image.
Before you begin
Prepare the following:
- Access to a cluster with a suitable NVIDIA GPU hardware instance
- A Hugging Face token if the model is gated or you need higher Hub rate limits
- A Bearer token or client certificate for endpoint authentication
The image and model versions must be compatible. Review the TensorRT-LLM support matrix before using a different model or image tag.
Configure the deployment
From Create deployment, select Inference, then configure the container:
| Field | Example value |
|---|---|
| Add or Select Image | nvcr.io/nvidia/tensorrt-llm/release |
| Container Tag | 1.3.0rc22 |
| Container Port | 8000 |
| Protocol | HTTP |
| Health check path | /health |
The image might appear as nvidia/tensorrt-llm/release in the prebuilt image list. The tag above is a validated example; use a tag that supports your selected model.
Add command
The TensorRT-LLM release image does not start an inference server by itself. Under Optional Details → Add command, enter:
/bin/bash -lc 'trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host 0.0.0.0 --port 8000 --config /etc/engine/config.yaml'
Replace the model identifier with the Hugging Face model or TensorRT-LLM engine path you want to serve. Wrap the full trtllm-serve command in /bin/bash -lc '...' so the image shell initializes the TensorRT-LLM environment before the server starts.
| Argument | Why it matters |
|---|---|
/bin/bash -lc '...' | Loads the image environment before running trtllm-serve |
trtllm-serve <model> | Starts the OpenAI-compatible server |
--host 0.0.0.0 | Required so CCluster can reach the server |
--port 8000 | Must match Container Port |
--config /etc/engine/config.yaml | Points at the mounted config file |
Config file
Add a config file with these settings:
| Field | Value |
|---|---|
| Filename | config.yaml |
| Mount path | /etc/engine |
Use the directory as the mount path, not the full filename. CCluster mounts the file at /etc/engine/config.yaml.
For the TinyLlama example, use:
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
max_batch_size: 8
max_num_tokens: 2048
max_seq_len: 2048
trust_remote_code: true
kv_cache_config:
free_gpu_memory_fraction: 0.8
enable_block_reuse: true
enable_iter_perf_stats: true
Tune max_batch_size, max_seq_len, and free_gpu_memory_fraction for your model and GPU memory. For TensorRT-LLM 1.3.x, keep these options at the top level; do not wrap them in pytorch_backend_config.
Model credentials
Under Optional Details → Environment Variables, set HF_TOKEN to a Hugging Face token when the model is gated or private. You can store the token in a Vault object instead of entering it directly.
Every endpoint must also be protected with a Bearer token, an mTLS client certificate, or both. See Securing Endpoints.
From here the deployment follows the standard General Inference flow — select the cluster and hardware, click Deploy, and wait until the deployment is ready. Model download and GPU initialization can take several minutes. See General Inference for details.
Test the endpoint
Copy the endpoint URL and Bearer token from the deployment detail page:
export ENDPOINT='https://<endpoint_url>'
export CCLUSTER_TOKEN='<your_bearer_token>'
# Check readiness
curl -sS -o /dev/null -w "%{http_code}\n" \
-H "Authorization: Bearer $CCLUSTER_TOKEN" \
"$ENDPOINT/health"
# Find the served model identifier
curl -sS \
-H "Authorization: Bearer $CCLUSTER_TOKEN" \
"$ENDPOINT/v1/models"
# Send a chat completion request
curl -sS "$ENDPOINT/v1/chat/completions" \
-H "Authorization: Bearer $CCLUSTER_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"messages": [{"role": "user", "content": "Say hello."}],
"max_tokens": 64
}'
Use the exact model id returned by /v1/models in subsequent requests. The first request can take longer while the runtime finishes warming up.
Troubleshooting
| Symptom | Resolution |
|---|---|
| Deployment does not become ready although the server started | Confirm the command includes --host 0.0.0.0. |
Logs report Is a directory: '.../config.yaml' | Set Mount path to /etc/engine and Filename to config.yaml. |
Logs report unsupported pytorch_backend_config arguments | Remove that block and place supported options at the top level of the YAML file. |
API returns 401 | Send the deployment's Bearer token in the Authorization header. |
API returns 503 or no healthy upstream | Wait for the deployment to become ready, then check the host and config mount settings. |
| Model download is slow or fails | Set HF_TOKEN and confirm that it can access the model. |