Securing Endpoints

NVIDIA CCluster does not expose public endpoints. Every LLM Serving and General Inference deployment must be protected, and clients must authenticate on each request. You secure a deployment with at least one of the following:

• Bearer token — an Authorization: Bearer <token> header on each request.
• mTLS client certificate — a TLS certificate the client presents on each request.

Endpoint bearer tokens (from your Vault) are not the same as service account credentials. Service accounts authenticate automation to NVIDIA CCluster platform APIs; Vault bearer tokens protect calls to your deployed endpoints.

You choose the protection method when configuring a deployment. You may enable both for defense in depth.

Bearer token access

Generate a Bearer token from your Vault, then include it in the Authorization header of every request.

Using curl command

curl -X 'POST' 'https://<endpoint_url>/openai/v1/chat/completions' \
  -H 'Authorization: Bearer <your_bearer_token>' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "role": "user",
      "content": "what is the meaning of life?"
    }
  ],
  "model": "meta-llama/Llama-3.2-3B-Instruct",
  "max_tokens": 512
}'

Using OpenAI client library

from openai import OpenAI

client = OpenAI(
    api_key="<your_bearer_token>",
    base_url="https://<endpoint url>/openai/v1",
)

mTLS certificate access

Select the Make it a private endpoint? option when configuring a deployment. This generates a TLS certificate upon deployment, which you can download. Access to the endpoint will then require the http client to use this certificate. Here are a few examples:

Using curl command

curl -X 'POST' 'https://<endpoint_url>/openai/v1/chat/completions' \
  --cert <path to TLS certificate> \   # Downloaded certificate
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "role": "user",
      "content": "what is the meaning of life?"
    }
  ],
  "model": "meta-llama/Llama-3.2-3B-Instruct",
  "max_tokens": 512,
  "n": 1,
  "presence_penalty": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "temperature": 0.7,
  "top_p": 1
}'

Using httpx library

import httpx

client = httpx.Client(cert="<path to TLS certificate>")

Using OpenAPI client library

import httpx
from openai import OpenAI

client = OpenAI(
    api_key="no_key",
    base_url="https://<endpoint url>/openai/v1",
    http_client=httpx.Client(cert="<path to TLS certificate>"),
)

What's next

Managing Vault Objects

Learn how to generate and store NVIDIA CCluster tokens and other vault objects.

Clients

Learn how to interact with the NVIDIA CCluster programmatically

Python SDK Reference

Learn how to access the NVIDIA CCluster using Python.