Private Inference Endpoints

By default, endpoints created for both LLM Serving and General Inference deployments are publicly accessible. If you prefer restricted access, you can make the endpoint private by selecting the Make it a private endpoint? option. This setting generates a TLS certificate upon deployment, which you can download. Access to the endpoint will then require the http client to use this certificate. Here are a few examples: Using curl command

curl -X 'POST' 'https://<endpoint_url>/openai/v1/chat/completions' \
  --cert <path to TLS certificate> \   # Downloaded certificate
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "role": "user",
      "content": "what is the meaning of life?"
    }
  ],
  "model": "meta-llama/Llama-3.2-3B-Instruct",
  "max_tokens": 512,
  "n": 1,
  "presence_penalty": 0,
  "stream": true,
  "stream_options": {
    "include_usage": true
  },
  "temperature": 0.7,
  "top_p": 1
}'

Using httpx library

import httpx

client = httpx.Client(cert="<path to TLS certificate>")

Using OpenAPI client library

import httpx
from openai import OpenAI

client = OpenAI(
    api_key="no_key",
    base_url=”https://<endpoint url>/openai/v1”,
    http_client=httpx.Client(cert="<path to TLS certificate>"),
)