Serverless Endpoints

​

The Problems Teams Face

​

The CentML Solution

• Text Generation Quality: Does the model generate coherent, fluent, and contextually relevant text?
• Task-Specific Performance: How well does the model perform on specific tasks, such as conversational dialogue, summarization, or question-answering?
• Common Sense and World Knowledge: Are there any obvious gaps or inaccuracies in the model’s knowledge or understanding? Is this due to a knowledge cut off date?
• Bias and Stereotyping: Does the model perpetuate or amplify biases and stereotypes present in the training data?
• Adversarial Robustness: How well does the model handle adversarial inputs or attacks, such as those designed to elicit nonsensical or biased responses?
• Explainability and Transparency: Can the model’s outputs or decisions be explained or justified in a way that is understandable to humans?
• Overfitting and Memorization: Does the model rely too heavily on memorization of the training data, rather than generalizing to new or unseen inputs?

​

Prerequesites

​

Accessing the Chat UI

​

Configuring the Endpoint

• Region: The geographical region where the model is hosted.
• Model: The specific AI model to be used for generating responses.
• Temperature: Controls the randomness of the generated response. Higher values increase the likelihood of more diverse and creative responses, while lower values result in more predictable and deterministic responses.
• Max tokens: The maximum number of tokens (words or characters) that the model will generate in its response.
• Top p: A parameter that controls the sampling of the next token. It represents the cumulative probability of the top tokens to be considered. A higher value means more tokens are considered, resulting in more diverse responses.
• Top k: Similar to Top p, but it directly limits the number of top tokens to be considered when generating the next token.
• Stop: A specific string that, when generated by the model, will cause the response to be truncated.
• Presence penalty: A penalty applied to tokens that have already been used in the response, discouraging the model from repeating them.
• Frequency penalty: A penalty applied to tokens based on their frequency of use in the response, discouraging the model from using the same tokens repeatedly.
• View (UI or API): The interface through which the model will be interacted with. UI refers to a user-friendly graphical interface, while API refers to a programmatic interface accessed via any combination of Python, Javascript, and cURL commands.
• System prompt: A predefined prompt or instruction that is passed to the model along with the user’s input, influencing the tone and direction of the response.

​

Accessing the Serverless Endpoint Programmatically

​

Using the Code Snippets (cURL)

 curl -X POST "https://api.centml.com/openai/v1/chat/completions" \
  -H "Authorization: Bearer <your serverless token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "What is CentMLs value prop?" }
    ],
    "max_tokens": 2000,
    "n": 1,
    "stream": false,
    "temperature": 0.7,
    "top_p": 1,
    "top_k": 40,
    "frequency_penalty": 0,
    "presence_penalty": 0.5,
    "stop": []
  }'

​

Using the Code Snippets (Python)

from openai import OpenAI

client = OpenAI(
    api_key="<Your API Key>",
    base_url="https://api.centml.com/openai/v1",
)

# Define your question
user_question = "How does CentML improve your AIOps?"

completion = client.chat.completions.create(
    model="meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": user_question}  # Add the user's question
    ],
    max_tokens=2000,
    temperature=0.7,
    top_p=1,
    n=1,
    stream=False,
    frequency_penalty=0,
    presence_penalty=0.5,
    stop=[]
)

print(completion.choices[0].message)

​

Using the Code Snippets (JavaScript)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "<your API key",
  baseURL: "https://api.centml.com/openai/v1",
});

async function main() {
  const userPrompt = "How can CentML improve my time to value?"; // Change this to your desired command or prompt

  const completion = await client.chat.completions.create({
    model: "meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8",
    messages: [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": userPrompt }, // Add the user's prompt or command here
    ],
    max_tokens: 2000,
    temperature: 0.7,
    top_p: 1,
    top_k: 40,
    n: 1,
    stream: false,
    frequency_penalty: 0,
    presence_penalty: 0.5,
    stop: []
  });

  console.log(completion.choices[0].message.content); // Log the response from the model
}

main();

​

Invalid API Key

​

Data Policies

​

Prompt Training

​

Prompt Logging

​

Moderation

​

Additional Considerations When Leveraging CentML’s Serverless Endpoint Offering

​

Concurrency

​

Model Availability

​

Token Limits

​

Requesting a New Model

​

Model Request Ticket Best Practices: What to Include

• Use Case Description: A detailed enough explanation of the use case and how it aligns with the business goals.
• Requirements and Constraints: Any specific requirements or constraints related to the request, such as performance, scalability, or security considerations.
• Evaluation Criteria: Information on how the success of the request will be measured or evaluated.
• Current Limitations: Details on how currently available models have failed to meet your requirements.

​

Example New Model Support Request

Description:
We are reaching out to request the addition of a new Large Language Model (LLM) to your serverless API. Our team is currently evaluating various LLMs for our natural language processing (NLP) use cases, and we believe that integrating this new model will enhance our capabilities. Business Use Case: Our primary use case is to leverage the new LLM for text classification, sentiment analysis, and language translation tasks. We anticipate that this model will provide more accurate results compared to our current models, enabling us to improve our customer experience and gain a competitive edge. Specifically, we plan to use the new model to:

• Analyze customer feedback and sentiment on our platform.
• Classify and route customer inquiries to the relevant support teams.
• Translate user-generated content to facilitate global communication.

Limitations: While the current hosted LLMs show promise, our testing has revealed limitations that impact their suitability for our use cases. Specifically:

• Lack of domain-specific fine-tuning affects text classification accuracy.
• Sentiment analysis is biased towards certain emotional tones or language styles.
• Language translation struggles with domain-specific content, idioms, and colloquialisms.

We’re concerned that these limitations may compromise the accuracy and reliability of our customer-facing applications. We’re looking for alternative models or customizations that can address these issues. Concurrency Considerations: We understand that serverless APIs are designed to handle variable workloads, and we expect our usage to be moderate. Initially, we anticipate an average of 10 requests per minute, with occasional spikes to 50 requests per minute during peak hours. We believe that the serverless API can handle this concurrency level without significant performance degradation. Future Plans: After evaluating the performance of the new model, we anticipate that our usage will grow, and we may eventually require a dedicated endpoint to handle our workload. We are working towards assessing the model’s performance and scalability, and we expect to migrate to a dedicated endpoint if our request volume exceeds 500 requests per minute. We would like to request guidance on the process for migrating to a dedicated endpoint, should it become necessary.

​

CentML Serverless Endpoint Specifications

​

What’s Next