Setup an AI Gateway

AI Gateway is a service that let's you use with different LLM models from a single endpoint. We can use AI Gateway along with App Config to easily test different LLM models with our LLM application. Some notable AI Gateway are:

For this guide, we'll use Portkey as our AI Gateway.

Setup Portkey

You can setup Portkey locally, or use the hosted version at portkey.ai. Once you have Portkey setup, continue with the following steps.

Add Portkey to your project

npm install portkey

Using Portkey to call OpenAI models

import { Portkey } from 'portkey'

class ChatbotAgent implements Agent {
  async chat(
    content: ConversationRequestContent,
    context: ConversationContext,
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content
    // Create Portkey client
    const portkey = new Portkey({
      Authorization: 'Bearer sk-xxxxx',
      provider: 'openai',
    })
    // Call the API
    const response = await portkey.chat.completions.create({
      messages: [{ role: 'user', content: userMessage }],
      model: 'gpt-3.5-turbo',
    })
    return {
      message: response.choices[0].message.content,
    }
  }
}

Using Portkey to call any LLM model with App Config

To be able to easily swap different LLM models in your application, you can use AppConfig along with Portkey. This will allow you to use any LLM model without changing your code and just updating the AppConfig.

import { Portkey } from 'portkey'

interface AppConfig {
  model: string
  provider: string
}

class ChatbotAgent implements Agent {
  async chat(
    content: ConversationRequestContent,
    context: ConversationContext<AppConfig>,
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content
    const { model, provider } = context.appConfig
    // highlight-start
    const portkey = new Portkey({
      Authorization: 'Bearer sk-xxxxx',
      provider: provider,
      // ...additional authorization params
    })
    const response = await portkey.chat.completions.create({
      messages: [{ role: 'user', content: userMessage }],
      model: model,
    })
    // highlight-end
    return {
      message: response.choices[0].message.content,
    }
  }
}

Using AppConfig to swap LLM models allows you to easily run experiments with different variations of your applications locally, or run A/B tests in production without changing your code.