Logo

Model Providers

With Palico, you own the implementation details of your LLM application. As such, you can use any LLM model provider to build your application. Here are some common LLM model providers that you can use with Palico.

Portkey

Portkey is an AI Gateway that allows you to connect to multiple AI models and providers using a single API. You can setup Portkey locally, or use the hosted version at https://portkey.ai.

Installation

npm install portkey

Example Usage

import { Portkey } from 'portkey';

class ChatbotAgent implements Agent {
  async chat(
    content: ConversationRequestContent,
    context: ConversationContext
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content;
    // Create Portkey client
    const portkey = new Portkey({
      Authorization: "Bearer sk-xxxxx",
      provider: "openai",
    });
    // Call the API
    const response = await portkey.chat.completions.create({
      messages: [{ role: 'user', content: userMessage }],
      model: 'gpt-3.5-turbo',
    });
    return {
      message: response.choices[0].message.content,
    };
  }
}

AWS Bedrock

To use a model hosted on AWS Bedrock, you can either use the AWS Bedrock SDK or use an AI Gateway like Portkey to connect to the AWS Bedrock model.

Installation

npm install @aws-sdk/client-bedrock-runtime

Usage

import {
  BedrockRuntimeClient,
  InvokeModelCommand,
} from "@aws-sdk/client-bedrock-runtime";

class ChatbotAgent implements Agent {
  static readonly client = new BedrockRuntimeClient({ region: process.env.AWS_REGION });

  async chat(
    content: ConversationRequestContent,
    context: ConversationContext
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content;
    // Create request parameters
    const payload = {
      anthropic_version: "bedrock-2023-05-31",
      max_tokens: 1000,
      messages: [{ role: "user", content: [{ type: "text", text: userMessage }] }],
    };

    // Call the API
    const apiResponse = await client.send(
      new InvokeModelCommand({
        contentType: "application/json",
        body: JSON.stringify(payload),
        modelId: "anthropic.claude-3-haiku-20240307-v1:0",
      }),
    );

    // Parse the response
    const decodedResponseBody = new TextDecoder().decode(apiResponse.body);
    const responseBody = JSON.parse(decodedResponseBody);

    // Return the response
    return {
      message: responseBody.content[0].text,
    };
  }
}

GCP Vertex AI

If you are using a model hosted on GCP Vertex AI, you can directly call the GCP Vertex AI model using the @google-cloud/vertexai package from within your Agent, or use an AI Gateway like Portkey to connect to the GCP Vertex AI model.

For the most up-to-date information on how to call GCP Vertex AI via the GCP Vertex AI SDK, please refer to the npm package docs.

Installation

npm install @google-cloud/vertexai

Usage

import {
  FunctionDeclarationSchemaType,
  HarmBlockThreshold,
  HarmCategory,
  VertexAI
} from '@google-cloud/vertexai';

class ChatbotAgent implements Agent {

  async chat(
    content: ConversationRequestContent,
    context: ConversationContext
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content;
    // Create VertexAI client
    const client = new VertexAI({
      project: "PROJECT_ID",
      location: "LOCATION",
    });
    const model = client.getGenerativeModel({
      model: textModel,
      // The following parameters are optional
      // They can also be passed to individual content generation requests
      safetySettings: [{category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE}],
      generationConfig: {maxOutputTokens: 1000},
    });


    const response = await model.generateContent({
      contents: [{role: 'user', content: parts: [{text: userMessage}]}],
    });
    return {
      message: response.contents[0].parts[0].text,
    };
  }
}

Ollama

Ollama lets you run various LLM models on your own machine. You can setup Ollama by following the Ollama Installation Guide.

Installation

npm install ollama

Usage

import { Ollama } from 'ollama';

class ChatbotAgent implements Agent {
  static readonly client = new Ollama({
    apiKey: process.env.OLLAMA_API_KEY,
  });

  async chat(
    content: ConversationRequestContent,
    context: ConversationContext
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content;
    const response = await ollama.chat({
      model: 'llama3',
      messages: [{ role: 'user', content: userMessage }],
    })
    return {
      message: response.choices[0].message.content,
    };
  }
}

OpenAI

You can use OpenAI models by directly calling the OpenAI API using the openai package from within your Agent, or use an AI Gateway like Portkey to connect to the OpenAI model.

For the most up-to-date information on how to call OpenAI via the OpenAI SDK, please refer to the npm package docs.

Installation

npm install openai

Usage

import { OpenAI } from 'openai';

class ChatbotAgent implements Agent {
  static readonly client = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });

  async chat(
    content: ConversationRequestContent,
    context: ConversationContext
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content;
    const client = new OpenAI({
      apiKey: process.env['OPENAI_API_KEY'],
    });

    const response = await client.chat.completions.create({
      messages: [{ role: 'user', content: userMessage }],
      model: 'gpt-3.5-turbo',
      maxTokens: 1000
    });
    return {
      message: response.choices[0].message.content,
    };
  }
}

Anthropic

If you are using an Anthropic model, you can directly call the Anthropic model using the @anthropic-ai/sdk package from within your Agent, or use an AI Gateway like Portkey to connect to the Anthropic model.

For the most up-to-date information on how to call Anthropic via the Anthropic SDK, please refer to the npm package docs.

Installation

npm install @anthropic-ai/sdk

Usage

class ChatbotAgent implements Agent {
  static readonly client = new AnthropicClient({
    apiKey: process.env.ANTHROPIC_API_KEY,
  });

  async chat(
    content: ConversationRequestContent,
    context: ConversationContext
  ): Promise<LLMAgentResponse> {
    const { userMessage } = content;
    const response = await client.messages.create({
      messages: [{ role: 'user', content: userMessage }],
      model: 'claude-3-opus-20240229',
      max_tokens: 1000
    });
    return {
      message: response.content,
    };
  }
}

Was this page helpful?