GCP Vertex AI
If you are using a model hosted on GCP Vertex AI, you can directly call the GCP Vertex AI model using the @google-cloud/vertexai
package from within your Agent
, or use an AI Gateway like Portkey
to connect to the GCP Vertex AI model.
Calling GCP Vertex AI via GCP Vertex AI SDK
For the most up-to-date information on how to call GCP Vertex AI via the GCP Vertex AI SDK, please refer to the npm package docs.
Installation
npm install @google-cloud/vertexai
Usage
import {
FunctionDeclarationSchemaType,
HarmBlockThreshold,
HarmCategory,
VertexAI
} from '@google-cloud/vertexai';
class ChatbotAgent implements Agent {
async chat(
content: ConversationRequestContent,
context: ConversationContext
): Promise<LLMAgentResponse> {
const { userMessage } = content;
// Create VertexAI client
const client = new VertexAI({
project: "PROJECT_ID",
location: "LOCATION",
});
const model = client.getGenerativeModel({
model: textModel,
// The following parameters are optional
// They can also be passed to individual content generation requests
safetySettings: [{category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT, threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE}],
generationConfig: {maxOutputTokens: 1000},
});
const response = await model.generateContent({
contents: [{role: 'user', content: parts: [{text: userMessage}]}],
});
return {
message: response.contents[0].parts[0].text,
};
}
}
Calling GCP Vertex AI via Portkey
Make sure you have Portkey setup by following the Portkey Integration Guide. For the most up-to-date information on how to call GCP Vertex AI via Portkey, please refer to the Portkey Documentation.
Usage
class ChatbotAgent implements Agent {
async chat(
content: ConversationRequestContent,
context: ConversationContext
): Promise<LLMAgentResponse> {
const { userMessage } = content;
// Create Portkey client
const portkey = new Portkey({
apiKey: "PORTKEY_API_KEY",
virtualKey: "VERTEX_AI_VIRTUAL_KEY"
});
// Call the API
const response = await portkey.chat.completions.create({
messages: [{ role: 'user', content: userMessage }],
model: 'gemini-pro',
maxTokens: 1000
},
{
authorization: "VERTEX_AI_ACCESS_KEY"
});
return {
message: response.choices[0].message.content,
};
}
}