Basic Usage
const handler: Chat = async ({
userMessage,
// 1. destructure the stream object
stream,
}) => {
// 2. call llm with streaming
const response = await openai.chat.completions.create({
model: "gpt-3.5-turbo-0125",
stream: true,
messages: [{ role: "user", content: userMessage }],
});
// 3. read the response stream
for await (const chunk of response) {
if (chunk.choices[0].delta.content) {
// 4. stream chunks back to the client
stream.push({
message: chunk.choices[0].delta.content,
});
}
}
};
You can then use the Client SDK to read the stream from other services such as an API or NextJS application.
const responseStream = await palico.agent.chat({
agentName: "my_agent",
userMessage: "Hello",
stream: true,
});
for await (const chunk of responseStream.readChunks()) {
// Do something with the chunk
}
Learn more about Client SDK.
For agent-based application, you can stream intermediate steps back to the client. This can be useful for debugging, or for providing more context to the user. Here’s an example of an agent streaming intermediate step when a new tool is called:
const handler: Chat = async ({ userMessage, stream }) => {
// ...
const response = await agentExecutor({
onToolCall: (toolCall) => {
// streaming tool calls back to the client
stream.push({
intermediateSteps: [{
name: `Function Call: ${toolCall.name}()`
data: { input: toolCall.parameters, result: toolCall.result },
}],
})
},
// ... other parameters
})
// ...
}
The IntermediateStep
object contains the following fields:
A name or description of the intermediate step.
Data associated with the intermediate step.