Introduction
What is Palico?
LLM application by default has a lot of performance issues (accuracy / hallucinations, latency, cost). To improve its performance, developers need to be able to experiment with many variations of their build layer. This involves testing different prompt techniques, LLM models, RAG pipeline, and more. Since you need to trial-and-error so many components of your application, iterative development is key to building a performant LLM application.
Building an LLM Application has three core activities:
- Building: The components used to create the application such as prompt, LLM model, RAG, etc.
- Experimenting: Testing different variations of the build layer to improve the performance of the LLM application
- Deploying: Moving the LLM application to production, integrating with other services, and monitoring its performance
Palico is a framework that standardizes the lifecycle of building, experimenting, and deploying an LLM application such that you can maximize the speed of the iterative loop for developing an LLM application that meets your performance targets.
Components of a Palico Stack
Agent
Agent is an interface
that you wrap around your LLM Application. It helps you structure your LLM application for rapid experimentation.
An agent generally consists of your logic for building your prompt, and calling an LLM model. As this is defined in-code, you can virtually use any external libraries or services such as LangChain, LlamaIndex, Portkey, etc. to help you build your LLM application.
class MyLLMApp implements Agent {
async chat(
content: ConversationRequestContent,
context: ConversationContext
): Promise<LLMAgentResponse> {
// ... your LLM application logic
}
}
Read more about Agents
Palico Studio
A UI Control Panel that runs locally on your machine during development to help you quickly test your application, compare responses, run experiments, and review traces.
Learn more about Palico Studio
Playground
As part of Palico Studio, we provide few UI tools to help you quickly test your changes.
Chat with your Application
You can chat with your LLM application to quickly test your changes.
Compare Responses Side-by-Side
Quickly compare responses from different variations of your LLM application.
Experimentation
Experiments helps you imperically evaluate the performance of your LLM application. Palico provides tools to help you run, manage, and analyze experiments.
Learn more about Experiments
Experiment Management
You can run and manage experiments from Palico Studio.
Experiment Analysis
You can analyze the results of your experiments from Palico Studio.
Tracing
Palico comes with built-in tracing out of the box. We provide our own tracing, but you can also add any custom tracing with OpenTelemetry.
You can view traces from Palico Studio.
Learn more about Tracing
Deployment
Docker
You can compile your Palico application into docker containers and deploy it to any cloud provider.
Client SDK
You can connect to your Palico application from any external service using our Client SDK or REST API. Read more about this in the Client SDK section.
Version Control
An LLM Stack is constantly evolving as you experiment with different variations of your LLM application. All your changes and experiments are automatically versioned and stored in git. This allows you to easily understand the history of your LLM application and collaborate with your team.
Getting Started
You can get started by following the installation guide or get familiar with all the core concepts of Palico by running through the tutorial.
The role of Palico vs other tools
How does Palico compare to evaluation frameworks
Evaluation frameworks are often just a tool to help you grade the response of your LLM application. They may provide some proprietary observability and tracing tools. They however do not help you structure your LLM application development for rapid experimentation. They do not help you build your LLM application. They do not help you deploy your LLM application. The responsibility you have when using an evaluation framework is you need to build your own experiment management system that helps you scale your experimentation process across your team. You will have lots of fragmented tools that you need to integrate together to get a full picture of your LLM application.
We are a more integrated framework that helps you build, scale experimentation, and deploy your LLM application. We provide a more integrated experience for your team to work on LLM applications.
How does Palico compare to libraries like LangChain or LlamaIndex
LangChain and LlamaIndex are more like libraries that helps you achieve different tasks with LLM Development. For example, they provide you tools to connect with different LLM providers, connect to vector database, create evaluations, and more. They are more like a swiss army knife that helps you achieve different tasks with LLM Development. It's up to you to use these tools to structure your LLM application development for maximum productivity.
We are a framework (not a library) that has strong opinions on how you should structure your LLM application development. Our opinion is biased towards accuracy improvement through rapid experimentations. With our framework, you have a standard process for how you build your LLM application, measure accuracy, and run experiments. These processes all work together to maximize experiment-ability of your LLM application so you can reach your accuracy goals faster.
As we are a framework and LangChain or LlamaIndex are libraries, you can directly use LangChain or LlamaIndex within our application to help with tasks such as calling LLM models or managing your RAG layer. Our framework will help you streamline your experimentation process.