Project Structure
Palico App expects a specific project structure to be able to run your application. The structure is as follows:/agents
: This directory contains all the agents that you have created. To create a new agent, just create a new directory with the agent’s name and add anindex.ts
file in it/eval_tests
: This directory contains different testsuites that you want to use for experiments. To create a new testsuite, just create a new directory with the testsuite’s name and add anindex.ts
file in it/workflows
: This directory contains different workflows that you want to use for experiments. To create a new workflow, just create a new directory with the workflow’s name and add anindex.ts
file in itinstrumentation.ts
: This file configurations for tracing and logging. You can just leave it as is for now.main.ts
: This file is the entry point of your application. You can just leave it as is for now.
Agent
Agents are the core building blocks of your Palico App. Agents are just an encapsulation of your LLM application. Agents are generally your prompt logic and a call to your LLM model. In an LLM application, your call to an LLM model is the main non-deterministic part of your application. As such, for a given agent, you should have a single LLM model call. This will let you systematically improve the accuracy of that LLM component of your overall application. For applications with multiple LLM models, you should have multiple agents and orchestrate them using workflows.Tracing
Tracing helps you understand the runtime behavior of your LLM application. You can log events, errors, inputs and outputs, the time it took to run different parts of your application, and more. We provide traces through OpenTelemetry, which is a standard for distributed tracing.Experiments
You should treat your LLM application as a scientific experiment. In a have indepdenent variables, (LLM model, your prompt, context to your prompt, etc) and dependent variables (accuracy / metrics). Experiments helps you test your LLM agents by modifying the independent variables and measuring their effects on dependent variables. In an experiments, you often run multiple tests to measure the effect of multiple independent variables on your dependent variables. Experiments helps you run, manage, and analyze these tests. Experiments management in done through Palico Studio.Test Cases
To systematically improve the accuracy of your LLM application, you need to create some way to measure it’s accuracy. This is where test cases come in. Test cases are just a suite of tests you can run against your LLM model to measure it’s accuracy. You can create multiple test suites to measure different aspects of your LLM model, or have different test suites for different LLM agents.Metrics
It’s hard to have a single number that indicates the accuracy of your LLM application. So instead, we want to be able to measure different attributes of an LLM Agents response. This can be things like:- For JSON mode, is the schema correct?
- Does the response contain the correct response?
- Is the response professional?