Description
EvalCore is designed to help developers know when their AI gets worse before users notice. It offers snapshot testing for AI behavior, allowing users to record how their LLM app or agent behaves and then replay that recording in continuous integration (CI) on every change. This process is offline, deterministic, and completely free.
Read More
With EvalCore, developers can easily check if changes such as modifying a prompt, swapping a model, or updating a dependency have caused any issues. The tool records AI behavior and checks every change against the recorded data, ensuring that any regressions are caught early. The setup is straightforward, requiring only a small binary that works with any programming language, eliminating the need for a test harness or SDK.
The first run of EvalCore calls the real model and records every request and response to a local SQLite cassette. This cassette is keyed on a hash of the canonical request, allowing for easy tracking of tokens and costs. Each subsequent run in CI replays the cassette, ensuring that identical inputs produce identical outputs, thus blocking regressions effectively. This means that developers can run tests without needing network access or API keys, making the process more reliable and efficient.
EvalCore is compatible with various models and APIs, including OpenAI, vLLM, and Ollama, as well as custom REST APIs and shell commands. This flexibility allows developers to ship their first evaluation in minutes without any dependencies. The tool is open-source under the Apache-2.0 license, making it accessible for anyone looking to improve their AI applications.
- CategoryAI Code Review & Testing
- Pricing ModelFree
- DeploymentOffline
- PlatformsmacOS, Linux, CI runners
EvalCore's Core Features
Snapshot testing for AI behavior
Offline replay
Deterministic results
No SDK or test harness required
Records requests and responses
Compatible with OpenAI and other APIs
Local SQLite cassette storage
Free to use in CI
How to use EvalCore?
Configure: Point a target at your app and list the dataset.
Run: Execute the command to record the AI behavior.
Replay: Use the recorded cassette in CI to check for regressions.
Commit: Save the cassette for future tests.
EvalCore's Use Cases
- Regression Testing
- Behavior Monitoring
- Model Comparison
- Integration Testing
- Cost Tracking
