EvalCore

Description

EvalCore is designed to help developers know when their AI gets worse before users notice. It offers snapshot testing for AI behavior, allowing users to record how their LLM app or agent behaves and then replay that recording in continuous integration (CI) on every change. This process is offline, deterministic, and completely free.

With EvalCore, developers can easily check if changes such as modifying a prompt, swapping a model, or updating a dependency have caused any issues. The tool records AI behavior and checks every change against the recorded data, ensuring that any regressions are caught early. The setup is straightforward, requiring only a small binary that works with any programming language, eliminating the need for a test harness or SDK.

The first run of EvalCore calls the real model and records every request and response to a local SQLite cassette. This cassette is keyed on a hash of the canonical request, allowing for easy tracking of tokens and costs. Each subsequent run in CI replays the cassette, ensuring that identical inputs produce identical outputs, thus blocking regressions effectively. This means that developers can run tests without needing network access or API keys, making the process more reliable and efficient.

EvalCore is compatible with various models and APIs, including OpenAI, vLLM, and Ollama, as well as custom REST APIs and shell commands. This flexibility allows developers to ship their first evaluation in minutes without any dependencies. The tool is open-source under the Apache-2.0 license, making it accessible for anyone looking to improve their AI applications.

EvalCore's Core Features

• Snapshot testing for AI behavior

• Offline replay

• Deterministic results

• No SDK or test harness required

• Records requests and responses

• Compatible with OpenAI and other APIs

• Local SQLite cassette storage

• Free to use in CI

How to use EvalCore?

1. 1

   Configure: Point a target at your app and list the dataset.

2. 2

   Run: Execute the command to record the AI behavior.

3. 3

   Replay: Use the recorded cassette in CI to check for regressions.

4. 4

   Commit: Save the cassette for future tests.

EvalCore's Use Cases

• Regression Testing
• Behavior Monitoring
• Model Comparison
• Integration Testing
• Cost Tracking

FAQ from EvalCore

• What is EvalCore?

  EvalCore is a tool for snapshot testing AI behavior, allowing developers to record and replay AI interactions.

• How does EvalCore work?

  It records requests and responses to a local SQLite cassette, which can be replayed in CI.

• Is EvalCore free to use?

  Yes, EvalCore is completely free to use in continuous integration.

• What programming languages does EvalCore support?

  EvalCore works with any programming language as it does not require a specific SDK.

• Can EvalCore be used with OpenAI models?

  Yes, EvalCore is compatible with OpenAI models and other APIs.

• What is the purpose of the SQLite cassette?

  The SQLite cassette stores recorded requests and responses for deterministic replay.

• How do I set up EvalCore?

  Simply point a target at your app, list the dataset, and run the command to start recording.

• Does EvalCore require internet access?

  No, EvalCore operates offline and does not require network access for replay.