Description
Attest is a comprehensive testing framework designed specifically for AI agents, available for both Python and TypeScript environments. It allows developers to rigorously test the behavior of their AI agents, verifying their outputs, tool interactions, and any modifications they make to external states. The framework emphasizes deterministic checks, enabling developers to capture agent traces and model exchanges, then replay them against the agent's real code in a controlled environment.
Read More
The core workflow of Attest involves three main steps: capture, replay, and gate. First, `attest capture` records the agent's trace and model interactions, redacts sensitive data, and drafts initial behavior contracts. These contracts define expected behaviors, such as specific tool calls or changes to a 'world state.' Next, the `replay` mechanism runs the agent's actual code against these recorded model responses through a local proxy. This eliminates the variability of live model calls, making tests consistent and repeatable in CI/CD pipelines. Finally, the `gate` step enforces policies based on test results, blocking merges if critical contracts are violated or unexpected behaviors occur.
Attest supports a layered approach to testing, starting with free and instant deterministic checks (schema validation, constraint checks, trace and behavioral checks, content pattern matching) before moving to more resource-intensive, model-based evaluations like embedding similarity and LLM-as-judge assertions. This tiered system allows for efficient testing, with model-based checks only invoked when necessary. The framework also offers advanced features such as multi-agent testing, simulation mode with mock tools and seeded faults, and experimental continuous evaluation for monitoring score drift.
With a robust adapter system, Attest integrates with popular AI providers like OpenAI, Anthropic, Gemini, and Ollama, as well as frameworks such as LangChain, CrewAI, and LlamaIndex. It runs within standard unit-test runners like pytest for Python and Vitest for TypeScript, making it easy to incorporate into existing development workflows. The framework's ability to turn synthetic incidents into regression tests, coupled with features like mutation testing and an HTML viewer for detailed failure analysis, positions Attest as a critical tool for building reliable and robust AI agents.
- CategoryAI Code Review & Testing
- Pricing ModelOpen Source
- DeploymentCloud
- PlatformsWeb
Attest Framework's Core Features
AI Agent Testing Framework for Python and TypeScript
Capture AI Agent Traces and Model Exchanges
Strict Replay of Recorded Model Responses in CI
Deterministic Checks for Agent Behavior
Behavior Contracts for Tool Calls and World Effects
CI Policy Enforcement and Merge Gate
Multi-Agent Testing for Delegation Chains
Simulation Mode with Mock Tools and Seeded Faults
Continuous Evaluation and Drift Detection (Experimental)
Layered Assertion Evaluation (Deterministic to LLM-as-Judge)
Framework Adapters for LangChain, CrewAI, LlamaIndex
Provider Adapters for OpenAI, Anthropic, Gemini, Ollama
Mutation Testing for Test Suite Effectiveness
HTML Viewer for Detailed Failure Reports
Local ONNX Embeddings for Cost-Free Semantic Checks
How to use Attest Framework?
Install SDK: Add `attest-ai` and `pytest` for Python, or `@attest-ai/core` and `vitest` for TypeScript.
Create Agent: Define your AI agent, ensuring it records LLM and tool calls for trace capture.
Write Test: Use `expect()` to chain assertions and `attest.evaluate()` to send them to the engine.
Run Tests: Execute tests with `uv run pytest -v` for Python or `npx vitest run` for TypeScript.
Capture Incident: Use `attest capture` to record agent traces and model exchanges from real or synthetic incidents.
Replay in CI: Run tests against recorded model responses in your CI pipeline to ensure deterministic outcomes.
Gate Merges: Configure CI policies to block pull requests that violate defined behavior contracts or baselines.
Review Reports: Use `attest view` to generate an HTML report for detailed analysis of test failures and traces.
Attest Framework's Use Cases
- Regression Testing AI Agents
- Verifying Tool Interactions
- Continuous Integration for AI
- Evaluating Agent Outputs
- Debugging Multi-Agent Systems
- Simulating Agent Environments
- Monitoring Agent Performance
- Cost-Effective Testing
- Ensuring Business Invariants




