Description
evalship offers a comprehensive quality control solution specifically designed for Large Language Model (LLM) evaluations. It integrates directly into your GitHub workflow, acting as a vigilant guardian for your LLM development process. The core purpose of evalship is to identify and address flaws in your evaluation setups, ensuring that your LLMs perform as expected and do not introduce regressions.
Read More
Upon setup, evalship thoroughly reviews your existing evaluators, judges, and prompts to pinpoint weaknesses such as evals that incorrectly pass or rules that lack proper test coverage. It then monitors every pull request, providing immediate feedback on what broke, why it broke, and the likely cause. This includes flagging untested AI calls and separating genuine regressions from flaky eval results by analyzing historical performance on your default branch.
The platform works seamlessly with various eval frameworks, including those that produce JUnit XML, promptfoo JSON, or evalship JSON, such as pytest, Vitest, Jest, RSpec, Go, Java, PHPUnit, Rust, Langfuse, Braintrust, and LangSmith. It doesn't run your evals directly; instead, your GitHub Actions workflow or coding agent executes them using your API keys, and evalship processes the results.
Beyond identifying issues, evalship helps optimize your prompts and models. Through your coding agent, it tests different changes on your machine, such as cheaper models or clearer rules, and reports which changes yield statistically significant improvements. This allows teams to make data-driven decisions to enhance model performance and cost-efficiency. evalship also provides weekly reports summarizing eval performance and highlighting critical issues caught before they reach production.
Pricing includes a free tier for public repositories and one private repository with one user, offering 100 PR analyses and 30 days of eval history. Paid plans (Team, Pro, Business) offer increased users, private repositories, PR analyses, and longer eval history, with options for monthly or yearly billing. A launch offer provides significant discounts for early adopters. evalship ensures security by not storing your source code or API keys, only processing diffs and eval results.
- CategoryAI Code Review & Testing
- Pricing ModelFreemium
- DeploymentCloud
- PlatformsWeb
evalship's Core Features
Identifies flaws in LLM evaluations during setup
Comments on every pull request with detailed breakdown of eval failures
Auto-optimizes prompts and models based on performance data
Works with any existing eval framework (JUnit XML, promptfoo, evalship JSON)
Provides 24+ quality checks for LLM evals
Separates regressions from flaky evals using historical data
Detects untested AI calls in pull requests
Blocks merges that break evaluation rules (configurable)
Unblocks pull requests once issues are fixed
Emails weekly eval reports summarizing performance and issues
Supports monorepos and multiple eval workflows
Integrates with Langfuse, Braintrust, and LangSmith without requiring a switch
Offers a free review of evals at setup
Allows coding agents to check changes before opening pull requests
Ensures security by not storing source code or API keys
How to use evalship?
Set up: Connect your GitHub repository and ask your coding agent to set up evalship, which reviews your evals and creates a setup pull request.
Monitor pull requests: evalship automatically watches every pull request, commenting on changes that break evals or introduce untested AI calls.
Review reports: Access detailed reports for every eval's inputs, outputs, and history to understand what broke and why.
Optimize: Use evalship's optimization feature to test prompt and model changes with your coding agent, identifying improvements beyond chance.
Receive reports: Get weekly email reports summarizing eval performance and highlighting critical issues caught before merging.
evalship's Use Cases
- LLM Evaluation Quality Control
- Automated Pull Request Review
- Prompt and Model Optimization
- Regression Prevention
- Untested AI Call Detection
- Flaky Eval Management
- Compliance and Liability Risk Mitigation
- Developer Workflow Enhancement
