Description
DoGBench is a benchmark tool designed to evaluate AI agents in their ability to write and maintain user-facing software documentation. The tool assesses whether agents can recognize when documentation requires updates and produce patches that meet expert review standards. Each patch is scored using rubrics validated with project maintainers, focusing on progress towards expert standards rather than performance relative to human experts.
Read More
The benchmark includes 292 tasks sourced from real open-source projects such as Helm, PostHog, Mautic, and Doc Detective. Of these tasks, 205 require documentation changes, while 87 require no changes. The evaluation uses a stratified 117-item held-out split, consisting of 82 tasks needing updates and 35 requiring abstention.
DoGBench reports separate scores for update decisions and patch quality. The scoring rules capture both delivered patch quality and correct abstention, using the harmonic mean to combine them. A critical failure caps a patch's score at 60 out of 100, with each rubric criterion having equal weight.
Agents often struggle to determine when documentation needs updating, sometimes documenting internal changes that don't affect users or missing necessary updates due to lack of existing coverage. Despite polished writing, agents can omit essential steps or place patches on pages readers are unlikely to find.
The tool is valuable for documentation owners who wish to discuss technical writing challenges with their teams or leadership. It provides insights into where AI agents fall short and emphasizes the importance of expert review to ensure documentation accuracy and completeness.
- CategoryAI Code Review & Testing
- DeploymentCloud
- PlatformsWeb
DoGBench's Core Features
Evaluates AI agents for documentation updates
Uses rubrics validated with project maintainers
Scores patches based on expert standards
Includes 292 tasks from open-source projects
Separates scores for update decisions and patch quality
Addresses task-completion gaps and technical inaccuracies
Provides insights for documentation owners
Emphasizes the importance of expert review
How to use DoGBench?
Evaluate: Assess agent's documentation capabilities
Score: Use rubrics to score patches
Analyze: Review task-completion gaps
Discuss: Share insights with documentation teams
DoGBench's Use Cases
- Documentation Updates
- Expert Review
- Technical Writing
- AI Evaluation
- Open-Source Projects
