Oscar - Git at Google

Oscar is fundamentally an experiment, with its future direction and learnings yet to be fully determined. The initial prototype, the @gabyhelp bot, has already demonstrated numerous successful interactions within the Go issue tracker. Currently developed under the Go project, Oscar may potentially be spun out into a separate project in the future. The project's concrete goals include reducing maintainer effort in resolving issues and change lists (CLs/pull requests), decreasing the effort required to address forum questions, and enabling more individuals to become productive maintainers. A key non-goal is automating coding; the focus remains on automating maintainer toil.

The approach centers on building a reusable and extensible architecture that any software project can adapt to create customized agents. Oscar aims to address maintainer toil, which is not unique to the Go project. Key identified capabilities include indexing and surfacing relevant project context during contributor interactions, using natural language to control deterministic tools, and analyzing issue reports, CLs/PRs, and group discussions to improve them in real-time, label, and route them appropriately. LLMs are integral to these capabilities due to their strength in natural language processing, while deterministic code handles the execution. The project emphasizes using LLMs for semantic analysis and natural language-to-program translation, relying on deterministic code for the rest.

One crucial capability is indexing and surfacing related project context. Software projects are complex, and maintainers cannot always retain all relevant details. Oscar uses LLM embeddings to create a project search engine, indexing documentation, issues, and discussions. When a new issue arises, an agent can use this index to find related context, such as similar past issues or relevant documentation, and reply with links. This has shown benefits like surfacing duplicate issues, providing context to contributors, and saving maintainers time by pointing them to relevant past discussions or bug reports. The agent's immediate interaction after a report is filed is also vital for engaging reporters while they are still available and engaged.

Another key capability is using natural language to control deterministic tooling. As projects grow, the number of tools increases, making it hard to track and use them. Oscar envisions facilities for editing issue comments, adding labels, or assigning people based on natural language requests. LLMs are adept at translating natural language intentions into executable forms like tool invocations. Preliminary experiments with Gemini show promise in selecting and invoking tools. This approach, which uses LLMs to convert natural language to deterministic tool use, is considered more effective than a pure LLM "magic wand" approach, as it allows for deterministic execution and easier verification.

The third important capability involves analyzing issue reports and CLs/PRs. Beyond linking related issues, Oscar plans to incorporate semantic analysis, such as identifying the primary topic of an issue for appropriate labeling (e.g., "performance"). It also aims to determine if more information is needed for a report to be useful, such as requesting a reproduction program link or inlining code. The agent could potentially run sandboxed execution tools to identify bug-containing releases or use git bisect to pinpoint bug-introducing commits. These analyses are most effective when performed immediately after a report is filed, leveraging the agent's 24/7 availability.

The first prototype, Gaby ("Go AI bot"), operates within the Go issue tracker, posting as @gabyhelp. Its source code is in internal/gaby. Gaby indexes content from Go issues, Go documentation, Gerrit code reviews, and Go project/golang-nuts discussions, enabling it to respond to new issues with relevant links. Gaby's architecture is designed for flexibility, allowing it to run on various hosting services, use different LLMs, storage layers, and vector databases. The current prototype uses GCP Cloud Run with Google's Gemini LLM and Google Cloud Firestore. The project also acknowledges the existence of @gopherbot, a long-standing deterministic agent in the Go project, and aims to eventually merge @gabyhelp into @gopherbot by rebuilding @gopherbot as an Oscar agent, seeking a lighter-weight configuration method using natural language.