Description
TaskMatrix is an open-source project hosted on GitHub, with mirrors under Microsoft's GitHub organization. It's the current name for the approach published in the paper "Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models," and it still carries that lineage: the main script is named visual_chatgpt.py and its official Hugging Face demo Space is titled "visual_chatgpt." The core idea hasn't changed: connect a conversational model like ChatGPT to a set of specialized visual foundation models so the conversation can send and receive images, not just text.
Read More
The core insight is a division of labor. ChatGPT, or another large language model, acts as a general interface that understands a request in plain language, while separate foundation models act as domain experts for a specific visual task, such as detecting an object, segmenting it, or inpainting a region. TaskMatrix formalizes this with a template system: a template is a pre-defined execution flow that lets ChatGPT chain multiple foundation models together to complete a more complex task than any single model could handle alone. The InfinityOutPainting template is one example, combining an image-captioning model, an inpainting model, and a visual-question-answering model so a user can ask to extend an image to a specific size and get an outpainted result without any additional training.
In practice, a common workflow is text-guided image editing: a user says "find xxx in the image" or "segment xxx in the image," and TaskMatrix uses GroundingDINO to locate the object by its text description, Segment Anything to generate a precise mask, and Stable Diffusion inpainting to edit only that masked region. The project also supports Chinese-language conversations.
TaskMatrix can be tried without installing anything, through a Hugging Face Spaces demo or a Google Colab notebook. Running it locally requires cloning the repository, creating a Python environment, installing the listed foundation models, and providing an OpenAI API key for the ChatGPT side. Which foundation models a user loads is configurable, since each one has a different GPU memory footprint, from lightweight models that need no dedicated GPU memory to image-generation models that need several gigabytes each. The project is aimed at developers and researchers experimenting with multi-model, conversational image editing rather than end users looking for a finished consumer app.
- CategoryComputer Vision Tools
- Pricing ModelOpen source
- DeploymentSelf-hosted, Cloud
- PlatformsWeb
TaskMatrix's Core Features
Sends and receives images inside a ChatGPT conversation via connected visual foundation models
Text-guided object detection and segmentation using GroundingDINO and Segment Anything
Mask-based inpainting with Stable Diffusion to edit only the requested part of an image
InfinityOutPainting template that extends an image to a larger size without additional training
Template system for chaining multiple foundation models into one reusable task flow
Chinese-language conversation support alongside English
Runnable through a Hugging Face Spaces demo or Google Colab notebook with no local install
Configurable model loading so only the foundation models a task needs run on a given device
Getting Started with TaskMatrix
Try the demo: Open the Hugging Face Space or Colab notebook to test TaskMatrix without installing anything
Set up locally: Clone the repository, create a Python environment, and install the listed requirements
Add an API key: Provide an OpenAI API key for the ChatGPT side of the system
Load the models you need: Start the app with a --load flag listing which foundation models to run and on which device
Chat and edit: Describe an image request, or say "find xxx" or "segment xxx" to detect and edit an object in an uploaded image
TaskMatrix's Use Cases
- Conversational image editing
- Image outpainting
- Multi-model research prototyping
