Description
The UNITER research code repository, presented in the ECCV 2020 paper 'UNITER: UNiversal Image-TExt Representation Learning', offers a comprehensive set of tools for multimodal AI research. This repository facilitates finetuning and inference for a range of vision-and-language tasks, including Visual Question Answering (VQA), Visual Commonsense Reasoning (VCR), SNLI-VE (Visual Entailment), Image-Text Retrieval for datasets like COCO and Flickr30k, and Referring Expression Comprehensions (RefCOCO, RefCOCO+, RefCOCO-g).
Read More
UNITER is built upon a universal image-text representation learning framework. The code supports both UNITER-base and UNITER-large pre-trained checkpoints, with options for in-domain pre-training. The repository includes scripts for data preprocessing, model training, and inference. It leverages external libraries such as PyTorch, HuggingFace Transformers, OpenNMT, and Nvidia's DeepLearningExamples, with image features extracted using BUTD.
For ease of reproduction, a Docker image is provided, requiring specific NVIDIA driver and Docker versions. The setup involves mounting source code and data directories into the container. The repository details quick start guides for tasks like NLVR2, demonstrating data download, Docker container launch, finetuning, and inference/evaluation procedures. Customization is supported through command-line arguments and JSON configuration files, with options for multi-GPU training using Horovod.
The project also outlines steps for downstream tasks such as VQA, VCR (including a second-stage pre-training option), Visual Entailment, Image-Text Retrieval (both zero-shot and finetuning with hard negatives for Flickr30k and COCO), Referring Expressions, and in-domain pre-training. Users are guided on downloading specific datasets and running the respective training and inference scripts. The repository is licensed under the MIT license.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud, Self-hosted
- PlatformsLinux
UNITER's Core Features
Official research code for ECCV 2020 paper 'UNITER: UNiversal Image-TExt Representation Learning'
Supports finetuning for NLVR2, VQA, VCR, SNLI-VE, Image-Text Retrieval, and Referring Expressions
Provides pre-trained checkpoints for UNITER-base and UNITER-large models
Includes scripts for data preprocessing, model training, and inference
Offers a Docker image for simplified reproduction and environment setup
Supports multi-GPU training via Horovod
Includes code for zero-shot and finetuning of Image-Text Retrieval tasks
Facilitates in-domain pre-training and VCR second-stage pre-training
Getting Started with UNITER
Download pre-trained checkpoints and task-specific data using provided bash scripts.
Launch the Docker container for a consistent and reproducible environment.
Integrate model training by running finetuning scripts with custom configurations.
Perform inference on downstream tasks using dedicated inference scripts.
Evaluate model performance using provided evaluation scripts or by submitting results to leaderboards.
Customize training options via command-line arguments or JSON configuration files.
UNITER's Use Cases
- Visual Question Answering
- Visual Commonsense Reasoning
- Image-Text Retrieval
- Referring Expression Comprehension
- Visual Entailment
- Multimodal Pre-training
