Description
This GitHub repository, locuslab/convmixer, hosts the implementation of ConvMixer, a neural network architecture detailed in the ICLR 2022 paper titled "Patches Are All You Need? 🤷" by Asher Trockman and Zico Kolter. The primary code for the ConvMixer model is located in `convmixer.py`. The project leverages the `pytorch-image-models` (timm) framework for training and includes modifications to this framework to support ConvMixers.
Read More
Key modifications within the `pytorch-image-models` framework include the addition of `timm/models/convmixer.py`, updates to `timm/models/__init__.py`, and the integration of a "OneCycle" learning rate schedule with corresponding scheduler files. While the OneCycle schedule is noted, the authors suggest that standard cosine schedules could also be effective for training ConvMixers.
The repository offers pre-trained model weights for several ConvMixer variants, including ConvMixer-1536/20, ConvMixer-768/32, and ConvMixer-1024/20, with details on their kernel size, patch size, and file size. Instructions and example commands are provided for evaluating these models on datasets like ImageNet-1k, aiming for high accuracy. For instance, ConvMixer-1536/20 is expected to achieve 81.37% accuracy on ImageNet-1k validation.
Training ConvMixers is also detailed, with an example command for distributed training on a multi-GPU setup using ImageNet-1k. The training command specifies hyperparameters such as batch size, number of workers, optimizer, epochs, learning rate schedule, augmentation techniques (e.g., RandAugment, CutMix, MixUp), and input size. The repository also includes a simplified, tweetable version of the ConvMixer architecture in Python.
Additionally, a separate repository for training ConvMixers on CIFAR-10 is linked, and advice is given on adjusting training parameters like `--scale` for smaller input sizes. The project is open-source under the MIT license, encouraging community contribution and further development.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud, Self-hosted
- PlatformsWeb
ConvMixer's Core Features
Implementation of ConvMixer neural network architecture
Based on the research paper "Patches Are All You Need? 🤷"
Includes code for training ConvMixers
Provides pre-trained model weights for evaluation
Supports training on ImageNet-1k and CIFAR-10 datasets
Leverages the `pytorch-image-models` (timm) framework
Includes a "OneCycle" learning rate scheduler implementation
Offers example commands for model evaluation
Includes example commands for distributed training
Contains a simplified, tweetable version of the ConvMixer model
Model weights available for download
Open-source under the MIT license
Getting Started with ConvMixer
Access Model: Clone the GitHub repository to your local machine.
Set Up Environment: Install necessary Python libraries, including PyTorch and `pytorch-image-models`.
Load Model: Instantiate a ConvMixer model using the provided Python scripts.
Evaluate Model: Use the `validate.py` script with downloaded weights and a validation dataset.
Train Model: Execute the `distributed_train.sh` script with specified dataset paths and training parameters.
Integrate: Adapt the `convmixer.py` code for custom training loops or inference pipelines.
ConvMixer's Use Cases
- Image Classification
- Model Evaluation
- Custom Model Training
- Research and Development
- Transfer Learning
- Benchmarking
