Description
Youtu-Parsing-Omni is a compact omni-modal parsing model designed to handle a wide range of inputs, including document pages, natural images, charts, flowcharts, geometry figures, audio clips, and videos. This model produces a structured JSON output that encompasses both perception and cognition tasks. The perception tasks include layout elements, text, tables, formulas, bounding boxes, timestamps, ASR, OCR, acoustic events, and camera motion. Cognition tasks involve generating captions, narratives, and reports.
Read More
The model operates using a unified schema, known as OmniSchema, which allows for a consistent output format across different tasks. It features a single Youtu-Omni-Encoder that replaces traditional separate towers for pixels and audio, enhancing text reading capabilities. The model is initialized from a pre-trained text language model, and it utilizes thin modality-specific stems to map pixels and log-mel frames to tokens. A shared bidirectional Transformer contextualizes the inputs, and modality-specific mergers and projectors feed the tokens to the Youtu-LLM decoder.
Youtu-Parsing-Omni is known for its strong performance despite its small size, achieving state-of-the-art results on OmniDocBench v1.6 and competitive results on OmniParsingBench. It is particularly effective in specialized tasks such as chemical-structure recognition and music-score recognition. The model is easy to serve, with a vLLM plugin and pinned serving settings included.
The model's audio-visual fusion capabilities allow for cross-modal alignment within the encoder, ensuring that frames and audio chunks are processed in temporal order. This approach enhances the model's ability to handle complex inputs like videos with synchronized audio and visual elements. Youtu-Parsing-Omni is available for download on platforms like Hugging Face and ModelScope, making it accessible for developers and researchers seeking a robust parsing solution.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
Youtu-Parsing-Omni Highlights
Omni-modal parsing model
Unified schema for JSON output
Single Youtu-Omni-Encoder
State-of-the-art performance
Audio-visual fusion capabilities
Compact model size
Easy to serve with vLLM plugin
Supports diverse input types
Getting Started with Youtu-Parsing-Omni
Access model: Visit Hugging Face
Authenticate: Set up credentials
Set up environment: Install dependencies
Integrate via API: Use provided examples
Optimise: Adjust settings for performance
Youtu-Parsing-Omni's Use Cases
- Document Parsing
- Image Analysis
- Chart Interpretation
- Audio Processing
- Video Analysis
