Description
ITZMA is a Rust-based vector database library engineered for efficient vector search directly on object storage. It provides a scalable, durable, and cloud-native solution for AI applications that require their data to be readily accessible without the overhead of an intermediary database server. By embedding ITZMA within your application process, you can interact directly with object storage like S3, GCS, Azure, local disk, or even in-memory stores, treating it as the durable source of truth for your vector data.
Read More
The core architecture of ITZMA is designed for cloud environments, allowing multiple processes to read and write to the same object storage bucket concurrently without requiring a coordinator, lock, or leader. This is achieved by leveraging the inherent `put-if-absent` and `conditional update` functionalities of object stores, ensuring data correctness and consistency. A write operation is confirmed in the bucket before the call returns, guaranteeing durability.
ITZMA supports both dense and sparse vector search. Dense vectors can be indexed using HNSW or SQ8 algorithms, chosen per segment, with candidates reranked on exact vectors to ensure precise results. Sparse vectors, represented as (term id, weight) pairs, are stored alongside dense ones, enabling exact sparse queries with an optional inverted index. Additionally, it offers filtered search capabilities, allowing users to define typed attributes (Bool, Number, Text, TextSet, NumberSet) when creating a namespace and apply filters independently or to narrow vector queries.
Key features include atomic batch writes, support for various object storage backends, and a flexible indexing strategy that can adapt to different access patterns (e.g., HNSW for cached reads, SQ8 for uncached reads). The library also provides Python bindings on top of its memory-safe Rust core, making it accessible to a broader developer community. Maintenance tasks like compaction and indexing are explicit, giving developers full control over their data management. ITZMA is built by Sergio Marcial, a principal software engineer with extensive experience in distributed systems, aiming to provide a precise and durable vector search solution.
- CategoryVector Databases & Retrieval
- Pricing ModelOpen source
- DeploymentLibrary
- PlatformsWeb
ITZMA's Core Features
Vector search on object storage
Rust library with Python bindings
Scalable and durable data storage
Cloud-native architecture with no server
Supports dense vector search (HNSW, SQ8)
Supports sparse vector search with inverted index
Filtered search on typed attributes
Concurrent writes from multiple processes
Atomic batch upserts and deletes
Compatible with S3, GCS, Azure, local disk, memory
Explicit compaction and indexing
RAM and NVMe read cache
Read-only health report
NumPy and Arrow input in Python
How to use ITZMA?
Install ITZMA: Add the Rust crate from crates.io or install the Python package from PyPI.
Create a namespace: Define a namespace with fixed dimensions, metric (L2, cosine, dot product), and typed attributes.
Write documents: Send batched upserts and deletes to the namespace, which are stored in the object bucket.
Query data: Perform dense or sparse vector queries, with optional filters, to retrieve closest hits.
Manage maintenance: Explicitly call compaction, garbage collection, and indexing as needed for optimal performance.
ITZMA's Use Cases
- Building AI Applications
- Cloud-Native Data Storage
- Real-time Retrieval
- Dense and Sparse Vector Search
- Filtered Vector Queries
- Distributed Data Management
- Rust and Python Development
