Description
Flide is an advanced inference infrastructure specifically engineered to accelerate Jev-style decision models. Its core innovation lies in its ability to perform exact state forking, allowing a single input state to be processed for multiple questions in one forward pass. This means the state is read only once, and then efficiently forked, ensuring that every question receives the precise answer it would obtain if it were asked in isolation.
Read More
This unique architecture provides substantial performance gains, with Flide demonstrating up to 9.5 times faster inference compared to widely used serving engines like vLLM and SGLang. The performance advantage becomes even more pronounced as the number of questions associated with a given state increases. For instance, with 64 questions on a 2k-token state, Flide achieves a 9.5x speedup and 2.2x higher throughput with 64 clients.
Beyond speed, Flide prioritizes accuracy, staying within 1.1 points of a float32 reference, making it one of the closest engines to a ground truth reference. It supports a variety of models, including chat models based on Qwen3.5 and Gemma 4, as well as OneJev models. Flide also handles hybrid models, where Gated DeltaNet layers ensure each question starts from the exact state of the shared context. The system offers a `--batch-invariant` option, guaranteeing that a request yields the same bits regardless of how it is batched.
Installation is straightforward via pip, and it supports serving with prebuilt kernels for various NVIDIA GPUs, including A40, A10, RTX 30, H100, and H200. Flide exposes the same `/v1/systemone` API as vLLM and SGLang, providing probabilities for every option in the answer. It supports multimodal inputs, including text, images, and video, depending on the backbone model. Features like FP8 quantization for Qwen3.5 on Ada, Hopper, and Blackwell GPUs, and a session cache for resuming agent states, further enhance its capabilities.
- CategoryMLOps & Model Deployment
- DeploymentSelf-hosted
- PlatformsWeb
Flide's Core Features
Exact State Forking: Processes one state for all questions in a single forward pass.
Accelerated Inference: Up to 9.5x faster than other serving engines.
High Throughput: Achieves 2.2x throughput with 64 clients on specific benchmarks.
Float32 Reference Proximity: Stays within 1.1 points of a float32 reference for accuracy.
Batch Invariance: Ensures consistent results regardless of batching with `--batch-invariant`.
ForkAttention Mechanism: Each question reads its state in place, as if asked alone.
Exact on Hybrid Models: Supports Gated DeltaNet layers for precise context handling.
One Flat Stream Processing: Runs requests as a single token stream replayed by CUDA graphs.
vLLM/SGLang Compatible API: Uses the `/v1/systemone` API standard.
Prebuilt Kernels: Optimized for NVIDIA A40, A10, RTX 30, H100, and H200 GPUs.
Model Backbones: Supports Qwen3.5 (0.8B, 2B, 4B, 9B, 27B) and Gemma 4 (E2B, E4B, 12B, 26B-A4B, 31B).
Multimodal Inputs: Handles text, images, and video inputs.
FP8 Quantization: Runs Qwen3.5 in FP8 on Ada, Hopper, and Blackwell GPUs.
Session Cache: Keeps earlier states for agents to resume from (e.g., `--session-cache-gb 40`).
Frontend Processes: Supports running HTTP, rendering, and tokenizing in separate processes for many clients.
How to use Flide?
Install Flide: Use pip to install the Flide package.
Start a Server: Run `flide serve` to start the inference server, optionally specifying `--host`.
Configure Models: Serve chat models like Qwen3.5 or Gemma 4 directly, or merge Open-Jev adapters.
Send Requests: Interact with the server using the `/v1/systemone` API to send states and questions.
Process Multimodal Inputs: Include image paths in the state and send media for models supporting images.
Flide's Use Cases
- Accelerated Decision Models
- Multimodal Inference
- Consistent Batch Processing
- Agent State Management
- High-Performance LLM Serving
- Research & Development








