Description
Matrix New York is a sophisticated tool that leverages Google Street View imagery to create interactive video world models of New York City. It is designed to simulate vehicle control and trajectory using a combination of 3D vehicle rendering and kinematic bicycle models. The tool retrieves the nearest Google Street View panorama for a given location, removes near-field vehicles, and inserts a 3D vehicle rendered under a chase camera. This process allows users to drive pretrained interactive video world models through the city, offering a unique perspective on urban environments.
Read More
Matrix New York features three distinct models: Matrix-Game-3.0, Matrix-Game-3.5, and PanoWorld. Each model receives the same Street View observation and controller trajectory, differing only in the action adapter used. Matrix-Game-3.0 uses a keyboard and camera yaw actions, Matrix-Game-3.5 employs a camera-to-world trajectory, and PanoWorld utilizes a translation trajectory over a full 360° panorama. These models are designed to run on a single GPU without fine-tuning, ensuring efficient performance.
The tool's interactive viewer allows users to select a location on a map or list to play its rollout, with the panel below the video displaying the controller input at the displayed frame. This feature provides a comprehensive view of how the models operate and interact with the virtual environment.
Matrix New York is particularly useful for urban planning, traffic analysis, and virtual reality applications, offering a detailed and immersive experience of New York City's streets. However, it is important to note that the tool has limitations, such as the composited ego vehicle being a 2D layer that does not interact with generated content. Additionally, large heading changes away from intersections may steer the camera into facades, limiting the tool's effectiveness in certain scenarios.
- CategoryAI Video Generators
- Pricing ModelOpen source
- DeploymentCloud
- PlatformsWeb
Matrix New York's Core Features
Google Street View integration
3D vehicle rendering
Kinematic bicycle model
Interactive video world models
Single GPU operation
Three distinct models
Interactive viewer
Apache-2.0 license
How to use Matrix New York?
Select location: Choose a location on the map or list
Retrieve panorama: Get the nearest Google Street View panorama
Remove vehicles: Eliminate near-field vehicles
Insert 3D vehicle: Render a 3D vehicle under a chase camera
Simulate control: Use kinematic bicycle model for vehicle control
View rollout: Play the rollout in the interactive viewer
Matrix New York's Use Cases
- Urban Planning
- Traffic Analysis
- Virtual Reality
- Vehicle Simulation
- Educational Tool
