AI agents orchestrating classical tools to build controllable, physically accurate virtual worlds
At Siggraph this week Nvidia set out a vision for the future of computer graphics: neither traditional rendering nor generative AI alone is enough.
A key theme is the limitation of today’s neural graphics and generative models: while they can create images and videos in seconds from simple prompts, they struggle with fine-grained control and artistic direction. Nvidia’s answer is a hybrid approach — one in which AI agents orchestrate established graphics and physics tools, using classical techniques as building blocks inside AI-driven workflows.
Nvidia presented 21 papers of groundbreaking research in this area. “All are grounded in 3D and physics, and all are directable by creators,” said Rev Lebaredian, VP of physical AI simulation at Nvidia during a press briefing.
Among the highlights is ArtiFixer, an AI model that turns messy real-world 3D captures into clean, complete virtual scenes.
“This is a model that can take a rough, very incomplete and noisy 3D scan of the real world and turn it into a clean and complete 3D scene,” said Lebaredian. “It fixes situations where you have noisy or missing input data when you’re making your Gaussian Splat.”
ArtiFixer also includes a new method for predicting photoreal global illumination straight from a scene’s geometry — without tracing a single ray.
While ArtiFixer is useful for VFX and games, Nvidia expects major uptake in real-world applications such as architecture, city planning, GIS, and preparing environments to train and test robots.
To make AI agents practical in production, Nvidia is promoting Model Context Protocol (MCP), which allows agents to work with existing DCC tools including Adobe, Blender, Unreal Engine, SideFX Houdini, Foundry tools and Canva without a web of custom plugins.
MCP lets AI systems move data between applications, automate tedious tasks, and effectively serve as a natural-language interface for users with little or no 3D experience.
In Unreal Engine, Nvidia says MCP opens the door to assistants that can reason over scenes, assets and project state.
In Blender, a lightweight MCP server offered through Blender Lab provides a natural-language interface to Blender’s Python API, documentation and complex setups.
Omniverse, disaggregated: atomic libraries for physical AI
Nvidia is extending its modular, agent-friendly philosophy to Omniverse, its long-running environment for physics-based, USD-centric simulation of the physical world.
Originally designed as a more monolithic platform, Omniverse has already shifted toward a microservices and cloud-based model to address heavy compute requirements and ease integration.
Now Nvidia is going further, by expanding its Agent Toolkit with new Omniverse libraries — a collection of software components that give AI agents tools and skills to add physical AI capabilities to existing applications and prepare 3D content for simulation.
These Omniverse libraries expose core capabilities – from RTX rendering to physics and sensor simulation – in a form that is ‘agent friendly’, making it straightforward for AI systems to call them, and can be embedded in existing products, not just in Omniverse-branded applications.
“We’re providing low-level, core, atomic Lego pieces you can use to assemble simulations of the physical world,” said Lebaredian.
Nvidia is already showcasing several concrete integrations including PTC Onshape: using Omniverse’s CAD-to-sim-ready asset generation so mechanical designs can be turned into simulation-ready USD assets for robotics and other physical AI use cases.
Main Image caption: ArtiFixer turns messy real-world 3D captures into clean, complete virtual scenes.
