NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI

NVIDIA has unveiled its latest innovation, the NVIDIA Cosmos™ 3, during the GTC Taipei event. This new foundation model is designed for physical AI, leveraging a groundbreaking mixture-of-transformers architecture that seamlessly integrates vision reasoning, world generation, and action prediction into a single system. The launch signifies a major advancement in the capabilities of AI to interact with and understand the physical world.

Cosmos 3 stands out as the world’s first fully open omnimodel, capable of natively processing and generating a variety of media, including text, images, video, ambient sound, and actions, all while maintaining high physics accuracy. This capability significantly reduces the time required for training and evaluating physical AI systems, compressing what previously took months into mere days.

A New Architecture for Physical AI

The architecture of Cosmos 3 addresses a critical challenge in the field of physical AI: the ability of robots, autonomous vehicles (AVs), and vision agents to generalize their understanding of the real world with limited training data. Traditional models often struggle with fragmented simulation stacks, which can hinder their effectiveness in real-world applications.

Cosmos 3 employs a mixture-of-transformers architecture that combines a reasoning transformer with an expert generation transformer. This innovative design allows the model to comprehend the interactions between objects, their motion, and spatial-temporal relationships before generating video and action trajectories. The training process for Cosmos 3 utilized one of the largest multimodal physical AI datasets, which includes billions of samples across various domains such as text, images, video, sound, and action trajectories. This extensive dataset empowers developers to create robust physical AI systems while minimizing data requirements and training costs.

In terms of performance, Cosmos 3 has achieved leading results on several physical AI benchmarks. It ranks first among open models in categories such as Artificial Analysis, Physics-IQ, PAI-Bench, and R-Bench for world generation accuracy. Additionally, it excels in action policy benchmarks like RoboLab and RoboArena, as well as in vision understanding on the VANTAGE-Bench and TAR leaderboards.

Cosmos Coalition Accelerates Open World Model Development

Alongside the launch of Cosmos 3, NVIDIA introduced the Cosmos Coalition, a global initiative aimed at fostering collaboration among world model builders and AI developers. This coalition includes notable members such as Agile Robots, Black Forest Labs, Generalist, LTX, Runway, and Skild AI, all working together to advance the development of open world models across various industries.

The coalition's mission is to enable faster innovation and broader interoperability by allowing members to contribute models, research, and evaluation techniques while utilizing Cosmos 3 technologies, training tools, and NVIDIA DGX™ Cloud infrastructure for large-scale training. By building in an open environment and sharing resources, the coalition aims to drive rapid advancements in the field of physical AI.

The Cosmos platform serves as the backbone of NVIDIA’s physical AI stack, streamlining training and evaluation workflows across multiple sectors. The platform has been enhanced with new datasets focused on robotics, physics, human motion, autonomous driving, warehouse safety, and spatial reasoning. Furthermore, it introduces new physical AI agent skills, including neural scene reconstruction, defect-image generation, and video augmentation, which are crucial for the development of advanced AI applications.

Various industry leaders are already leveraging the Cosmos platform, including Agile Robots, Doosan Robotics, LG Electronics, Samsung, and Skild AI for robotics, as well as Li Auto for autonomous vehicles. Companies such as Centific, Fogsphere, Linker Vision, Milestone Systems, and Yuan are utilizing the platform for vision AI agents to enhance industrial AI and smart spaces applications.

Availability and Future Prospects

The Cosmos 3 Super and Cosmos 3 Nano models are now available, with the Cosmos 3 Edge model set to be released soon for real-time inference capabilities. Developers interested in exploring Cosmos 3 can access it on build.nvidia.com, download open models from Hugging Face, and customize models using Hugging Face Diffusers and resources available on GitHub. The models can also be deployed as NVIDIA NIM™ microservices, facilitating integration into various applications.

Model builders and software providers can expedite access, customization, and deployment of Cosmos for key reasoning and synthetic data generation workloads. This is made possible through physical AI agent skills available on GitHub, alongside inference services and cloud infrastructure partnerships with companies like Baseten, CoreWeave, Microsoft Azure, Nebius, Deep Infra, and Classmethod.

As NVIDIA continues to lead in AI and accelerated computing, the introduction of Cosmos 3 and the Cosmos Coalition marks a significant step forward in the evolution of physical AI, offering developers unprecedented tools and capabilities to create intelligent systems that can interact with the physical world effectively.