Ray tracing is a powerful technique that can produce highly realistic and visually stunning results, with accurate reflections, refractions, and other advanced lighting effects. However, it comes at the cost of significant computational complexity, often requiring minutes or even weeks to render a single frame. On the other hand, rasterization is a much faster approach, making it the preferred choice for real-time applications like computer games. While rasterization may not achieve the same level of visual fidelity as ray tracing, it can still produce satisfactory results in many cases.

The key advantage of ray tracing is its ability to simulate the true behavior of light, leading to more accurate and realistic visual effects. This includes high-quality reflections, refractions, and complex lighting interactions that are difficult to achieve with rasterization. Ray tracing can also handle a wider range of camera models, such as fisheye lenses, and support advanced features like rolling shutter effects.

In contrast, the primary advantage of rasterization is its speed and efficiency. By simplifying the rendering process, rasterization can achieve real-time performance, making it well-suited for interactive applications like games. Rasterization also tends to be less memory-intensive, which is an important consideration for graphics hardware with limited resources.

The choice between ray tracing and rasterization ultimately depends on the specific requirements of the application. For applications that prioritize visual fidelity and realism, such as high-end visual effects or architectural visualization, ray tracing may be the preferred approach. For real-time applications that require fast performance, rasterization is often the better choice.

Gaussian Splats is a fast and efficient technique for rendering virtual worlds, but it comes with some limitations. While it can achieve real-time performance and handle scenes captured from the real world, it struggles with certain aspects of rendering.

One of the main limitations is the quality of mirror-like specular reflections. The Gaussian Splat representation does not capture these reflections well, and they often appear distorted or lacking in detail. Additionally, many of the beautiful ray tracing effects, such as refractions and volumetric caustics, do not translate well to the Gaussian Splat approach.

Another issue is the high memory consumption of the Gaussian Splat representation, which can be a problem on graphics hardware with limited memory. This can be a particular challenge for complex scenes or high-resolution virtual worlds.

Furthermore, Gaussian Splats do not support advanced camera models, such as fisheye cameras, or camera tricks like rolling shutter. This can limit its applicability in certain use cases, such as training self-driving cars, where these camera features are important.

Overall, while Gaussian Splats offer a fast and efficient way to render virtual worlds, they lack the ability to faithfully reproduce some of the more advanced rendering effects that are possible with ray tracing techniques. This trade-off between performance and visual quality is an ongoing challenge in the field of computer graphics.

Scientists at NVIDIA had an insane idea: to combine the speed of rasterization with the quality of ray tracing. This seemed impossible, as these two approaches are often seen as oil and water - they don't mix. However, the researchers managed to pull off the impossible.

They took the concept of Gaussian Splatting, which is incredibly fast, and added something they call "secondary rays". This allowed for proper light bouncing and the introduction of high-quality reflections and refractions in real-time. The result is a virtual world running in real-time with stunning visual effects that were previously only achievable through slow ray tracing.

This work, dubbed the "3D Gaussian Unscented Transform" or "3DGUT", supports advanced camera models like fisheye lenses and rolling shutter, making it highly useful for applications like self-driving car training. Compared to previous techniques, 3DGUT delivers a significant improvement in visual quality, especially for objects closer to the camera.

The best part is that the researchers didn't just keep this technology to themselves - they have made the source code freely available for all to explore and build upon. This open approach to research is truly inspiring and empowers the community to push the boundaries of real-time rendering even further.

The new 3DGUT technique offers significant improvements over previous methods. When compared to an earlier approach that could extend Gaussian Splats to fisheye cameras, the 3DGUT method shows a huge difference in quality. The previous technique struggled with artifacts and distortions, especially for objects closer to the camera. In contrast, the 3DGUT approach handles these cases much better, delivering a high-quality result.

The key advantage of 3DGUT is its ability to combine the speed of rasterization with the realism of ray tracing. By incorporating secondary rays, the technique can simulate realistic reflections and refractions, which were not possible with the earlier Gaussian Splat-based methods. This allows for the creation of virtual worlds with a level of visual fidelity that was previously unattainable in real-time.

Furthermore, the 3DGUT technique supports a wider range of camera models, including fisheye cameras, and can handle camera tricks like rolling shutter. This makes it particularly useful for applications such as training self-driving cars, where these advanced camera capabilities are crucial.

Overall, the 3DGUT method represents a significant advancement in the field of real-time rendering, bridging the gap between the speed of rasterization and the quality of ray tracing. The open-source availability of the code further encourages exploration and experimentation by the research community.

This new research work offers subsurface scattering support for Gaussian Splats, a fast rendering technique that represents a scene as a collection of small Gaussian "bumps". While Gaussian Splats are incredibly fast and can handle real-world captured scenes, they previously lacked the ability to render high-quality translucent materials like skin, marble, or milk.

The proposed method extends Gaussian Splats to support subsurface scattering, allowing for the realistic rendering of translucent objects. Furthermore, it also enables relighting capabilities, where the appearance of an object can be adjusted to match different lighting environments. Users can even edit the materials, seamlessly transitioning between different appearances such as skin, glass, or wax.

This advancement is particularly exciting as Gaussian Splats are a relatively new technique, being around 2 years old. The combination of open research, human ingenuity, and a touch of AI has now made it possible to achieve reasonably high-quality translucent objects within these fast virtual worlds. This paves the way for next-level computer games and virtual experiences, as well as improved capabilities for self-driving car training, which often requires handling of complex camera models and rolling shutter effects.

The research presented in this work has demonstrated remarkable advancements in the field of virtual world rendering. By combining the strengths of rasterization and ray tracing, the 3DGUT technique offers a real-time solution that delivers high-quality reflections, refractions, and support for various camera models, including fisheye and rolling shutter. This breakthrough has the potential to revolutionize the development of computer games and virtual environments, as well as enhance the capabilities of self-driving car systems.

Furthermore, the integration of subsurface scattering techniques into the Gaussian Splats framework further expands the realism and visual fidelity of these virtual worlds. The ability to edit materials and relighting capabilities add to the versatility and creative potential of this approach.

The open-source nature of these advancements, with the source code made freely available, empowers the research community and enthusiasts to explore and build upon these innovations. This collaborative spirit and the accessibility of these cutting-edge techniques are truly inspiring, paving the way for the next generation of immersive and visually stunning virtual experiences.