When you think about Ray Tracing, most people think about graphics. They think about light, shadows, and crepuscular rays. But the technology can be used for so much more – including simulating how sound travels in a game.
“Ray Tracing is commonly used for indirect lighting,” Kasparas Eidukonis, Senior Audio Programmer, says. “Like when you shine a light into a room – maybe it bleeds out through the door and highlights the areas outside, maybe it bounces on walls and creates different colors of light based on the materials it hits.”
“For audio, it’s essentially the same. Light can be described as a wave, just like sound, it just behaves differently when it hits a surface. We shoot out rays that bounce around and propagate sound. And the more rays we shoot out, the clearer an idea of what it should sound like we get.”
Light can be described as a wave, just like sound, it just behaves differently when it hits a surface.
Let’s imagine you sit opposite another person at a table. If you talk, you can clearly hear each other. The sound waves don’t have an obstacle that separates the two of you. Place a lamp on the table, the sound changes slightly. Flip the table over and suddenly you have something between you and the person on the other side, muffling sounds. Is there a door? Is it open? You can keep building on the scenario, and everything you add will bounce waves – or rays, in the case of Ray Tracing – back to your ears, all in different ways. That’s sound propagation.
Now teleport yourself to the middle of the Kinglor Forest. Consider all the trees, plants, rocks, animals, flowing water, birds, enemy bases, and machinery. Imagine what they do to the thousands of rays the game sends out every frame and what it means for the audio experience.
“It’s super important for immersion to get the sound propagation right,” Simon Stevnhoved, Senior Sound Designer, says. “We are evolutionarily trained to be aware of our surroundings, and our ears can pick up on the slightest changes in our environment. When sound propagation works well, you might not pick up on it consciously, but when something is off, it quickly becomes noticeable. You know instinctively that if you make a loud sound – like firing a gun next to a canyon – the sound will reflect a certain way.
We are evolutionarily trained to be aware of our surroundings.
The team started out by using propagation tech adapted from Tom Clancy’s The Division 2, such as the Slapback, Bubblespace, and Obstruction systems. But, as work on Avatar: Frontiers of Pandora progressed, they realized that they needed to both improve what they already had and build completely new tech for the game. While looking for ways to improve sound obstruction, GPU Ray Tracing was integrated into Snowdrop, and the audio team saw their opportunity to jump on board.
“The Division 2’s solution wasn’t built for a jungle environment,” Kasparas says. “It was designed for a city environment, with blocky buildings and props. In Avatar: Frontiers of Pandora, the geometry is much more varied, with trees and leaves and bushes.”
So, the team got to work on building a small prototype for raytraced sound obstruction, which worked well for walking around the Kinglor Forest, hiding behind trees and tree trunks, and listening to water sounds. They were happy with what they had but knew there were a lot of improvements that could be made. Some environments worked better than others, for example. The team ended up with a hybrid system, mainly relying on the Ray Tracing solution, but still utilizing hand placed data which was already available, such as volumes for inside spaces and doors.
“In the old system, we would have a custom-built solution for each type of environment, for each type of plant, rock, cliff, terrain,” Kasparas says. “The advantage we were seeing with the Ray Tracing solution is that we could do much more with the rays. We wouldn’t need to build those custom systems, instead it would see all those things naturally. We didn’t need to edit a rock and say, ‘this rock is solid’ or ‘this bush is semitransparent’ every time – naturally the bush will have gaps and the rays can pass through it better than the rock.”
Another bonus of utilizing Snowdrop’s Ray Tracing system as a basis was that the team could move the process from the processor (CPU) to the graphics card (GPU), allowing them to create and sample a lot more rays, faster. Moving parts of the process to a separate piece of hardware also frees up the CPU to do other things. And being released exclusively on the current generation of consoles – and PC – helped, as there’s suddenly a lot more memory and power to take advantage of.
For an early prototype of the system, the audio team tried a very realistic approach to sound propagation – trying to mimic how sound propagation works in the real world as closely as possible. While it created an impressive soundscape, it also became too overbearing to be useable in a game.
You still need to tell a story; you still need to understand gameplay. It needs to be approachable.
“You would end up with sensory overload,” Alex Riviere, Audio Director, says. “It’s a lot of sounds: from insects to birds, foliage rustling with the wind, cues for gameplay and navigation, to name a few. It became too much to take in and understand. It sounded great, but any new player would be overwhelmed in that situation. So, we had to take a step back and iterate on the sound mix to make it less overbearing. You still need to tell a story; you still need to understand gameplay. It needs to be approachable.”
There’s still a lot of exploration left in the audio field. But now, we have the opportunity to enter the various vast (and virtual) environments and landscapes of Pandora and immerse ourselves in the diverse sounds of the flora and fauna you can only find there.
Avatar: Frontiers of Pandora™ © 2023 20th Century Studios. Game Software excluding 20th Century Studios elements: © 2023 Ubisoft Entertainment. All Rights Reserved. Avatar: Frontiers of Pandora™ and the 20th Century Studios logo are trademarks of 20th Century Studios. Licensed to Ubisoft Entertainment by 20th Century Studios. Ubisoft and the Ubisoft logo are registered or unregistered trademarks of Ubisoft Entertainment in the U.S. and/or other countries.
