SIGGRAPH 15 – Physically Based and Unified Volumetric Rendering in Frostbite

作者：

Sebastien Hillaire – Electronic Arts / frostbite

sebastien.hillaire@frostbite.com

https://twitter.com/SebHillaire

• introduction
  

Physically based rendering in Frostbite

• See [Lagarde & de Rousiers 2014]
  
  • http://www.frostbite.com/2014/11/moving-frostbite-to-pbr/
    
• Huge increase in visual quality
  

基于物理渲染的效果非常好！

Volumetric rendering in Frostbite was limited

• Global distance/height fog
  
• Screen space light shafts
  
• Particles
  

体素渲染在这里还是受到限制的，主要受限于这三点

Real-life volumetric 真实的体素

我们期望做到的就是自然界中的这些 云与大气层，雾，光线散射等效果

• Related Work
  

Billboards

Analytic fog [Wenzel07]

Analytic light scattering(散射) [Miles]

特点：Fast，Not shadowed，Only homogeneous media

http://blog.mmacklin.com/2010/05/29/in-scattering-demo/

http://research.microsoft.com/en-us/um/people/johnsny/papers/fogshop-pg.pdf

http://indus3.org/atmospheric-effects-in-games/

Screen space light shaft 屏幕空间的光轴

• Post process [Mitchell07]
  
• Epipolar sampling [Engelhardt10]
  

特点

• High quality
  
• Sun/sky needs to be visible on screen
  
• Only homogeneous media 均匀介质
  
• Can go for Epipolar sampling but this won’t save the day
  

Splatting（泼溅）

• Light volumes
  
  • [Valliant14][Glatzel14][Hillaire14]
    
• Emissive volumes [Lagarde13]
  

This can result in high quality scattering but usually it does not match the participating media of the scene. （这种方法已经很常用了，但是相对独立处理）

Volumetric fog [Wronski14] 体积雾

• Sun and local lights
  
• Heterogeneous media
  

allowing spatially varying participating media and local lights to scatter.

spatially 参与 （scatter）散射，此做法与这边作者的想法一致

However it did not seem really physically based at the time and some features we wanted were missing.

缺点是不是很符合物理规则

• Scope and motivation
  

Increase visual quality and give more freedom to art direction!（更好的视觉效果）

Physically based volumetric rendering （物理）

• Meaningful material parameters
  
• Decouple(去耦合) material from lighting
  
• Coherent(一致性) results
  

We want it to be physically based: this means that participating media materials are decoupled from the light sources (e.g. no scattering colour on the light entities). Media parameters are also a meaningful set of parameters. With this we should get more coherent results that are easier to control and understand.

Unified volumetric interactions（交互）

• Lighting + regular and volumetric shadows
  
• Interaction with opaque, transparent and particles
  

Also, because there are several entities interacting with volumetric in Frostbite (fog, particles, opaque&transparent surfaces, etc). We also want to unify the way we deal with that to not have X methods for X types of interaction.

This video gives you an overview of what we got from this work: lights that generate scattering according to the participating media, volumetric shadow, local fog volumes, etc.

And I will show you now how we achieve it.

先秀结果（视频见投影片）

• Volumetric rendering
  

• Single Scattering
  

As of today we restrict ourselves to single scattering when rendering volumetric. This is already challenging to get right. （单看一条）

When a light surface interact with a surface, it is possible to evaluate the amount of light bounce to the camera by evaluating for example a BRDF. But in the presence of participating media, things get more complex. （一条光线与物理世界的交互是很复杂的）

1. You have to take into account transmittance when the light is traveling through the media（考虑光源到物体的传输介质影响）
   
2. Then you need to integrate the scattered light along the view ray by taking many samples（物体表面整合散射过来的光）
   
3. For each of these samples, you also need to take into account transmittance to the view point（考虑光从物体到相机的传输介质的影响）
   
4. You also need to integrate the scattered light at each position（相机各个位置收集所有散射结果）
   
5. And take into account phase function, regular shadow map (opaque objects) and volumetric shadow map (participating media and other volumetric entity)（考虑相位函数，普通阴影贴图（不透明的物体）和体积阴影贴图（与会媒体和其他体积实体））
   

公式里面存在两个积分标识的就是上面2，4两条解释的散射整合。

求和表示的是sample光线

• Clip Space Volumes
  

Frustum aligned 3D textures [Wronski14]

• Frustum voxel in world space => Froxel J
  

As in Wronski, All our volumes are 3d textures that are clip space aligned (such voxels become Froxels in world space, Credit Alex Evans and Sony ATG J, see Learning from Failure: a Survey of Promising, Unconventional and Mostly Abandoned Renderers for ‘Dreams PS4′, a Geometrically Dense, Painterly UGC Game’, Advances in Real-Time Rendering course, SIGGRAPH 2015).

Note: Frostbite is a tiled-based deferred lighting（平铺的延迟光照）

• 16×16 tiles with culled light lists
  

Align volume tiles on light tiles

• Reuse per tile culled light list
  
• Volume tiles can be smaller (8×8, 4×4, etc.)
  
• Careful correction for resolution integer division
  

This volume is also aligned with our screen light tiles. This is because we are reusing the forward light tile list culling result to accelerate the scattered light evaluation (remember, Frostbite is a tile based deferred lighting engine).

Our volume tiles in screen space can be smaller than the light tiles (which are 16×16 pixels).

By default we use

Depth resolution of 64

8×8 volume tiles

720p requires 160x90x64 (~7mb per rgbaF16 texture)

1080p requires 240x135x64 (~15mb per rgbaF16 texture)

• Data flow
  

This is an overview of our data flow.

We are using clip space volumes(使用裁剪空间体素) to store the data at different stages of our pipeline.

We have material properties(材料特性) which are first voxelised from participating media entities.

Then using light sources of our scene(场景光源) and this material property volume(材料特性体素) we can generate scattered light data per froxel. This data can be temporally upsampled to increase the quality. Finally, we have an integration(积分) step that prepares the data for rendering.

1. Participating media material definition (对应图上第一部分)
   

Follow the theory [PBR]

• Absorption 𝝈𝒂 (m^-1) 【吸收】
  

Absorption describing the amount of light absorbed by the media over a certain path length

• Scattering 𝝈𝒔 (m^-1) 【散射】
  

Scattering describing the amount of light scattered over a certain path length

• Phase 𝒈 【相位】
  

And a single lobe phase function describing how the light bounces on particles (uniformly, forward scattering, etc.). It is based on Henyey-Greenstein (and you can use the Schlick approximation).

• Emissive 𝝈𝒆 (irradiance.m-1) 【自发光】
  

Emissive describing emitted light

• Extinction 𝝈𝒕 = 𝝈𝒔 + 𝝈𝒂 【消失】
  
• Albedo 𝛒 = 𝝈𝒔 / 𝝈𝒕 【返照光】
  

Artists can author {absorption, scattering} or {albedo, extinction}

• Train your artists! Important for them to understand their meaning!
  

As with every physically based component, it is very important for artists to understand them so take the time to educate them.

(美术需要相关物理知识！)

Participating Media(PM) sources

• Depth fog
  
• Height fog
  
• Local fog volumes
  
  • With or W/o density textures
    

Depth/height fog and local fog volumes are entities(实体的) that can be voxelized. You can see here local fog volumes as plain or with varying density(密度) according to a density texture.

下面解释 数据结构及存储。

Voxelize PM properties into V-Buffer

• Add Scattering, Emissive and
  Extinction
  
• Average Phase g (no multi lobe)
  
• Wavelength independent 𝝈𝒕 (for now)
  

We voxelize(体素化) them into a Vbuffer analogous(类似的) to screen Gbuffer but in Volume (clip space). We basically add all the material parameters together since they are linear. Except the phase function which is averaged. We only also only consider a single lobe for now according to the HG phase function.

We have deliberately(故意) chosen to go with wavelength independent(波长无关) extinction(消失) to have cheaper volumes (material, lighting, shadows). But it would be very easy to extend if necessary at some point.

Supporting emissive is an advantage for artist to position local fog volume that emit light as scattering would do but that do not match local light. This can be used for cheap ambient lighting. (自发光是可选项)

1. 1 Froxel integration (对应图上第二部分)
   

Per froxel

• Sample PM properties data
  
• Evaluate
  
  • Scattered(稀疏的) light 𝑳𝒔𝒄𝒂𝒕(𝒙𝒕,𝝎𝒐)
    
  • Extinction
    

For each froxel, one thread will be in charge of gathering scattered light and extinction.

Extinction is simply copied over from the material. You will see later why this is important for visual quality in the final stage (to use extinction instead of transmittance for energy conservative scattering). Extinction is also linear so it will be better to temporally integrate it instead of the non linear transmittance value. (线性的 Extinction就够了)

Scattered light:

• 1 sample per froxel
  
• Integrate all light sources: indirect light + sun + local lights
  

Sun/Ambient/Emissive

Indirect light on local fog volume

• From Frostbite diffuse SH light probe
  
  • 1 probe(探测) at volume centre
    
  • Integrate w.r.t. phase function as a SH cosine lobe [Wronski14]
    

Then we integrate the scattered light. One sample per froxel.

We first integrate ambient the same way as Wronski. Frostbite allows us to sample diffuse SH light probes. We use one per local fog volume positioned at their centre.

We also integrate the sun light according to our cascaded shadow maps. We could use exponential(指数) shadow maps but we do not as our temporal up-sampling is enough to soften the result.

You can easily notice the heterogeneous nature of the local fog shown here.

Local lights

• Reuse tiled-lighting code
  
• Use forward tile light list post-culling
  
• No scattering? skip local lights
  

We also integrate local lights. And we re-use the tile culling(平铺剔除) result to only take into account lights visible within each tile.

One good optimisation is to skip it all if you do not have any scattering possible according to your material properties.

Shadows

• Regular shadow maps
  
• Volumetric shadow maps
  

Each of these lights can also sample their associated shadow maps. We support regular shadow maps and also volumetric shadow maps (described later).

1. 2 Temporal volumetric integration (对应图上第二部分)
   

问题：

scattering/extinction sample per frame

• Under sampling with very strong material
  
• Aliasing under camera motion
  
• Shadows make it worse
  

As I said, we are only using a single sample per froxel.

aliasing （下面两个视频见投影片，很明显的aliasing）

This can unfortunately result in very strong aliasing for very thick participating media and when integrating the local light contribution.

You can also notice it in the video, as well as very strong aliasing of the shadow coming from the tree.

解决：Temporal integration（时间积分）

To mitigate these issues, we temporally integrate our frame result with the one of previous frame. (well know, also used by Karis last year for TAA).

To achieve this,

we jitter our samples per frame uniformly along the view ray

The material and scattered light samples are jittered using the same offset (to soften evaluated material and scattered light)

Integrate (集成) each frame according to an exponential(指数) moving average

And we ignore previous result in case no history sample is available (out of previous frustum)

Jittered samples (Halton)

Same offset for all samples along view ray

Jitter scattering AND material samples in sync

Re-project previous scattering/extinction

5% Blend current with previous

Exponential moving average [Karis14]

Out of Frustum: skip history

效果很明显，先投影片视频。

仍然存在问题：

This is great and promising but there are several issues remaining:

Local fog volume and lights will leave trails when moving

One could use local fog volumes motion stored in a buffer the same way as we do in screenspace for motion blur

But what do we do when two volumes intersect? This is the same problem as deep compositing

For lighting, we could use neighbour colour clamping but this will not solve the problem entirely

This is an exciting and challenging R&D area for the future and I’ll be happy to discuss about it with you if you have some ideas J

1. Final integration
   

积分

Integrate froxel {scattering, extinction} along view ray

• Solves {𝑳𝒊(𝒙,𝝎𝒐), 𝑻𝒓(𝒙,𝒙𝒔)} for each froxel at position 𝒙𝒔
  

We basically accumulate near to far scattering according to transmittance. This will solve the integrated scattered light and transmittance along the view and that for each froxel.

代码示例

One could use the code sample shown here: accumulate scattering and then transmittance for the next froxel, and this slice by slice. However, that is completely wrong. Indeed there is a dependency on the accumScatteringTransmitance.a value (transmittance). Should we update transmittance of scattering first?

Final

Non energy conservative integration: （非能量守恒的集成）

You can see here multiple volumes with increasing scattering properties. It is easy to understand that integrating scattering and then transmittance is not energy conservative.

We could reverse the order of operations. You can see that we get somewhat get back the correct albedo one would expect but it is overall too dark and temporally integrating that is definitely not helping here.

So how to improve this? We know we have one light and one extinction sample.

We can keep the light sample: it is expensive to evaluate and good enough to assume it constant on along the view ray inside each depth slice.

But the single transmittance is completely wrong. The transmittance should in fact be 0 at the near interface of the depth layer and exp(-mu_t d) at the far interface of the depth slice of width d.

What we do to solve this is integrate scattered light analytically according to the transmittance in each point on the view ray range within the slice. One can easily find that the analytical integration of constant scattered light over a definite range according to one extinction sample can be reduced this equation.

Using this, we finally get consistent lighting result for scattering and this with respect to our single extinction sample (as you can see on the bottom picture).

• Single scattered light sample 𝑆=𝑳𝒔𝒄𝒂𝒕(𝒙𝒕,𝝎𝒐) OK
  
• Single transmittance sample 𝑻𝒓(𝒙,𝒙𝒔) NOT OK
  

è Integrate lighting w.r.t. transmittance over froxel depth D

Also improves with volumetric shadows

You can also see that this fixes the light leaking we noticed sometimes for relatively large depth slices and strongly scattering media even when volumetric shadow are enabled.

Once we have that final integrated buffer, we can apply it on everything in our scene during the sky rendering pass. As it contains scattered light reaching the camera and transmittance, it is easy to apply it as a pre-multiplied colour-alpha on everything.

For efficiency, it is applied per vertex on transparents but we are thinking of switching this to per pixel for better quality.

• {𝑳𝒊(𝒙,𝝎𝒐), 𝑻𝒓(𝒙,𝒙𝒔)} Similar to pre-multiplied color/alpha
  
• Applied on opaque surfaces per pixel
  
• Evaluated on transparent surfaces per vertex, applied per pixel
  

Result validation

Our target is to get physically based results. As such, we have compared our results against the physically based path tracer called Mitsuba. We constrained Mitsuba to single scattering and to use the same exposure, etc. as our example scenes.

Compare results to references from Mitsuba

• Physically based path tracer
  
• Same conditions: single scattering only, exposure, etc.
  

The first scene I am going to show you is a thick participating media layer with a light above and then into it.

You can see here the frostbite render on top and Mitsuba render at the bottom. You can also see the scene with a gradient applied to it. It is easy to see that our result matches, you can also recognize the triangle shape of scattered light when the point lights is within the medium.

This is a difficult case when participating media is non uniform and thick due to our discretisation of volumetric shadows and material representation. So you can see some small differences. But overall, it matches and we are happy with these first results and improve them in the future.

This is another example showing very good match for an HG phase function with g=0 and g=0,9 (strong forward scattering).

Performance

Sun + shadow cascade

14 point lights

• 2 with regular & volumetric shadows
  

6 local fog volumes

• All with density textures
  

PS4, 900p

You can see that the performance varies a lot depending on what you have enabled and the resolution of the clip space volumes.

This shows that it will be important to carefully plan what are the needs of you game and different scenes. Maybe one could also bake static scenes scattering and use the emissive channel to represent the scattered light for an even faster rendering of complex volumetric lighting.

• Volumetric shadows
  

Volumetric shadow maps

We also support volumetric shadow maps (shadow resulting from voxelized volumetric entities in our scene)

To this aim, we went for a simple and fast solution

• We first define a 3 levels cascaded clip map volume following and containing the camera.(定义三个跟随相机的体)
  
  • With tweakable per level voxel size and world space snapping
    
• This volume contains all our participating media entities voxelized again within it (required for out of view shadow caster, clip space volume would not be enough)
  
• A volumetric shadow map is defined as a 3D texture (assigned to a light) that stores transmittance
  
  • Transmittance is evaluated by ray marching the extinction volume
    
  • Projection is chosen as a best fit for the light type (e.g. frustum for spot light)
    
• Our volumetric shadow maps are stored into an atlas to only have to bind a single texture (with uv scale and bias) when using them.
  

Volumetric shadow maps are entirely part of our shared lighting pipeline and shader code.

Part of our common light shadow system

• Opaque
  
• Particles
  
• Participating media
  

It is sampled for each light having it enabled and applied on everything in the scene (particles, opaque surfaces, participating media) as visible on this video.

（这边可以看PPt效果视频）

Another bonus is that we also voxelize our particles.

We have tried many voxelization method. Point and its blurred version but this was just too noisy. Our default voxelization method is trilinear(三线性). You can see the shadow is very soft and there is no popping(抛出) visible.

We also have a high quality voxelization where all threads write all the voxels contained within the particle sphere. A bit brute force for now but it works when needed.

You can see the result of volumetric shadows from particle onto participating media in the last video.

(See bonus slides for more details)

Quality: PS4

Ray marching of 323 volumetric shadow maps

1k particles voxelization

Point lights are more expensive than spot lights because spot lights are integrated slice by slice whereas a full raytrace is done for each point light shadow voxels. We have ideas to fix that in the near future.

Default particle voxelization is definitely cheap for 1K particles.

• More volumetric rendering in Frostbite
  

Particle/Sun interaction

• High quality scattering and self-shadowing for sun/particles interactions
  
• Fourier opacity Maps [Jansen10]
  
• Used in production now
  

Our translucent(半透) shadows in Frostbite (see Andersson11) allows particles to cast shadows on opaque surfaces but not on themselves. This technique also did not support scattering.

We have added that support in frostbite by using Fourier opacity mapping. This allows us to have some very high quality coloured shadowing, scattering resulting in sharp silver lining visual effects as you can see on this screenshots and cloud video.

This is one special case for the sun (non unified) but it was needed to get that extra bit of quality were needed for the special case of the sun which requires special attention.

Physically-based sky/atmosphere

• Improved from [Elek09] (Simpler but faster than [Bruneton08])
  
• Collaboration between Frostbite, Ghost and DICE teams.
  
• In production: Mirror’s Edge Catalyst, Need for Speed and Mass Effect Andromeda
  

We also have added support for physically based sky and atmosphere scattering simulation last year. This was a fruitful collaboration between Frostbite and Ghost and DICE game teams (Mainly developed by Edvard Sandberg and Gustav Bodare at Ghost). Now it is used in production by lots games such as Mirror’s Edge or Mass Effect Andromeda.

It is an improved version of Elek’s paper which is simpler and faster than Bruneton. I unfortunately have no time to dive into details in this presentation.

But in the comment I have time J. Basically, the lighting artist would define the atmosphere properties and the light scattering and sky rendering will automatically adapt to the sun position. When the atmosphere is changed, we need to update our pre-computed lookup tables and this can be distributed over several frame to limit the evaluation impact on GPU.

• Conclusion
  

Physically-based volumetric rendering framework used for all games powered by Frostbite in the future

Physically based volumetric rendering

• Participating media material definition
  
• Lighting and shadowing interactions
  

A more unified volumetric rendering system

• Handles many interactions
  
  • Participating media, volumetric shadows, particles, opaque surfaces, etc.
    

Future work

Improved participating media rendering

• Phase function integral w.r.t. area lights solid angle
  
• Inclusion in reflection views
  
• Graph based material definition, GPU simulation, Streaming
  
• Better temporal integration! Any ideas?
  
• Sun volumetric shadow
  
• Transparent shadows from transparent surfaces?
  

Optimisations

• V-Buffer packing
  
• Particles voxelization
  
• Volumetric shadow maps generation
  
• How to scale to 4k screens efficiently
  

For further discussions

sebastien.hillaire@frostbite.com

https://twitter.com/SebHillaire

References

[Lagarde & de Rousiers 2014] Moving Frostbite to PBR, SIGGRAPH 2014.

[PBR] Physically Based Rendering book, http://www.pbrt.org/.

[Wenzel07] Real time atmospheric effects in game revisited, GDC 2007.

[Mitchell07] Volumetric Light Scattering as a Post-Process, GPU Gems 3, 2007.

[Andersson11] Shiny PC Graphics in Battlefield 3, GeForceLan, 2011.

[Engelhardt10] Epipolar Sampling for Shadows and Crepuscular Rays in Participating Media with Single Scattering, I3D 2010.

[Miles] Blog post http://blog.mmacklin.com/tag/fog-volumes/

[Valliant14] Volumetric Light Effects in Killzone Shadow Fall, SIGGRAPH 2014.

[Glatzel14] Volumetric Lighting for Many Lights in Lords of the Fallen, Digital Dragons 2014.

[Hillaire14] Volumetric lights demo

[Lagarde13] Lagarde and Harduin, The art and rendering of Remember Me, GDC 2013.

[Wronski14] Volumetric fog: unified compute shader based solution to atmospheric solution, SIGGRAPH 2014.

[Karis14] High Quality Temporal Super Sampling, SIGGRAPH 2014.

[Jansen10] Fourier Opacity Mapping, I3D 2010.

[Salvi10] Adaptive Volumetric Shadow Maps, ESR 2010.

[Elek09] Rendering Parametrizable Planetary Atmospheres with Multiple Scattering in Real-time, CESCG 2009.

[Bruneton08] Precomputed Atmospheric scattering, EGSR 2008.