Reframing "Tonemapping" in Games
It's time for games to let go of the term "tonemapping." The terms "display mapping" or "display rendering" as part of a broader stage of "image formation" is a much more appropriate way to communicate the intention of this part of a rendering pipeline. Why is this important? Because accurately
describing what we're doing makes our goals more apparent and can shine a light on where some of our core assumptions may be wrong.
There is no "tone" being "mapped", there is no way to view the tristimulus data generated in a variable-dynamic-range buffer except through the display as a formed image.
Doing nothing counts as image formation when we submit it to the display, letting channels clip and presenting it as color is just very poor image formation.
Until we've completed the image formation step, we are not working with "color", we are working with open domain tristimulus data. We should stop calling this "color" prior to this step.
The fact that data produced in an average game scene mostly-sort-of looks like an image is purely coincidental due to the fact that we do our work with tristimulus representation and most of it falls within the 0-1 displayable range.
Even if the whole thing doesn't fall within that range, the graphics programmer is typically examining the data through graphics profilers like PIX which "tonemap" the data for us, furthering the illusion that the data is color and on its own is already an image. It isn't!
It's time to let go of the purely per-channel mappers of the past. ACES, Uncharted, Reinhard, etc, it's time we move on. We still respect their place in our history as effective high performance solutions at a time when full screen processes were more expensive than they are today, producing "good enough" results while we
explored and unpacked the advent of high-dynamic-range rendering and physically based shading. However, it's time to acknowledge that we can do better than a simple 2-3 line per-channel S-curve, and with modern hardware there is no good excuse not to. Give image formation the same reverence and attention to detail that we give to physically based shading.
The purely per-channel mappers effectively bake a look transform into the compression transform, in ways artists will never truly be able to account for. This leads to artists influencing their source art in DCCs to fit the mapper, rather than the other way around. Any perceived pleasing look provided by a per-channel mapper could just as well have been authored instead through a combination of source art and look transforms,
and why wouldn't we want that? If artists want those hue shifts, they can do that themselves, what job are we accomplishing that wouldn't be better done intentionally in a separate look transform? Take a bright asset and screenshot it with a per channel mapper in various lighting situations and see how very different the result looks in each scenario, and ask honestly how an artist is meant to account for that, and if
"good enough" is still really "good enough".
An alternative ideal might be some kind of "neutral" mapping, more specifically a focused reduction of techniques that noticeably alter hues from the source assets. I'm not talking about taking those same functions and slapping luminance into them instead, either. More something in the realm of desaturation with a preservation of lightness (turns out this is much harder than it sounds!). But that's just one example, this is an open area of research with so much to consider, I'm barely scraping the surface here.
And then, if/when we want to make it look "filmic" or more generally apply looks and grading, we apply additional transformations as separate steps. This can occur before or after the image formation, likely both! But it shouldn't be built into the display mapper. If you want your engine to provide a particular look by default, or at least options, then offer them separately.
I generally do not like to pull examples from published games or engines for a number of reasons, so please see this one from Tunic with nothing but positive intent, it was just a game I played very recently (and you should too, it's fantastic!).
Can you spot the members of the Notorious Six that have made an appearance here? Here's a hint:
But who could blame them? This is no doubt using the recommended default "tonemapping" from the engine. We see stuff like this in almost every game released, from proprietary and publicly available engines alike. Once you start looking for it you never unsee it - that strange shift in the way your character looks when they walk out into the sunlight, or the way the sky seems to
blow out to cyan when the auto-exposure kicks in.
We can do better! And we should, we've spent all this time and effort on the math to get that super sexy high quality lighting, only to frequently blow it when we resolve the result. Don't take my word for it though, follow some color science misfits (I mean this in a good way) in our space that have been doing
excellent work pushing this subject forward.
Presentations and/or code for examples of alternative display mapping techniques, which you should look to as a starting point rather than copy/pasting ACES again:
Advanced Techniques and Optimization of VDR Color Pipelines by Timothy Lottes.
HDR Color Grading and Display in Frostbite by Alex Fry.
"notorious6" display mapping demo from Tomasz Stachowiak and Gray Olson.
kajiya, from Tomasz Stachowiak/Embark that improves on the work from the above demo.
Tony McMapface from Tomasz Stachowiak.
AgX from Troy Sobotka ported to a simplified shadertoy.