Perceptual Lightness for Game Graphics
Overview
In recent months following conversations I had with some smart developers, I've found myself with a new interest in better color management in games. One tangential conversation related to this sent me down a rabbit hole about a perceptual phenomenon that I never really considered in dealing with graphics, which led to some neat utilities and considerations that I share below. My new home renderer is still in a very early stage, so rather than a game scene I used some helpful images from Martin Smekal on the ACES forums.The Helmholtz–Kohlrausch Effect
To save you the wiki click, "The Helmholtz–Kohlrausch effect (after Hermann von Helmholtz and V. A. Kohlrausch[1]) is a perceptual phenomenon wherein the intense saturation of spectral hue is perceived as part of the color's luminance." Or in other words, some colors appear brighter than others to our eyes, despite having identical luminance. We don't need to look any further than the image on the wiki page to know this is true:
Here, we have 5 colored boxes with the same luminance, and yet some of them appear brighter than others. We use luminance in shaders to do all sorts of things, but in doing so without accounting for HK, we're potentially ignoring a helpful piece of the puzzle. That's not to say that we've been using it "wrong" necessarily, but it does beg the question of whether there might be better (or at least complementary) tools for the tasks. To hammer this home, let's take a look at the following image:
And now the classic luminance dot product output:
Notice what happened to those blues and pinks? In the luminance output, they're showing up pretty dark, which you would expect given the contributions that the blue and red channels make towards luminance. Check out the piano and the floor in front of it, the central person's right leg, and the tea cup in the bottom right.
Accounting for the HK Effect
Not being anything resembling a color science expert, I started reading about ways to account for HK. In retrospect it seems obvious that people would create color spaces that are good at this kind of thing, but it wasn't until I read a few papers that I discovered them. The first one I came across that felt like what I was looking for was Predicting the Lightness of Chromatic Object Colors Using CIELAB (1991) by Mark Fairchild and Elizabeth Pirrotta. In this paper they describe the formulation of a model for calculating lightness in a way that accounts for the HK effect. I'm not going to regurgitate all of that research here of course, but the implementation is wonderfully simple, and the model is effective. The steps involved are:1) Transform the source color to CIELch.
2) Calculate the two new factors as described in the model.
3) Calculate the new lightness result, which I'll call FP-Lightness (FPL) to help disambiguate it from the rest of the terms with the same name.
Here's the shader source for how to do that. If we plug this in the output, by default we get something a little washed out, as lightness is roughly a cubic root of luminance in this model. To compare this with luminance, we'll pow(x, 3) the output and get something like this:
Tab back and forth between these 3 images to see just how much better many of these bright details are captured than with luminance. It might need some further tweaking for practical usage, but consider that in games, we often deal in higher saturation than we have in real life, which makes this lightness utility all the more helpful.
Applications for Lightness
Because of personal/professional work boundaries, I'm going to share the following potential applications for lightness as theoretical to save myself some trouble. Of course, all of this requires being okay with the additional operations required to get to something like FPL, it's hard to beat a single dot product!The first thing that came to my mind as an application of FPL (or similar calculations) is in auto-adaptive exposure in games. This is typically implemented by applying exposure adjustments towards some target based on global and/or local luminance of the game scene. Given what I mentioned above about our tendency to deal in higher saturation, we miss this component of HK by using luminance for the job. If we instead used HK-informed lightness as at least a complementary component to the equation, we would likely have a more pleasing result, and a system that is less likely to overexpose things we don't want.
Next is filtering. Sharpening and antialiasing come to mind, where we often make use of luminance to filter, and generally to discover areas of high vs low frequency details. Consider here that lightness may be a solid tool for discovering details that the average person's eyes would see, rather than purely based on emission from a display.
Another use case would be tonemapping, though being a challenging subject in itself, I think this would require a lot of exploration to get something that felt good for general purpose usage. This gets into my personal dogma about tonemappers, but to me the "right" way to tonemap involves moving away from the per-channel compression schemes that our industry is fond of, and instead towards better hue preservation (and separation of grading from compression) via mappings more akin to what Timothy Lottes described in his presentation Advanced Techniques and Optimization of VDR Color Pipelines, and Alex Fry in his presentation HDR color grading and display in Frostbite. In these models, some form of "lightness" (either luminance, or similar concept) is used in part for informing compression, and my instinct is that FPL or a similar model could refine this further.
Resources
If you search around the web for HK prediction, there's been quite a bit of research about this, and a number of different models. The one I've implemented above might not be the best one of the lot (I haven't tried them all), but it was a simple enough snippet that it felt worth sharing to demonstrate the impact this can have. Here are some related resources:Predicting the Lightness of Chromatic Object Colors Using CIELAB
https://www.academia.edu/13506981/Predicting_the_lightness_of_chromatic_object_colors_using_CIELAB
Prediction of the Helmholtz-Kohlrausch Effect Using the CIELUV Formula
https://onlinelibrary.wiley.com/doi/abs/10.1002/(SICI)1520-6378(199608)21:4%3C252::AID-COL1%3E3.0.CO;2-P
Apparent Greyscale: A Simple and Fast Conversion to Perceptually Accurate Images and Video
http://resources.mpi-inf.mpg.de/ApparentGreyscale/Perceptual_Greyscale.pdf