A Failed Adventure in Avoiding Temporal Anti-Aliasing
Image credit: SMAA: Enhanced Subpixel Morphological Antialiasing
Overview
Unlike my usual tutorialized posts, this one serves as a post-mortem and stream of consciousness on 5ish months of spare time here and there that I spent on trying to find a way to avoid a need to use temporal anti-aliasing in a production context. The bar I wanted to meet was getting to a point where I could say "yes, I would ship with this, without some form of main-pass temporal AA, without regretting it." As my title implies, and surely to the surprise of very few, I didn't get there. I did, however, feel comfortable with the solution I ultimately settled on, which I describe down below. Keep in mind while reading that often when I discuss TAA, I'm talking about main-pass TAA, meaning you use TAA to resolve your forward or deferred passes for the bulk of your scene geometry. I was never under the impression that there were other, better options for resolving stochastic techniques and Monte Carlo integration - for those techniques you simply need more samples and temporal accumulation is currently the best way to get there. I did, however, believe that with modern hardware I might find a happy medium where TAA was only used to resolve things on a per-technique basis rather than everything.Why Not Main Pass TAA?
TAA has many fantastic and far reaching benefits that I'm not going to spend time getting into here, there are plenty of resources for that. What I will say is I completely understand why people find it so attractive as an antialiasing solution, and in fact for many it may be their only reasonable option. My premise for avoiding main pass TAA largely came down to the following reasons:-Smearing/ghosting from games with TAA can cause motion sickness for some players, even in games with high framerate (though it's not as bad at 60+ fps). In fact, I'm one of them!
-Managing the balance of ghosting and other undesirable artifacts (like flickering from limited history) is a pain that basically never ends and needs plenty of per-game tuning.
-FX and tech artists are sad when their transparent effects, uv animated objects, etc look bad because TAA messes with them.
-Purposefully high frequency details in materials like from normal maps or glints tend to get lost in TAA along with sharpness, even after returning some of it with post-TAA sharpening.
-The above points usually mean that artists and programmers are spending non-negligible amounts of time fighting TAA throughout production, sometimes in especially undesirable ways (like inconsistent shading functionality).
-TAA quickly becomes a crutch upon which many features rely on to look at all reasonable. This is a positive, speaking to TAA's versatility and ability to explore techniques that are otherwise not possible. It's also a con, because we start trivially banking on solving problems that might be solved in different ways that don't introduce some amount of sharpness loss and smearing/ghosting. Or further, people get too comfortable and start grossly undersampling things assuming TAA will make it good enough.
-When TAA is a cornerstone of the main pass (and often the whole renderer), it leaves no options for user preferences. Indeed, games are increasingly released without AA options at all apart from TAA, and some won't even let you just disable it because it is so integral to the game.
Clearly there were reasons to want to avoid TAA, or at least to avoid putting all eggs into the TAA basket.
What Other Options Are There?
In a renderer absent TAA, what options do we have to solve the different kinds of aliasing we're going to run into? Most common issues will fall into categories of geometric aliasing, specular aliasing, material aliasing (ie runtime texture blending), and temporal aliasing. We have MSAA (as a part of the main pass) that tackles spatial anti aliasing along geometric edges, but it only does that. I haven't tried this myself, but presumably it would also be good for more than that, in the case of object/texture space shading. FXAA is a post process that tackles some degree of geometric, specular, and material aliasing, but with the complex shading we have today, it struggles a lot. We have LEAN, CLEAN and Toksvig AA prefiltering for handling specular highlights, thereby reducing specular aliasing. We have runtime SAA which is good at filtering the NDF, reducing specular aliasing as well (though it needs to be combined with some smart filtering of other contributors to specular aliasing like reflections). We have SMAA and its evolutions - 1x, T2x, 1Tx, S2x, 4x, Filmic, which to varying degress (depending on the mode) tackles all the primary aliasing issues. Finally, we still have SSAA if we want to take the problem and hit it with a large but effective hammer.MSAA Isn't Dead
Side topic. I see this take now and then that MSAA is dead, and I don't agree with it. There are enough great looking games that ship successfully with MSAA (combined with other techniques) to know it still holds value. Not everyone runs deferred renderers, not everyone ships games full of micropolys, and not everyone subscribes to full on shader graph style material authoring. Some studios have the ability to ensure tight control over their pipelines to inherently minimize aliasing issues that MSAA does not address. Some even use it is a utility for object/texture space shading, though usually selectively for things like hair. That's not even to mention mobile and VR spaces, where MSAA is essential and sees plenty of usage. Regardless, if you have a forward renderer (which is my personal preference) MSAA is likely readily available for your main pass, and it's probably not as expensive on performance as you remember from older hardware. With considerations for tonemapping, and a wide filter in the resolve, MSAA does quite a good job (and don't forget about the underrated alpha-to-coverage!), the key is that you don't simply stop here. MSAA must be augmented with other AA techniques and authoring considerations in order to be successful nowadays, but if implemented at all, you've already provided your players with MSAA as an option if they struggle with issues introduced by temporal techniques. To me this is a huge deal when it comes to accessibility, and that is too easily overlooked.What I Tried
With the plethora of games we have today that exclusively use TAA, it is genuinely a relief to the eyes when I jump into games like Overwatch and Destiny 2, which make use of straight up SMAA instead. Images are sharp, silhouettes aren't overly soft, and they just look so clean.
Of course, there are certain cases where these games drown in aliasing, but I can't help but wonder if (as an industry) we've lost sight of how good games can look without TAA. What I wanted to try was a combination of techniques that could reasonably stack up to TAA while maintaining image clarity and addressing the major issues. Being a forward rendering fan, I started with an MSAA resolve like I mentioned above. Next was hitting specular aliasing, and I was largely satisfied with runtime geometric SAA when combined with careful management of specular radiance usage in materials, though there was still temporal specular aliasing when in motion. Then... I got to material aliasing from runtime material blending, a common operation when dealing with layered materials, and while you can manage this while things are still, this (like specular) can fall over in motion, and after a lot of time spent I just couldn't see any way to deal with this without a temporal component. To save on text, what I ended up doing was selective edge detection + pattern recognition ala morphological anti-aliasing, and blending alternating jittered frames of this. Sound familiar? It was basically just a poor person's SMAA T2x, but I liked how much sharpness was retained in the image when in motion. I wondered if this could be combined with the MSAA I was already using (though had disabled while trying this), and sure enough when looking through the SMAA github I found S2x (which uses MSAA 2x) and 4x (which is T2x + S2x). I've been fiddling with these methods since then, and feel hopeful about it.
End Result
In the end I think I'm mostly settled on MSAA + runtime SAA + general authoring considerations + some form of temporal SMAA. MSAA because it's a great way to get extra samples where they matter, runtime SAA to help with specular without needing to involve extra data, and temporal SMAA because I failed in my quest to avoid temporal anti-alasing! I did, however, manage to avoid straight-up TAA. It's simply needed in order to help manage temporal aliasing in materials and specular. There have been great advancements in this technique since the early T2x implementation, with Activision for example having shared their excellent Filmic SMAA and its evolution over time. Truth be told I'm not sure why more studios aren't adopting this methodology, since the morphological component makes for a massive improvement in sharpness/clarity over the more common style of TAA. Likely this just comes down to TAA being used to resolve lots of other effects onto the main scene and relying on that, and for me to handle this I would need to selectively use TAA on a per-pass basis and merge results with the main scene (which costs performance), but frankly that's what I would prefer to do anyway. Additionally, what I love about this mixture is that you can offer users a choice of MSAA 2x/4x/8x, SMAA 1x, T2x, S2x, 4x, Filmic and whatever evolution of you make of that. That's a lot of choice! Great for accessibility, while still providing paths that include temporal anti-aliasing. To me, Activision's work with Filmic SMAA in CoD is simply the best of the state of the art solutions available, and I appreciate that they still offer their players choices for their AA options.Passing Thoughts
-I wonder if more bread-and-butter TAA implementations could be made friendlier by at least offering players control over the number of subsamples/contributing frames. Some kind of sliding bar between accepting more aliasing and flickering vs accepting more ghosting and softening.-I wonder how much of a contributing factor TAA plays in the anti-30fps audience, considering TAA at 30 fps can look and feel pretty bad depending on how it is used.
-I get the feeling the high variability in displays nowadays can make TAA feel worse than it otherwise would with the average player who (probably) never bothers to tweak their display settings. Settings like OverDrive meant to combat ghosting/trailing artifacts at different refresh rates can introduce their own kinds of ghosting/trails on top of whatever the TAA'd image looks like, and no doubt there are many who have no idea what this phenomenon is or that it is caused by their display and they can change the setting.
-I've been fortunate enough to be able to snag a 3080, and there are a number of games I'm now able to comfortably play at 1440p with just SSAA. Maybe we'll see more of this in the not-so-distant future?
Resources
http://www.iryoku.com/smaa/https://github.com/iryoku/smaa
https://research.activision.com/publications/archives/filmic-smaasharp-morphological-and-temporal-antialiasing
https://research.activision.com/publications/2020-03/dynamic-temporal-antialiasing-and-upsampling-in-call-of-duty
https://therealmjp.github.io/posts/msaa-overview/
https://therealmjp.github.io/posts/msaa-resolve-filters/
http://www.jp.square-enix.com/tech/library/pdf/ImprovedGeometricSpecularAA(slides).pdf
https://bgolus.medium.com/anti-aliased-alpha-test-the-esoteric-alpha-to-coverage-8b177335ae4f
http://graphicrants.blogspot.com/2013/12/tone-mapping.html