RESOURCE: Unreal Engine Integrated Cel Shading (Engine Fork)
Unreal Engine really wants you to make PBR stuff. It’s really great at it, but if you’re like me, you probably wish for more flexibility in art direction implementation. A great area of exploration in Unreal with regard to stylization is the use of cel-shading.
Typically, cel-shading in Unreal Engine is achieved by way of post process materials that, to crudely summarize, take the final image output & estimate image luminance as a gradient, quantize or threshold that gradient, and use that as a mask to scale the luminance of the base color buffer.
While this approach canwork, it is riddled with caveats and restrictions. (doesn’t play nicely with translucent surfaces or fog, light coloration is lost, other post process effects like exposure or bloom can muddle with the result, the list goes on.)
The ideal scenario would be to dig deeper into the rendering pipeline, to affect how surface self-shading is applied, but this step in the process is pretty heavily obscured from most users and requires a lot of digging & engine customization to solve for.
A couple years ago, a really great blog post from user One3y3 on the UE forums gave substantial breakdowns on how to modify the engine to add a custom shading model, and where to look to make edits to the engine shader files to allow for more custom lighting responses.
Piggybacking off of some of this information and extending the ethos further, I was able to create an engine fork that includes a custom cel-shading model with an accompanying material output node, similar to something that Single Layer Water has, only specific to cel-shading.
The original build for this was done on 5.2, but I’ve since migrated it to 5.5 as well. With 5.8 providing a custom stylized shading model that expands even further, the future of more custom stylized rendering in Unreal looks bright, however for those who are interested in learning more about how these things function under the hood, I’ve linked my fork of 5.5 that has this custom shading model built-in.
