Cartoony fog (depth + sobel based?)

I would like to recreate a 2D-like depth fog effect in 3D.

The regular fog simply adds the fog color on top of the object's texture based on its depth in the scene. But in this 2D style, you don't really add a gradient based on depth, but rather add a solid color to entire (and closed) areas of the image (think of it as making a selection in MSPaint, and using the bucket tool on it). This is possible in 2d because you can place hills and objects in a way that covers the depth transition between them (so they look more like layers, than like a 3D enviroment)


(In this image, you can see how everything is one color or another. There is no transition)

The first thing I tried was to make a fog shader with a posterized depth gradient (so that it would look like bands of color, instead of a gradient), but the cut from one color to another could happen anywhere in the world (in the middle of a flat ground, a few steps ahead of you, for instance. Or cutting the arm of the monster in the picture, since it's a bit closer to the camera), giving confusing silhouette information (the whole point of this effect is to make silhouettes stand out, like in the previous picture).
Instead, the cut should somehow follow the silhouette of the objects within some distance from the camera.

https://gyazo.com/adaa8757cf593021bb0e2a655a8ee24b
(Here you can see how the cuts don't always look good by just posterizing the depth gradient)

After thinking about it for a long time, the only solution I could come up with is to break the picture into areas using a sobel depth effect (creates a black texture, and adds white in high contrast areas (depth contrast, in this case). It's commonly used for outline effects), and somehow filling each section with a solid color, based on the overall depth of that section (again, like the fill selection tool in MSPaint).

https://gyazo.com/8f7e761e308126acf034538881e7c8d3
(This is an approximation of how it would look with a sobel effect applied)

I think it is possible to achieve, but I don't have enough programming knowledge, so I would appreciate some guidance.
To put it simple, I think the steps would be:
-Render a sobel depth texture
-Assign a single depth value to each enclosed section of it (convert gradients to an overall value per-section)
-Assign a fog color to those values (a simple gradient mapping should be enough)

If you know how to do this, or if you can come up with a better approach, please let me know.
Thank you

 

Using sobel is a nice idea, except it won't actually work because that's not how sobel works. Sobel finds edges, but doesn't guarantee enclosed shapes.

This is what sobel will really do.


What you're talking about is more something like a flood fill. This isn't something a normal shader can do, you'd have to do it with a compute shader which makes this much more complex. There's a lot of difficult issues to deal with here too.

You're best options going to be going with a stylized fog shader, something that's halfway between the posterized, hard shapes and the smooth transitions. A fully hard edged look like you want will always have the problem of objects popping in as they get closer. The other thing is generally you only want two or three steps between the foreground and the background colors that aren't separated by equal, linearly distanced steps, you want a foreground, a mid ground, and one maybe two levels of background.

See this video at around 12 minutes in to see what they did for Firewatch.

There's a couple of stylized fog image effects that have been released on the asset store and elsewhere for free that offer similar control to what Firewatch did.

 

I decided to follow your advice, and I made a custom fog shader, similar to Firewatch's. In my game there will only be 3 color layers, so instead of a ramp texture, I made it so that you can select the colors and smoothness from the inspector.
I also tried a few other effects, so here's how the process went:

http://imgur.com/GqVWdZn
Here I made the 3 layers.
The layer "0" is the gameplay area (where the player will be focused on most of the time), so it has texture detail multiplied by the layer's color (green).
The layer "1" is out of combat range, but the player might still need to tell what's what, so a solid color like this wasn't good enough (that's what the next pictures are all about).
The layer "2" is too far away to care, and should work only as a landmark, to orientate yourself in a general direction (a montain at the horizon, a castle in the distance...).

http://imgur.com/7GpL6oG
First thing I noticed was that, when moving, the transition from layer 0 to 1 was running over objetcs very fast, taking my attention away from the gameplay area (the effect was similar to a silhouette moving at the edge of your vision cone).
I made the transition smoother, so that the line wasn't that obvious.

http://imgur.com/MB2qUS1
But on larger objects (like entire buldings, their roofs...) you could still see a straight line (even though smooth) transitioning all over the surface.
So I multiplied the transition gradient by a world-space noise texture, which made it look like a dissolve shader as you approach objects. It looks much less artificial

http://imgur.com/xPMdJyU
Then I tried a fresnel on layer 1, so that the player would be able to tell what the silhouettes were.
But I didn't like the effect (it looked like pillow shading), so I scrapped it.

http://imgur.com/CM4ot54
Instead, I decided to go with a regular outline

http://imgur.com/8yxa7tx
And finally, I made it so that areas above you get into layer 2 sooner than if in front of you.
The reason behind this is that in my game, those areas are out of your reach (you can't jump too high and get onto a rooftop, for instance), so even if you're close to the base of that building, the roof shouldn't look like part of the gameplay area at that time (layer 0)


After that, I had a lot of fun trying out color palettes
http://imgur.com/a/70v3g
(The grass didn't take the shader, because it was Unity's grass system, which can't use custom shaders, as far as I know. But that gave me the idea of making important objects/enemies keep their shading when in layer 2, so that they pop out a bit)

Anyways, thank you for your help