Edge Detection via Shader not Image Effect

Hello Everyone,

So I've been playing around with the unity edge detection image effect script but cant get it to provide the look i want. First I can only get the sobel depth and sobel depth thin to work, the other options show no edge detection at all, no matter what i change the values to.

The Sobel Depth look ok, but pretty much misses any edge that doesn't have a huge distance difference (which is like 75% of the angles).

I would like to highlight EVERY edge of a given object, and I'm looking to do this in the shader. Ive done some googling and some forum searching, but i always end up getting pointed in the direction of the edge detection script that i don't want to use.

Does anyone have any ideas how to go about achieving this within the shader itself? Perhaps some links for reference? Im pretty sure i get the basic idea, im just new to shaders so im not sure how to go about it.

I think i need to in my first pass create a secondary texture that i store the normal of the texel in. In a second pass i need to read this texture and somehow determine the texels around it and check for variations in the normal values. Variations outside a certain range mean its an edge, and i should change the pixel color to my edge color.

Ill be honest though, i don't get shaders enough to understand passes, or how to get values from a texture outside the current on screen texel im processing in the fragment function.

Well, hopefully someone can shoot me off in the right direction, thanks a head of time.

 

There are a few main ways to go about drawing outlines / edges on objects.

Fresnel - Use a clamped dot(view normal, view direction).
Pros: Works in shader without a separate pass. Will highlight almost every edge on smooth shapes. Cheap. Works on transparent objects.
Cons: Difficult to control line thickness as its dependent on the angle of the surface to the camera rather than the depth and the roundness of the surface. Impossible to have consistent line thickness. Doesn't work well on hard edged shapes or low poly meshes. Lines are only drawn on the "inside" edge.

Push Hull / Mesh Doubling - Duplicate the mesh and expand along surface or view normals, full front faces. Unity's toon outline shader works this way.
Pros: Cheap on low poly characters. Works on pretty much every device. Gives a lot of control for line thickness. Lines are outside the edge.
Cons: Lines still won't be constant, and only works on smooth meshes. Hard edges will create holes in the outline. Thicker outlines can make smaller details and concave edges loose their outline as its dependent on the outline mesh not intersecting other geometry.

Edge Detection (Depth based) - Test depth between multiple screen pixels.
Pros: Draws edges on everything that's opaque. Precise pixel size control over line width.
Cons: More expensive than other options, but constant cost regardless of geometric complexity. Will "edge" surfaces that are simply angled away from the camera because the depth discontinuity between pixels is high enough on glancing angles without heavy depth biasing and then will miss seemingly obvious edges because of the bias. Each pixel of thickness increases the cost exponentially. Doesn't

Edge Detection (Depth & Normal based) - Test depth and normal between screen pixels.
Pros: Everything above, plus catches edges depth only solutions don't.
Cons: Even more expensive.

Geometry Shader Edge Detection - Test adjacent polygons to find corners, construct outline geometry.
Pros: Catches every single edge, guaranteed. Complete control over outline thickness. As close to hand drawn edging as you can get.
Cons: Super, super expensive, especially on high poly objects. Also, doesn't work at all in Unity since they don't provide adjacency information to the geometry shader.


Geometry shader based outlines are kind of the holy grail of outlines, but as I said they're stupidly expensive (a 100,000 polygon model can bring a modern computer to it's knees), and they don't work in Unity.

So, lets go back to Edge Detection. The depth & normals (Roberts Cross as it's called in Unity's Image Effect) method is really quite good, but requires you're using the deferred rendering path, or your camera has to render depths and normals for all opaque scene geometry. Obviously that's not working for you for some reason.

Unity's Sobel implementation misses a ton of edges because of how it handles the issues of depth discontinuity between pixels on glancing angle geometry. The depth disparity needs to be greater and greater the further away something is so edges disappear on small details as they get further away. This works for the style a lot of people are going for when doing outlines. You can write your own by not increasing the disparity required in the distance, but you might start running into issues of depth precision and that glancing angle issue I mentioned before. However, unless you need really thick outlines, this really is the best you can get.

If you don't want to use an image effect, you can do it as an extra pass in the shader. This can let you limit outlines to a specific object instead of applying it to an entire scene. This is actually something I've been working on lately. You can only do the depth based edge detection when doing this, unless you're using deferred, but it offers a bit more control than the other options.

 

Hey, Thank you for the reply.

I mainly want to detect the edges of geometry that is cube shaped, which is why i was assuming some sort of normal test would be the best approach.

What you are talking about in your last paragraph is exactly what I would like to achieve, i just have no clue where to even begin.

 

If you've just got a cube and you want to highlight all of the edges, there's a bunch of ways to do that. The easiest way is to just have another texture with the lines in it. None of this fancy shader stuff.

Most of the techniques above are more about detecting outline and overlap edges rather than the internal edges. Since internal edges like this are "constant" in that the don't depend on the camera facing people generally just draw them into their textures. The above are to catch the edges that are view dependent.

You could do something like an image effect using the normals, but that's spending a lot of resources on something that doesn't require it.

Some alternative approaches would be using distance fields or modified barycentric coordinates stored as vertex colors or UVs. Ultimately these options are all pretty similar to just drawing the lines in a texture as they require authoring the content ahead of time rather than something that can be done 100% in shader. Well, you can do wireframe stuff purely in shader using geometry shaders, there's a free example on the Asset store, but it's not quite what you want.

 

@bgolus

I guess i should of clarified about the shapes of the mesh. They are "Cube-ish". Think of tetris for example. All the shapes were made from smaller cubes. This is essentially how the mesh is generated. If i were to put the edge in the texture i would be outlining the smaller cubes, not the face edges of the mesh. Does that make sense?

 

Then your only option, really, is to use an image effect of some kind. If the surface is made up of separate cubes, there's nothing a regular shader, or even a geometry shader, can do as these do not know about other faces.

If you only want to render edges on a specific object you'll need to render out a mask and that object's normals to a full screen texture and then do an image effect pass on that. There's another thread that has a nice image effect for getting all the hard edges that just popped up too.

http://forum.unity3d.com/threads/image-effect-edge-detect-normals-colours-rel.310280/

 

@bgolus

YES YES YES! You are the man(or women)! That is exactly the effect i wanted to achieve! Thank you.

 

I landed here on my search for edge detection for specific objects, and then decided to make depth&normals edge detection based on a DepthNormals RenderTexture I render first. In this video I'm explaining what I did:

Sorry for the necro, but I think it's worth it, it would have helped me

 

I came across this post looking for a way to do a 'sonar' effect (like in Batman: Dark Knight), a kind of neon edge detection. But I have done work with this type of effect before, and I thought I could offer advice to anyone who's working on this now. There are other ways to do this, and might be better suited to you depending on what you're doing, but you could just put the 'lines' in the texture, seen in games like Xbox 360's Borderlands 2. You can get the same effect either using a texture or with a cell-shading shader. Unity is already able to do this edge detection without you having to do anything extra, just take a look at the Unity Editor's scene-view Shading Mode tab's 'Shaded Wireframe' setting. So you could easily use line renderer to add that in the LateUpdate(). But doing it through a texture is nice too, you can have multiple textures for switching it off and on or for different variations of edge detection. Unity has a cell-shading example project that demonstrates cell-shading in Unity. Also, texture-based cell-shading can look better than 'rendered'. Look at this 'rendered' version: http://ws.cis.sojo-u.ac.jp/~izumi/U...n/Images/manual/script-EdgeDetectEffect-0.jpg
PS, For my 'sonar effect', I just found a good way to do it: http://www.codeavarice.com/dev-blog/tutorial-burning-edges-dissolve-shader-in-unity

 

Anyone have the links to the original scripts posted in the video, the link doesn't work anymore? : (

 

but this version requires image effects though?

 

(Slightly old topic, but related additional question..)

Does other 3d software/CAD use different outlines from those listed above?

For example, any thoughts what they might be doing:
- 3ds max vray outlines: https://docs.chaos.com/display/VMAX/VRayToon (looks really good)
- Catia/Solidworks (those 2 main screenshots) https://tutorial45.com/catia-vs-solidworks/

 

I believe SolidWorks draws outlines by drawing the outlines directly. Likely calculating them on the CPU and then generating bespoke geometry for the lines.

That Vray plugin I suspect works exactly like the depth + normal post process outlines, just using subpixel resolution data.

I know some high end toon outline plugins do some more complex stuff where they detect overlaps and edges during rendering and then generate geometry strokes based on the detected lines.

 

https://www.reddit.com/r/Unity3D/comments/taq2ou/improving_edge_detection_in_my_game_mars_first/
I want to try this. Does anyone know how to do it

 

The short version is you effectively write your own deferred renderer pipeline. You can't use any built in shaders for anything, because all of your shaders output color values to one or more render targets that you assign to the camera. Then in a post process you take those render targets and write back to the frame buffer.

Obra Dinn does something similar as well. As does Manifold Garden.