Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Outline Shader With Outline Inside of Mesh

Discussion in 'Shaders' started by dalanchoo, Jun 29, 2020.

  1. dalanchoo

    dalanchoo

    Joined:
    Jun 6, 2014
    Posts:
    44
    Hi. I need help writing an outline shader. The nature of my problem requires that I need the border to be inside the mesh, not external to it.

    To give a quick overview, I am making a strategy game. The user is inside a giant sphere. The sphere is composed of hexagons. My game creates territories by grouping several hexagons into a single territory. Each territory will be owned by a given player, and the color of that territory will be the color for a given player. Adjacent territories will be often owned by the same player, and hence be the same color.

    I need some sort of outline to show the extents of each territory. My first thought was to just find an outline shader written by somebody else and use it. I have tried using the shader provided with the oculus grab object code.

    The first issue is that since I am inside a sphere, the outline that would surround a territory would actually be on the exterior of the sphere behind other territories. Maybe it is possible to make the outline draw in front of everything else so the user can see it and look nice. I don't know.

    From what I understand, outlines shaders basically consist of drawing a larger version of the object with the border color so that there is an outline. So my idea was to draw the outline as the size and shape of the original object, and then shrink my territory a bit and draw it so that I will see the outline. I think this would work with convex shapes, but my territories are jagged. Here is what it looks like when I try my approach.

    Thick Border.png

    I have added a really large border here for clarity. You can see the green territory actually extends outside of the bounds of the border. So, this technique is flawed and I am not sure how to fix this.

    How would I go about making a nice line around the edges of my territory?

    Here is my shader code. It is my first shader ever, so I am probably making a ton of stupid mistakes.

    Any help is greatly appreciated.
    John Lawrie

    Code (CSharp):
    1. Shader "Custom/InternalLine"
    2. {
    3.     Properties
    4.     {
    5.         _Border("Border Color", Color) = (1,0,0,0)
    6.         _Color("Color", Color) = (0,1,0,1)
    7.         _MainTex ("Texture", 2D) = "white" {}
    8.         _OutlineWidth ("Outline width", Range (.002, 0.9)) = .005
    9.     }
    10.     SubShader
    11.     {
    12.         Pass
    13.         {
    14.             CGPROGRAM
    15.             #pragma vertex vert
    16.             #pragma fragment frag
    17.  
    18.             #include "UnityCG.cginc"
    19.  
    20.             struct appdata
    21.             {
    22.                 float4 vertex : POSITION;
    23.                 float3 normal : NORMAL;
    24.             };
    25.  
    26.             struct v2f
    27.             {
    28.                 float2 depth : DEPTH;
    29.                 float4 pos : SV_POSITION;
    30.             };
    31.  
    32.             v2f vert (appdata v)
    33.             {
    34.                 v2f o;
    35.  
    36.                 float4 objectCenterWorld = mul(unity_ObjectToWorld, float4(0.0, 0.0, 0.0, 1.0));
    37.                 float4 vertWorld = mul(unity_ObjectToWorld, v.vertex);
    38.  
    39.                 o.pos = UnityWorldToClipPos(vertWorld);
    40.  
    41.                 return o;
    42.  
    43.             }
    44.  
    45.             sampler2D _MainTex;
    46.             float4 _BorderColor;
    47.  
    48.             fixed4 frag (v2f i) : SV_Target
    49.             {
    50.                 return _BorderColor;
    51.             }
    52.             ENDCG
    53.         }
    54.  
    55.  
    56.  
    57.         // Second pass
    58.         Pass
    59.         {
    60.             CGPROGRAM
    61.             #pragma vertex vert
    62.             #pragma fragment frag
    63.  
    64.             #include "UnityCG.cginc"
    65.  
    66.             struct appdata
    67.             {
    68.                 float4 vertex : POSITION;
    69.                 float3 normal : NORMAL;
    70.             };
    71.  
    72.             struct v2f
    73.             {
    74.                 float2 depth : DEPTH;
    75.                 float4 pos : SV_POSITION;
    76.             };
    77.  
    78.             float _OutlineWidth;
    79.  
    80.             v2f vert (appdata v)
    81.             {
    82.                 v2f o;
    83.  
    84.                 float4 objectCenterWorld = mul(unity_ObjectToWorld, float4(0.0, 0.0, 0.0, 1.0));
    85.                 float4 vertWorld = mul(unity_ObjectToWorld, v.vertex);
    86.  
    87.                 float3 vertToCenter = objectCenterWorld.xyz - vertWorld.xyz;
    88.                 float3 shrinkVect = vertToCenter * _OutlineWidth;
    89.  
    90.                 o.pos = UnityWorldToClipPos(vertWorld + shrinkVect);
    91.  
    92.                 return o;
    93.  
    94.             }
    95.  
    96.             sampler2D _MainTex;
    97.             float4 _Color;
    98.  
    99.             fixed4 frag (v2f i) : SV_Target
    100.             {
    101.                 return _Color;
    102.             }
    103.             ENDCG
    104.         }
    105.     }
    106. }
    107.  
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    Can't be done with a shader alone.

    Most outline shaders work by drawing the mesh a second time slightly larger, either using vertex normals or similar to the example you posted above by scaling the entire mesh. This works fine for nice, rounded 3D models. But it fails fairly spectacularly on flat shapes or anything with hard edges. This also really only works for drawing outlines as an "inside" line means scaling the interior mesh which you probably don't want if you're going to have any kind of texturing on your mesh.

    Really there are two classes of solutions: A render texture based effect, or a script generated mesh.

    The render texture based approach requires rendering each region into a full screen render texture, then running an edge detection outline on that. Depending on how many regions you have you might be able to render all of them as a solid color into the render texture. However there are two issues with this approach. First is it can be slow, especially for wide outlines, and second is the resulting outlines are in screen space which may not be desirable.

    The script generated mesh approach can go a couple of ways. The "basic" way is to actually generate a separate line mesh. This doesn't require any special shaders, but you have to be careful to keep the line slightly above the surface to prevent z fighting, or use a custom shader with an
    Offset
    to effectively do the same on the GPU.

    The "better" way is to encode the edge distance into the territory meshes you're already generating. For example, for a single tile area you could use the vertex color or extra UV component to color all of the exterior vertices black, and have a center point colored white, like this:
    upload_2020-6-29_0-43-22.png

    For more complex regions you may have to do some further geometry splits to get the data you need. Notice the bottom left connection:
    upload_2020-6-29_0-49-54.png

    Then in the shader you can use that gradient to find the relative distance to the edge and draw the outline with a simple
    if (edgeDist < outlineWidth) return _OutlineColor;
     
  3. dalanchoo

    dalanchoo

    Joined:
    Jun 6, 2014
    Posts:
    44
    Hi bgolus. Thank you so much for the insight!

    The approach that I will try first is the "better" approach that you listed last. I am however unsure of a couple of things.

    For a solitary hexagon, I understand that I would create an image with a hexagonal gradient like in your image. I would set the UV of each vertex to the corresponding corner in the image. Then the pixel shader would simply ask for the color of the image at that spot and compare it to the border threshold and draw it in a different color if it was dark enough.

    I don't understand what needs to be done to apply it to the group of hexagons in your second image. Do I need to generate an image that corresponds to the shape of the multi-hex territory, or am I able to use the image for the single territory cleverly assigning UV values? Would it be possible or desirable to use a square image with a gradient from black to white

    If I need to generate an image for each territory, would I encounter any issues if I had large territories consisting of many hexes? How large would a texture need to be?

    BTW, how did you generated the image in the bottom picture? It looks like I might have to generate that image programmatically for each territory.

    Thanks again for your help.
     
  4. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    You'd need to construct a mesh in a c# script of all of the hexagonal regions combined into a new mesh, and then color the single whole mesh's vertices and split geometry based for thin edge connections (or just always have all of the hexagons be made of 12 triangle instead of 6).

    A single image of the whole territory would technically work, but that's basically the same thing as the render texture based approach, but comes with lots of gotchas and caveats since this would mean you'd be doing it in local mesh UV space rather than screen space. Like how would you do a territory that wraps the whole sphere and not have a double-edge? Any kind of "clever" UV mapping of a square or hex texture isn't actually any easier as now you're trying to assign two components instead of one per vertex, and you'd still need to figure out where the edges are and split geometry occasionally.

    One other method I can think of is would be to keep each hexagonal area separate, and instead use material properties to tell the shader which edges are connected. You could then calculate an appropriate outline per hex tile. Maybe using multiple SDF functions to get a UV space distance to edge for the single hex tile and its attached neighbors.

    I manually built it in 3ds Max in a few seconds by copy/pasting a flattened cone.
     
  5. dalanchoo

    dalanchoo

    Joined:
    Jun 6, 2014
    Posts:
    44
    My territory does consist of a single mesh. This is what it looks like.

    MeshTris.png

    I bought a third-party plugin off the asset store that generates the hexagons for my sphere. My code selects a bunch of hexagons and calls CombineMeshes to combine them all into on.

    If I understand you correctly, I would basically need to have the mesh subdivided in such a way that hexagons on the border of the territory would have a vertex in the middle. The vertex in the middle would be white. The vertices on the perimeter would be black. Vertices in the middle of a hex would all be white and I would have to look out for special cases like in the lower left of your image and add an additional vertex there. Basically, subdivide the mesh in an ideal way. Do I have the jist of it?

    Before I knew anything about shaders, I had this idea of keeping the hexagons separate. Then I could have a bunch of hexagon pieces that had different combinations of edges with borders on them and select then and combine them in a way that the outside edge of the territory would have a border.
     
  6. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,329
    Yep.

    Yeah, that's honestly a total valid way of handling this too. You could have an atlas of "outlines" and use a secondary UV for each hex to map the outline atlas tile to the individual hexes. That'd totally work too and may be easier to implement.
     
  7. dalanchoo

    dalanchoo

    Joined:
    Jun 6, 2014
    Posts:
    44
    OK. Cool. You have given me a lot to mull over.

    Thanks for your help and I hope I don't have to harass you too much in the future :p