How do they do the painting in Splatoon?

I was watching my younger son play Splatoon yesterday, and it's got me thinking. First, about what an amazingly good game design it is (but that'd be a topic for the Game Design forum). And second...

How do they do that?!

In case you've been living in a cave, Splatoon is a Nintendo third-person shooter where, instead of ordinary weapons, all your weapons distribute paint in various ways. One team is one color, the other team is a different color.



This paint (or "ink" in Splatoon parlance) splatters very satisfyingly, and also wraps neatly around corners and edges.

So I woke up wondering to myself how I would tackle something like that. Things I know how to do:

1. Cast a ray onto a mesh and get the UV coordinates (as well as the 3D coordinates) of the point hit.
2. Copy a source texture into a target Texture2D (though this is a bit expensive).

But those edges and corners are the tricky part. If I had to do it this moment, I guess I would cast a pair of rays at each of 4 points around the center of the splat, and from the point hit, calculate where (in UV space) the center and scale of the splat would be. For example, I cast a ray 1 unit above and right of the aim point, and another ray 0.9 units above and right, and from the two UV points hit, extrapolate 10 times further to get where the center would be. Then I copy in my splat texture, and repeat for the other three corners, only skipping repainting if they work out to the same center.

But I foresee problems. If my two rays happen to straddle an edge/seam which goes to different parts of UV space, I could end up drawing a giant splat that spans things it shouldn't. And I suspect there are cases where edges wouldn't line up as neatly as I want this way.

Another approach would be to, for each hit, draw a large number of very small circular blobs of paint. And simply iterate over the whole splat area, casting a ray for each small blob. But that's a lot of ray-casting, and a lot of overdrawing in the texture, both of which seem like a Bad Thing for performance in a game where spraying paint everywhere is kind of the point.

Any other ideas? Is there some clever solution to this I haven't thought of?

 

Check this out... not much info there but possibly contact them...

https://forum.unity3d.com/threads/splatoon-like-game.383798/

Edit: just checked the profile, last seen 1 year ago...but there is a little info about approach in the text...

 

Keeping a low res texture in memory mapped like light bake UV's and writing to it wouldn't be that bad.

I'd probably go for setting vertex color with tri-planar projection just to avoid dealing with the seams.

In either case the splatter-ey-ness is just basic texture blend stuff.

 

I don't think the texture can be too low-res... the splatters have very rounded edges. You wouldn't want them to look pixely, even up close (since usually, "up close" is where they appear).

Can you elaborate on that? I don't quite understand this technique.

A bit of googling did turn up this thread, but it's light on detail. I gather that it's somehow using the vertex colors to index into a texture map, but I don't understand the "triplanar projection" part, nor why this is better than just using another UV channel.

But I have a feeling this may be the clever approach I've been missing!

 

OK, I found this explanation of triplanar mapping, which simply samples three textures based on the world coordinates of each point.

That's a cool trick and one I'm glad to add to my toolbox. But I still don't see quite how we'd use this to paint the environment in a Splatoon-like fashion. (Though I do now see the advantage of using colors rather than an extra UV channel: UV channels store only two scalars, while a color channel stores three or four.)

Boldly confessing my ignorance since 2011,
- Joe

 

Yes, you use world coordinates and blend between them based on the normals. Saves you from doing the UV's and solving the seam issue but results in more samples in your shader.

Instead of adding the "paint value" to a texture I'm suggesting you add it to the vertex color on the level mesh itself and use that as the mask between Team A's pain, Team B's paint, and the default appearance of the level.

 

This is basically the same effect as Portal 2's gel. You render color into a low resolution render texture that uses the scene's lightmap UVs, then peturb that texture lookup using a noise to hide the low resolution-ness of the underlying data. Advanced terrain shaders and vertex color based texture blending do the same thing to hide the low resolution control texture or vertex density.

Edit: there was a great post years ago where someone broke down exactly how the paint / gel was done in portal 2, but I can't find it anymore. The page might not even exist anymore.

However this NeoGAF thread has a great example: http://m.neogaf.com/showpost.php?p=116699387&postcount=235

 

But doesn't it also mean that you can't have 2 parallel surfaces that "overlap" when viewed from any of the three projection directions? In other words, if I had two towers right next to each other, both part of the same mesh, then this triplanar projection trick wouldn't work, because it would come up with the same texture coordinates for both boxes (except for the angles where they are side-by-side).

In fact, even a single box would (as I found the triplanar projection described above) end up with the same texture on opposite faces of the box, which is no good — painting one side shouldn't paint the other. To avoid that you'd have to go to 6-plane projection, no?

Man, I feel dense today, because I just don't see how that would work unless your level models are very polygon-dense. Picture a big box or wall; I'd expect it to be composed of 2 triangles. I shoot a little splat at some random place in the middle... what would I set for the vertex colors at the corners, to get a nice splat in the middle?

 

Thanks for that, that makes sense and is a great way to hide the low-res-ness of the texture you're using to keep track of the paint.

It doesn't really help with the problem of how to figure out WHAT texels to set... I've been thinking about this on and off all day, and still haven't come up with anything better than the multiple ray-casting idea at the top of this thread.

OK, this is the second time I've heard something like this, and the second time I've failed to understand it. Perhaps it's because I don't have much experience with lightmaps. Can you break this down for me, or point me to a reference that explains it? What does lightmapping have to do with it?

Thanks,
- Joe

 

At its most basic, lightmaps are just textures. Textures that are unity uniquely UVed for the entire scene, at least for static geometry.
https://docs.unity3d.com/ScriptReference/RaycastHit-lightmapCoord.html

Do a raycast, use that position to draw into your custom texture. You'll have to do multiple raycasts to deal with (literal) edge cases as just drawing a larger radius it might bleed into areas visually unconnected to where you hit. The alternative is you could use a 3D texture instead, but that might cause problems with occlusion, ie: paint a wall and the bank of the wall also gets painted.

 

So, raycasting and drawing into a texture is easy enough. (I was able to quickly adapt the code we use for painting zones in High Frontier.)



But it's the darn edge cases that are troublesome. You can see a seam in the cylinder at right, where it was cut for UV mapping; and there are similar discontinuities in the block at left (somewhat harder to see because I moved the mouse more and continued painting onto the next surface). Same problem where the block and cylinder meet with the ground.

I'm no longer certain that ray-casts are really the way to go. In Splatoon, most (maybe all?) weapons shoot blobs of paint that arc out of your gun, and splatter sometime later. They're certainly not ray-casting from the camera or player. They might be ray-casting from their own position as they move forward each frame, but it's just as likely they're doing a sphere sweep.

So, maybe a better way to think of this is as a sphere collision test. You could even drill down to individual triangles, painting appropriately-sized blobs of color on each triangle according to how they intercept the sphere. Unity doesn't provide such detailed collision info, but sphere/triangle intersection is pretty easy.

On the other hand, maybe I'm still approaching this the hard way.

 

Hmm this thread got me thinking.
I never thought about how this could be done, but if you guys are open for another idea...

What about re-meshing the entire scene, just like the nav-mesh does with its voxelizer?
You'd have essentially twice the vertices for the level, which would be pretty bad. A decimate filter like in blender could help.. maybe, but it would still be pretty bad.

As for raycasting, maybe they have some system that's heavily optimized for the game where they can indeed correctly identify all vertices in a given sphere and how to paint them. Then render every object in the level with a "paintable shader" that just blends between normal and the ink.

 

I wont be able to use fancy terminology here but give an idea of how I think it is handled.

What I believe they do is a liquid system(aka Particle system) that is coded to stick where it lands. after it does the initial splat animation it then becomes a object that can be interacted with (like swimming threw it)

 

I doubt thats what they do.
That would be a decal system. It would work (with a shader that maps in world coordinates), but imagine how many objects - and therefore drawcalls - that would cause.

Also it would have the same problems as what we already have. (no support for edges or when a wall meets a floor...)

 

No, there is no texture. You're writing into the mesh.

So what? Verts are cheap. Your paintable mesh doesn't have to be your collision mesh.

 

OK, so I think I get what you're saying, @brownboot67 (and it may be what you meant too @dadude123, though it wouldn't be twice as many vertices — it'd be hundreds of times as many).

Build your level out of high-poly models, even for flat surfaces. Something like no polygon more than 0.1 m apart from another polygon (whatever scale you pick here determines the smallest drop of paint you can apply). Then to apply paint, you set the vertex color — probably just filling in the red channel for one team, the blue channel for a different team, etc.

Then you'd use a shader that takes the interpolated vertex color and thresholds it, probably with the addition of some noise as described here.

It's a clever approach, though I'm not completely convinced verts are that cheap. But it would be very quick to apply paint, since you don't have to write to (and upload) a texture — all you have to do is set vertex colors. Oh, and finding the vertices to set is also very easy; assuming the blobs of paint flying around are spherical, it's just a point-in-sphere test, which is about as easy as it gets (though on the other hand, you'd have to do this for thousands of vertices).

Let's do some quick math... say we have a vertex every 0.1 m; that's 100 verts per square meter. The 10x10 plane in my image above would have 10,000 vertices. That 1x2x3 box would have (...doing math...) 19 m^2, or 1900 vertices. That's a lot but it's not completely outrageous. And maybe you could apply a LOD or two (a 0.2-m spacing would be 1/4 the verts, and a 0.4-m spacing would be only 1/16 as many).

This sounds worth a try!

 

OK, first quick test of that approach:



Wraps around edges and seams beautifully, and feels very fast. Rather blocky though, as you can see. This is of course with a simple threshold, with no noise, and no anti-aliasing where I paint the vertices.

Here's a close-up in shaded wireframe mode, so you can see what's going on...



Notice that the blockiness very much depends on whether the edge lines up neatly with the triangles in the mesh, or runs counter to it. In the floor strip above, it's counter to the triangles, and so comes out considerably worse than a strip running the other way. This is an issue I ran into with zone painting in High Frontier as well. We were able to mostly hide it there, though I no longer remember exactly how... we may have added an extra vertex at the center of each square, so there was no strong preferred direction, at the cost of 25% more vertices and 100% more triangles.

But it might also be that adding some noise and a softer edge to the vertex colors will hide the edge well enough.

 

It is outrageous
You'd definitely need some octree or something to narrow down the vertices.
And yes they're not that cheap.
I think that if you take a really good decal system and then make it so it expands the nearest decal instead of just blindly applying yet another quad (and that's the best case, when you only hit a flat plane!!), you'll get pretty decent results.

Limit how complex the "expanded decal" things can get at most.

Or if you go the mesh route, why not just add additional vertex attributes to all the models in the scene itself, instead of duplicating it?
I mean what you'd do is essentially copy the vertex positions, just so you have another vertex-color channel.
You could just reuse the existing vertex colors (if they exist, and if not then add them)... like, i don't get why you'd have a full copy of everything even if that copy is simplified.

The only downside wouldbe that your whole level and all its props have to be drawn with your custom shader.
And the bad part about that is that you have to essentially make a copy of the standart shader and then add support for blending towards your goo/ink.

That will cause the shader to become pretty expensive, but if its just on PC you'll likely never notice any problems ever.

In any case, if you really try this, let us know how it goes!

 

I dunno though... 10,000 polygons isn't unusual for a main character, and it's not uncommon to have lots of "main characters" running around these days. Here 10k polys is an entire section of the level. It might be OK.

But look again at the Splatoon screenshot at the top of this thread. There is SO MUCH PAINT all over everything, with complex edges and detail, all wrapped neatly around even curving shapes, and honoring the underlying normal maps... I just can't imagine doing all that with a decal system.

I'm not sure where you got the idea we were copying anything. There is no copying here... I just made a high-poly version of the models in my modeling app, and used that instead of the low-poly version. (But as @brownboot67 pointed out, we could still use the low-poly version for the collision mesh.)

Meh. Shaders aren't that hard. And I assumed from the beginning that a custom shader would be needed. It doesn't have to be as complex as the modern Standard Shader; it only needs to support the features I need to support.

Yeah, if I were going to actually do this, it would be for PC, since my main interest would be in playing it with a keyboard and mouse. I suck at aiming with a gamepad!

I don't know if I will actually make anything of it, though... mostly I just want to figure out how!

 

Yep, what you're doing is what I'm thinking. You can allow much wider interpolation space and let the shader do the interesting bits to define the shape. Like a quarter of the loops on what you have would probably look just as good with some decent noise and thesholding.

Verts aren't bad. Especially on PC/console. We're doing close to a million verts a frame on medium power mobile devices in our new The Walking Dead project at 30 fps.

If you aren't making a custom shader for each type of your assets you're doing it wrong. At the very least you should be packing your data more efficiently.

 

Noise for the win!



There's still a lot that could be done to make it look more interesting — in particular, making it appear glossy — and my current shader only supports one color. But I'm convinced that this is the way to go.

Thanks for the help, everyone!

 

Would you mind posting a simple example of how you got it to work to give others a head start? I don't need it now but it looks like something useful to get back to at a later stage....

 

I concur with @sngdan. I was trying to do something similar. Though I request you speak in terms a non-artist can understand...I've had very, very little interaction with shaders at all, much less making my own.

 

Looks really good. Everything else is just fine-tuning, like making a nice shader for some "wet effect" haha.
Should be very easy to do with shaderforge.

The small imperfections along object borders could be fixed with decals, and when the texture below has enough ink the decal can be removed. But it already looks so good that I doubt its neccesary, maybe it'd be even better to increase the vertex count.

@EternalAmbiguity don't you mean "in terms a non-programmer can understand" ?
Get ShaderForge, then invest a few days to learn how most shader effects are done. Then what's being done here will be easy for you

 

Shader and graphics code in generally is something neither artists nor programmers usually understand, sometimes including the people writing them.

 

Well, I would consider myself a very, very rudimentary programmer. I suppose you're right, writing a shader would technically be a kind of programming--but I have little experience with either programming or the artistry of shaders and such.

I'm just asking that it be clear enough that I know what to look up and research when reading the "answer."

 

Yes, that's a fair request. There have been several other people who have gotten as far as I have, but then disappeared without a trace before posting any details of their work.

So, though I can't do it this instant, I'll try to write something up before Wario shows up at my door.

 

Changing vertices can be reasonably fast, even on a high poly mesh. I include an example in MegaSplat about how to do this efficiently at runtime (10x speedup and no GC over a naive implementation). He's a gif from my website showing a ball painting the terrain as it rolls (modifying the texture and tessellation of the terrain in the process). After optimizations, this takes very little CPU time.



Also, you likely don't need your vertex count to be very high resolution at all with a decent shader..

 

I'd swear I saw a koopa troopa hiding behind a cactus as I returned home just now. So, maybe I'd better hurry.

So the key trick here is the shader, which looks at the alpha channel of the vertex color, plus a bit of noise, to decide where the paint is.

Code (CSharp):

1. Shader "Custom/Blobby Paint" {
2.     Properties {
3.         _MainTex ("Base (RGB)", 2D) = "white" {}
4.         _PaintColor ("Paint Color", Color) = (0.8, 0, 0.8, 1)
5.         _Freq ("Frequency", Float) = 5
6.         _Threshold ("Threshold", Range(0,1)) = 0.3
7.         _Amp ("Noise Amplitude", Range(0,1)) = 0.5
8.     }
9.     SubShader {
10.         Tags { "RenderType"="Opaque" }
11.         LOD 200
12.      
13.         CGPROGRAM
14.         #pragma surface surf Lambert
15.  
16.         #include "noise-simplex/noiseSimplex.cginc"
17.  
18.         sampler2D _MainTex;
19.         float4 _PaintColor;
20.         float _Freq;
21.         float _Threshold;
22.         float _Amp;
23.  
24.         struct Input {
25.             float2 uv_MainTex;
26.             float4 color : COLOR;
27.             float3 worldPos;
28.         };
29.  
30.         void surf (Input IN, inout SurfaceOutput o) {
31.             float noise = 0;
32.             float paint = IN.color.a;
33.             if (paint > 0.01) noise = snoise(IN.worldPos * _Freq) * _Amp + paint;
34.             if (paint + noise > _Threshold) {
35.                 o.Albedo = IN.color.rgb * 0.95 + float3(1,1,1) * noise * 0.05;
36.                 o.Gloss = 1;
37.                 o.Specular = 1;
38.             } else {
39.                 half4 c = tex2D (_MainTex, IN.uv_MainTex);
40.                 o.Albedo = c.rgb;
41.             }
42.             o.Alpha = 1;
43.         }
44.         ENDCG
45.     }
46.     FallBack "Diffuse"
47. }

It's referencing simplex noise shader code I from here. This version of the code takes the actual RGB from the vertex color, but I'll probably change it to instead pick one of several colors defined as shader properties, based on which channel of the vertex color is greatest. That'll work much better for team colors.

Anyway, then to apply a blob of paint, I simply check all the vertices in a mesh for their distance from the center of a blob, and set the vertex color accordingly.

Code (CSharp):

1.     public void ApplyPaint(Vector3 position, float innerRadius, float outerRadius, Color color) {
2.         Vector3 center = transform.InverseTransformPoint(position);
3.         float outerR = transform.InverseTransformVector(outerRadius * Vector3.right).magnitude;
4.         float innerR = innerRadius * outerR / outerRadius;
5.         float innerRsqr = innerR * innerR;
6.         float outerRsqr = outerR * outerR;
7.         float tFactor = 1f / (outerR - innerR);
8.      
9.         for (int i=0; i<verts.Length; i++) {
10.             Vector3 delta = verts[i] - center;
11.             float dsqr = delta.sqrMagnitude;
12.             if (dsqr > outerRsqr) continue;
13.             int a = vertColors[i].a;
14.             vertColors[i] = color;
15.             if (dsqr < innerRsqr) vertColors[i].a = 255;
16.             else {
17.                 float d = Mathf.Sqrt(dsqr);
18.                 byte blobA = (byte)(255 - 255 * (d - innerR)*tFactor);
19.                 if (blobA >= a) vertColors[i].a = blobA;
20.             }
21.         }
22.         mesh.colors32 = vertColors;
23.     }

The above code lives in a little Paintable class that I attach to any mesh that I want to be paintable. It's already grabbed and cached its own mesh, and the colors32 array thereof.

I find the meshes to call this on via Physics.SphereCastAll, as part of my projectile code. (My projectiles are little wibbly wobbly balls which burst on impact, calling the above ApplyPaint method on every mesh in range.)

So, there, that's basically all there is to it. Now hopefully I can .cdf jbHMR,Y-[

 

@JoeStrout thank you - appreciated.

 

Hello @JoeStrout, thanks for sharing this.

I tried to follow your examples but kept getting empty color32 arrays (when attaching the script to a standard Unity plane).

Any tips?

Best regards

 

@JoeStrout could you upload a package/project so we can see what's going on, getting lots of errors following what you've said.

 

Hey there,

thanks for sharing your code. It works just fine Now I stumbled over the problem you had with the blocky paint. While I wasn't satisfied with what the noise could do for the solution of this problem I tried out your suggested mesh layout. Sadly it turned out that this does not fix the problem, but leads from blocky diagonals to blocky straights. Is there any other way to smooth them out? Maybe with letting the edges of the paint color vertices which weren't part of the paint in the first place, but are in line with the rest? For the "peaks" in the picture that would mean that they would be connected and the valleys filled even if the vertex in the valley wasn't in the paint radius.

Unfortunately I'm not that experienced in shader programming and have no idea how to adjust your shader to implement this solution. Maybe someone with more experience can tell me if it's doable and even a good idea?

Greetz

Edit: Added another picture without noise and threshold
Edit2: I think my suggestion wouldn't actually be implemented via shader. I guess the algorithm just have to find the missing vertices somehow and set their color (no shader change involved)

 

Hey,

just a quick update: I managed to fix my problem with the blocky paint Here is how:

Firstly I used a grid with alternating triangles. While the standard grid is easy to create, you only have diagonals in one direction and can not paint triangles as needed, but if you alternate the triangles you get diagonals in both directions with which you can work (see picture below)

Secondly I used an other array, where I stored the neighbors per vertex (array of hashset so I don't have to worry about duplicates). You then just have to check some neighboring and color conditions to fill the gaps. There shouldn't be any performance issues since the max. number of neighbors a vertex can have (in this grid) is 8. It will need a little bit of memory though, but nothing to worry about ;D

There is still an issue with the shading nonetheless. The fade from black to red on the diagonals still has some "spikynes". But I think that is realy a shader "problem". Maybe I will check it out in the future, but for now it's fine for me (you maybe can tweek it out with a higher resolution grid).

In the images below you can see where my algorithm has added color to fill gaps. Image 2 and three have the standard noise and threshold applied, while image three is color only.

Greetz





Edit: So I fixed the (minor) issue with the black fading at the edges. Every time I paint a vertex I automatically paint the neighbor vertices in the same color, but with an alpha value of 0. In the end it only affects the vertices directly next to the paint without directly affecting the paint itself and leads to fading between the same color (instead of the color and black when no color for the neighbors is defined).

 

Hey,I just found this asset free now,last time I view this topic it's not free,now it is.
Cheers.

https://www.assetstore.unity3d.com/en/#!/content/86210

 

I was reading the post, it's possible to make this effect the same in the video, it starts at 0:22

 

Great job @JoeStrout!
Can you share a Unity demo project to try your experiment?
Thank you

 

Is there any documentation how you achieved that?

 

I documented the example code included with MegaSplat..

 

Ah okay, thanks anyway

 

I think it's related so I'll leave this here:
https://forum.unity.com/threads/color-bleed-effect.717866/#post-4797017