Question Best way to render object silhouetes as plain colours.

Hello there!
I am working in a 3D simulation for an academical research, and to train a neural network we are working with, I need to make renders of the scene where every type of object (for example, people, rock, houses, etc) is just a plain-color silhouette (this is called a "semantic label" in AI).
For example:


I need to do this with a Unity scene. There's the problem that some objects has special shaders on them (we use GPU instancing, for example), and the easy solution of applying an unlit material to each object is out of the question.

My first idea was to separate each type of object in different layers, and render each layer with a different color, but I'm not sure on how to do this, or if this is the best way to do it. I have researched a little bit, but I'm not familiar with this side of Unity.
Thanks!

 

If you need something that is performant and have time to implement, I would recommend MRT. Basically writing to both a regular color texture and a "semantic label" color texture at the same time. It would require basically using all totally custom shaders.

Another option would be rendering after the main camera to a second render texture where you replace each object's shader with a unlit color shader, but if that's out of the question, then first option sounds better...

There are plenty of other ways to do this as well, but in the end you will need to render to multiple textures and you will need to use different shaders for each texture you render to.

 

It doesnt need to be particulary performant. The idea its to record multiple images, not run the simulation in realtime with the semantic label activated. Obviously should be playable, but no need to be superperformant.

Could you link some info or expand on this? Sounds interesting. To replace each shader, I have to modify each shader individually, or can I centralize the substitution to say "I want this object use this shader now, and this other object use this other"?

The principal obstacle is that I need to render both images (the scene per se and the semantic label), so in the end i need the replacement system to be a switch that can be turned on and off easily (if not, I could just susbtitute all materials for solid colors).

What I would like to avoid is modifying every single shader/object individually, because this is a one-time experiment.
I never used replacement shaders, so I dont know which posibilities do I have. Performance is not an issue, so no problem in rendering two times.

 

Alright in this case I think there are a couple options that would be easiest.

If you have less than 20 or so tags, I would say just use the unity built in tag system. the problem is this only allows for a limited number of tags. Otherwise, you have another option that is almost as easy. What I would do is add a gameobject to each object and just have an enum as part of the gameobject. the enum specifies the semantic label.

Then you need another script that when activated, will create an array of all objects in your scene. the array contains both the object and its current material. Then you will loop through each object and set the material on the object to the material specified for that enum (semantic type). Then you will call Camera.Render() on your camera (make sure to set a render texture to the camera first).

Once that is done, set your camera's render texture back to null, and reset the scene's object's materials based on the array you made before.

Do this every frame and you should be good to go.

Lmk if that makes sense.

 

Thats a good method. I got lost thinking about fancy solutions that I forgot the easiest way was to just simply tag all objects and iterate through them.
The more elegant way would be to use Replacement shaders, but since I dont want to add a custom shader to everything (I use a lot the Standard shader), I think the built in tag system could do the work.
This way I dont have to touch the scene more that to put the proper tags on objects, and I can specify different versions of custom shaders we have.
Still have to see how to store both passes to a image file. Maybe there's a way to link this to the Unity recorder?

 

The last time I used the unity recorder was rough Maybe it will be better for you though.

I would recommend just outputting a long list of image sequences then running that through some kind of command line tool to convert to a video.