VertexDirt - Per vertex Ambient Occlusion baking

If you're referring to "Keep RGB, luminance to A" value in blend mode dropdown, - ignore that, I simply added a new blend mode I add to every VertexDirt release (keep RGB channels untouched, write AO to alpha - I use VD as part of my asset processing chain, and it's important to stop it from touching RGB channels which store material IDs in my workflow). I implemented it before I noticed the green bakes and reproduced same green bakes with both of the standard blend modes (overwrite, multiply) and with meshes which have no preexisting vertex data at all (like your sample wheel meshes - blend modes have no effect then).

If you're curious about it, it's just this inserted into ApplyColors:

Code (csharp):

1.  
2. else if (settings.blendModeIndex == 2)
3. {
4.     Debug.Log ("VertexDirt | Applying vertex colors using alpha-only mode");
5.     Color32[] originalColors = ch.originalMesh.colors32;
6.     Color32[] blendedColors = new Color32[ch.originalMesh.colors.Length];
7.     Color32 tempColor = Color.white;
8.     for (int b = 0; b < blendedColors.Length; b++)
9.     {
10.         tempColor = originalColors[b];
11.         blendedColors[b] = new Color32 (tempColor.r, tempColor.g, tempColor.b, vColors[b].a);
12.     }
13.     ch.colors = blendedColors;
14. }

First thing I did after discovering the issue was reverting all my customizations and reproducing the green bakes on a clean asset store release.

 

Just made a quick test by applying VertexDirt sampling configuration to my own camera and everything seems to be rendering correctly in every Unity viewport while exported PNG snapshots and bake results remain black and green.



So, something breaks down between camera setup and sampling, and it's not a shader or camera-related configuration. Maybe it's render texture format or something like that?

Upd: Hmm, nope, render texture looks fine - at least, in the editor inspector. Maybe it's something about Texture2D format and reading in VDSampler ?

 

I relieved that you added those enum to Control Panel Actually there are commented, experimental code lines there doing something similar, with similar look, so I worried maybe those things somehow activated.

It's green to me in Unity 5.6.3 too. My bad, I haven't tested on that exact version before.

 

Using TextureFormat.RGBAFloat in VDSampler fix the problem!

 

Thanks, this works! Might be a Unity bug, I don't see why ARGB32 or RGB24 shouldn't work for render texture sampling.

By the way, on another subject - after looking a lot at your baking method, I think I can propose an improved rendering mode which will work better for large-scale baking (architectural models and such). Your current approach is to sample the average color of 64x64 pixels where every pixel is either black (belonging to a surface point within the sampling range) or white (belonging to an out of range surface or sky). This approach works well for small props, but at large scale it might produce very abrupt shadow transitions where, for example, a corner of a distant building can have zero influence at a baked object at sampling distance N, but would heavily shadow it at sampling distance N+1 - because no matter the distance, occluders introduce pure black samples into the rendering.

In the real world, distance gives scattered light an opportunity to reduce the ambient occlusion even if no straight path to the sky exists in a given direction. This effect can be easily imitated by introducing a new setting (fade distance) and making only samples at sampling distance or below black, while lerping samples to sky color as they approach sampling distance + fade distance sum. That's literally linear fog, making this very easy to implement. To be more specific, implementation involves only this:

• Save all fog settings from RenderSettings
• Set fog on, fog start distance to sampling distance, fog end distance to sampling distance + fade distance sum, fog mode to linear and fog color to sky color.
• Use a different replacement shader that does nothing aside from outputting constant occluder color and utilizing UNITY_APPLY_FOG function.

That's it, now the sampling camera will see a smooth transition between occluder color and sky color, eliminating cases where faraway surfaces have too much influence. A user can set the start distance to very low value and end distance to a very far value to get a smooth look (e.g. a very humid day where ambient occlusion begins to be cancelled out early), or just utilize the fade distance as a way to prevent undesirable heavy shading from objects on edge of sampling distance, and you can completely replicate the current standard rendering mode by just setting fade distance to 0.01, turning the edge between occluder and sky color pin sharp.

Here is the shader for that approach - it has absolutely no need for multiple passes, more than one global color or any length calculations:

Code (csharp):

1.  
2. Shader "Hidden/VD-AMBIENTOCCLUSION-SMOOTH"
3. {
4.     SubShader
5.     {
6.         Tags
7.         {
8.             "RenderType" = "Opaque"
9.         }
10.  
11.         LOD 100
12.         Cull Off
13.  
14.         Pass
15.         {
16.             CGPROGRAM
17.  
18.             #pragma vertex vert
19.             #pragma fragment frag
20.             #pragma multi_compile_fog
21.  
22.             #include "UnityCG.cginc"
23.  
24.             struct appdata_t
25.             {
26.                 float4 vertex : POSITION;
27.                 float2 texcoord : TEXCOORD0;
28.             };
29.  
30.             struct v2f
31.             {
32.                 float4 vertex : SV_POSITION;
33.                 UNITY_FOG_COORDS (1)
34.             };
35.  
36.             fixed4 _VDOccluderColor;
37.  
38.             v2f vert (appdata_t v)
39.             {
40.                 v2f o;
41.                 o.vertex = UnityObjectToClipPos (v.vertex);
42.                 UNITY_TRANSFER_FOG (o,o.vertex);
43.                 return o;
44.             }
45.  
46.             fixed4 frag (v2f i) : SV_Target
47.             {
48.                 fixed4 col = _VDOccluderColor;
49.                 UNITY_APPLY_FOG (i.fogCoord, col);
50.                 return col;
51.             }
52.  
53.             ENDCG
54.         }
55.     }
56. }

And here is new code in the Dirt method of VertexDirt.cs:

Code (csharp):

1.  
2. // Caching fog settings
3. bool fog = RenderSettings.fog;
4. float fogStartDistance = RenderSettings.fogStartDistance;
5. float fogEndDistance = RenderSettings.fogEndDistance;
6. float fogDensity = RenderSettings.fogDensity;
7. Color fogColor = RenderSettings.fogColor;
8. FogMode fogMode = RenderSettings.fogMode;
9.  
10. // Applying our fog settings
11. RenderSettings.fog = renderingModeIsSmooth;
12. RenderSettings.fogMode = FogMode.Linear;
13. RenderSettings.fogColor = cam.backgroundColor;
14. RenderSettings.fogStartDistance = settings.samplingDistance;
15. RenderSettings.fogEndDistance = settings.samplingDistance + settings.fadeDistance;
16.  
17. // Rendering is done here
18. ...
19.  
20. // Restoring user fog settings
21. RenderSettings.fog = fog;
22. RenderSettings.fogStartDistance = fogStartDistance;
23. RenderSettings.fogEndDistance = fogEndDistance;
24. RenderSettings.fogDensity = fogDensity;
25. RenderSettings.fogColor = fogColor;
26. RenderSettings.fogMode = fogMode;

I implemented this as a new mode on my side, but since this approach can completely replicate old baking results if settings.fadeDistance is set to a low value, it might be a better approach to just move everything to this.

Thanks for the help, hopefully this would be useful advice.

 

Apparently it only works in DX11 API, and fails in DX9, with RenderTexture to texture2D conversion error messages spawning. Moreover, using RGBAFloat on Texture2D for that situation shouldn't be work well, or at all as the RenderTexture is not a Float...probably they messed up the TextureFormat enum in the Unity code itself as it seems other versions works as expected. I'm trying to figure out how to Debug that.

But to be sure, I have to test all Unity version from the supported oldest to the latest, and ifdef-ing the broken versions to some workaround or at least a display a warning prompt....
...unfortunately there are 39 (!!) versions released started from 5.1.0....probably I have to raise the number of the supported minimum version...

Thanks for writing about your approach, I have similar in my mind for a while, but I wasn't sure how to put in to the existing workflow. AO is not exactly a real physical thing, and I think it's beauty and universality comes from it's simplicity. But introducing a flag like 'smooth", as you mentioned may do the trick. I think about it.

Actually that two-pass shader which is in use right now just an experimental one trying to fix a baking bug, but I already changed it for a more simplified one VertexDirt 1.6.1 is on the way

The pleasure is mine! I appreciate the comments to the plugin, having feedback and great ideas are helps a lot!

 

Hi. I need to know if this asset is capable of generating an vertex based AO if two non combined meshes (as different game objects) intersect with each other? I would love to buy this at Sale right now. So a quick response would bei great.

 

Yes, it is capable for that

 

Hi, I'm a fan of this plugin. After I used this plugin,I encountered a little problem. The problem in this pic is that the borders of AO on the floor are very sharp. I think the reason is due to the vertex color of the floor. How can I get a better soft AO? Thanks a lot.

 

Hi! The floor probably not occluded in the other side of the wall. Please send me a sample project so I can give you advices.

 

I get a problem. When i try to bake a set of Gameobject with lots of mesh and high polycount. The editor keep holding more and more memory and then crash. Do you know why?

 

Hi!

Sorry to hear that.
The reason for the crash is probably because of lots of mesh and high polycount.
What is the mesh/polycount on the scene you try to bake?

 

i think it's >300K vertices
I have tried to remove some part of the plugin's codes, and i see the cam.Render() is the part, and then i don't know how to improve

 

Probably it's some sort of memory leaking while rendering the samples.
I'll have a look on this.

Thanks for reporting it.

 

I bought this asset and when I try to compile my application standalone, I get the following error:

Assets/Plugins/VertexDirt/Scripts/VertexDirt.cs(256,5): error CS0103: The name `Undo' does not exist in the current context

 

Hello!

Sorry about that, please comment out that line!

 

Hi. Just bought your asset, it performs great so far.
I would like to suggest some improvents to user interface though, namely the mesh-saving window. Currently it can't save meshes to a non-existent folder, we need to create the folder manually first, which gets quite annoying if I want, for example, to keep meshes for many levels in separate folders. Would be more user-friendly if you could integrate a standard folder selection dialog, instead of a simple text input field for the path. And it would also be nice if the window could remember the previously used folder path after being re-opened.

 

Thanks for buying! I noted the improvements, and I ket you know when they ready for release.
Best

 

Hello! Does it work with pro builder?

 

Can you share the vertexDirt settings you used for that result ?
I do use your plugin for archviz and VR visits of interior, but I find it difficult to get proper ersults for simple rectangle rooms..
Thanks !

 

Please write me an email!

 

Is this plugin compatible with HDRP?

 

Hello.

Bake are not, results are. It is soon fixed so you can bake in any SRP too.

 

Hi I tried to use this plugin in Unity 2020.1, URP, with a simple scene with 3 cubes, the progress time just keeps increasing and the process does not stop. The vertexdirt settings are at default.

 

Hi there, are you still supporting this?
I just bought it, because I needed a vertex AO solution.

This tool would, in my opinion, sell quite some copies if you updated it a bit and polish it.
See, the Quest has sold quite a number, and devs are in need of tools and tricks to better up the visuals for headset resident apps and games.
I find that vertex baked AO is a great way to increase the visuals quality with essentially no overhead added.

However, the tool feels a bit incomplete, probably because it wasn't thought to be a scene baking solution, but mostly a AO solution for isolated meshes.
Anyway, I left a review and I paste it here below to mark the improvements needed to make it a complete tool:

It does the job, it's bugless, but it's quite limited.

====================

In example, you can't even set up the number of samples to get, which would be a pretty quick value to expose on the tool window.

====================

It also lacks a shader for Cutout Transparent objects. In example, I have some bushes, they are static and part of the scenery, and I can't bake them.

====================

One last problem is that (for what I can see, at least), there is no way to revert back to the original mesh.
This means that if you update the mesh after baking, if you add detail because you see that it's not sufficient, the baker won't know it, and you will have to manually replace the mesh in the object.
The strange thing is that there IS a reference to the original mesh in the script that gets appended to the mesh, so the tool KNOWS which mesh is the original, and could recover it... but it doesn't.

This NEEDS to be automated, otherwise, if you happen to need to update the meshes in your level, it will be a REAL PAIN to deal with. It means that the tool can only be used when you are ABSOLUTELY SURE that you won't update your models anymore.

At UI level, this could be achieved easily by adding a button to reset to the original mesh. It's that easy.
One for the single mesh, and one for all the meshes in the scene that were affected by the tool.

On the other side, the rendering is blazing fast in nature, practically no overhead, and the baker is fast too, and you can bake the objects one by one, which is a great plus as well.

====================

All in all, a great small tool for those that need a FAST AO baked solution, both in baking and rendering, but that feels incomplete due to the points exposed.
I will wait for these 3 small features to give put there the 5th star! XD

P.s.: one last thing: the tool won't save even the last parameters you used to bake. You have to write the parameters down if you want to keep a uniform style with more baking later on.

Man, if you upgrade the tool with the suggested points, be sure I'll make you a good present with the money coming from my project.

 

For anyone who, like me, had the need to refresh the meshes, just remove the component VD COLOR HANDLER (script) and you will have your original mesh back.

 

Hello!

Thanks for the review, I appreciate it!
I'll think about your suggestions.

Best, Zoltán

 

Thank you!
Another problem: as default, the addon works on all the children as well.
This is ok, but... in some instances, this is not good.
I have a static mesh to which are appended a few components, and one is a Textmesh dynamic text that I need to change at runtime.
VertexDirt adds its component to it, introducing a "number of vertexes" error, and stopping the execution of further code on it.

 

Oh, I see, I have to handle that.