Search Unity

VertexDirt - Per vertex Ambient Occlusion baking

Discussion in 'Assets and Asset Store' started by ZFarago, Sep 11, 2014.

  1. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hello,

    VertexDirt is my hobby project at home, under development for a year now (well actually it could be done in weeks if I had more freetime :) ), and finally the alpha version is live at the Asset Store! Try it if interested.

    Asset Store Link
    PDF Overview
    Original WIP thread
    Sample Video: Spheres

    VertexDirt is a Unity plugin for baking many things (like AO) to vertex colors.
    The baking itself is a quick progress, but not realtime, so the method is to select some object, tune the parameters, hit the Bake button and wait for the results.

    Currently, the source is ready to bake many things, but actually only the Ambient Ocllusion Tool window exists yet. This means, if you are friendly with scripting (and shading), you can easily expand the basic tool to bake more exotic things. VertexDirt includes fully documented source code.

    Alpha version is for testing and feedback purposes, so feel free to comment when you had try. This is the time when I am going to spend to fix mistakes and bugs, so big new features will come after this stage.

    Screenies:




    Cheers,
    Zoltan.
     
    Last edited: Oct 7, 2014
    UnityLighting and TVishahan like this.
  2. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Any feedback guys? I would appreciate if you would write me your thoughts.
     
  3. TVishahan

    TVishahan

    Joined:
    Oct 21, 2013
    Posts:
    5
    Amazing! Excellent work you got there! However I would like to ask about this white shade at the bottom of the model, isn't it supposed to be dark? or am I doing it wrong?
    Screenshot 2014-10-07 14.24.34.png
     
    Last edited: Oct 7, 2014
  4. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Thank you TVishahan. You may try to move up the building a little bit, as the vertices on the bottom corners are to close to the ground or maybe lower than it. I hope it will help. It would be really helpful if you could send me a sample project with this exact bug.
     
  5. TVishahan

    TVishahan

    Joined:
    Oct 21, 2013
    Posts:
    5
    Alright so I move it up a little but now I don't get the occlusion from the bottom plane (or effect is not powerful enough), here is the sample scene.
    [UPDATE: Sent via email]
     
  6. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Please send me via email and remove it from here. I will check it later.
     
  7. TVishahan

    TVishahan

    Joined:
    Oct 21, 2013
    Posts:
    5
    Oh, by the way, do you plan on creating a textured shader and a lit shader?
     
  8. TVishahan

    TVishahan

    Joined:
    Oct 21, 2013
    Posts:
    5
    Alright, after taking your advice and implementing my own (partially) colour shader, It looks great!
    Screenshot 2014-10-08 05.42.09.png
     
  9. Warrior1424

    Warrior1424

    Joined:
    Sep 30, 2010
    Posts:
    984
    Hello!
    Everything works fine until I try to build a project.
    When I try to build my project, it stops it because there are compiler errors:
    This error only appears when trying to build a project, and never appears at any other time.
     
  10. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hi!
    This was unfortunately my mistake, the plugin's file structure is wrong. I have fixed this bug couple of days ago, but it is not live at the store yet. If the fix is urgent to you, please write me in private.
     
  11. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    New version is out, reported bugs fixed, two new sample scenes and several vertex color shaders included.
     
  12. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    VD Ambient Occlusion - different parameters, different results.
    If you change the sampling angle, you can modify the contrast of the results. Wide angles makes more details visible but with less contrast.
    Occlusion distance controls the range of the occlusion. If you are using really small values, a specific verticle only shaded by the closest neighbours. Using large values can affect a vertex from long distances.
    vd_ao_parameters.jpg
     
  13. TVishahan

    TVishahan

    Joined:
    Oct 21, 2013
    Posts:
    5
    That looks amazing, can't wait to try it out!
     
  14. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Well, this is already on the Store :)
    This picture is just a "cheat-sheet" for the good old AO function :)
    New features still under development, pictures soon.
     
  15. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Good news everyone who experiencing plain grey results: bug fixed, new version sent to the Asset Store. Should be out soon. :) Sorry for the inconvenience guys.
     
  16. Zomby138

    Zomby138

    Joined:
    Nov 3, 2009
    Posts:
    657
    Hello. I just started using this tool and it seems pretty good! I have hacked the Unity standard shader so it multiplies the diffuse colour by the vertex colour. This works really well and gives the objects a good AO look. I've managed to save the coloured meshes into prefabs so that they get used all round the scene successfully. All good.

    But unfortunately I have a problem. Everything looks good when I'm running in the Unity editor, but when I make and run a build, the meshes that have been modified by the vertex dirt plugin are all messed up :(

    It looks like it might be something to do with the enlighten light map, like the lightmap UVs are messed up or something. Do you know anything about an issue like this?

    I can't take any screenshots right now because I'm at work, but I might be able to take some tonight if you think that would help.
     
  17. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    hi.
    Sorry for the trouble,
    Would you please send me a sample project via private?
    I'm not sure what the problem is, maybe you're right and enlighten conflicts with VertexDirt, so I am going to find out.
     
  18. Zomby138

    Zomby138

    Joined:
    Nov 3, 2009
    Posts:
    657
    I'll make a small test scene tonight. Have you tested it with enlighten before?
     
  19. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Not much yet.
     
  20. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    I have tested it with enlighten. Working on to fix the problem.
     
  21. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Okay, problem fixed, soon in the asset store (testing needed under different unity versions, plus the review time)
    -completely rewritten VDColorHandler Component
    - correct prefab control, no need of saving meshes as assets
    - no mesh leaks, or missing mesh / colors anymore (rare think, but it could happened before)
    - works with static / dynamic batching (could messed up enlighten results on build before, see the quote)
    - works with enlighten realtime prebaking and normal baking.
     
  22. Zomby138

    Zomby138

    Joined:
    Nov 3, 2009
    Posts:
    657
    All sounds good!
     
  23. yoDon

    yoDon

    Joined:
    Aug 29, 2008
    Posts:
    44
    I'm trying to use your asset for the first time and am unable to reproduce the baking in your sample file in 5.3.4f1 without turning on legacyMode (which the code says is only intended for use with Unity 4.x projects).

    When legacyMode is off, the brightest value the code bakes into a vertex is about RGB(203,203,203). When legacyMode is on, the same settings bake RGB(255,255,255) into the non-occluded vertices. I'm using a distance of 100 and an angle of 100, but the darkening happens for all values when legacy mode is off. This extra darkening in non-legacyMode is problem because it makes the baked assets not compatible with existing lighting and scenes. I'm assuming I should just go ahead and use legacyMode but the code is pretty clear legacyMode is for Unity 4.x and I didn't know if this might be a sign of deeper problems.

    I'm also not finding any guidance in the docs or the code on when to use compatibilityMode, can you give any info on when one does or doesn't want compatibilityMode?

    (all of this is with VertexDirt 1.59.31)
     
  24. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hi!

    Please use the legacy mode switched on in these cases. It's a bug in the code. Don't worry about Compatibility mode.
    Sorry for the inconvenience.
     
  25. Alverik

    Alverik

    Joined:
    Apr 15, 2016
    Posts:
    413
    Hi, just a question. I'm interested in your asset, it looks amazing, but I'm using models painted in blender using vertex colors. Is there a way to use VertexDirt without overwriting a model's original vertex colors? like, just blending the new AO over the previous vertex colors? or is the process destructive?
     
  26. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hi Alverik,
    It is "destructive" right now, but really easy to change it to something else, like multiplying in code. (I can help you with that)
    Next version will be using multiplying as it was already a feature request :) But it has some other changes needed to be documented (writing docs is my weakness - sorry), so it is not available in the store yet.
     
    Alverik likes this.
  27. Alverik

    Alverik

    Joined:
    Apr 15, 2016
    Posts:
    413
    Sorry for the late answer. Anyways, thanks for your response, I really want to try the asset out, but due to personal reason's/budget I'll have to wait a bit. Hopefully I'll be able to catch the next version. If the AO can be multiplied over the original vertex color, then it will really be a God send!
     
  28. TheodoricSacron

    TheodoricSacron

    Joined:
    Nov 22, 2014
    Posts:
    2
    Hello - Avagy tiszteletem a kolleganak :)

    A friend of mine showed this plugin to me and recently I bought it as well.
    However I found a little error which makes our life a bit harder on our small project.

    I make small group of objects, put them in prefab and use vertexdirt and save prefab.
    So far so good in the editor everything is as expected.

    But when the code use the prefab and place it to the level there is nothing to render.

    The reason is in the asset browser when I open the prefab, all the models missing the Meshfilter's target - it's own model.
    If I put the prefab to the editor it works again, but the prefab itself doesn't have them, however if I set the proper models in the prefab in the asset browser, then everything works perfectly.

    However everything is lost again if I change something in the prefab and save again - and it happens a lot during development... :)

    So my question is, could you take a look at it, why it's breaking the connection in the Meshfilter component when saved in prefab?

    Also the AO works a bit funny. I have an object what doesn't generate any useable AO, not even in the deep corners. I tried many parameter variation, but nothing. I cleaned up the mesh in many ways what I know in 3D app, it seems okay in every way but still...

    Well beside these it's an awesome plugin, I can only recommend!
    Cheers!
     
  29. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Szia! :)

    when the code use the prefab and place it to the level there is nothing to render
    I'll check this prefabing bug, really sorry about this. Make procedural meshes and prefabs working together is an old task for me, and I almost thought I fixed all related bugs in the latest version :)

    I have an object what doesn't generate any useable AO, not even in the deep corners
    Some kind of geometry can have really strange looking AO. Please send a sample model via email and I'm going to check if I can figure out some sort of solution.

    Kind Regards, Zoltan.
     
    Alverik likes this.
  30. made_it_app

    made_it_app

    Joined:
    Mar 2, 2014
    Posts:
    4
    Hi, a small question prior to purchasing – is it possible to bake in runtime? Or alter the code to make it bake in standalone build? I realise there would be a lag, but technically - would it be possible or there are some Editor API involved on which the whole baking system relies?
    Thanks,
    Olaf
     
  31. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    TheodoricSacron:
    when the code use the prefab and place it to the level there is nothing to render
    I have an object what doesn't generate any useable AO, not even in the deep corners

    Both bugs are fixed, and available on Asset Store :)

    made_it_app

    Not tested for a long time, but is should work runtime, yes.
    The main VertexDirt class is a static UnityEngine class, so its available on runtime for other scripts.

    Just call VertexDirt.Dirt(Transform[]) from your script and the bake starts.
    To set the properties of the Bake, fill up the VertexDirt.settings structure with your parameters.
    This is a VertexDirtSettingsStruct class and you can find its parameters documented in the code of theVertexDirt class.
    The VertexDirtWindow editor class is a good example for accessing VertexDirt.

    And yes, the run will be freeze for the time the baking being calculated.
    I hope that helped :) If you have further questions, please write me via email.
     
  32. made_it_app

    made_it_app

    Joined:
    Mar 2, 2014
    Posts:
    4
    @ZFarago Thanks for the answers, on other question: does it support fragmented shaders or just surface surface (if both support vertex color)?
     
  33. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    It should work with fragmented shaders. Any kind of shader wich is supports vertexcolors, will work fine.
     
  34. ARQAlejandroMontes

    ARQAlejandroMontes

    Joined:
    Jun 21, 2015
    Posts:
    3
    Hello, I have a question prior-purchasing.

    Is it possible to achieve something like this, on this kind of geometry (little buildings)? I mean, for the small details and more.


    Thank you.
     
    Alverik likes this.
  35. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hi!

    Well, it is possible to get similar results, but need detailed geometry with good and precise topology.
    I suppose it would be easier to use lightmaps instead, but of course it will have a smaller memory footprint.

    I made a quick test to showcase this scenario.

    lowpolybuildings.jpg

    I hope this helps.
    Kind Regards, Zoltan
     
    Last edited: Nov 29, 2016
    ARQAlejandroMontes and Alverik like this.
  36. ARQAlejandroMontes

    ARQAlejandroMontes

    Joined:
    Jun 21, 2015
    Posts:
    3
    Hello,

    Did you model that quick scene just to answer my question? I think I love this plugin even without buying it yet haha. :) I think this result is perfect for what I need.

    One last question.

    I understand when you say "VertexDirt is a Unity plugin for baking many things (like AO) to vertex colors. (...) but not realtime," that means that it is not replacing the original lightmaps? Sorry if it's a confusing question, I'm not an expert on the subject.

    I have this situation.
    1. A scene (standalone build) with some objects, these already have their own lightmaps or baked textures for each INDIVIDUAL.
    2. The possibility of adding new objects (placeholder).
    3. But if I put two objects together, even though each has its own lightmaps, there is no AO for the relationship between the two.
    So, is it possible to generate that AO of both objects without overwriting the lightmap, a kind of combination?



    Thank you very much. Amazing support. :cool:
    Sorry my english.
     
  37. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Sure I did! :) I like to test VertexDirt in different cicumstances and it didn't took too mucj time either :)

    Well VertexDirt is an "offline" generator which means the results needs to be prebaked, as it is not able to generate the AO again and again in every frame. You can use the baked results in any kind of builds in every platforms of course.

    One more thing, VertexDirt is not offically supports baking in runtime, because I'm not tested it completely yet, but it works in theory (it will probably hang the game while baking)
    The main VertexDirt class is a static UnityEngine class, so its available on runtime for other scripts.
    Just call VertexDirt.Dirt(Transform[]) from your script and the bake starts.
    To set the properties of the Bake, fill up the VertexDirt.settings structure with your parameters.
    This is a VertexDirtSettingsStruct class and you can find its parameters documented in the code of theVertexDirt class.
    The VertexDirtWindow editor class is a good example for accessing VertexDirt.

    Probably you can update the AO in runtime with VertexDirt, but I'm afraid the updating itself will take at least a couple of seconds and hangs the game until. Other thing is the topology and placing of the buildings. Vertex AO (and not just VertexDirt) could be behave really bad sometimes in cases like that. With detailed object it should works fine, but the more vertex count the models using will lead to greater hangs while the AO updates.

    You may try to do some tests with image effects too like SSAO as they can produce realtime results.

    If you have further questions, please feel free to contact me in email and I give you more precise help.
     
    ARQAlejandroMontes and Alverik like this.
  38. ARQAlejandroMontes

    ARQAlejandroMontes

    Joined:
    Jun 21, 2015
    Posts:
    3
    I think it's a perfect solution for me because "VertexDirt can bake objects one by one" - I mean individual, I just read the documentation about "save baked meshes" and in any case I know it will be very helpful.

    Awsome, thank you!
     
  39. weeblegleep

    weeblegleep

    Joined:
    Mar 13, 2015
    Posts:
    1
    hello,

    This is a great tool, Thanks.
    I do have one error that i am not sure how to handle. Please see below:

    Mesh.colors is out of bounds. The supplied array needs to be the same size as the Mesh.vertices array.
    UnityEngine.Mesh:set_colors32(Color32[])
    VDColorHandler:Awake() (at Assets/Plugins/VertexDirt/Scripts/VDColorHandler.js:19)

    do you know what might be causing this? the meshes look fine, but i keep getting this error. Please let me know what might be wrong here. Thanks
     
  40. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hi!

    It looks the vertex array becomed shorter than the Vertex Color expects. If you have a mesh with a color handler and update the mesh, the verticle count can be different, depending of the mesh changes. Is the error shows in Play mode only, or always in the editor?

    You may try to Rebake that particular object and check if the problem resolved.
     
  41. Reanimate_L

    Reanimate_L

    Joined:
    Oct 10, 2009
    Posts:
    2,567
    @ZFarago can i ask a feature request?
    vertex color channel selector where we can select to which channel the vertex ao to be applied into the vertex data?
    Edit : would like to modify it but i'm not good with js :/
     
  42. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Sure thing! Will be in the next version!
    Until that, just throw me a mail and I help you to modify.
     
  43. Reanimate_L

    Reanimate_L

    Joined:
    Oct 10, 2009
    Posts:
    2,567
    sending you a PM
     
  44. tolosaoldfan

    tolosaoldfan

    Joined:
    Sep 14, 2012
    Posts:
    66
    Hi, the tool doesn't work on my Skinnedrendered mesh. Is it normal ? The console says :
    "Baking of 1 objects (0 vertices) finished in 0 seconds.
    UnityEngine.Debug:Log(Object)
    VertexDirt:Dirt(Transform[]) (at Assets/Plugins/VertexDirt/Scripts/VertexDirt.js:132)
    VertexDirtWindow:OnGUI() (at Assets/Plugins/Editor/VertexDirtEditor/Scripts/VertexDirtWindow.js:87)"

    If I apply on a badic object, it works (so the plugin works and is well installed).
    I tried also on several Skinned meshes with the same result.
    Version is 1.59.11
    Thanks
     
  45. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Hi! Skinned renderers not supported, sorry. I can add this feature in the future.
     
  46. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
  47. bac9-flcl

    bac9-flcl

    Joined:
    Dec 5, 2012
    Posts:
    816
    @ZFarago
    Glad to see the update to C#!

    Unfortunately, new version doesn't seem to work correctly for me. I'm using Unity 5.6.3 and the bake with the following settings in your sample wheel scene results in this:



    It's probably not an intended output (for some reason, green channel is the only one receiving an AO bake, and on some of my custom meshes I occasionally see pink-ish areas in the areas which should be deep black). I couldn't pinpoint the source of the problem yet, but it's probably related to some typo in Color or Color32 constructor somewhere, mixing up the channels.

    Checking the issue further with a custom shader, it looks like vertex AO is only ever written to Y (green channel) while X/Z (red/blue channels) remain at 0. Range of values written to green channel is capped somewhere around middle gray and never reaches 100% even on completely unoccluded surfaces. This also has a side effect of luminance written to W (alpha channel) being even darker. Not sure where green-only color is coming from, though.
     
    Last edited: Sep 23, 2017
  48. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    uhh sorry about that! Well, actually I'm a bit jealus because I've tested for weeks and never turned to Hulk to me...
    I look into the problem, and post an update to the store. Until that, please throw me a mail so I can send a patch immediately once it's done.
     
  49. bac9-flcl

    bac9-flcl

    Joined:
    Dec 5, 2012
    Posts:
    816
    Please send it to if you'll figure it out, thanks!

    I've checked it a bit more, and it seems to be a problem arising from bad texture content. Just uncomment the commented lines you left in OnPostRender method of VDSampler, and you'll see this:



    Just as expected, the values are always at 0 in red channel, 0 to ~0.5 in the green channel, and 0 in the blue channel for every single step. Maybe it's a problem with the shader that's used for the bake?
     
  50. ZFarago

    ZFarago

    Joined:
    Aug 13, 2012
    Posts:
    154
    Probably, but your Control panel window also have strange enum in the "Blend to existing colors" property, I have to investigate that too.
     
unityunity