Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

[Released] Beyond Magic: GPU Nodes for MapMagic

Discussion in 'Assets and Asset Store' started by CodersFromBeyond, Jun 29, 2019.

  1. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11


    [Documentation, Asset Store]

    GPU processing for MapMagic is out!

    Beyond Magic add-on will give MapMagic many long awaited improvements such as dramatic speed-boost, water flow node and other new features. Speed improvements are done by enabling GPU processing and alternatively by including an optimized code and parallel CPU processing support for the nodes. This will enable running MapMagic at a fraction of the time, with no loss of MapMagic features.

    Requires MapMagic!
    MapMagic is a procedural node-based terrain generation library, required for the add-on. https://assetstore.unity.com/packages/tools/terrain/mapmagic-world-generator-56762

    GPU processing and code optimization
    The add-on currently features GPU processing support and an alternative CPU optimization.
    All map nodes, as well as height, splat and grass outputs have been optimized.

    Simple to use!
    Automagically co-operates with MapMagic without separate setup. Contains one-click tool to automatically convert pre-existing MapMagic trees.
    Always back up your data before attempting to convert pre-existing scene.

    Expanded Functionality
    Erosion node has been expanded further with alternative algorithms. New map nodes Flow and Curvature added.
    The add-on is in active development with further features incoming!

    Requirements
    Parallel CPU boosts do not require compute shaders, works with MapMagic anywhere.
    GPU processing uses compute shaders! Platforms where compute shaders work:
    • Windows and Windows Store, with a DirectX 11 or DirectX 12 graphics API and Shader Model 5.0 GPU
    • macOS and iOS using Metal graphics API
    • Android, Linux and Windows platforms with Vulkan API
    • Modern OpenGL platforms (OpenGL 4.3 on Linux or Windows; OpenGL ES 3.1 on Android). Note that Mac OS X does not support OpenGL 4.3
    • Modern consoles (Sony PS4 and Microsoft Xbox One)

    Works with .NET Standard 2.0, .NET 4.x and .NET 3.5 equivalent (deprecated)
    MapMagic is obviously required as well.

    GPU vs CPU processing
    GPU processing handles numerical data slightly differently from CPU processing. We have worked hard to get the GPU to produce nearly identical floating point results compared to CPU version, but some floating point values will still round slightly differently (such as 1.00000000043 vs 1.00000000044). This will normally look identical, but some nodes (such as ruffle in erosion node) use these values as seed values, which means that even a fraction can produce different results in later nodes.

    Troubleshooting
    Q: I'm getting an error that MapMagicWindow is not a partial class.
    A: Make sure you have the latest version of MapMagic installed. Beyond Magic requires MapMagic v.1.10.4 or later to function.



     
    Last edited: Jul 5, 2019
    camta005 and Djayp like this.
  2. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
  3. Djayp

    Djayp

    Joined:
    Feb 16, 2015
    Posts:
    63
    CodersFromBeyond likes this.
  4. camta005

    camta005

    Joined:
    Dec 16, 2016
    Posts:
    207
    Very nice!
     
    CodersFromBeyond likes this.
  5. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    So I started going through my map's tree which currently can take a very long time to work, and right clicking each node to Convert to Beyond. I got a few nodes in and then the whole things died :( After converting a Blend node, the graph stopped working and stopped drawing properly and I see this error in the console.

    EDIT: Its constantly spamming the console. I'm going to try closing and restarting.

    Code (CSharp):
    1. NullReferenceException: Object reference not set to an instance of an object
    2. MapMagic.BlendGenerator2.OnGUI (MapMagic.GeneratorsAsset gens) (at Assets/MapMagic/Generators/MatrixGenerators.cs:966)
    3. MapMagic.BeyondBlendGenerator.OnGUI (MapMagic.GeneratorsAsset gens) (at Assets/BeyondMapMagic/Generators/BeyondBlendGenerator.cs:582)
    4. MapMagic.MapMagicWindow.DrawWindow () (at Assets/MapMagic/Editor/MapMagicWindow.cs:560)
    5. MapMagic.MapMagicWindow.OnGUI () (at Assets/MapMagic/Editor/MapMagicWindow.cs:340)
    6. System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at <23c160f925be47d7a4fd083a3a62c920>:0)
    7. Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
    8. UnityEngine.UIElements.UIR.RenderChain.Render (UnityEngine.Rect topRect, UnityEngine.Matrix4x4 projection) (at C:/buildslave/unity/build/Modules/UIElements/Renderer/UIRChainBuilder.cs:236)
    9. UnityEngine.UIElements.UIRRepaintUpdater.DrawChain (UnityEngine.Rect topRect, UnityEngine.Matrix4x4 projection) (at C:/buildslave/unity/build/Modules/UIElements/Renderer/UIRRepaintUpdater.cs:60)
    10. UnityEngine.UIElements.UIRRepaintUpdater.Update () (at C:/buildslave/unity/build/Modules/UIElements/Renderer/UIRRepaintUpdater.cs:48)
    11. UnityEngine.UIElements.VisualTreeUpdater.UpdateVisualTree () (at C:/buildslave/unity/build/Modules/UIElements/VisualTreeUpdater.cs:72)
    12. UnityEngine.UIElements.Panel.Repaint (UnityEngine.Event e) (at C:/buildslave/unity/build/Modules/UIElements/Panel.cs:636)
    13. UnityEngine.UIElements.UIElementsUtility.DoDispatch (UnityEngine.UIElements.BaseVisualElementPanel panel) (at C:/buildslave/unity/build/Modules/UIElements/UIElementsUtility.cs:240)
    14. UnityEngine.UIElements.UIElementsUtility.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr) (at C:/buildslave/unity/build/Modules/UIElements/UIElementsUtility.cs:78)
    15. UnityEngine.GUIUtility.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr) (at C:/buildslave/unity/build/Modules/IMGUI/GUIUtility.cs:179)
     
  6. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Reload of the map didn't help. My graph now seems to be broken and I can't figure out how to recover it :( Any thoughts?
     
  7. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    I'll message you, let's see what the problem is.
     
  8. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
  9. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    When switching over to GPU it gets even weirder. While now it appears to do -- something -- it seems very anomalous.

     
  10. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    Could you send us the .nodes file? Having hard time reproducing this.
     
  11. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    https://www.dropbox.com/s/2sqbmdd38o3n6p4/MM_BM_Erosion.nodes?dl=0

    I did some more testing and found a couple things.

    1) If I set "Erosion" to "Cliff" rather "MM-Like" then I get results similar to what MM was giving.
    2) The erosion node with "Generate Changed" rather than "Generate All" does not appear properly deterministic. Once it gets into a bad state as shown above, it will remain that way even if you switch it back to a known working state until such times as I Force Generate All to fix it up.

    I'm trying to nail down some consistent repro steps.
     
  12. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25

    Hi,

    Thank you for the .nodes files.

    I'm the main coder in Coders From Beyond and I have identified few of the problems and started working on the fixes.

    - Beyond Terrace Node GPU version with larger than 0.999f input values was not deterministic.

    If you need to contact me directly via Discord, please PM me.

    Br,
    Tommi
     
    CodersFromBeyond likes this.
  13. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Ok I need to walk back the statement about determinism. I don't think that is what's happening.

    But GPU is definitely causing very bad results for me. I am running with a GTX 1080ti.

    Parallel:


    GPU:
     
  14. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Ah ok so there was something going on in there. Trying all sorts of things (and waiting between them) makes it dififcult to be sure exactly what is happening.
     
  15. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Ok with that bit of information, I removed the Beyond Terrace and went back to a standard Terrace, and now I can generate everything else in GPU mode without anomalies. Although "MM-Like" still doesn't generate results like MM. I have to switch to Cliffs for that which seems wrong to me.

    So something about that Beyond Terrace is breaking the GPU flow.
     
  16. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    One last thing that I didn't notice originally because of the overlay MapMagic draws is seams. The Beyond Erosion seems to be generating far larger seems than standard MM Erosion does.

    Both these screenshots are with Safe Border set to 0, and "Margins" turned off in Beyond Erosion (What exactly is Margins supposed to do?) First is standard, second is Beyond.





    Sorry to barrage you with all these issues :)
     
  17. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25

    We are working on better seam blending, but it is still heavily in development.

    Erosion margins is option whether or not to use margins that the original erosion algoritm uses. All other nodes are calculated with the same resolution as the terrain (such as 512), but with erosion, additional 10 pixel margin is added to every side. This margin helps with border blending but GPUs are designer to work with power of 2 textures, such as 512. With margin the size is 532 x 532 and it is way slower to calculate that.

    When generating multiple terrains, it is best to use margins. And usually have some safe borders, such as 10.

    Tommi
     
  18. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Here are some erosion examples with different algorithms:


    As you can see, original and MM like are not quite identical, but some approximation of the original algorithm.

    But with tweaking the values, you can get pretty close.



    Tommi
     
  19. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Blend node fixed:

    - Beyond Blend Node GPU and Parallel version: Blending now works without base layer.

    We'll gather bug reports for few days and post a patch during weekend.

    Tommi
     
  20. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    I'm still continuing to see some very weird results out of the Beyond Erosion node. It is definitely a LOT faster which is great, and lets me experiment a lot faster, but I'm just getting wacky results more often than not. Such as:



    Compared to the same settings in standard Erosion:


    EDIT: Not sure whats going on with the images. They show fine when I'm editing the post, but not after I save. Here are links:
    Vanilla:

    https://i.imgur.com/kLpBcJ1.png

    Beyond:

    https://i.imgur.com/pJu1KBR.png
     
  21. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Can you provide the .nodes export once again, we will take a look :)

    In the meantime you can try to decrease some of the erosion values such as iterations count to see if that helps.

    Tommi
     
  22. Bodyclock

    Bodyclock

    Joined:
    May 8, 2018
    Posts:
    105
    Hi, Congratulations on the asset, it's already saving me a lot of time with iterating the possibilities and fine tuning. I have one issue with the Beyond Texture Input node. It produces a stepping effect on the terrain (see screenshots)
    Original Node
    Step2.png
    Beyond Node
    Step1.jpg
    This is with a 32bit EXR file using a BCH6 compressed texture. I've currently had to revert back to the original node.
     
  23. blacksun666

    blacksun666

    Joined:
    Dec 17, 2015
    Posts:
    156
    does this work with runtime infinite generated terrains?
     
  24. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Can you link a screenshot of the texture input node and parameters?

    And can can you check if GPU and parallel both produce stepping effect?

    You should still be able to use our Beyond node but select type: Original if you want to use the original.

    Tommi
     
  25. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Hi,

    Yes it works in runtime and with infinite generated terrains.

    There are few remarks that should be noted before using our add-on in runtime:
    • Be sure not to change dynamically between original, parallel and GPU nodes.
      • Most of the nodes produce seemingly identical results between different versions but there are still some floating point rounding differences between CPU and GPU versions.
    • Some of the nodes are close approximations of the original algorithms such as erosion and unity noise, which produce noticeable different results.
    • All of the generation and processing is done in main thread and in one frame when using GPU generation in runtime.
      • This means that game will freeze for the generation duration. For smaller MapMagic graphs this means duration of under a second but it is still noticeable.
      • This can be softened a bit with loading screens or dialogs during generation.
    • Our parallel version is better suitable for runtime generation. It uses all of the available cores to generate a terrain chunk, which can also produce some lag to the user.
      • This lag can be avoided by lowering the max processor core count in MapMagic settings.
    Tommi
     
  26. Bodyclock

    Bodyclock

    Joined:
    May 8, 2018
    Posts:
    105
    Hi Tommi,
    Screenshots as requested. Only happens in GPU mode so I'm using parallel.
    GPU.jpg
    Parallel.jpg

    Cheers
    Chris
     
  27. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Thanks for the screenshots. I'll start fixing bugs tonight.

    Tommi
     
  28. Bodyclock

    Bodyclock

    Joined:
    May 8, 2018
    Posts:
    105
    As a data point, the same happens with the Raw Input Node.
     
  29. Cosmoraph

    Cosmoraph

    Joined:
    Jan 25, 2017
    Posts:
    52
    Hi, I have a question. Is convert to GPU processing possible from the custom node? Or should I write my own compute shader?
     
  30. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Still seeing odd results in Beyond Erosion like the erosion node doesn't know how to handle sediment buildup like the Map Magic version does. The sediment will just stay in one space and build up into huge spikes in Beyond where in original it seems to fill in the cracks better and either vanishes before building up, or is forced to flow down regardless of fluidity. Not sure the specifics of how the original algorithm works. All I know is that with vanilla, low fluidity and high sediment amounts smooth out terrain very nicely, while in Beyond the same gives huge spiky messed up terrains or at best clumpy lava looking output.
     
  31. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    If you have your own custom node you will need to also write your own GPU code.

    However, we can provide empty GPU and parallel node template with proper commenting during next releases for making custom GPU nodes.

    All source code in the asset is included, readable and commented so it may serve as a good model for your own implementation.

    We can also provide support if you need help converting custom nodes to GPU.

    Tommi
     
    Last edited: Jul 2, 2019
  32. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    The sediment spread in original algorithm was not suitable for parallel execution so we did our best to find close enough alternative.

    You could try to use erosion with sediment output to our new flow node and use flow node output to substract from height. I will try to make sample scene with this behaviour for our next release.

    Did you try examples and parameters shown in this picture: https://i.imgur.com/fEr21cl.jpg

    If that doesn't work, you could try to make a simple scene with only noise, erosion and height output and try out some settings that could work.

    Br,
    Tommi
     
  33. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Interesting. I wasn't really sure how to use your new nodes as I didn't see any documentation on them. Thus I haven't yet experimented with them at all. I will try what you suggest though.
     
  34. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    There should be a Documentation folder in the asset.
    Additionally you can go to http://www.codersfrombeyond.com/beyondmagic/Manual.html to access the documentation online.
     
  35. jwvanderbeck

    jwvanderbeck

    Joined:
    Dec 4, 2014
    Posts:
    528
    Ah ok I missed that. I assumed because the Help link on your BeyondNodes goes to the original MapMagic nodes that you just didn't have docs yet. My bad!
     
  36. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Hi,

    I am unable to reproduce the stepping effect with some generated 32bit EXR height map files.

    Can you provide us your texture file to support@codersfrombeyond.com with screencapture of the texture import settings in unity.

    Tommi
     
  37. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    Thank you for helping us reproduce the issue, fix will be included in the next update.
     
  38. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    Version 1.0.1 has been submitted for Unity review, updates should be available soon!

    Following things will be included in the next update:

    Features:
    - Added more info prompts for context menu functions.
    - Added reminder to disabling multithreading where necessary.
    - Beyond node help button now links directly to online manual.

    Fixed bugs:
    - Beyond Terrace Node: GPU version with larger than 0.999f input values was not deterministic.
    - Beyond Blend Node GPU and Parallel version: Blending now works without base layer.
    - Beyond Height output: scale value is now saved.
    - Beyond Noise Node: Parallel and original versions now produce identical results (could be seen with erosion node with ruffle value greater than 0), gpu version nearly identical but slight floating point value differences remain.
    - Beyond Height Node: Beyond Height Node now works with Custom Shader Output nodes, custom material is not switched back repeatedly to built-in standard.
    - Beyond Texture Input & Beyond Raw Input: GPU generation works properly with 8k input.
     
    Djayp likes this.
  39. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    Version 1.0.1 is out with the first bugfixes!
     
    camta005 likes this.
  40. kilik128

    kilik128

    Joined:
    Jul 15, 2013
    Posts:
    837
  41. CodersFromBeyond

    CodersFromBeyond

    Joined:
    Apr 14, 2019
    Posts:
    11
    Hi,

    The renamed nodes are expanded version of the original nodes. So, for example, Beyond Slope would be an optimized version of the original Slope node. They can be used together with the other normal nodes though, and we have an optimized version for every map node and more. We also have a tool for hopefully converting nodes in your pre-existing scenes.

    We haven't actually tested GEMM so I can't say for sure.
    A custom node should theoretically work just fine with our nodes, but I am not sure what changes GEMM makes to MapMagic.
     
  42. kilik128

    kilik128

    Joined:
    Jul 15, 2013
    Posts:
    837
    ok sound good ! thank's
     
  43. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Good news!

    We have uploaded version 1.1 for review:
    • Beyond Noise node was updated to exactly match the Unity noise algorithm used by the original MapMagic Noise node!
    • We have also included a new original map generator node: Beyond Fractal.

    Tommi / Coders From Beyond
     
  44. kilik128

    kilik128

    Joined:
    Jul 15, 2013
    Posts:
    837
    you say Fractal ? we can see
     
  45. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    Version 1.1 is out!


    Tommi / Coders From Beyond
     
  46. tuskake

    tuskake

    Joined:
    Mar 5, 2013
    Posts:
    25
    It seems that our asset is not working with Unity 2019.2.

    We have a fix ready and we'll be posting an update to Asset Store soon.

    Tommi / Coders From Beyond