Search Unity

Assets [WIP][Sept 16th] Terrafirma - An Auxiliary Terrain Engine Replacement and Extraction Toolset

Discussion in 'Works In Progress' started by Murpenstien, Dec 31, 2016.

Thread Status:
Not open for further replies.
  1. HeadClot88

    HeadClot88

    Joined:
    Jul 3, 2012
    Posts:
    691
    Got a few Questions -

    1. Will this work with World Streamer?
    2. How Well does this handle performance wise with large worlds?
     
    Teila likes this.
  2. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Great questions, lol. :) Sorry in-advance this may be a long one..

    1. As of right now I haven't consciously designed any of this to work with that particular asset. Though if world streamer can load prefabs from disc. I don't see why it would have an issue loading a terrain generated with my system, though in regard to the specialized culling/lod stuff it does to unity terrains, I'm going to say that's not going to be compatible. I may be able to compensate on that end already though. I really have to look into this one more to be perfectly honest. When I have some spare time since your more familiar with World Streamer than I am, I can write up something for you about what's actually going on under the hood to give you a better idea of what's going to compatible and what's not.

    2. This system handles massive terrains very well. The only real bottle neck you'll encounter when using the memory schemes would be on the gpu's end. In terms of central processing performance this all comes down to how far out level handling is processing. So basically a 30000x30000 terrain having a radius of 2500 for the last lod distance to render up to will process just as fast as 5000x5000 terrain with same lod settings. In regards to all of this there's a lot to detail. The terrain in the demo I have posted is pretty good example though. I get the same fps (1000fps) on a 500x500 terrain object as I would in the demo which is running 4000x4000. For comparisons sake, I get around 400-500fps on the 500x500 using Unity Terrain, and 250-290-ish using Unity again on the 4000x4000.

    It can also generate 2000^2 unit parcels of terrain with no noticeable stutter,(Haven't really looked into how it performs with larger ones atm), so generating terrain in real time shouldn't be any issue, the real time mesh handling can generate terrains even faster, and in terms of processing it should be just as fast as updating a Unity Terrain when moving with the exception that world size is not going to effect your performance nearly as much. The only real limitation is going to come down to triangle counts composing the mesh along with memory. In the case of system ram though there's no reason you can't load in and load out terrain objects out side your cameras draw distance to reserve memory as you would traditionally.

    Hope this helps. :)
     
    antoripa and Teila like this.
  3. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,719
    The special LOD stuff is just a ring of low poly meshes substituted for the distant terrains. As your player walks, the low poly meshes are replace by your regular terrain/mesh as the chunks stream into view. I would think if your terrains are meshes that it should work. With your LOD system, one might not even need to use the low poly substitute meshes in the distance. :)

    I plan to test the two together soon after release.
     
  4. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Holy crap sorry for not responding sooner ! There should be no issue, though it'll depend on how you set things up. You'll most likely need to set up two objects, one for each camera. I'll set up a demo running split screen soon, just finalizing a few things at the moment. I have toyed with the idea already though, having one object being drawn to the scene view camera and the other being drawn by the game view. In regards to performance and memory though this should hypothetically be no problem for an xbox one to handle.
     
  5. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Thanks for the clarification ! :)

    Update
    Just a small update, I'm going to push back release a few days. Got snagged up having to implement a custom undo/redo scheme for the editing portion. To give an idea of what needs to be done, essentially I just need to finalize a few things with the editor interface and finish up the documentation. I figure this will take me no longer then a week.

    For the time being though i'll post an updated demo with tree handling later today. It'll also perform much better meshing wise as I've done alooooot of work under the hood. Things were a mess, but everything is now stable. :)

    In-regards to performance now there's a lot to address, I'll fabricate a super update later today and try cover everything I've added and changed since my last actual update, pictures, videos and demo included and lots of words!

    Would also like to apologize to anyone who's been in contact with me over the course of this projects development. This was partly an act of love and desperation for me. When I started this I was squirrelly at 140 lbs, as of now I look like sol 400 Matt Daemon at 117, lol. Basically all of my time and attention has went into this over the past few months, and though my correspondence has been lacking at times just wanted to put it out there that once I finish what I have to up I'll take the time to properly correspond with everyone and start working towards integrating all of the amazing systems that people of this community have come forward to me with thus far ! Thanks everyone for your interest, I honestly don't believe I would have realized half the functionality I've added since starting this work in progress thread without the interest and attention I've received from everyone, so in that regard thank you for pushing me and inspiring me to expand on what this was initially conceived to be to what it is today.

    Would also like to mention I'll start to post updates on a more regular basis again. The past couple weeks have just been.. "Stressful", to put it lightly, lol.
     
  6. tequyla

    tequyla

    Joined:
    Jul 22, 2012
    Posts:
    327
    Hello,

    is possible to dig the terrain (cave, etc..) ?

    ++
     
  7. ConAim

    ConAim

    Joined:
    Jan 17, 2017
    Posts:
    6
    I used MagicMap as well and with all Post Processing and Image Effect, the FPS dropped down to 45fps on 4000x4000 terrain. This is definitely a life saving if it can optimizing it..
     
  8. antoripa

    antoripa

    Joined:
    Oct 19, 2015
    Posts:
    1,158
    Take your time... better a stable first release that doing a rush
     
    magique likes this.
  9. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,719
    Yikes! Take care of yourself! Without you, there will be no Terrafirma and we all need it now. :)

    I am sure most of us have enough to do that a delay won't kill us...although it seems like rushing might kill you!!!
    :eek:
     
    one_one and antoripa like this.
  10. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Not at the moment, my focus for the initial release is to offer a complete rebuild of the Terrain system that comes with Unity 3D, so basically terrain generated from sampled 2D height fields. Once the first release is published however I'm going to be directing my attention towards generating terrains from scaler fields \ 'volumetric' datasets. At the moment I don't want get to side tracked however, and before I can integrate the marching cubes mesher for starters, shown some where in this thread already, I need to finish up my foliage handling.

    Basically my plans are to finish up everything I'm doing in-regards to mimicking unity terrains first so the majority of assets and systems that work with unity terrains should in theory work for the most part with my system. This also prevents anyone switching to my system from having to rebuild/restart work on of their scenes.

    To sums things up anyways, not at moment, in the future yes! I'm actually excited to move away from 2D terrain data (Wheres's the fun in that? ;) ) and dive into 3D data field handling. I really think there's a lot of potential here that has yet to be tapped into. I think the determining factoring though as to weather or not this feature becomes a necessity vs a novelty is going to ultimately come down to execution in terms of it's implementation. So rather the half a** the job trying to push the feature into this release I'm going to give it the tlc it deserves.


    Here's a few screenshots I was actually planning to update the main post with.

    In regards to post processing effects i dont have much in terms of assets besides what comes in the standard assets bundle.

    In this scene though i have 3 x 3 2000 x 2000 unity terrain objects being handled :

    Terrafirma top, Unity Terrain bottom in both images;


    Camera Distance 2000 units.


    Camera Distance 10000 units

    I kinda just smashed a bunch of effects onto the main camera until it looked "nice" lol.

    Currently using :

    -Bloom
    -Antialiasing
    -FXAA2
    -Color Correction Curves
    -Depth Of Field

    Probably not the most demanding list of effects but for reference here's the same terrain with out any effects :



    I'm sure our settings differ on our terrains internally, but there should be evident differences accross the board in terms of batching, fps, triangles, ect. Would also like to note set pass calls can be can cut further in this instance by merging controls. For what ever reason that setting isn't active.


    You are completely right..

    Just want to put this out there again sorry for the delays.

    At first all I wanted was skyrim on smart phones. We deserve nice things right, lol? Me and my team had a lot of work done on a project in terms of our worlds scenery and we hit a point where what we had set out to do simply wasn't going to happen using unity's solution to terrain handling. Around the time I started this thread I initially intended to offer only the mesher. After starting this however, it became apparent that the right direction to take would be to aim productions focus towards developing a full terrain handling solution rather than just a shell for post production. Also figure this could be a great way to offer over 8 years of work/research I've put into developing terrain handling solutions as well for working with volumetric data in future updates to come.

    To sums things up this is going to be great.. Delays suck but personally I'm happy I've decided to take the time to get this to the point it is today, I really to won't to produce a piece of crap, lol. And I want anyone who uses my assets first impression to be really great.

    So thank you for your patience. :)

    Lmao! Thank you for your consideration. For me though this sorta thing is kinda of typical. One thing always seems to lead to another. -_- One moment your maybe making meshes and next somehow, some way, your working on tree bill boarding procedures or something never really initially planned. Basically I get side tracked easily and really curse the universe for not adding more hours into a day. (Spin slower earth!! ) But yah, I really enjoy my job... So no worries. :)


    Also updated the interface :



    The Unity Button when glowing blue indicates the terrain handler is to produce a mesh that replicates the shape of your unity terrain being referenced. Essentially this is the less advanced interface and if you are simply looking for a solution that meshes your terrain data identically to how unity would, this would be your best bet.

    When the option isn't glowing all options are open to you.

    You also have a few options for data sourcing :

    You can input the dimensions of the terrain along with the height data width and height for storing editing changes, or inject any image (Greyscaled preferable ;) ) into the height map field. You can also continue to reference Unity Terrains.

    Undo/Redo

    Would also like to mention the undo/redo buttons (they can also be linked to a keyboard macro). I opted for handling editing history myself for a number reasons and think I found a solution that offers more potential then what is given with the default solution.

    As of now you can choose to have the logger persist when unity closes. You can choose the max amount of logs to store, it'll work in real time and since all the source code comes with the system it can be extended/modified to fit your needs easily.

    In regards to the logger working in real time, the editor is accessible in code during run time. Simply give it a point, brush index, strength, ect and it will modify the terrain at the given point. This can be used for handing real time river generation, burrowing effects (Something underground creating debris as it travels, like bugs bunny...?), yah, kinda reaching.

    I'm thinking of adding an option for animating undos/redos as well, think wizard stalagmites the terrain, then system undo does the changes. I'll (or you, lol) would need to set up some systems for getting the union / intersections of specific logged height fields though or something to that effect, more then possible though and majority of what would be needed should be available when the volumetric tool sets are done.

    Let me know of animated undos/redos would be something of interest anyways.

    Going to update the demo and give more detail into the tree handler. Grass seems to be less taxing, so I have a simple solution for alleviating a lot of resources dedicated to tree handling. Also plan to make a up small road map for where this project is going after first release. As of now though I'm going to update the main thread post to reflect where the system is today over the next couple days.

    Native Rendering :

    Also working towards native rendering I should mention as well. This should be another great improvement in terms of performance. Relying on Unity meshes is bottle necking my system, not badly but performance gains can be pushed even further with a more direct pipeline to mesh data for making modifications.

    Final notes :

    Should also mention for the most part everything is now done... *yay*. I'm going to spend some time with the system though and make sure everything is good and stable as well as finish up the last few things I need to tend in the meantime.

    Thanks again everyone.
     
    Last edited: Feb 12, 2017
  11. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Another quick update! I'll prep the Tree Handler preview for launch! Only thing that was stopping me was some minor batching things (Could of been done smarter, just lazy .. lol) and billboard coloring. Just noticed though that Unity isn't compensating the bill board colors to match the actual models. Least in my scene so, as a replacement, it's more then ready. (Billboards i'm producing are the same tone as Unity's)

    Yah, kinda one of those things where you end up putting more work into something then is needed.. Did the same with the mesher.. -_-

    Anyways I'm handling 40 000 trees with no hiccups so I'm pretty excited to preview the fruits of my labor, lol.

    So in that regard, i'll adjust my batchingz and get something ready to show off around this time tomorrow ! :)
     
    matteumayo and Teila like this.
  12. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,719
    I do the same exact thing. :) I sometimes wish I had four of me.
     
    Murpenstien likes this.
  13. AdamGoodrich

    AdamGoodrich

    Joined:
    Feb 12, 2013
    Posts:
    3,627
    Just saw this thread - looks very nice.

    I am the author of Gaia, and am working on Gaia 2.0. Gaia becomes real time, and will have erosion etc. Getting amazing generation performance on GPU.

    Would be happy to collaborate and get some one click integration going. Appreciate your probably busy trying to launch so reach out when you are ready / if you are interested.
     
    Mark_01, one_one, Knightmore and 4 others like this.
  14. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Lol ! Yah, each of us not having 4 clones is an obvious flaw in our design, haha.


    Thank you! I love your asset by the way! ;) That all sounds awesome in-regards to 2.0 as well ! :)

    In-regards to collaboration, I am very interested. But as you suggested, I am very busy at the moment finalizing everything. After launch though I will personally get in contact with you myself, and we'll see what needs to be done in order to get this happening. :)

    Thanks again, look forward to working with you in the future. :)


    Small Update :

    Figured i'd show off the height map generators functionality in regards to sampling images.



    Top images are unlit just showing the normal's, bottom images include lighting.


    Another mesh generated from a heightmap sourced on google.

    I figure this should make it much easier in regards to sourcing height maps for terrain templates. There is no image restriction. Only requirement is that an imported texture is readable and it's preferably set to RGBA 32 bit. Kinda make's me wonder though, why this wasn't an accompanying default option for Unity to begin with...

    Also, here's a small preview of tree handling, it's a major work in progress, kinda figured it was further then it was but i see now it obviously needs more polish. I'll explain below :

    Billboards close up in the left image. A band-aid if you will, till I toss something more elegant together ateast.. lol

    Obvious billboards - Still a work in progress


    Billboard stats; Grass for the record will use exactly the same technique with the exception that I'll be using true billboards that rotate with the camera via the batch meshes shader.


    As far as everything on the CPU goes that is done. Right now I'm breaking up regions into batch objects. Each object has a model representing all the bill boards in that region in a single mesh and has a container holding the actual mesh interpretation of the trees.

    The system only processes tree meshes in the object container when batches are with in the maxTreeObjectDistance and are touching a batch that isn't processing any fuly meshed trees. If the object is out side that distance it turns off the entire object container and skips processing it.

    In regards to billboards, billboard mesh renderers are on between the maxTreeDistance and partly before the maxTreeObjectDistance so trees can transition on correctly and off otherwise. Tree bill board object with a mesh are culled through the shader, which essentially discards pixels if a distance calculated through positions stored in the meshes normals is to close or to far from the camera. It also adjusts the pixels alpha channels as well for a pretty naive transition effect to supplement pop-in.

    Basically everything is good to go with the exception of the shader itself and my billboarding method, I've tried a few things such as traditional bill boards, switching images based on viewing angle, ect. Basically I need to give this problem the proper attention it deserves. Found an amazing solution to grass handling for the future, I'm eager to preview though. I learned a lot from tree related research, my billboards aren't really scratching my resources, even with culling so, I'm going to put that to all to good use. ;) I figure though I'm going to focus my effort from here till I can publish this on finishing the mesher. Unity3D vegetation works fine with my system already, and there's guy out there with even better solutions. So for now until I can write a more proper bill board shader/mesher i'll keep this all on the back burner.

    Anyways, I'm going to continue to finish things up. I'll get a small grass preview up, today or tomorrow or dare say it, the next day, lol? Sometime soon, nothing major, basically as much grass as I can push before it starts to impact resources. I'll be batching grass quads together on this as well to keep things.
     
    Last edited: Feb 14, 2017
    Teila, antoripa and LennartJohansen like this.
  15. AdamGoodrich

    AdamGoodrich

    Joined:
    Feb 12, 2013
    Posts:
    3,627
    With Grass you often don't want to have the billboard face the camera as it looks unnatural to see the grass following you when you wander around in the scene. So might be good idea to have a 'per grass' option to not billboard. Perhaps just pick the setting up out of the detail.

    I assume you are instancing trees as well. Would be cool to explore grass instancing as meshes as well with various settings to fade it out into the distance, or even to pseudo colour the terrain based on the grass that's on it. The unity grass system is pretty terrible in the way it does this.
     
  16. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Actually way ahead of you on the first part, since this is something I'm R&D'ing on the side a little more thought needs to go into placing the grass pieces, but as of now I have a couple of options. One would be traditional billboards, single quads rotating with the camera. The other is stationary quads slid through each other. I lock the bottom verts intended to be anchored and apply a wind effect to the rest. Below I've added a few things in terms of grass handling to give an idea of where that's at and what to expect. Gif below actually shows some foliage animated set up basically how you described. I also have an option for doubling the mesh planes as well, I'll get more into that one though later.

    The way I'm actually handling trees sad to say as of now is without instancing. I'm kinda on the fence as to how to handle my tree bill boarding at the moment so I've been mostly experimenting. Grass as of now is handled in massive patches. well depending on user input, and I currently fade and eventually cull grass and tree bill boards through the shader. In regards to blending the terrain/grass colors so to speak, that also sounds very interesting. From what I'm thinking it wouldn't actually even be that hard to do either. Should mention however vegetation is kinda in it's infancy, to perfectly honest though, haha.

    Another minor update... Grass systems are now running !

    GIF displaying wind below, About 1.6 mb : Link to GIF



    Definitely check out the gif. Considering the effort, I think It's looks really great for our first steps into proper grass handling, I'll post a few images below as well..






    The shader as expected handles shadow casting and receiving.

    Overall this will be faster then the tree handling scheme and should be the final step needed to go completely independent of unity terrains ! I'll post bench marks tomorrow, but so far displaying these meshes isn't hurting my frame rate in the slightest. :)

    To get an idea of how big we can have the system make our chunks :


    Lots of grass !! Single Mesh Object That Supports Wind Animations, culling, fading, ect. 51,000+ triangles total !!

    Sorry this really excites me, this whole system should be nothing for even much lower end systems to manage and can push out aloooot of grass.

    Should also mention as well depending on the target platform finding the sweet spot between batch size and triangle counts will be a bit of a factor.

    Note the image in the left isn't 'fading' from the shader. Also billboard positions will not be that uniform in real world circumstances. :)

    Also throwing in one last meshing option for LOD handling/mesh generation. It involves bit masks and shaders, and should be even faster then the current sets ups. I'll discuss this a bit more later, just putting this out there, in terms of everything to come. This is only the beginning!! :)

    Small Update :


    One Mesh 'Animated' via the shader.

    Kinda self explanatory, turned off shadows to show batch counts. This is a single mesh receiving a controlled wind effect. There'll be more to wind simulation when this feature is ready.

    Also gonna make those greedy triangles share their verts! ;)
     
    Last edited: Feb 16, 2017
    AdamGoodrich and Teila like this.
  17. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Minor Update :

    Just thought id show the performance impact of the new grass handler. It's currently drawing grass all the way to end of the terrain and applying a simple 'wind effect' to it. (Triangle count can be halved using the traditional billboard option)










    The shader is still a wip. But for the most part is complete.

    For now I plan to finish up the shader, get everything done in terms of the mesher and get the base system published. After that I'll come back to grass and tree handling and finish that up with the intentions of having that out a couple weeks into the systems first month of availability. Before I get to side tracked however I'm going to get back to working on getting the base system ready for submission after I work a little more on the shader portion. For now though I think I have I have a great foundation to continue off in the future. :)

    EDIT

    Did more work on the shader, it's really starting come along nice. I'll post a screen grab tomorrow/later with trees overlapping.

    EDIT :
     
    Last edited: Feb 17, 2017
  18. AdamGoodrich

    AdamGoodrich

    Joined:
    Feb 12, 2013
    Posts:
    3,627
    This + Gaia 2.0



    Will be super cool!
     
  19. Knightmore

    Knightmore

    Joined:
    May 11, 2012
    Posts:
    226
    Not just super cool... this will be amazing.

    I tried a lot of selfmade stamps the last days and especially with island this will be awesome as I could delete a lot of polygons which aren't part of the island but just the base below sea level.
     
    Goodgulf and antoripa like this.
  20. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Minor Update :

    Added in a LOD plane you can use for whatever. I tossed the water4 shader on it and aside from some visual artifacts, looks great ! Could make a great platform for ocean rendering with LOD support.





    This Plane Object also supports selective culling so you can for example, disable chunks within a terrain. ;)


    Should also mention the terrain system also support this feature, so chunks of terrain under water that would never be visible anyways can by disabled in the same way as well.

    Also I'm sure I mentioned the system is capable of pushing mesh quality, but I never really touched much on that. On the left is a mesh produced with terrafirma, the right unity. Both systems used the exact same height field. with the exception that terrafirmas mesh heights are interpolated between points to generate new data. This enables the system to generate more complex meshes from smaller height fields then what is possible currently with unity. I also added in the ground work for inputting custom height functions into the editor, so you can write your own normal calculations and height field processing functions.



    The systems will not only smooth your data but make a more accurate interpretation of it pronouncing hidden details other wise lost in during Unity's meshing process.



    Also updated the main post, not to substantial just streamline information and changed pictures and such. The terrain handlers interface is now complete as far as release is concerned, moving focus to the editor and everything should be ready very soon.

    Also planning to post a video soon showing how my vegetation system fairs vs Unity's in coming days, though it'll be an early alpha presentation meaning it'll be like in features and will serve more as a function test as far as grass is concerned anywas, trees are actually done.
     
    Last edited: Feb 19, 2017
  21. LennartJohansen

    LennartJohansen

    Joined:
    Dec 1, 2014
    Posts:
    2,310
    Looks great
     
    Murpenstien likes this.
  22. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Thanks ! :)

    Figured i'd show off the new shaders for tree bill boards. Finally don't look crud, haha. Just finished this up because who needs sleep right?



    In-contrast to before.... much better ! :)

    Should also mention they support wind simulation and now proper lighting and less noise. On my side anyways. There should be no issue in the future getting this running on via Unity.
     
    Last edited: Feb 19, 2017
    matteumayo likes this.
  23. jc_lvngstn

    jc_lvngstn

    Joined:
    Jul 19, 2006
    Posts:
    1,503
    Wow...very nice work!
     
    Murpenstien likes this.
  24. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Thank you very much ! :)

    Another minor update :

    In my opinion, photogrammetry is about to become a new standard and I've been looking at the same 10-15 scripts for over three months. I tend to go a bit crazy so I like to start little projects on the side adding in the the foundations for future features to keep my sanity, and this time around I was inspired watching some old battlefront tech demo footage.

    Now to start off tessellation by itself is nothing major to implement and can be done in about 5-10 lines in a shader file. Over the past year or two with the rise of VR, assets scanned via camera/lasers are becoming ever more prominent in our entertainment and in order for solutions such as mine to meet industry standards I felt it was time I started thinking about how I was going to go about handling this type of data and if I wanted to even support it.

    With that, i decided a great start would be the addition of tessellation for a number of reasons :
    -Not very difficult to get running.
    -Adds another layer on-top of the height map, allowing for the generation of more complex mesh topology.
    -Adds another layer of accuracy on-top of normal calculations, allowing players to not only interpret a more accurate visual interns of lighting but depth as well.
    -Adds a level of dampening/blending in regards to level transition pop-in.
    -Most hardware today support it. (Past 7 years worth of it anyways.. lol)
    -It's cool ! ;)

    Tesselation :

    Rendered in unity with 8 k texture; This is where I want quality to be in the future, for now I'm just using the normal map for heights as I set things up to build off. Though if it means anything this is the precursor shader to the one used in the images below, so I definitely got this. ;)





    Now works with my Terrain system.. Encountered the same issues others are obviously having about what do about passing height info onto 2nd, 3rd, ect pass for every 4 textures. Since I have all the control i'm willing to implement into my system though I have a few solutions.



    Works between meshes and mesh transition zones. (Crack free)




    Just sampling the normal map (Lack of height maps with the exception of the first image).

    Anyways back to work on the core stuff tomorrow. Today was a lazy day, but anyways rather then spending moneh, or writing a custom solution yourself if you plan to use my system. Know you should be covered right out of the box interms of tesselation, or at most a month after. (This isn't a big job to finish, just need to find time for it)

    Most i'll probably do with the feature until after release is get some much higher resolution normal maps and textures and really push this.

    Should also note the performance hit..



    Pictures speaker louder then words.... ;) Yah, they're may be a marginal one. Kinda hard to gauge on my end. But more then enough room to breath... *phew* lol

    Also note the end goal isn't far off, just needs to read from displacement maps now and communicate heights to other layers. Really excited about this one as well, Kinda been obsessing over the possibilities for awhile now.

    Edit



    Been obsessing over this all morning. Hot damn the future looks soo good... :')

    Should also put it out there when this all does release i'll have a set of shaders to accompany the terrain one that all meshes to blend materials with the terrain. I'll try and add shaders for both displacement mapped materials, and non displacement mapped materials.

    Another note, none of this will be compatible with unity terrains, but most unity tessellation solutions should be compatible with terrafirma.
     
    Last edited: Feb 20, 2017
    Hikiko66, matteumayo and Teila like this.
  25. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Minor Update :

    Got a lot of stuff done in terms of editor stability. Kinda hard to display these changes in an image or video. I did however start writing up a small splat system, still control map based, so I could get proper displacement mapping to work via height maps and still have normal map support; ao support, ect to come later. Needless to say, using the normal maps sucked and this looks a lot better. Doesn't cost another sampler either. :)



    So now terrains support sampling from height/displacement maps. Few more images below, all sourced off of a terrain mesh.



    Geometry placement is obviously mush accurate then using normal maps.



    Don't really have much in terms of resources at the moment, Ill get more textures in the future and set up an environment , vegetation free to preview the displacement mapping effect and post a demo. In contrast to flat textures however, this simple effect blows my mind. A lot of possibilities for ground clutter, foot print systems, ect. Really excited to say the least.

    Anyways, good portion through finalizing the editor. I figure the road map in terms of development will go as follows :
    1. Mesher release first.
    2. Grass and trees along with editor integration in update 2-4 weeks after release.
    3. Displacement mapping shader support to release sometime during vegetation development.
    4. Volumetric Meshing and interface/editor updates and enhancements.
    5. Knowing me and how easily I get side tracked, lots of good stuff...?
     
    Last edited: Feb 23, 2017
  26. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,574
    What's the expected submission date now? Everything is looking fantastic.
     
    Goodgulf likes this.
  27. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Thank you ! :) I'll be sending you a message in the next couple days.

    I'll address your questions below however as this update is somewhat related...

    Major updates in regard to performance :

    First off to anyone who has messaged me, sorry I've neglected my inbox, I have a lot on my plate at the moment.. :(

    As for submission date, I can't say for certain now recently thought of a solution that should make mesh updating considerably faster while making batch counts even smaller or I should say controllable.

    For example, the new system will store at the moment 64, 16 x 16 patches into a single buffer/mesh. This means we can effectively reduce 129 batches down to 1 which on a 512x512 terrain would translate to 8 batches total for the highest level rather then 1024 which is great news to say the least! That's not even a single percent (0.7% ish) of what it would be previously. Batching will be even better at lower levels.

    Where 129 comes from :
    - 33134 is the max vert count before triangle count hits 65000
    - 16 x 16 = 256 vertices,
    - 33134 / 256 = 129

    Where 0.007% comes from :
    - 32 divisions * 16 = 512 vertiices total on any given axis.
    - 1024 = 32 * 16 x 32 * 16 = old batch count
    - 1024 / 129 = 8 = new batch count
    - new batch count / old batch count = 0.007 or 0.7%

    None of this will actually impair stitching/meshing updating performance either, and should in fact make that even faster. I can say it'll take 25% the time to modify meshes or roughly or 400% percent faster in these cases.

    I'm am also in the midst of adding a new lod transition routine that should mimic Unity's algorithm with subtle differences if any. This comes down to how I calculate the viewport projection matrix height among other things vs how unity is doing it, I'm guessing I'm on par as it's kinda trivial but until I get around to switching the levels with all the calculations I just finished the generators for I can't say for certain at this moment. Should mention this and the first update are going to really improve quality and configurability of lod transitions, in a lot of ways and will really push what's possible to new lengths.

    So updating the mesher should take around 1-4 days I figure. Could be relatively simple or could be a major pain. I have everything all mapped out on paper ready to go though and it seems concrete. The meshbuffer needs to be updated to store more then one mesh and the lod handler needs to be designed to read the new data and modify it along with the editor.

    The lod system updates I figure I'll be done tomorrow so, a week-ish from now I'll release another demo anyone can play with show casing the new system. I imagine the figures from the previous preview should be double if not more by the time i'm finished.

    So I figure submission should be anywhere from 3 weeks to 5 I want to say. I'll be doing a testing period as well once I move my attention toward the documentation.

    Final Notes

    This has become alot bigger then I'd had initially intended. Editing support seemed essential however, and the new optimizations are a few things I'd rather not put off. Especially the first one. It's really is one of those things that seems like it should have been done from the start but anyways.

    Photogrammtry Asset Inclusion through routine updates.

    Few other things I'd like to mention, I'll be including photogrammetry assets with the package that wth time should build up to be quite alot in terms of quantity, all related to terrain obviously. ;) I have access to a pretty decent camera (Above 8 k) for surveying, agisoft photoscan ,and as I live in rural Canada. I have more then enough bush/environments to sample with a few members of the team I am working with who are already more then interested in helping with the cause. So I'll start release assets with the package around the time I dive into displacement map support and terrain texture blending.

    Should also mention these assets will be high quality, so this should be a very valuable addition that I hope by the time this is all said and done, should largely out value the terrain handler itself, considering i've seen single photoscan rocks go for $40-90 !! lol

    It's where the startup im directing is currently moving as well so this is a definite addition.

    Small Edit

    The new batcher can actually pack 129, 16 * 16 patches
     
    Last edited: Feb 25, 2017
  28. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Just put together a small preview sample of the quality were striving for in terms of both included and future available commercial assets. The demo as well will also have an option for displaying displacement maps, so I'm going to put together some textures to ensure that features shines in the right light as I find the quality could be a little better in the assets I previewed earlier.

    Anyways here's a mesh we produced from ground right out side the front door at my house, lol.

     
  29. antoripa

    antoripa

    Joined:
    Oct 19, 2015
    Posts:
    1,158
    cool
     
    Murpenstien likes this.
  30. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Just finished the new LOD handling system. It works basically how Unity handles terrain LODs, using a technique referred to as GeoMipMapping if anyone comes upon this and would like to learn more.


    Patch Gradient Based Level Adjustment



    Essentially we check each patch for largest rate of change for levels greater then zero. Once we have the biggest gradient we can then calculate a minimum distance before the patch can be rendered. Using all of that we then compare the distance between the camera to the patch and compare that with the minimum render distance values for each level.

    Over all it's a far more intelligent way to control level transitions.

    The old straight distance method will still be present along with a square shape distance setting.

    I also plan to make the new LOD handling scheme layer-able with the old one as well. This will adjust levels based on distance first and bump up/down levels based on the pre-calculated minimum render distance generated from biggest gradient value for the patch.

    Would also like to mention you can choose the number of LOD's to use and there's a setting similar to Unity's pixel error.

    Anyways, final notes. I'll have a new demo up very soon, just going to finish up the new batching methods.
     
    Last edited: Feb 27, 2017
  31. bgrz

    bgrz

    Joined:
    Mar 22, 2015
    Posts:
    42
    This looks really nice. If you need someone to test this in VR, I'm here ;)
     
    Murpenstien likes this.
  32. jc_lvngstn

    jc_lvngstn

    Joined:
    Jul 19, 2006
    Posts:
    1,503
    Very very nice.
    You posted some screenshots showing awesome performance. How about on the fly generation at runtime? For example, let's say I wanted to procedurally generate the heightmap data, or load terrain data from disk at runtime. Any thoughts or details on this?
     
  33. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Hey, first of thank you !. Hope you enjoy 'informative' responses, lol.

    In terms of on the fly generation, it's not bad to say the least. I admittedly lost some performance due to negligence focusing on run-time performance, but not much has changed aside from bunch of "get" calls to the front end bogging things down. Localizing what I need and benching the generation portion however is on the list of things to do.

    I'll set up a demo at some point though popping in terrains as segments 1km*km patches in scale and have the system push out height fields of 512*512 generated with noise functions to give any idea of where that stands, considering this definitely isn't the first inquiry, lol. I'll make the generation distance controllable as well.

    Not exactly sure when I can make that available however as things are sort of a mess with integrating the new batching system.

    So though I haven't tested this, I don't see much of an issue getting something going. Considering for the sake of editing sensitivity we need to make multiple edits to not only the meshes behind the terrain but the entire height map needs to be copied, modified, than re injected before mesh updates once every frame at small enough intervals for smooth height transitions.

    Also, the two functions that I use for modifying the terrain are basically the same as the ones I use for generating it with a few minor exceptions related to setting up the data for the first time, mostly related to creating objects. And as of now it grabs the patches somewhat naively (getting around to that soon), so I've had times on terrains with big partitions/patches where I'm redrawing every patch that makes up the terrain and along with in some cases five meshes per patch while also making the same adjustments to the back-end unity terrain when that's present.

    Few minor updates :

    -Added in two new schemes for real time generation that will be way faster then it is currently, the first method using stored tri-index's is working great and is compatible with all settings when under this option and reduces the work 'tessellating' the terrain, bringing performance back down to levels around the pre-generated mesh options..

    -The next option is somewhat of a new breed for my system. It won't be compatible with shaders not included with the system with out a few minor modifications. I'll include templates, as well as guide to modifying shaders to work with the system. The only work on the cpu's part will be generation and mesh switching depending, leaving displacement, normal calculations, stitching, ect to the gpu. Instancing will also be an option, so in general this will be the fastest option in overall even in terms of generation as only a max of five meshes will be needed over all, and one gameobject per patch.

    There's going to be a few options and in general these two setups will most likely be the fastest, in terms of transition handling.

    There's a lot to detail so i'll get to all of that either gradually through small updates or all at once in a huge one sometime very soon. As of now my priority is on the new batcher, but as things progress i'll regular post updates. I'll leave everything I've done prior to this as well to give a great breadth of options for handling 2d meshes before moving onto 3D solutions.

    -Also, I figure before volumetric handling rolls out, i'll include a volumetric modeler for making organic looking assets as well to accompany displacement mapping. The shader should if everything goes as planned should support texture blending so adding terraces, caves, ect, should be a lot easy for static meshes. I'll include an option for LOD generation as well, though it'll only be mesh interpretation in the case of these models, but should enable pretty high poly counts if needs and they should also blend seamlessly with the mesh object.

    New Batching System :

    I've built the new buffer and it's working fantastically, to the say the least. Block read/writes are working great, it's still half integrated and needs to be expanded, but I'm currently using it as holder for the old buffers I had in place (One for each mesh). Next step is going to be adding another layer, which will contain all the active meshes in a batch, in something I call a level buffer, these buffers will be partition-able to hold any number of inputted meshes, and will act as the cross road between the cpu and the gpu, reducing mesh set related calls, dramatically.

    I'm currently writing up the second portion of the integration, but idle performance alone will be doubled/trippled in most circumstance with the major reductions in draw calls, as well as a reduction in time spent modifying meshes.

    Anyways I'll most likely be finished the batcher pretty soon, so tomorrow or the next day i'll post an update on that.
     
    matteumayo likes this.
  34. jc_lvngstn

    jc_lvngstn

    Joined:
    Jul 19, 2006
    Posts:
    1,503
    Thank you for the info and details :) No rush though, it sounds like the system will be awesome!
     
  35. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Update

    Finished the new batching system!

    Control Circumstance - Unity 3D Terrain


    Terrafirma - Before Top ; After Bottom


    I managed to pull another 500 frames from the system and reduced batching dramatically. Though I should mention batching is somewhat naive, so frustum culling isn't functioning as of yet, as patch geometry behind the camera could be attached to a batch with geometry right in-front of it. More control over batching is what i'm working on now but just wanted to show off the benefits I'm seeing using the new data hierarchy I implemented.

    So starting with a unity terrain we were able to manage 500 fps, in the beginning Terrafirma offered a modest 100% boost in resource reduction at 1000 fps and now with the new batching system were pulling off 1500 fps; That's 300% the speed I was initially getting using unity terrains. As of now I'm more then impressed with the results, and I figure things will be even better when I finish the mesher up. I still have a few things to implement but overall thus far things are shaping up nicely.

    It's also amazing how you can be working on something for awhile thinking it's an optimal solution, telling yourself, "There's no better way I can achieve this", yet 4 months in and your finding better ways to manage things wondering why you didn't do it that particular way from the beginning...

    Anyways more I'm overhauling the mesh representation of the terrain as of now, and it should be a lot faster in regards to transitions. I'll update that soon as well also have a couple sample meshes with textures, to give an idea of what to expect from our 3D scanned assets. We recently set up a small studio and have been getting fantastic results.

    I'll provide a mesh or two tomorrow, they'll be raw exports so tri-counts are in the realm of 200k and there are not normal maps provided.

    Ill file host it tomorrow, but scan are coming out great. Have a look below.



    By the end of this year I hope to have at least 50-100 usable assets to go along with the system added in through updates.

    With that said however, I want to clarify that while I would like to start pushing mesh quality in the future, this definitely will not be mandatory.

    By the time the system is done there should be no reason you can't run a terrain identical to a unity mesh on a mobile device with better fps, and take the same mesh, have it tessellated by the cpu to have double the vert counts, then procedurally tessellated again by the gpu and displaced to get a mesh quality suitable for vr all with the same object.

    Essentially in terms of options for terrain rendering there should be an optimal solution that doesn't sacrifice in quality and one that can also offer a boost in speed.

    I'll actually start testing on android devices sometime this week again and see how open gl 3 devices take to tessellation. I'm somewhat of a performance junkie, so it would be interesting to see what would be feasible in terms of cross compatibility of features between more powerful systems in scale to lower powered platforms.
     
    AmBe_, eskovas, one_one and 2 others like this.
  36. Frednaar

    Frednaar

    Joined:
    Apr 18, 2010
    Posts:
    147
    Can't wait to test it, have you thought about releasing a beta (unfinished) product, so we can start testing and give you some feedback on different use cases?
    Megasplat did a beta at a discounted price, and I think it really helped Jason (one major feature I requested, reading from colormaps, is actually now implemented and is a big plus for the product...)

    Truth is I can't wait no more....
     
    one_one and Knightmore like this.
  37. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Yes that's actually the route I've decided to take. Personally I would like to get the editor in on the first release, but if it comes down to it I'll release the mesher first and add the editor in next along with displacement mapping then move focus towards vegetation handling. I'm not exactly sure about how i'm going to about this yet in terms of specifics, but I should have something together for submission relatively soon.

    In terms of the mesher, i'm trying to put as much tlc into as possible, it''s been kind of a mess this past week but things are coming together much better then I had initially expected.

    I recently implemented native vertex and indices pointers. So sadly some code will be in a dll, i will however provide the source code for all of that in c++ files. I know I could've written the whole system native side, but honestly. I feel in terms of accessibility it would be best to keep as much as possible open to developers. It really sucks looking at graghics api and such for the first time for anyone who is use to working with c# exclusively.

    I'll have more on native rendering support anyways, really impressed however. Much better then the alternative anyways, lol (mesh.vertices = verts, vs, ect). Thank you Unity. :)

    Anyways everything's wrapping up nicely, so you can expect something very soon. Even if compensations are needed to be made. At the very least I can offer a very performance conscious mesher with an independent editing interface, in that case however you'll be stuck using unity vegetation, or one of the many other fantastic assets coming out or that have been already released to handle your foliage.

    It'll be nice as well to have something functional out that I can regularly update and maintain as well.. Anyways I am very excited to work with the community and get the system out into the wild. Big things to come. :)
     
    Knightmore, magique and one_one like this.
  38. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Update

    -As of now in terms of R&D related to meshing. The system is now complete with some minor work ill be finishing over the next couple days. This is really big update, so bare with me..

    -The whole meshing system has been revamped. Heights, normal's, and uv's are now adjusted via the terrain material which has a lot of benefits. Normals and heights are pre-calculated and drawn to a texture which is now handed off to a shader. There's an option to instead have the shader calculate the normals in real time, I honestly see no reason to use it aside from memory considerations but aside from that it's faster to have the normals pre-calculated to/modified through the shader.

    -In terms of performance, there are essentially speed boosts across the entire board. I have two schemes currently. One that's instance based, that uses a total of five meshes max for batching/lod adjustments ect, that stitches through the shader, and another that's much the same as my older efforts with all the previous options included, with the exception that a lot of the, "meat" has been scraped off leaving only position adjustments to be processed on x and z axis while stitching from the cpu, among many other minor things.

    -Mesh Generation is ridiculously faster, especially when using the instance based method. Updating the entire height map at once, is now instantaneous. I can adjust the scale, tiling, and offset of the height map via the shader for example and these changes are applied with no drop in fps. Here the only issue you'll encounter is having to generate at the very least the height map before passing it to the system for procedural generation.

    -The system now is also more manageable in terms of storage. It can now store height maps, normal maps, along with texture related materials to later generate a terrain object with no drop in fps on the hard drive. There's also an option to generate a patch pool to avoid instantiation calls as well.

    I'm still integrating the batching system and the gradient LOD scheme, but performance increases should be evident by the screen shot below. Take note of the differences in vertices/triangles; More then Unity, but still faster ;) Currently using the radial scheme with long draw distances (Almost all meshes are highest detail to level), for no reason other then It was set like that when I took the screen shots.

    Batching and gradient system will be done today.



    This 500 fps increase is modest, but it's only the beginning. The batcher and gradient scheme will drop geometry and batch numbers considerably, pushing things further. As of now the system is completely free from spikes when moving the camera at any speed as well, having the stitching portion dramatically revamped, from doing over less then, i'd say, 90% of it's previous processing. Essentially. I can't see it getting any better then this.. Expect a demo in the 48 hours. :)

    Road Map for release is looking at 3 weeks to be safe. Tomorrow work on the editor begins. Thing's have gotten much more simplistic, so I expect re-visioning it should be nothing and major speed boosts are expected as well as were no longer redrawing meshes, only changing the height maps, textures fed to the shader.
     
    Last edited: Mar 8, 2017
    bgrz, Frednaar, one_one and 3 others like this.
  39. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Minor Update; Demo Should be live today/tomorrow-ish. Had some issues with the stitching not lining up when modifying verts via the shader, but managed to fix it. Also recently finished the gpu instancing scheme. I figure once, i finish up the non instance variant. Performance wont differ much, but as of now the system is running at 250% fps of the Unity Terrain before batching (Majority of the new scheme is finished, should be done in a few hours).



    Essentially halved resources on the main and render threads. As of now I'm pushing batching via naive terrain divisions. The new scheme i'm finishing up on now though is extremely simple, but will group meshes quite effectively with not much, if any noticeable work on the cpus end. I'll preview that later today when I finish that up. Should also mention as well that, I'll be incorporating the gradual scheme today. It's mostly a copy paste job.

    Basically what I'm getting down to is everything has gotten a lot faster. In terms of both GPU instancing methods and non. And in regards to the batching schemes I'm finishing up now as well and with the integration of the gradient scheme things are only going to go further from.

    Should also mention this mesh costs only a mere 1.0 mb to store as well. ;)
     
    matteumayo, zmaxz and magique like this.
  40. antoripa

    antoripa

    Joined:
    Oct 19, 2015
    Posts:
    1,158
  41. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Hey, just responded. Very sorry for my negligence. If anyone need's to get in-contact with me in the future, try mrochon1@laurentian.ca or better yet msg me on facebook at https://www.facebook.com/dr.murps

    I'll try and make more of habit in regards to checking my inbox here, and I'll try and post my contact info as directly as possible. If you getting ahold of me is ever a priority however, facebook would be your best option, followed by sending me an email.

    I should also have a demo up and running sometime today, the batcher is up and running now so some things to note..

    1. Requires at most 7 meshes when instanced; and 7 mesh templates handed off when not;

    2. Unlike, from what I can tell, Unity. It can batch the highest level.

    3. Options for the max batch dimension for each level, going up to a max dimension of 128 (width/height of a batch);

    4. Batching has no negligible effect on fps and meshes are never generated when using gpu instancing.

    From the brief tests I just ran I was able to condense meshes from 256, down to 86. My other batching scheme condenses meshes more so, but for now I'm holding off on the native mesh pointer scheme until later, and focusing on this method as it's much lighter, in terms of mostly memory and object generation/editing.

    So I should note, i'll be recycling the old mesh based system for my marching cube/surface net methods. My pipeline is great, but for 2D mesh handling it was a little above an beyond in terms of complexity in-contrast to simply switching meshes and having the shader do the work. I'll have the option open to have the cpu handle everything in the future for 2D schemes, but that will probably come with volumetric integration, as I'm really priming for development/beta release.
     
    antoripa likes this.
  42. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Small Update :

    The Batcher is now integrated, i'm just finishing up a few things and I figure sometime tonight, I'll have a new demo available. In regards to the actual system. This is the last day of work in regards to optimizations and getting the mesher ready. It's been a longer road then initially anticipated. But in it's current form, I am very proud of what I've been able to produce and feel it's now at the quality level any future user deserves when making an investment.

    Here's a small preview till the functional demo is ready later today to give an idea of how far batching can push push down draw count's:



    This map unbatched is divided into 16x16 patches. It should be apparent that this next, much needed layer, reduces drop calls substantially. It's not quad tree based as I typically find the complexity factor of such structures to taxing when making the amount of changes needed to handle this every frame, but something much simpler and in my opinion quite effective. Stitching does work. And as mentioned before it only takes 7 generated mesh instances to handle the entire terrain mesh.

    I'm really excited to get the demo out there anyways. Thank you everyone who's been following this thus far for your patience and support. The end is now near and things are better then I initially ever could have imagined. :)

    Edit; the map is actually 32x32 batches, unbatched. Sorry. xD
     
    LennartJohansen, antoripa and magique like this.
  43. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Major Update :

    This is potentially the biggest thing for the system. At this point it is costing close to nothing to handle terrain meshes with TerraFirma.

    There isn't a single aspect of the system that isn't faster now. So over the next couple of weeks I'll be posting a steady stream of updates, featuring videos and functional demos along with more development previews.

    The meshing system is now entirely done from here on out until after release and performance is now better than I ever would have expected it could be.



    Below is a comparison of a blank scene, one with a terrafirma mesh, and another with Unity3D..

    Scene No Meshes :



    TerraFirma :



    Unity3D :



    Any differences between shading comes down to the normals used and material settings, but both TerraFirma and Unity3D are using standard surface shaders, so any visual differences is superficial..

    Before I dive in here's a small edit to show the system with proper shading.



    Reason I showed the scene empty was to show how little everything is costing now. I'm matching unity in terms of partitions/terrain chunks and letting the batcher handle the rest in terms of drowning out draw calls.

    Everything now works smoothly and to say it's a thousand time better would be an understatement.

    Changes :
    -
    Editing is now ridiculously fast; Update heights as fast as you can edit a standard texture. The shader will reflect changes instantaneously. I've played around an even animated the terrain, with a sprite sheet walk cycle. Didn't skip a beat. :)

    -Mesh Generation is also fast, especially when pooled. (7 meshes max, potentially more in the future)
    -Generation goes as follows :
    -Rip Heights from Unity or Create blank texture 2d;
    -Generate 7 Meshes
    -Distribute Patches
    -Assign/Generate Material
    -Transitioning levels and drawing meshes is now costing next to nothing ! :)

    There's alot more to detail but I figure, i'll get to that when I update everything with the demo later today. I'll also be distrubuting test copies, so i'll be sending pm's to a few indivuals this week in regards that as well. If anyone is interested in testing the system, let me know.

    Thank.
     
    Last edited: Mar 14, 2017
    antoripa, matteumayo and one_one like this.
  44. jc_lvngstn

    jc_lvngstn

    Joined:
    Jul 19, 2006
    Posts:
    1,503
    Amazing. You know that there are people just waiting to take this baby for a ride :)
     
  45. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,574
    Sounds great. Have you been able to get MegaSplat working with it during all this?
     
  46. bgrz

    bgrz

    Joined:
    Mar 22, 2015
    Posts:
    42
    Glad to see it progressing nicely. I'm curious about the following:

    I can't discern the pattern by which these layouts get generated, can you write a few words about that?
    Is that the optimal layout, and how so? Why aren't some 2x2 groups of smallest rectangles joined into one?
     
  47. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    I know, I really can't wait to share fruits of this efforts with everyone. The end of first phase development is finally near. I'm extremely sorry for delays to anyone waiting patiently to give the system a go. I assure everyone though the whole effort is better off for it. :)

    Not at this moment, though I've had brief correspondence with Jason and it is happening. And unbeknownst to him as of now, he'll be receiving a test copy in the coming days so we can begin working towards that. So if he's still following keep an eye out for a PM from me over the next few days. ;)

    Should also mention to Adam Goodrich, i'll be sending him files as well. I think i have something now that will not only support, but cater to real time erosion. The idea of dynamic worlds that change with simulated natural forces, excites me. A day and night cycle is one thing, another is having the landscape change over time as well. :)


    This may help, the batcher isn't able to grab those regions as the mesh theres a transitions/difference in level in one the quadrants.



    This should help give a better idea as to whats going. As of right now I'm using a scheme I dubbed binary batching. Essentially I generate 7 meshes, each progressing in axis vert counts in powers of 2. (1,2,4,8,16,32,64,128).

    Essentially the batcher groups patches by level an ensures any given batch has a row/column axis count that is a power of 2 going up to 128, with plans to support any number of power of 2 (256,512,ect).

    I'll have another batcher scheme as well that will allow mesh row/vert counts to be an even number, not just powers of two, and will support batching meshes that as well will not need to have a power of 2 row/column axis count. This will come at the cost of either having to generate plane meshes on the fly or storing all instances of meshes with row/column counts that are even numbers going up to the max row/column count a single mesh can have that you decide to allow. Essentially though you'll be able to compress draw calls further. 128 is about half way to the max mesh size we can store in a single buffer.

    Max batch sizes are also level dependent as you'll see below.



    Basically at the lowest level a single 128 x 128 patch contains enough points to represent the entire level.
     
    AdamGoodrich and antoripa like this.
  48. bgrz

    bgrz

    Joined:
    Mar 22, 2015
    Posts:
    42
    Yeeeeah... kinda but not really :D I just had a flashback from The Witness while trying to figure it out :confused: I'll take a look at it once I get my hands on it, hopefully soon!
     
  49. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    Bahaha, :D

    It's not actually that complicated. I'm finishing the demo up now, and should have it posted later today. Just finishing up some UI related things. Once I'm done i'll post a detail description of how the system works.

    Some minor updates :

    Would also like to mention I have around a weeks worth of work left. Everything came together really nicely. As of now there isn't a single portion of the system I've benched that isn't faster then unity, that includes batching and handling mesh transitions at any speed.

    I'll be posting four demos over the next couple of weeks.

    1. A straight system comparison; Terrafirma Vs Unity

    2. Animated Height Maps; Height map is updated every frame.

    3. Procedural/Infinite Generation.

    4. Split Screen/Multi Viewport

    The system is now completely different. Better, but different. ;)

    I now have two options :

    Instanced and non;

    Both options can opt to have normals calculated via the shader or through the cpu.
    -Cpu method is a lil slower for updates, but could be potentially faster on lower end systems when handling static worlds.

    The optimal triangle method has been sidelined till I can i bring it back non invasively via native rendering stuff.

    I have menial interfacing to do as of now anyways, and the editor should take a day to finish as things stand now.

    So, i'll be spending the next couple weeks, sending out early copies to developers, getting demos ready and uploaded, bench-marking systems more deeply and covering that here, along with prepping commentated tutorials.

    Basically I figure submission for review should be in the region of two weeks tops.

    I'm going to get some rest for tonight, but in the morning i'll finish the UI demo and get that posted. :)
     
    magique, Teila, bgrz and 5 others like this.
  50. Murpenstien

    Murpenstien

    Joined:
    Nov 12, 2013
    Posts:
    104
    First demo will go live in the afternoon sometime after I wake up. :)

    Still working on the add pass shader and need to copy over the gradient based transition stuff. (two functions) But so far, matching unity 1 to 1 in terms of the smallest partition size, we have quite a leap. The first demo a month or two back was relying on having larger partitions to get a lower batching count. While it works, lod transitions were more noticeable. Below however, I ensured terrain patch count was exactly the same as unity's. Draw call reductions now come from a combination of controlling the max number of patches as well as how many can be batched. I figure I can push this down further, with the gradient based method so I'll re introduce that play with the settings and submit the demo asap when I start the day tomorrow.


    Add Pass is "highlighting" everything below it. Other then that, shaders are finished. :)

    Terrafirma Top Unity Bottom


    Also wanted to show mesh contour is still preserved, producing an identical mesh to unity.



    I switched triangle winding order to show how the meshes overlap. As before, the position vertices take are identical. So, until after I get to displacement mapping, I'll be most-likely relying on the built in Terrain Collider vs Meshes as it's obviously a lot faster to test against the height map, vs the actual geometry.

    Anyways, aside from the shader, and communicating stitching to neighboring object the meshes is finished. The editor shouldn't take much time either as it's no longer updated meshes via the cpu, and is instead just modifying a texture. So I figure a day or two and that will be finished as well.

    Thanks. :)

    Meshing performance demo to be posted later today.
     
    Last edited: Mar 22, 2017
    jdraper3, magique and LennartJohansen like this.
Thread Status:
Not open for further replies.