Search Unity

[RELEASED] World Streamer ! Forget about your game memory usage and create big world!

Discussion in 'Assets and Asset Store' started by NatureManufacture, May 13, 2015.

  1. Alex3333

    Alex3333

    Joined:
    Dec 29, 2014
    Posts:
    338
    I wrote you a post, but you are silent. You can allocate a little time on Skype and help us with your asset ?? we can not get it to work properly without lags !!
     
  2. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    857
  3. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,598
    I let him know. :) so soon as he can, he should be able to help Alex3333
     
  4. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,598

    Be sure to test it with out anything in your game....like no grass, trees nothing at first..... also what is in your scene right now, what is on your profiler ... so we, and he can see it.
     
  5. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Atm we are really overloaded every day because of unity 2018 HD and LT SRP. Its HUUUGE work on our side to support every render in every packs but we will!

    So...what I can offer is advise after you will send me:
    - streamer screenshots (layers and setup)
    - profile screenshots (from build not editor)
    - typical terrain screenshots (what is inside what data, resolution)
    - list of additional systems that you use like VS (if engine not handle to load 1 component async or it allocate too much memory you will get spike)
    - you load terrains or simple meshes?
    - you baked colliders and batched data? ( It reduce number of objects to search so GC work much faster)
    - how many objects you got ? you could find it at memory profiles. Object = script or game object, game object component.
    - this spikes are at build?
    - AND MOST important what device? mobile or standalone? Different devices have different power to load data. Big companies have whole team which work to distribute data so it will load smoothly (move between sectors, there was even article about this connected to Tom Clancy the division)

    Because if you got spikes they porobably are from GC. List of probably resons could be big, even your mistake in script which alocate memory - leak. It could be something simple or deep inside the project. I
     
    recon0303 and Bartolomeus755 like this.
  6. Mr_Dan_O

    Mr_Dan_O

    Joined:
    Oct 12, 2013
    Posts:
    345
    @NatureManufacture I have a great suggestion for the version 2 of WS - not sure if anyone said that before but I think it is good.
    GPU Instantiate!! What if WorldStreamer stream objects via GPU instead of CPU? We can then use it for vegetation and lots of smaller objects very efficiently.
     
  7. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    There is Vegetation Studio which spawn foliage, made by our friend / person with which we co-op on new assets.
     
  8. Mr_Dan_O

    Mr_Dan_O

    Joined:
    Oct 12, 2013
    Posts:
    345
    Yes that is right but I what I mean is a technology/concept that you could spawn the whole terrain and other objects via GPU with.
     
  9. SonGoku10c

    SonGoku10c

    Joined:
    Dec 9, 2017
    Posts:
    75
    Hi.
    When will the other video tutorials go online?
     
  10. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,598
    may want to ask something in general. The ones up there cover how to set it up, which ones would you like to see? This will help Bart knowing what your after. /cheers.
     
    NatureManufacture likes this.
  11. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Which one? because all general things are done I think.
     
  12. Vagrod

    Vagrod

    Joined:
    Aug 4, 2017
    Posts:
    63
    Hi! I'm still having troubles with terrain async loading. I have 64 scenes with 1000x1000 terrain pieces, and profiler was showing Loading.LockPersistentManager huge spikes that drops fps from 90 to 12 for about two seconds. I read that unity may lock async calls while other async calls are active, so I re-wrote Streamer script: I got rid of a while loop and made a chain load mechanic (AsyncOperation.OnCompleted triggers the next async load). I see no more LockPersistentManager spikes, now I see garbage collector 150ms (on scene unload I guess) and Culling 150ms (on load).
    Terrain does not contain any objects on it. Plain terrain with VegetationSystem script turned off. Terrain resolution is 1024, realtime GI. Indirect resolution 0.1.
    At this point, I doubt that seamless terrain streaming on unity is possible.
    gc.jpg cull.jpg
     
    Last edited: Apr 9, 2018
  13. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Why you got culling? Culling not appear in such way specialy at empty scene with terrains. I dont feel like re-wrote this was good idea:D Any custom shading on terrain? RTP and few others could generate spikes with unproper setup - exacly at GC. How many tiles you got loaded at 1 time? What's the unloading distance?
    Edit:
    Btw when you rewrote this did you take into account terrain neighbours? They could create such spike from terrain re-stitch etc. Stitching could also hold terrain references in memory.
    -1st spike from stitch
    - 2nd gc from holding references.
    Just short idea
     
    Last edited: Apr 9, 2018
  14. SonGoku10c

    SonGoku10c

    Joined:
    Dec 9, 2017
    Posts:
    75
    I mean these:

    UPCOMING TUTORIALS:
    6. World looping and small planet construction.
    7. Floating point fix system.
    8. Collider streaming solution.
    9. Local area updater introduction: tool for big worlds and teams.
    10. Workflow with other unity assets like World Creator, GAIA, Gena, Terrain Composer, World Composer, Weather Systems, Horizon Systems.

    Are the other ones still up-to-date? the are already 1.5 years old. I think WorldStremer has changer much or?
     
  15. Vagrod

    Vagrod

    Joined:
    Aug 4, 2017
    Posts:
    63
    I have terrain culling scripts on terrains, and I have vegetation system on terrains which is using culling -- but vegetation script is turned off right now on all terrains.
    I modified streamer to check if multiple active async-loading scenes are actually cause locks (LockPersistentManager issue). I have RTP on these terrains -- and where can I get/look at the RTP "proper setup" than?
    I have streaming grid of 1000 units for cell, 8 cells around the player, so it's loading 2-3 terrains at a time usually.

    I carefully replaced the loop with chained loading, so all the logic is totally intact)
     
  16. OneManBandGames

    OneManBandGames

    Joined:
    Dec 7, 2014
    Posts:
    207
    Just wanted to mention that I have the same problem with Loading.LockPersistentManager spikes. My setup is 7744 scenes with 128 x 128 terrain pieces. (I get away with such small pieces since my game uses an isometric perspective, so only a small part of the game world is visible with fixed perspective.)
    I did not do much yet about it since I felt it might be falling into premature optimization, I have noticed several entries so far for spikes during terrain loading, and some of those have "magically disappeared" again while things changed in the game. I will keep an eye on this thread, if there is any testing that I can do from my side, or any info I can give you let me know and I will be happy to help out & maybe make that spike disappear for me as well.
     
  17. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,598
    Those are no different as they use Unity terrain.... actually I believe some of the tuts use, TC or WC.. either way they are all Unity terrain, those are NO different than what is set up now with current tutorials.. as far as those go. I did this a few weeks/months ago actually.

    I tested the tutorials, to set up.. from videos and docs.. and they still work, some changes happen but nothing from the set up stopping anyone from setting it up....

    Now if you added Multi player, and a bunch of custom stuff that is another story.
     
  18. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,598

    I suggest giving the info, as Bart asked the other person, knowing how many tiles are loaded at once. what is the over all terrain size of the terrain. Do you have other assets in the scene? other scripts, etc?
     
  19. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    The probem with RTP is - it hold all terrains in list to manage snow and other global features, so you actualy don't unload anything and it throw lags during unloading process. There is manager which operate at many terrains at the same time and hold them in memory (that was the main problem with few terrain systems at store). We had to for example rebuild CTS for Crowfall MMO to co-op with terrain streaming which is simmilar to world streamer. (update is available on store). I don't personaly use RTP as I'm focused on own solution but hmm you could ask Tom to take a look at it and update system (He is good and really smart person so he will handle this). As magic thing to handle RTP as I remmeber was idea to remove all managers etc and leave only terrain materials. Maybe somone have other way as there were few users of RTP here. The second idea was to move RTP managers to main scene so there will be only 1 manager for whole world - I don't know how to handle different texture sets there. I personaly used RTP long time ago. Im not an expert in this area anymore specialy when this is third party asset.
     
    Last edited: Apr 10, 2018
  20. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Nope there are no big changes during this time only small fixes. System was cleaned and it's in very stable version. Yeah we could finish this tuts, but as we move assets to SRP system for unity 2018 we are really bussy:(.
     
  21. OneManBandGames

    OneManBandGames

    Joined:
    Dec 7, 2014
    Posts:
    207
    Yes, you are right, I need to provide that info for a proper support post. My intention was just to note "Hey, I see that one too", not to ask NatureManufacture for support since I currently still have spikes that I probably can get rid off by myself first. If I run out of all options I will post over here again with all the info.
     
  22. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    With streaming there is always everything about what to load and how and how much we could at once. Anyway i will check this what you mention myself.
     
  23. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,598

    ya I under stand that but not everyone will have the same issue, to many assume it is, what may work for the other person may not for you, so having as much info as possible so they can look into the issue is all i'm saying /cheers
     
  24. Alex33333333

    Alex33333333

    Joined:
    Jan 3, 2017
    Posts:
    3
    Hi! I have a problem. My terrain is really heavy, so i can't activate all terrain pieces at the same time in hierarchy, because unity will crash. So most of pieces are disabled. When i hit "Split scene into a virtual grid" nothing happens but when i make part of pieces active everything starts working. So, is there a way to make it work with disabled gameobjects? What i need to add to the source code?
     
  25. Vagrod

    Vagrod

    Joined:
    Aug 4, 2017
    Posts:
    63
    You can use
    Selection.objects
    and iterate through gameobjects that are selected in the Hierarchy (enabled or disabled, does not matter).
    To save your ram, close Scene view and Game view before enabling terrains. I ended up upgrading my notebook to 48G of ram in order to use all my terrain pieces.
     
  26. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Or use local updater which is designed to work on worlds where work scene would crash engine;)
     
  27. Redrag

    Redrag

    Joined:
    Apr 27, 2014
    Posts:
    127
    I have started to look at World Streamer mainly because of the floating point problem. (Although having the ability to make bigger worlds will be very useful in the future.) The manual does not say much about how the world coordinates are reset "World Mover will restart your world's position". I assume this means basically moving everything in relation to Unity's coordinate system. So could you tell me how rigidbody physics is handled? Using the free floatingOrigin script I get issues...
     
  28. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    It moves all world to 0,0,0
    All phisics objects need our script -it is for objects that are: not player, physic, not streamed. Everything should be fine.
     
  29. InGameGames

    InGameGames

    Joined:
    Mar 3, 2018
    Posts:
    33
    Hi, I was thinking of buying this tool to use along with Procedural Planets and World Creator to work on a project set primarily in space. It's based on colonization and you run the colonization effort from a space station at a la grange point between a planet and it's moon.

    I was wondering whether or not it would be practical to use Procedural Planets to generate the planet and moon, then use World Creator to build out landing sites with World Streamer allowing the user to swap between monitoring the inside of the Space Station through the bulkheads, the ground level colony and the planetary system overview as a whole seamlessly.

    Only the ground level colony and the inside of the space station would be close to scale with the planetary system view faking it due to LOD limitations on the Procedurally generated planets.
     
  30. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    4,348
    Consider that you can't rotate terrain if you use them.
     
  31. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    I would say unity terrain is not suitable for this kind of project.
    How about curvature of planets ?
    You know you could stream what you want etc:) just make good design - structure of maps.
     
  32. BrokenVector

    BrokenVector

    Joined:
    Feb 2, 2017
    Posts:
    32
    Hi,
    I'm having some issues with optimization.i get a good 2-3 seconds of 2 fps lag when terrain tiles are loaded. The tiles are 512 and I have the range set to 1 for loading and destroying. The lag exists even without any vegetation, just plain terrain. I am using CTS with 6 textures. Any ideas? I can upload my terrain settings if that helps.
     
  33. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Check your console for errors.
     
  34. sharkapps

    sharkapps

    Joined:
    Apr 4, 2016
    Posts:
    138
    I built a minimum scene with terrain only and the major streamer seems to be working fine, but I am having problems with the Ring streamer for the LOD1 terrain meshes. Only 1 of the LOD1 terrains seems to load. Please have a look at the video for more details. Any idea how to fix?

     
  35. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    4,348
    Wait a moment, something I might not have consider, do this works with 3d coordinate? I mean not only XZ (or XY or whatever 2d coordinate system one might use)?
     
  36. BrokenVector

    BrokenVector

    Joined:
    Feb 2, 2017
    Posts:
    32
    The only error I am getting is the "Can't remove terrain because CompleteTerrainShader (Script) depends on it" which was brought up before, but I don't recall that it caused lag for others. Below is a snippet of my terrain specs (note: I'm using VS):
    upload_2018-4-24_21-31-6.png
     
  37. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Are they on build settings?
    Yes it works on 3d but unity terrain streaming is rather 2,5d as this terrains coudn't be rotated but etc.. with mashes you are free.
    Check profiler because log to console kill every app if its error.
    VS could have gc issue now but check profiler for spikes - what generate it.
     
  38. sharkapps

    sharkapps

    Joined:
    Apr 4, 2016
    Posts:
    138
    @NatureManufacture Yes, all of the scenes from both scene collections are verified in the build settings.

     
  39. sharkapps

    sharkapps

    Joined:
    Apr 4, 2016
    Posts:
    138
    my ring streamer issue turned out to be a change I made when testing some performance enhancements. it is now working fine for me.
     
    NatureManufacture likes this.
  40. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    857
    Just something to remember for anyone reporting spikes and what not when loading and unloading, that definitely happens when playing in the editor, you have to also make sure to make a build and test it there. There are a good bit of things that are different between editor play and actual standalone build play, such as the batching and what-have-you.
     
  41. Alex3333

    Alex3333

    Joined:
    Dec 29, 2014
    Posts:
    338
    Good afternoon. who faced the problem ?! When the pieces are unloaded, objects with solid bodies are fed into pieces (ground). I tried on them to hang the script PhysicCullingSystem but it did not help. 0 (.
     
  42. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Check how this was made in our demo. We have physics balls there.
     
  43. BrokenVector

    BrokenVector

    Joined:
    Feb 2, 2017
    Posts:
    32
    It seems to be TerrainData.AwakeFromLoad that is soaking up all the memory. Attached is a screenshot.
     

    Attached Files:

  44. Chevvvv

    Chevvvv

    Joined:
    May 15, 2018
    Posts:
    1
    would this work for networked games?
     
  45. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Screen from editor not build ?
    Yes but floating point fix need additional attention, also client and server should separated app.
     
  46. baaleos

    baaleos

    Joined:
    May 12, 2018
    Posts:
    3
    Can anyone confirm if there are any tutorials on how to implement this for Network behaviour - EG: Multiplayer games?
     
  47. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    No we didnt make any tutorial. We only mention how to solve floating point fix in client server messages. There are many network systems and they work different. All networking is on developer side.
     
  48. sharkapps

    sharkapps

    Joined:
    Apr 4, 2016
    Posts:
    138
    I am wondering how would you use Vegetation Studio in the ring-streaming scenario. I can spawn VegetationSystem when each LOD0 terrain scene loads, but I don't know what to do about LOD1 scenes which are low poly meshes made with Terrain2Mesh. When using Unity terrain trees, everything is fine, because Terrain2Mesh can export the billboards along with the mesh. When using Vegetation Studio, vegetation is only generated on LOD0 terrains. Any ideas?
     
    MarkusGod likes this.
  49. BrokenVector

    BrokenVector

    Joined:
    Feb 2, 2017
    Posts:
    32
    I'm not totally sure what you mean, but this is the Profile Editor in my "Game" scene, the second image is the Profile Editor in my my main working scene that is split to generate different scenes, and the final image is the working scene being played in Game mode.

    Playing the Working scene that has the actual terrains is super slow to load, so it must be something to do with the rendering of the terrain. I cleared the preloaded shaders, played the streaming scene for awhile and saved those but it didn't help.

    Thanks for the help on this.
     

    Attached Files:

  50. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,624
    Because vs spawn realtime objects at the terrain. Did you bake vegetation?

    I see only profiler and ui spikes here.