Search Unity

[RELEASED] World Streamer ! Forget about your game memory usage and create big world!

Discussion in 'Assets and Asset Store' started by NatureManufacture, May 13, 2015.

  1. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,615

    I know for Networking They used ULink. I was talking about the Streaming aspects. ULink sucks, but the issue is alot of people have very few choices for MMORPG's ... Unless they make there own custom solution..... I won't make large games like MMO's in Unity personally. Anything with alot of people I would rather use Unreal or an in house engine . Every Unity game that I have seen that has alot of people in the same location , suffers badly.


    Unity needs to make games like MMO's better.. Which I know they planned to fix some of the many issues related to large MMORPG's Its too painful of development making those type of games in Unity , from what I have seen and played. and experienced myself.
     
  2. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    I made this system mostly for MMO and GV at the beginning, because I was Black Eye Game (Gloria Victis) owner until 1st January 2016. Now World Streamer is a very universal system, for each type of game. Anyway it works well with all multiplayer games.
     
  3. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    @NatureManufacture maybe since the performance in editor is 1000% less efficient compared to play mode -when all of your other developments are out of the way and you have time interest you maybe you can add a temporary solution to make working in the editor fast as well.. I want to be able to work on massive scenes/worlds in both editor and or play mode. I am investigating a possible solution that will help with one area rendering of models and only of areas that the camera sees "Polygonal Culling" then later when I add texture streaming like granite it should help out even more.
     
  4. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    This is not possible to compare editor and build this is totally misunderstood. Editor is used to create games not to play them. In every engine aspect editor is slower, specialy when it do additional calculations which are already made at build. If you will use tips from manual you will don't notice any spike during streaming even at editor. It will be slower like everything in engine editor what is compared to build. In tutorial video I get 300-500 fps without spikes, but ofc at build it will work much faster. So if something is faster at build, then it is. You cannot jump over engine architecture

    I worked at 6gb scenes with 10 milions of models with world streamer and everthing was just fine. World Streamer tool called "Local area updater" was also dedicated to this kind of scenes.

    I check this Polygonal culling" and hmmm it's nothing so much special, it work like occlusion culling and it's also baked. Probably it's better then Umbra occlusion culling but still nothing special in ofc only my opinion. From my experience with good LODs and billboard as last LOD this kind of system is cool add-on but not the core of optimization.

    Granite is cool, but Amplify texture 2 too:) check both:). I used Amplify texture for 2 years and I was very happy. It work great with unity editor, its easy to find own mistakes with it but... ofc there is but.. It not support objects with alpha transparent :( So probably Granite here is much better. When I tested Granite first unity version with his CEO, about 1,5 year ago, It didn't support transparent object. I don't know how it looks now. Basically unity load textures async (option in quality settings), so if you left few textures as transparent and unstreamed nothing bad will happend. Ofc both systems gives you ability to use unlimited texture resolution what is extremely cool!
     
    Last edited: Sep 2, 2016
  5. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,794
    Photon maybe? Unet is a pain still and bolt won't work for us. It is only peer to peer.
     
  6. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,615
    I can't see why not, But I have not used Photon Pun in for ever,
     
  7. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,615
    Ya that is what I thought you had said to me before. :)
     
  8. runningbird

    runningbird

    Joined:
    Sep 3, 2009
    Posts:
    377
  9. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,794
    We have been through all this before...Ulink, no support, Atavism, lots of promises and closed code, etc. We also do not want peer to peer or a lobby based multiplayer game.

    At this point, we want stability, not a new company with little to show yet.
     
    recon0303 and runningbird like this.
  10. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    I find the more geometry I add to Unity's hierarchy, it slows down when navigating around in the scene view so I wish that somehow real time culling could also be sent to Unity's scene view camera with some kind of slider to change the radius visibility (maybe disabling all the geometry/particle effects etc outside that radius -better than having to disable a lot of objects in the scene manually. -would be awesome if possible.

     
  11. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Thats why you should use Local Area Updater. You simply load only specific area or layer not whole world. You work on it very quick without performance drop and you could update your world by only 2 buttons. If you need I could show you how easy is this;). Im always open to help.
     
  12. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,615
    Hello Teila what are you looking for just wondering? I hear you on all points that was our issue, that is why for most of our Multi Player games we use other Engines. We plan to make a custom solution at one point in the future for some game games for Unity.... That seems the way to go with Unity sadly for MMO's and games like RUST, Shroud of the Avatar and Gloria Victis. I know some of them went custom for the same reasons most of us dislike about the other tools currently.

    We are making a mobile game with just Co-Op, and Bolt for us fits perfect for that.. But sadly Bolt is not for large MMO's at all.

    Sorry Bart if I went off Topic a bit.
     
    Teila likes this.
  13. Stanchion

    Stanchion

    Joined:
    Sep 30, 2014
    Posts:
    266
    You can use Bolt for a game with dedicated servers.
     
  14. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,794
    Bolt won't work for us. It has limitations. I don't think Bolt is recommended for more than 64 players.
     
  15. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    Sounds good!..this is what I was basically asking about, some kind of system to make working on big projects easier in the editor wondering if you might have something like this built in. I was working with many scene files loaded & saved in a master scene and would disable which ever under it I needed to but these still add up and require manual loading/unloading. Is there already a tut of Local Area updater in one of your videos?

     
  16. recon0303

    recon0303

    Joined:
    Apr 20, 2014
    Posts:
    1,615

    nope its not. From what I been told. and seen. If you are going to have a large amount of people in the same area..I would NOT use Bolt or any of them, I would make a custom solution, or switch engines. To be honest I would be using Unreal 4 . Or get your hands on someone to make you an engine or buy a license from someone that support this.


    But Unity is still doable, with a custom solution and/or with a very experienced Networking programmer on your team..

    Only MMORPG that I know of that have been made with Unity that are released or are near release are Shroud of the Avatar, which I thought used ULink, no idea what they use now...Gloria Victis, which is using ULink, but I know they wanted to move to something else..

    Then you have Crow Fall.. Which I think they are using a Custom Solution .. Albion Online is using Photon Pun. I think.. or was. There may be more, so maybe reach out to one of them, to get some more info.
     
    Last edited: Sep 3, 2016
  17. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,794
    We are using Photon Server which we will customize at the moment. We may go back to our custom server but were having issues and want to get our first test "episode" out soon. Once that is running, my network guy will probably look at the custom stuff again. Just need something stable to get something out.

    We are not making a massively massive giant game, so not worried about using Unity. Anything over 64 is considered an MMO...so that is the only language we can use.
     
  18. Stanchion

    Stanchion

    Joined:
    Sep 30, 2014
    Posts:
    266
    You can have over 64 players with Photon Bolt the same way you can with Photon server, interest management.
     
  19. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    I will make it today or tommorow, world update in 2 minutes in 2 different ways is first on my list. In manual you also have info about it:)
     
  20. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    It was my fault, first started watching the tutorial before reading that you have a system to make working in the editor
    faster but I really appreciate you taking the time to explain it in a video, I usually default to videos so I get a more of a solid understanding of the workflow but they don't always cover everything.

    Also I currently have two terrain technologies I want to use with different positions in the transforms and sometimes will overlap -Voxeland overtop Unity Terrains I have tiled baked areas of Voxeland which has its LODs and want to use your LOD system on the unity terrains.. is this possible?

     
  21. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    LOD system - you mean blend terrain neighbours?
     
  22. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Here is new tutorial, this time with world updating:
     
  23. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    When I bake Voxeland it makes HiRes and lowRes meshes and splits/tiles them as neighbors.
    Awsome! thanks for the new video :)
     
  24. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Hmm it works between unity terrains, I don't know how it will work with voxeland. It depends how it marks neighbour objects.
     
  25. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,736
    Can you elaborate on this? Are you referring to multi-terrain culling? Or some special terrain culling even for single terrain object? And for your estimated cpu/gpu improvements you say 90-10%, but that doesn't make sense. Do you mean 10-90%?
     
  26. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    It's our terrain culling system (script). We turn off terrain detalis which are invisible. It works only with terrain in tiles. This is not for single terrain. 10-90% depends from number of terrain tiles that you have loaded/active in your scene. If you have many small terrain tiles you will remove 90% cpu usage from unity terrain objects. If you have only 9 terrain tiles loaded/active you will remove 10-40% cpu usage (it depends from number of terrain tiles that you have curently at your view).
     
  27. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,736
    Thanks for the clarification.
     
  28. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    No problem:) There will be also video which will show power of this solution:)
     
  29. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    I haven't read the full doc yet but I was wondering if there is a way to unstream objects that are hidden inside or behind a bigger object at close range but they will only appear only if the camera sees them. I know you have your own culling system implemented as you say for hiding parts of the terrain the camera can't see but I don't yet have details on its full potential. I have Unity Pro but it's culling system makes everything in range appear even if objects are hidden. I was also looking at using instantOC but looking at the FAQ it states that Unity's OC is better unless people don't like the bake times etc.

    I'm using the "fracturing and destruction" kit and it generates it's own prefabs. I've also contacted the developer of this asset to see if he could optionally add a script to hide all objects inside a fractured mesh and only until it is impacted that the inside fragments of the particular prefab becomes active but I am not sure which method would be better your culling system or the idea I sent to the other developer.
     
    Last edited: Sep 8, 2016
  30. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    You could use our collider streaming for rooms buildings and interiors. You will not load specific area unitl player will hit the collider. It also could be saved with layer (matriosha construction - streamed object that could stream an object). It work with looping and floating point fix system.
     
  31. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    That's awesome! thank you! and the fragments don't need to be renamed small, medium etc? saved me a lot of trouble!
     
  32. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Nah you just create an object, then attach our script. Next step drag and drop objects into this collider streamer and done. Please read more about collider streaming in manual :). There is whole paragraph about it;)
     
  33. mrdl2010

    mrdl2010

    Joined:
    Jun 1, 2016
    Posts:
    23
    Stupid Unity Notification email being late made me miss the 50% sales by 4 hours. I put this on wish list fgs. Need to wait for another sale ...
     
  34. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    When it makes meshes it seems they are only side by side and each chunk is basically LOD management and position. What if each chunk were to be relabeled Terrain? would it keep the position of each regardless? How would you manage terrain if it were all just meshes rather than unity terrain?
     

    Attached Files:

  35. Hazor

    Hazor

    Joined:
    Feb 5, 2013
    Posts:
    25

    Same, did Unity removed the monday sale based on weekend sales?
     
  36. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Unfortunately yes.

    Basically we use unity functions to manage LODs. Unity terrain could hold info about neighbour terrains. Simple mesh don't have such features because unity terrain is a matrix, simple mesh is not and it not behave like matrix.

    Trick and Tip:
    To avoid holes between terrain lods and models you could simply move this further mesh a bit down on Y. Terrain which is used as LOD0 should be a bit higher 1-1,5 unit.You will mask transition between lods if there is no possibility to use terrain neighbour script, between objects, Player will not notice that you move terrain which is far away by 1-3 units, when it become closer. Look at our advanced demo.We replace simple mesh by unity terrain even in short distance. There is no possibility to connect mesh LOD with unity terrain LOD but we put meshes a bit lower then unity terrain and we dont see any holes between objects.
     
  37. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Screens from upcoming tutorial: "3. Terrain split and adjusting it to streaming in very short time".

    We had to create new big/heavy terrain for terrain tutorials.

     
    Last edited: Sep 13, 2016
    John-G, mrdl2010, Teila and 1 other person like this.
  38. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,736
    I have a question about large faraway objects. In some cases for larger outdoor areas I will want to have imposters for large faraway objects and have them visible for long distances. So, I'm wondering if World Streamer has a way to handle this. From what I've seen in the videos it seems like these objects would stream in only when within a certain range. The easiest way for this to work would be to simply be able to mark certain objects as not part of the streamer so they are always in the scene. Then I could use typical LOD groups to keep them as imposters far away and full objects when up close. Another way would be to have a built-in screen-space size algorithm in World Streamer that allowed these larger objects to be streamed in much sooner. I would mainly use this for large structures and landmarks. Is this sort of thing supported in World Streamer?
     
  39. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Thats why you World Streamer have layers. Different objects could be streamed in different way, you also could manage this in runtime. So system handle this. You could stream small objects in small distance, big in big distance etc..If you stream in layers you will save a lot of cpu,gpu,ram,vram. You even don't have to refresh LODs of invisible objects - this cost cpu.
     
  40. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,736
    Fantastic. I already own World Streamer I've just never gotten to the point where I am ready to start using it. But I'm going to be needing it soon and the thought of this scenario occurred to me as I was planning a level. Thanks for the quick response. World Streamer is definitely going to be a key part of my workflow.
     
  41. Apolyon6k

    Apolyon6k

    Joined:
    May 5, 2013
    Posts:
    32
    Hi, I recently bought WorldStreamer and tried to work with it but for some reason Unity (5.4.0f3 Personal) always crashes if the player leaves the first loaded terrain tile. Is there a specific rule to not have scripts or trigger collider in work scenes or what could be the problem I am having? (hopefully everything set up correctly by manual and tutorial video)

    Any help is appreciated, I am streaming my development on Twitch (Twitch name same as here, times are 18:00-22:00 CET and 10:00-14:00 CET during the week), if you want to have a look and maybe help out live :) [this is not an ad for my stream]

    [Edit] After a few tests I changed all layer sizes to the terrain tile size and it no longer crashes for me. Having a layer with a different size, causes the editor and game to crash in mentioned Unity version and latest WorldStreamer from assets store.
     
    Last edited: Sep 13, 2016
  42. mrdl2010

    mrdl2010

    Joined:
    Jun 1, 2016
    Posts:
    23
    Really looking forward to this.
     
    Ascensi likes this.
  43. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    Ok so it sounds like I can just stream Voxeland like normal meshes.. they aren't the main terrain and would stick partially out of the ground anyway.

    One last question for now.. what is the most optimal size for multi terrains for specific terrain sizes/resolutions .. you seem to choose 100 in both the documentation and the video tutorials. For example I currently have terrains 1000x1000 @ 4097 resolution.. I could them tile smaller.. I'm using World Creator Pro so I have some flexibility.. Here is a sample.. I'm using 4 tessellated displacement terrain textures@ 4k each. Will be using texture streaming and compression asset later.


     
    Last edited: Sep 14, 2016
  44. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Hi we found the issue with our terrain culling system but also with terrain neighbour script.
    Here is upcoomming update info:

    upload 16.09.2016

    - World Streamer Tutorial 2 (done)
    - World Streamer Tutorial 3
    - World Streamer Tutorial 4
    - Local area updater boundary gizmo fix (done)
    - Additional function to turn on/off streaming layer (done)
    - Terrain neighbour script fix and terrain streaming improment by 500%
    - Terrain culling system quick fix (it could generate editor crash)


    It depends from your camera view and your optimization process.

    In my upcomming tutorial I just cut 10.000x10.000, 4k heightmap resolution terrain into 625 x 625 or 1250 x 1250 parts.I want to have 2-4k far view on terrain at my camera. Im going to use 1250x1250 or 625x625 terrain tiles as base (HQ terrain for close range) and then I will render only low polly mesh to 2-4k view. In this tutorial I just found that our terrain neighbour script took 90% of terrain streaming performance what is total mistake! We also find small error with terrain culling system. Anyway we will fix this today and upload new version with tutorial. Btw cool terrain!
    So basically just compare your camera view with information about how far you need your terrain to be HQ. The distance that you want hold HQ terrain is your terrain tile size.
     
  45. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    I don't know how long it takes for the asset to get approved on the store, can I send you my invoice and get an update link? I haven't told world streamer to split any scenes yet and I have a massive multi terrain tile to be processed but don't want to give it issues first time go with your asset. I'll send you a PM and go from there.

     
  46. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    I will send you it tommorow. We have to fix one small thing, and everything will be done. We had problems with unity (engine bugs decrease performance).
     
  47. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    Update is ready to upload. I also send you it by mail. I will upload new version to assetstore as soon as I finish video tutorial (today). This tooks a bit long because we had to handle few engine bugs which eat performance by unity terrain. (not related with streaming).
    At tutorial scene we have very heavy terrain 19-20 fps. After streaming we get 70-180 and we didn't finish, also we lose some fps on unity bugs (about 50%)
     
    Last edited: Sep 16, 2016
    Ascensi likes this.
  48. Ascensi

    Ascensi

    Joined:
    Sep 7, 2013
    Posts:
    577
    So aside from resolution, is there any difference in how many terrain tiles sizes loaded within camera range?even if they were all 2k res I'm wondering what might be optimal visually & for performance. Might be good to test with many flat terrains, no textures etc just to see if its better to load 16 small terrain tiles to make say 10 Miles or whatever vs 8 tiles or even 4 tiles to make the same 10 miles using the same camera view.

    Just like the post request below mine, I would also like to see a tutorial if possible working with World Creator -it's what I'm using.

    Since the latest update I've been getting a compiling error with unity name space: Assets/WorldStreamer/Scritps/PlaceTargetWithMouse.cs(7,18): error CS0101: The namespace `UnityStandardAssets.SceneUtils' already contains a definition for `PlaceTargetWithMouse'

     
    Last edited: Sep 19, 2016
  49. Bengals007

    Bengals007

    Joined:
    Sep 11, 2016
    Posts:
    11
    Do you have any tutorials or docs on how to integrate World Streamer with World Creator?
     
    Ascensi likes this.
  50. NatureManufacture

    NatureManufacture

    Joined:
    May 20, 2014
    Posts:
    1,691
    I just finish one that use world creator terrain. I should available it today.

    Unity have few bugs and in my opinion if you have a lot of data (like I had in demo) you should use 9-25 HQ tiles (loading range 1-2). In that way you will get best performance. If you are going to use system that optimize trees and grass like unature and https://www.assetstore.unity3d.com/en/#!/content/43129 you could use more tiles.
    In my upcoming tutorial I use HQ terrain up to 1300 unity units view (3 tiles each 650) and then I use low poly mesh up to 6500 units view. So i get 6500 units view with 80-120 fps without optimizing anything (trees colllider take a lot of cpu). Terrain streaming depends on data that you have on it. If you will hold only necessary data it could be very cheap. For example: remove additional colliders from speedtree, unsued textures and trees from tiles,unsued grass objects. Like I said i didn't do this and i get very good performance, most cpu is eaten by tree colliders so probably when I will optimize them I will get 200-400 fps. Also if you will our use terrain culling system, you will get some additional fps even during streaming. We found that when we turn on and off tree it generates engine crash o_O. If only this feature will work we will get much much better perofrmance and streaming will be also much much faster. They repair and brake this tree managment from version to version....Thats why we put this as option because in many projects it could work.

    Hmmmm.. Script in your project have been duplicated. You could simply remove our script Assets/WorldStreamer/Scritps/PlaceTargetWithMouse.cs. This script was used only for test/tutorial scenes. Standard asset folder in world streamer catalog is only for ethan controler at demo scenes.
     
    Last edited: Sep 19, 2016
    Ascensi likes this.
unityunity