[RELEASED] World Streamer ! Forget about your game memory usage and create big world!

Okay, I've removed all grass and trees from my terrains, and disabled the LOD mesh terrain streamer, so my build only has the Unity terrains being loaded/unloaded (9 of them around the player). And I still get these huge spikes of TerrainData.AwakeFromLoad (three spikes of 246ms each). I found this thread but it wasn't very helpful: https://forum.unity.com/threads/reducing-terraindata-resource-loading-overhead-per-frame.323846/

Maybe I'll try 16x16 terrain grid with smaller heightmap resolutions (512 instead of the current 1024)....has anyone experimented with those sizes/resolutions to see what effect going smaller has?

EDIT: @sharkapps: My unity terrain streamer had almost exactly the same settings as yours, but a few minor differences, so I changed mine to match yours....but I still get these huge spikes. But I see on the other thread that your terrains are smaller, so that's the next thing for me to try.

 

@gecko verify that your terrain textures do not have Read/Write enabled, and if they already don't, then yes, it's possible using smaller terrains and resolutions will help.

 

I feel like moree terrains generates more problems and always advice to keep 9.

 

@sharkapps I did have a bunch of textures marked Read/Write (I guess for MicroSplat), so toggled that off and was very hopeful that would help a lot....but alas, no better. So the next step would be reducing the terrain sizes to 512....but I respect Bart's advice about not going above 9 terrains...

So now I've watched again the Unite Berlin video (at 32:50) about terrain improvements coming in 2018.3, and seeing how 2018.2 just released, I'm more hopeful that 2018.3 will be released earlier (Sept/Oct) than later (Dec). Apparently these improvements will boost loading performance tremendously, and also possibly even eliminate the need for mesh LOD terrains. So I'm thinking maybe I should just set this aside for a month and see what the 2018.3 beta can do by then. No sense pouring a ton of time into this now if that'll solve many of my problems in the not-too-distant future.

 

Terrain lod will never be eliminated. Most open world or big world games use this solution.
Witcher, star wars batlefront 2 and 1. There are articles.about what each lod contain. I would say star wars push this to limits.s
Specialy when they convert all splats into streamed pixels so they got unlimited amount of splats and resolution. Ahhh

 

@NatureManufacture This sounds really interesting. I'll have to go searching for the articles. Maybe after you're done with SRP, you can start selling a splat streamer?

 

It's a virtual texture solution like farcry? They literally bake the splat and decals down to texture pixel for streaming.

 

Yes all big games use this solutions ofc... builded game size is huge counted in many gb because of.this texture arrays.

Amplify texture could stream such data but...first someone must bake splats into epic size array. This basically is easy but without amplify texture or graphine system is usless.

 

That makes me think, could we just bake things, in a temp buffer (possibly circular) over multiple frame, down in real time and discard all useless texture? Philosophically, that's what we are already doing on screen but the screen as a fixed acceptable size. I need investigate a bit more, might be another bogus idea ...

 

I always think how much custom solutions I should use and what unity will make in update?

We could move WS into json and speedup it but we waste time when unity will upgrade something etc...We try to keep standard unity data and solutions, rest it up to devs. Thats why we marked asset as avanced unity devs only in last few days,

Anyway such terrain conversion is interestig thing. When we create cts we feel like 1 lod is missed for background terrains and unity didnt support more... Only lod0 and basemap (lod1). Ofc lp mesh change that. Hmmm you always can create texture from existing custom or standard terrain render, extract it and stream by streaming system. Big companies, I would say all chose same way why? because it's best You have only 1 texture with pixels streamned from array. Unlimited resolution and splats? what else you need?

 

Good afternoon. For a long time I could not understand what was my problem. unfortunately now I can not write a specific function that killed all the performance in the build. all the tips that the developer of this asset gave me did not help. I did everything right but loading and unloading was disgusting !!! I struggled with this for several months. recently I decided to go through the project. I began to remove the system one by one to find the culprit! and came to the conclusion that my project is to blame - TRANSLATOR TEXT! I use the sides of the text translation. To my surprise there was no limit! I was happy about this and at the same time was in sorrow! )) I could not think that it was in it. After removing it from the project (temporarily), everything became fine with me. After that, I had a problem with a lot of objects in the scene (the solution was written in the documentation), I divided everything into several layers and now everything seems to be normal))) maybe you use such an attribute or similar function that does the check each time a piece is loaded cards. After the project is transferred to a new version of the engine, I will try to find this function and send a screen. suddenly someone will help and guide the correct way. (exactly the same problems were like gecko) I have not been on the forum for a long time

 

I didn't meant to tell you what to do! I was thinking aloud following the statement made before. I need to implement (pcg) planetary terrain in future task (got the spherical projection down only,still need LOD and big scale) and this kind of discussion allow to anticipate potential solution to explore.

 

@NatureManufacture Have a look at this Unite video.

 

Wow.

Just. Wow.

I was on the bandwagon (not long ago) that Unity clearly needed to get its act together on the game-making front.

Now, it appears to me, they have covered just about everything they could possibly cover in games with Unite this year alone.


I never thought they'd officially tackle something like world-streaming too! (especially in addition to their new rendering-pipelines / streaming-animation accessibility, new input-system, new terrain tools -- and even their ECS-spearhead efforts!)

The Unity team totally has my unwavering respect now -- and if they even deliver on half of these, I'll be proud of them for that accomplishment alone -- but! -- if they somehow manage to deliver everything they've promised -- well, let's just say that it's a great time to be a game developer!

There are clearly great days ahead of us!

Thanks for sharing this @D4N005H ! -- I totally appreciate it!

 

Hey there @NatureManufacture I am not sure if this would help WorldStreamer any, but based on what it says in the article, if it does, it would be pretty nice!

https://medium.com/@neuecc/zero-ove...om-async-statemachine-with-c-7-0-6808695733ba

 

I have issue - I have Scene (one Unity Terrain, but not streamed) with multiple buildings and trees. I followed the tutorial, created one streamer for trees and another one for buildings. I use mobile shaders. Streming works fine, but when next scenes are loading, animation is very choppy. When it finishes, it's good. I tried to modify various values in Streamers, but nothing helps ;/ is there any solution for that? @NatureManufacture

 

I'm sorry that I did not unsubscribe before. A couple of days ago I became a dad. Here are the screenshots of what I commented on in the translator. it gave me huge peaks in my project. After that, everything became normal in my project. Line 55 commented. And also lines 171-180. it was in my project that made great peaks !!!! drip in this direction who has large peaks. I hope I'll help those of you. a screenshot attached. If I understand correctly. he each time started cleaning the cache and also unloaded unnecessary assets. and it happened every time you unloaded the scenes !.

 

Congratulations!

 

What about the preservation of information about the objects?
I want to implement the system as in the games from Bethesda, when the objects left by the player keep their positions after the player goes very far, and then returns to the place where they left these objects and they do not disappear anywhere. How can this be implemented with your asset? Maybe you have already made this system?

 

@NatureManufacture

With Enviro, it automatically put the Enviro / Skybox object into the scene's lighting but if I click it there is no prefab in the project for it, so I am not sure how to put it into the World Streamer Lighting manager box so I can apply it to all the scenes? Do you have any idea how to accomplish this?

 

This is non streamed part - put it on game scene.

 

Oh, I thought I needed to put it in the World Streamer Lighting Manager so that I could apply it to all of the scenes to be able to bake the lighting? If that is not the case, what is the Lighting Manager supposed to be used for? Is there another scenario in which you are supposed to utilize it?

 

Light manager is used to setup baked lightmaps, gi.
The thing is that skybox unity take from first scene - game scene.

 

Hello,

I'm experiencing strange culling when attempting to use the "Terrain Culling System" scripts on my terrain gameobjects. No matter which direction i face the camera, the terrains are simply not culled correctly. I'm wondering if I'm missing something? I simply attached the script to the same objects as my terrains and it could be the direction the camera is facing is getting culled (rotating the camera seems to give odd results)

Thanks

 

How many camera objects you got at the scene? Sorry for late replay, I just back from holidays,

 

This reminds me -- I am still not clear on how to handle culling on huge multi-scene setups where the world can get very large.

I know a "master" scene that loads all the other scenes is the current Unity workflow for multi-scene setups, but how does this handle culling for the individual scenes?

Is the scene's OCC ever unloaded with the scene, and does its OCC load upon loading the scene? -- Or is the OCC an ever-present entity?

 

Because OCC is area builded from boxes. Objects only mark if current box (area part) is visible or not. Anyway OCC for openspace is missing idea at all because of size. There are other tricks to do this.

 

Any hint at how you suggest to handle this OCC stuff then? -- I'm not sure I follow what you're suggesting (in terms of workflow to handle OCC for huge open worlds). Maybe some links to articles that show what you suggest might be more helpful?

 

- For small scenes :load all data into game scene for a moment to bake OCC and unload it after this operation
- For big open scenes there is no OCC that can handle such data and there is no reson to do this. Maybe fake solution with occ closed portals manualy placed at the scenes behind big walls etc
When you stream by distance or colliders (for building interiors, caves etc) you dont need occ at all at big scenes.
Simply unity don't handle work well with occ for openscenes. Baked data for big world is missed idea. I would rather seach for something what is realtime calculated with new job system.

I made MMO and also hit such wall with occ long time ago. At the end I didnt use OCC at all even at towns. Only for npc - unity runtime culling based on spheres. You could try such thing on models (we use it at terrains btw)

 

For using objects bounding box center (instead of pivot - which might not be ideal e.g. when adding large Unity terrain tiles, since their pivot is at their corner):
Go to line ~845 inside SceneSplitterEditor.cs, and change from

Code (CSharp):

1. string itemId = GetID (item.transform.position, layer);
2.  
3. GameObject split = null;
4. if (!splits [layer.layerNumber].TryGetValue (itemId, out split)) {
5.     split = new GameObject (layer.prefixScene + itemId);
6.     SceneSplitManager sceneSplitManager = split.AddComponent<SceneSplitManager> ();
7.     sceneSplitManager.sceneName = split.name;
8.  
9.     sceneSplitManager.size = new Vector3 (layer.xSize != 0 ? layer.xSize : 100, layer.ySize != 0 ? layer.ySize : 100, layer.zSize != 0 ? layer.zSize : 100);
10.  
11.     sceneSplitManager.position = GetSplitPosition (item.transform.position, layer);
12.     sceneSplitManager.color = layer.color;
13.  
14.     splits [layer.layerNumber].Add (itemId, split);
15.  
16.     Vector3 splitPosId = GetSplitPositionID (item.transform.position, layer);

to this

Code (CSharp):

1. var itemPosition = item.transform.position;
2. var itemCollider = item.GetComponent<Collider>();
3. if (itemCollider != null)
4. {
5.     var itemCenter = itemCollider.bounds.center;
6.     itemPosition = itemCenter;
7. }
8.  
9. //string itemId = GetID (item.transform.position, layer);
10. string itemId = GetID(itemPosition, layer);
11.  
12. GameObject split = null;
13. if (!splits [layer.layerNumber].TryGetValue (itemId, out split)) {
14.     split = new GameObject (layer.prefixScene + itemId);
15.     SceneSplitManager sceneSplitManager = split.AddComponent<SceneSplitManager> ();
16.     sceneSplitManager.sceneName = split.name;
17.  
18.     sceneSplitManager.size = new Vector3 (layer.xSize != 0 ? layer.xSize : 100, layer.ySize != 0 ? layer.ySize : 100, layer.zSize != 0 ? layer.zSize : 100);
19.  
20.     //sceneSplitManager.position = GetSplitPosition (item.transform.position, layer);
21.     sceneSplitManager.position = GetSplitPosition(itemPosition, layer);
22.     sceneSplitManager.color = layer.color;
23.  
24.     splits [layer.layerNumber].Add (itemId, split);
25.  
26.     //Vector3 splitPosId = GetSplitPositionID (item.transform.position, layer);
27.     Vector3 splitPosId = GetSplitPositionID(itemPosition, layer);

(Replacing any use of item position with item bounding box center where available.
So far this seems to work in my case.)

 

Thanks we will add switch option

 

This is pretty much what I do. no reason to. Keep everything small even for larger worlds... People think big in terms of making a game even for open world...They need to keep in mind to keep everything smaller... due to the amount of data being loaded at one time. something for OCC. So I agree with you here for sure this works really well. I use some of my own solutions for faking and such. which works well.
\


Unity has always been terrible for large worlds and MMO's sadly... I personally use Unreal , Cry or some other engine. .but With World Streamer it make it possible with Unity.. it just takes some experience and being inventive for things such as OCC...

 

Would you mind either elaborating on your solution's use with terrains?

I've never considered not using OCC before, but it does seem like an interesting approach to the visibility problem...

The issue I'm having is trying to figure out the best way to handle distant mountain ridges (while still allowing them to be visible and seamless from nearly anywhere in the world.)

 

Hi. New to World Streamer and cant figure something out. I followed the video and the system works fine but I need to adjust the size of the scene streamer. even at a load range of 1x1 its so big it loads half my world which defeats the purpose. The world is set up to be proportionate to the standard unity FPC prefab? Have I missed something? Need to find a way to bring down the size other than going through and make all my objects bigger. That would take forever.

 

When using the scene splitter, whatever you put in for the two sizes of the terrain streaming layer are what determine how big each individual scene are going to be, not the grid size (aka the 1x1 setting).

 

Use low poly meshes for far distance like we did in video and tutorials?

 

Lol sorry! That sentence was from a previous reply I started but forgot to delete! --

I began to ask this because there would be lots of distant (small) scenes if you're traveling across a wide-open area (e.g. coming from an open, flat desert, looking out at an extremely wide mountain ridge in the distance). Maybe I'm wrong, but I'm not sure ring-streaming would be adequate here. If all scenes are arranged in a grid pattern, this would mean that nearly all of the scenes would need to be loaded, so there would likely need to be a secondary mechanism in place here.

As far as I'm aware, World Streamer's solution to this problem is "distance-streaming" -- is that correct? The only real question I'd have about that is that I wonder if there's a way to mark the loading so that there will be a stand-in mesh version of a large selection of individual scenes? World Streamer needs to be able to tell whether or not you'd be able to see distant terrain chunks (so that the "mesh-only" representation(s) of this selection of scenes can "pop" into existence early and you can hide it at will or when it would be occluded, and as you get closer, the ring-streaming system starts populating those areas with "real" meshes and gameobjects).

These larger terrain masses could probably have multiple "children" scenes that contain the actual gameobjects that are loaded in as normal when the player approaches them, and the "mesh-only" version of the scene could handle the LOD and be loaded/unloaded in groups of these scenes (manually, via a call to WorldStreamer).

Is this a problem you'd be willing to include a solution to in WorldStreamer?

Also, I meant to ask you if you'd be willing to clarify your solution about this Job system method you mentioned (as well as the method you took with your terrains).

 

Hey thanks for the info. I understand how that works I might not have been to clear in my question. I need to find a way to make the grid loading system that follows my player. I have the terrain sizes correct for each terrain section but the grid that follows my player even at a load distance of 1 is to big and loads half my world. Thanks again.

 

Lol think I get it now.

 

A. Use Low poly high Quality Meshes. B. Use Imposters. C. Use a Shader. Like Horizon On.

 

Im not sure if I fully understand but you want to manage ring streaming by view?

Job system will not change anything in VS as its in relation to many processes. It not affect streaming and asset loading what is true bottleneck.

 

Yep, I forgot to mention that they would actually be interactable, meaning the player can walk to these mountain ridges and climb them like in Zelda BotW...

But yes, Imposters are probably the best way to go about this (if not flying too high).


@NatureManufacture:

Speaking of Imposters... could each "big" scene have an optional "Imposter" sub-scene assigned to it that could remain visible at all times (and be unloaded via an extra HUGE ring? -- And maybe be view-dependent?)

 

Eh?? So if you are interacting then they are no longer distance. terrains, objects etc. So you need to treat them as such. I wouldn't be using Imposters for anything you interact with...it would look terrible...Now your LOD's yes. I would. but LOD 0 and possible LOD 1. You would need to use a mesh. or it won't look right. Keep this in mind and experiment.

 

You could put imposters as additonal layer - like low poly terrains have.
You always could modify code it's open so rest it up to users and their goals.

 

That's perfectly fine by me -- Where / what scripts would I look at to modify the code to insert this imposter "layer" behavior?

 

you need to becareful with imposters, I been using them long before Unity had them.. so typically people use them as LOD's. as far as the one up close, just use a mesh. Last, you need to look at how heavy the shader that you are using is.. I have Imposter made by Amplify and been using it extensively... It can look terrible up close. So keep this in mind. I see people screaming about this..I personally would only use Imposers for your LOD's.. and use it as such for an Open World game or I doubt you will be happy.. but go for it. experiment... If your going for realism.....You aren't going to be able to use them for the up close LOD. it will look terrible. I think what Bart means, is you could set an imposer for the up close version. like you can have for LOD's.

What is done, for Low poly terrains is, you make a high poly version of it. (Retopo it) ...Then bake it to a Low poly to keep the detail.. ..Then you have different sets of LOD's for those terrains base on a distance.. So the terrain still will look and perform good.

 

Good afternoon. I have a problem. I decided to combine the mesh in each scene. I managed to combine objects with a size of 500 by 500. I got 81 pieces with combined objects. After combining, steel objects with zero coordinates. after that I decided to create a virtual layer for the scenes. I wrote the prefix and the name of the scenes and clicked create. but in the end, instead of put 81 scenes, I got only 1 virtual scene. how do I make it so that I can normally make a virtual layer and then generate scenes ?! I'm waiting for your help.

 

Objects pivot is in 0 0 0 ? Change center view to pivot.view in scene.

In games with massive enviro devs rather use crosses for trees and LP baked data for buildings etc.

Check SplitScene scripts.

 

I wasn't talking about buildings. I know what is used. not my first game hehe. but those are the three most common techniques in the AAA environment. anyways

But as far as buildings, in larger worlds.. in general NOT distance related
People should be using also modular meshes.. as well as LOD's. and baking the data.. ...I can see this a lot with meshes where people don't optimize them at all.. properly. They have double and triple the amount of tris with out changing its silhouette or changing the over all detail.. Some companies as I stated use shaders.. for terrains, rocks, buildings... They bake fake exteriors.... this is very common. for Distance stuff.





Distance stuff is what I think your talking about, yes, we fake that for buildings all the time. Many times, we use a very high quality bake.... but for terrains, we do the same thing.. trees, yes, in the distance, some may use crosses, I personally don't like it, not a fan of how it looks.
Distance buildings, is easy.... as heck...


in any game I see that uses it, but it can be done properly with low poly meshes.. from a high poly baked version. that is Retopo. I do this for even Android. for many years. I learned this stuff when I worked at EA years back. Works great. Naughty Dog, also uses these techniques.. as well, as shaders. So with proper planning and optimizing people shouldn't have issues. I never have with any game.

 

Looking to use this with a project aimed at switch and VR that has a large, open world setting. Terrain is made with Gaia and roads made using easy roads V3. Are there any known issues that world streamer has with these platforms or assets?

Also, for the terrain streaming feature, is it possible to revert the terrain back to normal once its been split up for streaming? I'd like to be able to quickly iterate my terrain and test with world streaming features.