[RELEASED] World Streamer ! Forget about your game memory usage and create big world!

Gi is saved per scene so you have to bake it for each scene or load all scenes by local area updater and then bake it - it should bake it for all scenes at once. Ofc you could make it in parts.

 

Hi:

I have made all the steps indicated in this video:

But I do not see the results. At the end of the first part, there is no type of change. In the aerial view, the whole terrain continues to be rendered, even though it has been divided into 16 parts.

After the second part of the tutorial, the zones are rendered but I do not see any improvement in CPU, GPU or FPS.

I'm with UNITY 2017.3.0f3. I do not know what more information I can give them so they can help me.

Thank you very much.

 

Few screens from your setup etc? From streaming. Maybe you left big terrain (no split) too at the scene.

 

I found my problem !!!! with low performance when the chunks are loaded and unloaded !! after 6 months I did it)))

 

Share this to everyone maybe someone had the same

 

Which problem are you referring to? Glad to hear you solved it, but we would much rather hear what you did to solve it! I know you mentioned some performance spikes in the past, but you didn't mention what the source of the spikes was.

 

@NatureManufacture

I've posted on this topic here in response to issues with "baked" Occlusion Culling using Unity's default recommendation of loading it all in a single "loader" scene and doing the baking (in that scene) of your entire world -- especially when your hardware cannot load that entire world to do the Occlusion Culling baking in the first place.

Since I got no response, I wanted to ask if your system has something in it that handles the "too big to load the entire scene in one go for baking Occlusion Culling", or is there another method your system requires (or you suggest) instead of doing it the "Unity way" for very large/heavy worlds that cannot all be loaded into memory in a single scene?

 

Sorry for late reply-we was at unite

Ok so why you want to bake oclusion culling in big world? Its rather for small scenes with wall that hide whole data behind them.
For big world even if you bake it it will be heavy, really heavy on cpu.

There are few dynamic ways to hide object behind object. You could use occlusion portal to fake occlusion culling and force cull in many places. I don't remember if it need bake probably yes but it need rather occ grid even empty. So you could mark area at empty game scene, then bake grid for occ and then put occlusion portals at scene (not sure if portals must be there during bake). I know its fake solution but I even setup 0.5mln triangles scene on mobile vr with this. Sometimes even occ don't handle rules and you could force culling by portal. Its even dynamic. So you will get occ without huge amount of baked data information and baking.

Anyway big worlds rather use other solutions then baked occ. It took too much space, cpu usage and you have to bake it everytime after changes.

 

Would you mind making an example / tutorial / video showing how to set this up properly?

I'm actually not very familiar with the technique of occlusion portals (or how I might use them with World Streamer). For example, I'd like to know how "occlusion portals" work with something like Zelda: Breath of the Wild's open environments? Particularly when using mesh-based terrain and LODs?

 

Just bake empty occ with occlusion portals. They must have checked that they arr closed. Then put object behind it and it should be culled. I could check how flexoble is it in monday, but i feel that dynamic occ could be usefull beside this that occ for huge scenes is rather mistake in unity.

 

My terrain has 4096 x 4096

Actually, my objective is to reduce the amount of RAM it uses. Now it exceeds 4Gb when it starts and goes up to 4.6Gb

I wish it did not reach 3Gb.

I followed the videos 3,4 to try to achieve it, but without success.

I expose my objectives to know if I'm on the right way with this tool or I should attack the problem in another way.

Thanks a lot!

 

Is this something that can be automated in WorldStreamer?

I have my terrains based on (and snapped to) grid coordinates (each in separate scenes), so perhaps occlusion culling should be part of the streaming process?


Seems like one could automatically generate the occlusion-culling / portal data on a scene-by-scene basis (i.e. let WorldStreamer open each scene, bake Occlusion, then save it someplace for reference, then feed that reference data into WorldStreamer at runtime to manage alongside its other streaming data).

Is this something you'd be willing to add? -- I think Map Magic does something like this with its "pinned" terrains, but it's too specialized for what I'm doing. That's why I bought World Streamer instead.

 

Its hard to say what you made wrong without know the project and process.

But how many scenes should it load around to get proper data?
Such occ will be overlayed many times because you have to load bigger area to get proper occ,

 

I have not asked what I do wrong. I will investigate that. I just ask if following those tutorials I can achieve that objectives. If those tutorials are not appropriate for that purpose, I do not waste any more time trying. That is my question. Thank you.

 

I'm not sure I follow this, but regarding occ, see below the concept of scene LODs:

Just use a more "distant" version of the "ring-streaming" concept.

Each scene loaded should have its own occlusion culling data loaded too (to have occlusion behave as if it were all just one big scene). -- The idea I have in mind is that the scenes to be loaded are not huge themselves (but if they are, they should at least be aligned to a universal grid across the world and loaded/unloaded based on the scene camera location.)


There should definitely be a concept of "LOD scenes" too. These would be scenes that consist of necessary gameobjects only (usually two or three objects, most not even rendered) and super-low-poly versions of scene geometry (with super blurry textures on a big combined / simplified mesh meant to be seen from far away). This will make it so that you can use the existing ring-streaming system to manage very distant scenes in nearly the exact same way you manage which objects are loaded (with entire scenes instead of single gameobjects). These "scene LODs" would be loaded from the outer-ring of the camera's view area. -- Essentially, where object-streaming stops, the new "scene LOD" streaming system would take over (if the user has specified any scenes to become "scene LODs" of course.)

The scene LODs would have a "fade" distance where they would be replaced by the real scenes they consist of, and perhaps occlusion culling for these "distant" scenes should be managed with very large grid boxes. The user can take care of that through baking occ manually (and generate it for World Streamer themselves) for those types of scenes.

 

Terrain heighmap is big but how about its real size? If terrain is big, you could get nice effect.

Seams like big thing. Nothing easy to do in few days. We atm overloaded like hell because of srp changes.

 

SRP is making life hell for a lot of developers right now, so go ahead and do what you must first!

I can definitely wait a bit for this. I mainly need the ability to create streaming worlds w/very distant geometry (like Zelda BotW or Xenoblade Chronicles X). These LOD- and occ- streaming possibilities are necessary for this. Doing occlusion culling on a per-scene basis would finally make this a technically-feasible option for me (and would probably be convenient to implement) in the end using your current ring-streaming system. So thanks for taking the time to hear me out!

 

Hi, I have a very large scene, with 3 large cities in it (no terrain). Currently when they are all on its slow, so I have only one on at a time (and use a train journey to turn on new city and turn off old city, but Id like to run between them. Will this asset help stream parts of a single scene? or is it just for streaming multiple scenes to feel like one? Also does it do anything in editor or just at run time ?

 

Hey,
Can work in oculus go???

 

@NatureManufacture

I have a question about using R.A.M or Cascade with WorldStreamer, what is the best way to go about putting the actual river into a streaming layer? You can roughly see the size of my terrain tiles in the green under the castle. They are 256x256 but the river stretches across I believe 6 tiles total. How do I do about setting up the scene splitter to include the rivers and eventually the lake that it is leading to? I know that you can adjust the overall X andY size in the scenesplitter, so I have to make that the size of the overall size of the river, or is that supposed to stay as the same size as each individual tile?

The same question for when I do make a lake in the area you see there, it will also cover multiple tiles.

Thanks,
-MH

 

That is pretty much what the Ring Streamer is for? That is how I use it. I used Mesh Materializer to create a low poly mesh version of all my unity terrain and then only have Unity terrain directly around the player, then anything past the first grid level is all the LOD mesh terrain.

 

It will help you. Anyway check our video tutorials for full knowledge about it and our manual doc.

Yes.

I would suggets to keep river with terrain layer or big objects
In new version (it's almost done need 1-2 weeks just for release) R.A.M will have direct streaming support so you could cut it into parts in points manualy or automatically by distance.

 

Maybe I am doing something wrong, but I use the same settings for all layers, granted I only have 3 layers currently, as I want to make sure I have everything setup right with terrain and rivers before I start to do other things. I have a terrain layer of 256x256, a LOD terrain layer of 256^ that is a Ring Layer for lower poly mesh terrain, then a layer that is 256^ for the castle. Since the river is not currently cut at all, is there a different way to cut the "big" layer? I thought you just put larger objects into their own layer, but the size of it is still the same size as the terrain cut of 256^ that it is sitting on? I am guessing its probably best to just wait for the update?

I see you said about 2 weeks, but then also 2 weeks of Unity waiting time also? Do you release updates anywhere outside of Unity for users to download and not have to wait for Unity approval time?

Thanks,
-MH

 

Try to desync terrain loading from big object because it will load all data at once every 256 units. Set 200 or 300 etc for castle.

No you will not have to wait for pending review with our updates.

 

Hi unity fans !
I've got a strange issue : in editor everything is working fine, streaming my terrain is also fine. But as soon as I make a build and run it, I only got a black screen, even if I can hear all the sounds of my world.
I tried with and without the loading tool. Any ideas of what's happenning ?

Thanks,
Aurélien

 

In build setings scene with streamers and camera should be on the top

 

OK, I'll check that tonight

 

Allright, it was already done. But I made it work by uncheking the MSAA option on the camera.

 

Ah ok, unsupported image effect

 

I'm trying to eliminate some major stutters when loading/unloading terrain tiles and terrain LOD meshes with ring streaming.

I see huge GFX.WaitForPresent spikes in the profiler, and found this in the WQ manual:

If texture streaming is slow/generates spikes please called "GFX wait for present" adjust "Async Upload Time Slice" and "Async Upload Buffer Size" in Quality settings.

Okay...but any suggestions now to adjust them? Make them bigger or smaller? How much bigger or smaller?

 

Um
a) setup in project about how much async time could unity spent for loading (check unity doc and experiment a bit)
b) unity 2018.2 got mipmap streaming so problem should be gone at all
c) does textures are big?

 

a) Okay, sure....but would you suggest making those values higher or lower? Just as a starting point.
b) Oh, well that's good news! Though I'm going to wait to upgrade until 2018.2.1.
c) The only thing I've set up so far is an 8x8 multi-terrain array, with Unity terrains loading around the player (so 9 loaded at a time), and mesh LOD terrains beyond. The Unity terrains have 1024 resolution height and splatmaps, the mesh terrains have 512 resolution textures (single texture, standard shader). Are those big enough to cause these problems?

I am also seeing big "GC.Mark.Dependencies" spikes in the profiler, but I don't understand how my scene (with 64 objects, mostly simple meshes, and some Unity terrains) would be so egregiously heavy to cause them....

 

So test it on build
At editor you got spikes from gc editor memory allocation etc.

 

@NatureManufacture I've got exactly the same issues with lagging on scene loads. Your videos make things look so smooth and easy but in practice that isn't what's happening.
- could you be more specific re. settings and set-up?
- ideally do more advanced videos (integrating the kinds of tools we all use e.g. Vegetation Studio)

... and please talk through your videos, explaining the process and why some options are better than others. I realise English may not be your first language, or you're shy about your voice but your customers honestly don't mind. We just want as much detailed information as possible. You're obviously super talented and knowledgeable about environment design (I've bought a bunch of your tools and asset packs!) It would be great if you passed on some of that knowledge

 

It's all about it. How to stream assets and don't kill unity pipline. The worst thing is every project have own problems and own resons. Sometimes leaking script somewhere at scene or bad setup in editor.
1st and last thing - never send me any info from editor, only build.... because editor allocate data and stream unprepared unbached data which is batched at runtime ..So such data is usless because it only shows what editor do.

Anyway everyone have everything at manual which should be well known by user before start.

Problem with third party assets integration and movies is that most info is outdated after few months .It's best to ask Lennart

We currently overloaded by SRP changes and store publisher changes and to be honest it killed our pipline for 1-2 months;/

I tried to make speedlevel design 2 days ago but.. just everything explode on store server side and there was big changes and atm im uploading about 6 packs over again...I lost few days because of such situations... I hope I willl record something tommorow or today in the afternoon. Anyway no promisses.

Ah and everything is team work not 1 person We have artists, programmers, designers, pr etc etc. We work on few engines.

 

Thanks for the prompt response @NatureManufacture
I totally understand how SRP changes have been a big burden on all you developers.

And yes all reports of spikes/lag are based on builds. I know that additive async scene loading doesn't work properly in Editor.

I'm on my second read through the manual, I'll keep persevering!

Not expecting any answer soon, but any help or guidance you can give would be very much appreciated

 

All these tests are in a build.

 

Okay, so here's what I've got: I'm testing a Windows build with a scene set up with World Streamer ring streaming, with 64 additive scenes. It's set up to load scenes with Unity terrains around the player (so 9 loaded at a time), and mesh LOD terrains beyond. The Unity terrains have 1024 resolution height and 512-rez splatmaps, the mesh terrains have 1024 resolution textures (single texture, standard shader). There's not much else in the scenes, just a few character models.

I increased the "Async Upload Time Slice" and "Async Upload Buffer Size" from 2 and 4, respectively, to 4 and 8. No idea if that was adjusting them in the right direction or not -- again, any guidance on that would be super helpful. I've also enabled Static batching and Prebake collision meshes and Optimize mesh data, per the manual.

Yet I'm still getting significant slowdowns/stutters when the loading occurs.

On the CPU:
• GC.MarkDependences spikes to 56ms several times.
• TerrainDataAwakeFromLoad spikes to 259ms.

On the GPU:
• Camera.Render spikes to 132ms several times.

Profiler screenshots are attached. Any suggestions?

Oh, and this is on a computer with a 1060 GPU and i5 8600K CPU, with an SSD. So if it's slow on this machine...

thanks
Dave

 

What terrain shader or unity standard?

 

MicroSplat

 

Try with unity standard shader for test.

 

Okay, i did a build with the Unity default terrain shader. CPU and GPU spikes aren't noticeably better,. Any suggestions?

EDIT: And actually, that TerrainData.AwakeFromLoad is still there at 246ms, even without MicroSplat.

 

Terrain is filled by grass etc? How dense how patches of grass look?

Basically this is only stats how engine "async" load terrain data ;/ in your project.

Btw what engine version? Sometimes 1 engine version could be retarded but i woudn't count on this.

I will not watch on gpu until cpu is not fixed, beacuse cpu loads and refresh data for gpu.

Ah please show me also streamer setup.

 

There is grass planted on the terrains, but it's set to zero density/distance on all of them. Does that still affect loading if they don't render?

I'm using 2018.1.7.

Here are the streamer setups -- just these two in the scene so far:


thanks
Dave

 

To be honest? Your terrain setup is nothing special.

Not sure if they affect. Btw try to stream only terrains without background low poly. And try to stream low poly without terrains. We will see what will hapend.

This will show up which layer gives S***ty performance.

 

I also suggest to, run this with out any grass, or vegetation, to see what your getting. I Personally wouldn't be using 2018...right now either...Until its been a bit more mature to many other factors...I have seen to many issues with 2018 + with many assets. and just Unity in general.

So I stick to 2017.4 until I see it mature.

 

Can you show your scene generation Which shows all of your layers, for each streamer, Major, Minor and so forth. I only want to see terrains. for now.

If you find an issue with the LOD as Bart is asking, did you optimize those meshes for the LOD ? how many verts are they?

 

Thanks for hopping in here, @recon0303

I'll try removing all grass and doing a build, and as Bart suggests, doing tests with only mesh terrains, and only unity terrains. That may be helpful to narrow down the problem....but the scene is already pretty basic and stripped down, just two streamers (Unity terrains and LOD meshes) in an 8x8 grid -- I posted the screenshots of those two streamers above, so not sure what else I could show you. The mesh LODs are about 4300 verts/8000 tris. We do plan to optimize those more, but does that seem excessive?

(And why 2018? We needed some features of 2018 (specifically with Mac Metal) so had to upgrade.)

 

Ofc its basic but 1 stupid thing could kill streaming like wrongly distributed foliage or just foliage sometimes. There was always problem with foliage at terrain that in many engine versions it kill streaming.

We stream milions of points in point viewer for scans but stupid unity terrain and foliage could kill streaming just because... :/ of engine version or bad setup.

Recon is veteran hehe with such cases.

 

Hi @gecko, @recon0303 @NatureManufacture and @jeromeWork.

The advice to just stream terrains only is the same advice I received early on and I am grateful for it. It helped me zero in on some not so obvious things that were causing lag during streaming.

I posted in your other thread, but here is where I am currently:

I have LOD0 terrains streaming without lag (and even with trees):

My current problem is garbage collection that only happens when streaming the LOD1 terrain meshes:

My streamer setups:




Any help appreciated, as always.

Best regards,
Tony