Far objects optimization system - idea

I don't know, maybe this is crazy, or maybe this is an idea after all, that somebody could like and try to implement.
This solution would allow to render a scene much farther than we used up to now with a "simple" trick, which I am going to explain below.


Inspiration: impostors
I took inspiration for this from the impostors solution that allow drawing thousands of objects after they have been rendered onto quads and placed as billboards.
I thought "why not do this in general, to everything that is far, and unify it?
Is it possible? Maybe.
How?


The system
First of all, another "skybox" is needed for the system.
The theory of the system is "simple": far objects, in example objects that are farther than 1000 units from the POV, are not rendered in the world scene, they are rendered as part of the texture on the extra skybox.
These far objects are NOT rendered at each frame: they are rendered in turn, because they don't need to be rendered at each frame, as their change in visuals is very subtle, because the POV is moving slowly.

One downside is that also objects that are outside of the camera frustum need to be always rendered, or a quick turn of the POV would show a sudden "jump" of the whole scenario in the far view.

Still, rendering the far objects every X frames instead than at each frame would be a huge advantage.
In example, the rendering of the extra skybox can be done one face at a time, so that the objects are rendered every 6 frames.
More optimized and complex methods of rendering can be developed, like in example to update more often the part of the skybox that is in front of the camera, and less often the part that is outside the frustum of the camera.

So, to recap:
- Far objects are rendered not as part of the frame, but on an extra skybox. The "real" skybox is never rendered, and it is replaced by the skybox "updated" with the far objects rendered on it.
- The rendering of an object does not happen at every frame, but in example every 6 frames (1 face of skybox is rendered at each frame)
- All the far objects in the scene must be rendered on the skybox, to avoid problems when the POV turns suddenly.


Advantages
- Performance for large scenes: thanks to the reduced number of objects to be rendered at each frame in the distance (1/6th), it is possible to add many more object to the scene.
- Impostors are a thing of the past, no more management of thousands of objects, pooling, "popping" of the 3D objects out of the impostors, etc.
- No extra shaders and other problems etc.: all the scene is rendered as usual, the only real mechanism going on is render to texture of the far objects, that's all.
- Gone are the days of crazy LODding systems with grouping etc.
- Gone are the days of the objects popping out of thin air!


Possible problems
- Fast moving POVs: this system is only good for slow moving POVs, like a FPS or even a car racing game, bu it wouldn't hold for spaceships or planes because the horizon objects would change too quickly. Even moving the POV vertically at a speed of 10 meters per second would only produce a difference of maybe 2 or 3 pixels every 6 frames at 2000 units distance, only noticeable in particular clean conditions: during gameplay it wouldn't be noticed, exactly like are barely noticeable the terrains LOD jumps.
- Fog shouldn't be a problem, as when the skybox texture is rendered, it is done with all the scene stuff present, included filters.
- LARGE particle systems: they are animated, and if the animation is fast, the object changes much, they wouldn't really look good.
- Cloud systems with moving clouds? Maybe they could be rendered as near objects to solve the problem.


Is this system nearly ready?
As far as I know, the solution should be almost ready, it's called REFLECTION PROBE.
Just render a reflection probe with far objects, one face per frame, at skybox resolution, and use it as a skybox.
Then, exclude the objects farther than the chosen distance from the scene rendering. Voilà.


Is this stupid?
Now, maybe I'm just crazy or presumptuous, because this system is so simple that it would be strange nobody would have thought of it before.
But you know: Carmack said that the texture streaming system he developed first, also was something that had to exist since years, when he first drafted his solution... so, you never know.

I'm waiting for some wizard out there to implement this idea!
I will accept 10% of the royalties!

 

I believe that is what GTA does, and some the newer TT Lego games use a similar technique.

 

Cool! So why we don't have this in Unity yet?
As I wrote, it should be pretty easy to implement!

 

Because it's a gameplay specific feature that you can implement yourself.

 

I didn't mean it to be part of the Unity base assets, I meant "in Unity" in general, like, in the Asset Store. I wrote in there that I'm waiting for the royalties, thought this was clear.

 

As @Murgilod pointed out, it’s pretty gameplay specific. It’s one thing of those things that isn’t very difficult to build, but each case is going to vary a lot. It would be challenging to create an asset that is both general and optimized.

For instance we did something similar on a game I worked on, but our level progression was linear (almost on rails). So we baked all our distanced model/building imposters rather than generate them at runtime.

 

I don't agree with the statement that it's so gameplay specific and that it would be challenging to create a generic optimized asset.
In example, if you had this asset ready, you would have saved all the work of building the impostors.
And the asset is IN NATURE general, because there's really nothing specific, apart the fact that it's good for game that... may have the player look at a distance, which happens on MANY games today, and probably on 99% of the FPS and adventures out there.
In other words, it's a common problem, and this system would solve it.

 

Why would anybody send you royalties? Just make it yourself.

 

Why so much polemics in this forum?
Don't you think that if I could make it I would have done it already?
I'm just proposing an idea, and the royalty thing is just a joke.
Also, in case you don't know, it's not unusual to sell ideas. It's called "patent".

People, please keep this constructive.
If you have some suggestion or constructive critic, it's more than welcome.
I would love to see somebody who KNOWS technical aspects on the matter (shaders and coding) write what he thinks about this solution.
Other stuff I'm not interested, and it only ruins the thread.

 

It probably would work for very specific camera angles only. Imagine a game where you could climb up a tower, such as assassins creed. You can look quite in the distance, but that's very little you could actually bake in the skybox.

The approach would also still need to render distant objects in certain intervals. It would render a lot of stuff per frame, basically everything to the horizon, even if you update one face per frame only. If you don't LOD and group those objects, I would assume you end up with tens-of-thousands of draw calls and gazillions of triangles per frame. I guess there still needs to be a LOD and grouping system in place for that matter.

I also believe that the pop, when objects get out of the skybox and become 3d again, isn't trivial to hide, just like with LODs.

 

I agree about the fact that a LOD system would still be needed, but I don't agree with the pop out of the skybox: we are talking about stuff that is at 1000 units or more from the POV, you only see a bunch of pixels at that distance.

 

I have done it twice in production, and a couple of different tech prototypes as a proof of concept. The technical part is nothing fancy. Some blitting, some camera rigs, dynamic (or just simple shapes) mesh building for the projection cards. Nothing too fancy shader wise unless. There are effects like blinking lights (one i did was for a hologram style interface so custom shaders for that). But really like 95% of it is planning. Level and camera movement design. Plan it out to optimize and cheat where you can. Find the places where imposters are more expensive than the real thing. Planning where and when transitions happen to lessen pops, and performance hits. Mainly just planning.. that will drive the tech specs.

 

Very much this.

A current project also has optimised distant objects. The approach is different (game specific!), and there's not actually any code specific to this. We planned out how we're going to build the world in a way that supports all of our use cases, and we wrote code for that, and optimised distance rendering is just one of the things achieved from the process as a whole.

 

THat's not new and it's something I had considered too.

But think about the impact, skybox are already optimized and trim down on a per game basis, a cubemap have "consistent" density and when you go above certain screen resolution it mean you need hirez skybox which is impracticable, it mean more bandwidth, fillrate and memory.

Also why update the face you aren't seeing? it's already slow update, so update them only when they come to view. You only see 3 faces at a time, most of the time 2. And generally most of the resolution needed is around teh horizon, you could probably skew the cubemap to sample a bigger angular resolution around the horizon and less at the pole.

It's actually much cheaper to have imposter which can have only local impact on resolution and only occupy a small part of the screen, reducing fillrate. Also pop in tend to happen well into the parallax range, especially for dynamic objects, it won't prevent pop in because you will have a dead zone between the far distance where you bake into the skybox and the distance at which it's practicable to have LOD.

And while we don't use cubemap, it's generally implemented even simpler with a distant rendering on a second camera, generally lower rez, or "matte painting" card that are like small gbuffer. The camera technique mean you only need to render just what's in view, no need for a cubemap cost,. And matte painting do the trick just well. I will also experiment with a technique of light field, but I doubt the angular resolution is good enough, will tell after experiment.

BTW I'm looking for a mathematical equation that tells me the parallax relative to pixel and motion displacement for visual optimization purpose, if anyone know.

 

You basically imagine a triangle (camera frustum). Get the width of the frustum at Z. This is the width that the parallax plane represents. Do the same for the near plane. Get the ratio of near and far plane width. However much you move the near plane, you multiply that by the ratio for the far plane.... I think.

Basically, you just apply the same amount of movement to the near and far plane while imagining the far plane is much larger. But who knows man, I could be way off here. I'm sure you could google this.

 

Yeah I know about the high level stuff, I also know there was a tan somewhere, I also have tried to find the math by hand too, but my brain has been resistant to even simple math lately, I think I have some burn out syndrome. I google it too, Haven't found much (yet?) except this, https://docs.unity3d.com/Manual/FrustumSizeAtDistance.html , which should be trivial to pipe in the data for pixel as it's must be a division of the size of the frustrum vs world object, I can tell you that, but my brain REFUSED to make that connection into proper math, It's crazy, it's driving me crazy, it literally disconnect when I tried.

I'm struggling because a lot of thing I OUGHT to do need math, I used to be good at it, brb gonna cry

 

Isn't this just impostors on a large scale?

That's not a polemic.

https://www.google.com/search?q=polemic

 

It is, and it gives more advantages: first of all you don't have to draw thousands of billboards. Second, you only have 2 pools, near and far objects. Third, you don't have objects popping up in the terrain: all the far scene is rendered together.

 

I don't see any reason why you would have to use thousands of billboards now.

Same here. I don't see any reason why you wouldn't be able to have just two pools with meshes for near objects and entire sections of the scene (eg a city block in a game like GTA V) set up as impostors for far objects. Having a pool of individual impostors would be best for performance though with traditional impostors.

While I could see this being an advantage for a scene where every object is its own impostor I'm not positive it would be an advantage at all if you had entire sections of your scenes set up as their own impostors. I haven't played too much with impostors though.

For the record this is the impostor solution I'm looking at. There is an open source alternative but it wasn't too impressive.

https://assetstore.unity.com/packages/tools/utilities/amplify-impostors-beta-119877

 

From the screenshots it seems very intricate to use or at least to develop.
And if I'm not mistaken, it also takes quite an amount of memory to store the objects images from different POVs.

"I don't see any reason why you would have to use thousands of billboards now."
That's how trees impostors work: they are replaced with billboards facing the camera.
If you have thousands of trees, you have thousands of billboards.
With my system, you have no billboard, it's all comprised in the skybox rendering.

 

That's assuming that you LOD your trees one at a time. That might be a reasonable approach depending on the use case*, but there's also nothing to stop things being LOD'd in clusters.

A nearby mountain could be rendered as a piece of terrain with billboards for each tree. A mid-distance mountain could be rendered as a base mesh with clusters of trees. A distant mountain could be rendered to a single sprite.

* We're talking about a similar performance cost to a particle system.

 

True that. And why not simplify the whole thing by just drawing all on a skybox?
Elasticity I guess, but still, I feel that the solution I suggest has quite some advantages in simplicity both of use and creation.

 

Because in my case I need to be able to render it effectively when flying around at speeds where that solution wouldn't work. We'd end up re-rendering to that skybox so often that it'd probably be more of a cost than a benefit.

Hence the game-specific nature of this that people are talking about.

Our approach is pretty darn simple to use and create so far. And it has to be, because we're a small team making a huge game.

That's something else to think about. Sure, the concept of "put stuff in the skybox and update it automatically" is a simple one. But how do you manage that? The rendering part is easy, but what about the impact along your authoring pipeline to get there?

 

I mentioned that my solution is only good for slow to average speed moving POVs.

Correct me if I am wrong: isn't the camera already sorting through objects with the near and far rendering planes?

 

In additional to this, would objects in the skybox still be able to contribute depth?

A lot of post processing effects rely on depth information to function properly, such as deferred fog to name a simple one. Unity's fog solution even comes with an option to exclude the skybox, would this mean then mean objects rendered to the skybox don't pick up fog anymore?

 

Well, the far objects would anyway be rendered with all the effects, including those using the Z buffer, so I don't think there should be any problem here... but I may mistake.
The skybox wouldn't "contribute" to depth: it would only be an unlit skybox, as the normal skybox, only where far objects are rendered over it instead than using thousands of impostor billboards.

 

The best way is to just start building it, specific to your needs. Try a few different options to find what works best for you. It’s a common technique, though the approaches differ from game to game.