How a longtime programmer develops in Unity... Me and my unorthodox way

I'd expect the simple code version would end up at least as fast as a standard Unity approach. If not, then if I go ahead and completely replace the collision system I'd expect to be at least a bit faster. The codebase in this example is all highly simplified with no layers of abstraction to run through. So if you think of it from that perspective even a C# program... just pure code... is incredibly fast these days. That's the main thing. When you take it on yourself to write code for everything the only time spent is on your own code and not unknown costs hidden behind blackboxes. Well other than the cost of the rendering, audio playback and any other "must have" stuff in their systems.

These objects are all ultra lightweight. They are GameObjects of course (no way to get around that as far as I know... I did spend a bit of time looking into the possibility of setting up an array of SpriteRenderers alone but stopped on it because in my cursory research it seemed like GameObjects are always needed for Transforms, etc). But not one of these GameObjects has a monobehaviour script attached to it. Only the GameManager gets the callback from the Unity C++ api to it's Update() method. So the overhead of all of those calls every update has been reduced from the ~900 we would see in a typical Unity implementation down to only 1. Instead of a complex animation system being used for such simple animation everything is a very simple bit of code to update the frames and set the appropriate sprite images. And so forth. Add on top of that no costly abstraction through interfaces, events and so forth.

That all being said, I am not sure exactly how many objects his performance test version is using. For all we know he may have 4,000 enemies on-screen. Still I'd be very surprised to ever see a game developed in the typical Unity way using the modern best practice layers of abstraction running as quickly as a game developed in a highly simplified straightforward way for the reasons mentioned above.

 

Maybe the image post effects on @Arowx's version weigh heavy. For a fair comparison such variables need to be eliminated first imho.

Must... not... get... sidetracked... ^^

 

That's a great point! Yes indeed Unity people in particular seem to love visuals and FX and use them often. These things may be more expensive than many people realize. I'd guess he is using some kind of shader here. For a rasterline effect I would just make an image and overlay it on the screen. So it would be one more sprite, 1 more draw call.

 

I would imagine the post would be constant cost and wouldn't increase with the number of sprites. My guess is that it's stuff like all the rigidbodies and the like. There are also like 'oncollisionstay' and stuff which is probably getting fired a half million times.

 

Okay, I thought it seemed like arow's version was also doing collisions between the player and enemy bullets.

Mine wasn't so I just added that.

While I was in there I also increased the player and enemy projectiles.

This one has:

660 enemies (12 rows x 55 per row) 12x original game
144 explosion particles 2x
102 barrier chunks same
64 hit particles 4x
36 player projectiles 12x
16 enemy projectiles 2x (and about 16x in reality due to timing changes)
1 mountain same
1 ground same

For a total of 1024 sprites. A good number. Can't imagine a retro style game ever needing more than this. Certainly shouldn't considering that is massively more than was available.

Anyway the important thing here is now the player and enemy projectiles are also colliding with each other which means more collision checking, more looping to find and deactivate projectiles that have been hit. It also means just hold down spacebar and you basically have a shield. No need to move you can just blow up the barrier directly above rather quickly.

Here is the WebGL Build
And here is the WebPlayer Build

@frosted I didn't add an FPS counter because from my experience in doing so in Unity the numbers fluctuate wildly and in the end the info is rather meaningless.

You will definitely notice any performance issues while playing. Not sure if this will do it but possibly just adding in all of these additional collision checks will. Here it makes absolutely no difference in how the game feels or plays except it is a whole lot more exciting and feel pretty overpowered.

 

Just a quick note:

There are a few things I neglected in my code of the original game. Mainly forgetting to add the reset methods to every manager. This means sometimes when you restart gameplay you may see some projectiles flying or particles fading out. lol

There is always something "more" with this stuff. Not sure I will worry about it updating it now. Just keep in mind if anyone does follow my approach be sure to add reset methods in the object managers and call them from the GameManager when GamePlay state is set. All the Reset methods need to do is loop through the objects in the relevant list and set them to Inactive state.

Alright now I am done playing around with this. Thinking about digging through Atari VCS games online and choosing one of those to build a game around.

 

Oh I should say if anyone happens to actually like this development style and wants to work on some games with me let me know.

I'd prefer it to be one of the regulars on the forum but since most of them are no doubt already busy working on their own grand ambitions I'm willing to consider working with anyone. You just need to be able to accept programming everything in this manner and when I say everything I mean only whatever parts you would be programming. This kind of development approach makes it super easy to split up the programming work. Interested in doing the collectibles? Cool then you write the CollectiblesManager. Interested in bouncy springy things for a platformer game? Cool you do the InteractiveSpringy thing manager responsible for all such objects. And so forth.

Obviously the Vampaders game can be a sort of template. AudioManager, WorldQuery, WorldScanning, etc are easy to drop in any project. Although I think I may go ahead and switch to my own collision checking system that is something I'd write and you'd only need to call methods.

Of course, just a little test game first. Very important to see how we get along, if we work well together, etc. Oh and you really should appreciate all games including (and even dare I say particularly) retro games.

Anyway, just message me on the forums if interested.

 

It's kind of funny, as programmers we're so conditioned to look at stuff like inner loops and be like "omg bad performance!" it's that scary O(n^2) that we all know is the worst possible thing. But the results here kinda speak for themselves.

I honestly wouldn't have suspected @GarBenjamin's version to scale so well.

... Not that you really need performant code for a space invaders. In fact I'd say that this 'load' test is totally irrelevant. But it's still interesting.

 

Well like you said. Mine is certainly not a perfect implementation by any means. There is plenty of room for improvement in both clean-up/condensing the code and also focusing on performance. The thing I like is that with this approach if there is a performance issue well that is my fault or at least more directly under my control. I wrote the code. I can optimize the code.

The reason this kind of approach scales effectively not just with "more" is because of (a) everything is ultra lightweight and (b) everything added is horizontal instead of vertical. Say I added in powerups, it involves only a new manager and list of game objects. What you see in the code is what you get. On the other hand, things like callbacks to every monobehaviour (with an Update method) every frame, some kind of callback (I presume somewhere) for animation every frame and all of the overhead of abstraction developers add firsthand... I think it's easy for folks to lose sight at just how much overhead is really being added.

They think I will just run through this interface and my code becomes very flexible... but they don't think what actually happens in that scenario? They think I will fire off some events and let only the objects that need to receive those get them. And they don't think about what is going on behind the scenes to make that magic happen. I'd say it is every bit as much if not far more overhead than a few hundred or even thousand loop iterations.

Actually, thinking about it... I think one of the Unity people actually wrote a piece about the overhead... hang on...

Here... scroll down to Calling 10,000 Updates section. That is basically a perfect illustration of one of the reasons why I prefer to develop the way I do. Because although this performance hit is known these kind of performance hits are all over from other usage of abstraction. It is just so common these days in development and particularly in C#.

 

630 invaders, each with a bomb that is dropped at random intervals, so probably 20-30 at any one time.
Player's ship with 10 shots.

Thanks for spotting this bug!

Here is a view from the Profiler in Record mode (not deep).



Note how much stress my approach, under heavy load, is putting on the physics system.

I am also getting this Sfx warning : Ran out of virtual channels. Sound will not be played.
Each bomb and bullet plays it's own launch sfx.

Also note the GC Alloc column, using GetComponent<>() is the main cause of this under the Physics 2D row.

 

I really want to thank both of you guys. @Arowx and @GarBenjamin for going above and beyond with these examples.

@Arowx, would you mind trying it without the rigidbodies - or just one attached to the fleet? Also make sure the aliens aren't colliding with each other. Although some of that performance is due to over reliance on the physics system, I think you should really be able to get better numbers than that. And wtf @ 5k allocation per frame?? something funky is happening there.

I think the sound channels is why I would get stutter - I would always blow up a whole row at once before you capped the shot frequency.

 

OK, I've had a play with my version and removing the Rigidbody2D from every Invader and just adding one to the root InvaderWave object, has provided a massive boost in performance (orange bar in Profiler Deep mode).

I have dug into the deep profiling, note my use of GetComponent<>() is causing GC Hits, every frame.

New Version Grouped Rigidbody -> https://dl.dropboxusercontent.com/u/19148487/SpaceInvaders/siPerformanceTest2/index.html

 

20kb GC Alloc?

The fast way this works is that you have one rigidbody and collider per alien, since Box2D can optimise them out then. I'm not sure why there's any allocs really. Why does space invaders need OnCollisionStay?

 

No problem. For me the key thing... biggest take-away so to speak... is that Unity is highly flexible. You can use the normal "Unity Way" ultra minimal on your own development and just let their systems do the bulk of the work, you can go to the other extreme and write code to handle all of these things yourself and use the minimal amount of the built-in systems or you can use any combination in between. I think this is a pretty big deal glad they aren't forcing any one way on people.

@Arowx I never cared for those RigidBodies. Seems like I remember reading that without having RigidBodies on any moving GameObjects you suffer on performance but I've just not seen that firsthand. And it only seems to complicate things from what I can see. They serve a purpose I am sure just not one that has any value I can see. But maybe that is more due to the way I develop. Anyway, that's great you were able to get a massive performance boost from dropping all of those. That actually makes far more sense to me that removing hundreds of components would increase performance than not having those hundreds of components decreasing performance.

I really wish there was an option (definitely make it default to True) "Enable Physics" that a person could turn off. I've always thought it was overkill to have such a thing in a game engine anyway when it all can be programmed on our end easily enough.

 

@Arowx out of curiosity, why don't you kill GetComponent in FixedUpdate? Calling GetComponent every frame is known to be a performance killer. Why not simply cache the references in Awake?

I realise this was never meant to be about performance. But now that you have started down the performance track, might as well do it properly. The built in physics system is unlikely to ever catch up to @GarBenjamin's system. But you can get closer.

 

I've seen similar numbers from profiling in the editor. The GC alloc stats were WAY off compared to the real build. I don't understand why.

https://docs.unity3d.com/Manual/CollidersOverview.html
Scroll down to the matrix. You need the rigidbodies for some messages to even trigger, it seems.

 

I'm not I'm only calling it when a collision occurs e.g.

Code (CSharp):

1. void OnCollisionEnter2D (Collision2D c) {
2.         Destroyer d = c.gameObject.GetComponent<Destroyer>();
3.  
4.         if (d)
5.         {
6.             d.Destruct();
7.          
8.             Destruct();
9.         }
10.     }

But I have updated the Destructible class to obtain it's linked to components on awake, as opposed to OnDestruct().

I can't get overy why Unity is allocating 0.6 kB on nearly every GetComponent<>() call?



If you look at the bottom of the memory profile, the purple line shows GC Allocs, the large spikes are the Side colliders being hit and using GetComponent<>() to find what hit them.

The Smaller spikes bombs, bullets and shields being hit.

 

Yeah I read the docs a lot back when I first tried out Unity. I just never liked the collision event system so that was one of the first things I axed and replaced it with casts. Maybe that is it though. Just need it if using it for collision triggers. Oh yeah I remember that now. I had to mark one side as Kinematic and the other as a trigger. Something like that.

 

Ha ha. You guys are still digging into this performance thing?! I was doodling some graphics for another game. Maybe I should revisit the Vampaders code and do the refactoring to eliminate duplicate methods as well as focus on some optimization too. lol

 

Sometimes it takes a village to write an efficient Space Invaders.

 

That's the point of doing a stress test to find potential areas of optimization.

 

Think of it more along the lines of debugging and profiling the 'standard Unity way' of doing things, or at least my take on the 'Unity way'.

 

Its also worth noting that you are probably beyond the point of sensible optimisation. The game should perform well with the maximum possible number of objects on screen. Going beyond that is often a waste of time.

 

As kind of irrelevant as performance tuning is to a game like space invaders, it's still kind of interesting. Using colliders and stuff really shouldn't be that slow. Looking at these numbers has definitely made me wonder about a few things in my own project.

 

proper use of collision layers will cut down physics expense.

 

In the original version the bombs and bullets were using non kinematic rigidbodies, movement was handled by force application. I could see scaling up the number of bombs/bullets both incurring cost, although, again, it shouldn't be too much.

Also in the original - as @hippocoder mentioned the collision matrix didn't seem to be properly set up.

The Unity-centric approach should not be this slow with under 1000 entities. . . I hope? If it is, I'm a little concerned honestly.

 

Keep in mind this is also for a WebGL game which may well be the slowest build offered by Unity. I am not sure how it compares to mobile. Just that WebGL builds seem pretty slow to me. Surely in a desktop build @Arowx 's Unity-centric approach would be running fast and smooth... ?

I do agree if a desktop build performs similarly something is seriously wrong.

 

Yeah I get that. The thing is with a Unity-centric approach it seems like you could only do just so much. This is actually a great example of what I was saying about relying on these auto-magical systems and ending up fighting against them spending a lot of time just trying to figure out what is going on and if there is anything that can be done about it. But yeah I agree it is good to get it as fast as possible.

 

I've managed to reduce the GC allocs triggered by GetComponent<>() from highs of 20kb to about 3-4 kb, I used SendMessage() to trigger a response in the collider. (No major speed up at least playing in IDE).

Not sure what is still triggering the allocations, assume it's the SendMessage() calls or the OnCollision event.

Also trimmed the Physics2D layer array to a bare minimum. This has drawn down the Physics2D time, but starts to highlight high times in BehaviourUpdate() and Animators.Update (7ms)

I'm surprised the Animators are chewing up 7ms as it's only running sprite based transitions even though it's on 600+ objects.

Major performance fix in physics performance changing the Invader Mover code from Transform.translate to Rigidbody2D.MovePosition()

A bit better Performance Test -> https://dl.dropboxusercontent.com/u/19148487/SpaceInvaders/siPerformanceTest3/index.html

Updated Version -> https://dl.dropboxusercontent.com/u/19148487/SpaceInvaders/index.html

 

Great timing @Arowx I just came here with an update as well.

I got burn-out looking at Atari 2600 game videos so decided to do a light optimization pass on Vampaders and then jack it up a little more...

1,056 enemies (16 rows x 66 per row) 12x original game
144 explosion particles 2x
102 barrier chunks same
64 hit particles 4x
36 player projectiles 12x
32 enemy projectiles 2x (and about 16x in reality due to timing changes)
1 mountain same
1 ground same

For a total of 1,436 sprites. And now I am done despite knowing collision checking may well be a major opportunity for performance improvement. At the last I could do a single call for all collision checks instead of the current 3 separate checks per object. Knocking out 2/3 of the calls might just make a difference. And possibly just replacing the collisions completely with my own overlappingrects checks would be even faster.

BUT... I don't see the point in it really. Knowing that even a WebGL game can have over 1,000 sprites on-screen at the same time with about 50,000 collision checks running per frame... yeah I can't see coming close to that ever. Especially when anything that would have such requirements would be a desktop game not WebGL.

Saying that, I am actually impressed with the WebGL speed. I guess I need to stop saying WebGL performance is so bad. They must have made some major improvements at some point because it seems to perform on par with the WebPlayer build.

Here is the WebGL Build
And here is the WebPlayer Build

@Arowx I'll check out your update now.

 

@Arowx Performance is definitely a little better in your last build. Well done!

And the actual game update itself is really nice. That title and overall display is pretty sweet. Really awesome job on it!

 

Just in case anyone gives a who... the optimizations I made were simply changing from Lists to Arrays to hold the objects and getting the Transform InstanceID and storing in the object containers. Originally I figured that call would just be a straightforward property read but since I have no clue what they are doing or how anything is stored internally I decided to just store that in the object containers and do no GetInstanceID calls at all during update.

Ha ha. It's funny really. With all of the advancements in tech over the years ultimately optimization is the same things today as it was 30 years ago. Precalculating & storing data for fast retrieval. Using ultra simple code the CPU can burn through quickly. Removing work from inside loops. Breaking things down so as little as possible is happening in any one frame. And so on. Exactly the same stuff as I was doing on the C64. Has to be one of the few areas that has stayed the same and likely always will until we get some new paradigm in computing.

 

Well I hope the Vampaders code helps those of you who were interested. Also from @Arowx 's work as well. And I definitely want to say "thank you" to him for jumping in and showing the ultra minimal code Unity Way and then spending time analyzing, optimizing and sharing his strategies and results here.

I think this thread has probably serves its purpose. Of course, if anyone has anything to ask about Vampaders project just message me or post here. Either way works for me.

 

I'll have to duplicate the work I did the other day at the new forum making the source, web game and performance tests easier to find. I'll do it sometime this week and put them into that third post from the top again.

 

Duplicating my previous answers:

IT happens because you're profiling a game from within the editor. While within editor mode, there are extra allocations going on for debug purposes. In case of GetComponent, IIRC, unity allocates some sort of "fake componnent" to report that you have a problem in case you try to access a component that isn't there. Either that or something along those lines.

Because of this, do not trust GC data from editor profiler, instead build a game and attach profiler to it, you'll see very different numbers.

 

IMO, 1000 objects with collision checks between should be a joke for a modern system, unless you hit fillrate/state switching bottleneck while rendering them.

I remember that few years ago out of boredom I wrote a test "billiard" particle system test program that had at least 20000 potentially colliding objects moving at the same time. You can run those kind of massive collision queries using sweep and prune algorithm which utilizes temporal coherence. As long as you don't put all the objects into the state where they ALL overlap each other, you'll have very small number of collision queries happening. In order to have sweep and prune in your game you'll need few sorted lists (one list per coordinate axis), every frame you'll need to re-sort them using adaptive sorting algorithm (insertion sort works well), then you'll need some extra memory to store collision candidates. Complexity is significantly lower than numObjects^2 number of collision queries. The system can also be used to utilize binary-search like queries to find all objects within a given AABB region. Also, you need to know maximum object size.

 

I agree with you. There is no doubt I can probably get about as many sprites moving around colliding in 2016 in Unity as I could many years ago in C# & XNA or a decade to 15 years ago in Blitz. It's just something I've accepted over time. The more powerful computers become the more crap that gets put into everything to basically nullify the hardware progress.

Just Windows itself will be a massive amount of overhead compared to something like DOS. But it also is providing a lot of support for things we don't need to worry about mainly all of the device drivers. The same happens with these modern dev tools. They have so much stuff crammed into them these days there is no way they can be as fast as something more streamlined such as Blitz or even XNA. But again someone might need that stuff at some point so it has to be in there.

We can't see the true power of modern computers because there are always more and more layers of crap between what we are doing and the end hardware. I just develop this way because it is way more logical and faster for me to to develop. The performance benefits are just an added bonus that I was not going after. If I was I could have made many optimizations.

That all being said keep in mind these are tests running only in WebGL and the Webplayer inside a browser. Performance is much greater on desktop. So that may compare much better to the older tests I did years ago. Still the performance tests done in this thread at least illustrate the kind of things I have mentioned before. Where just dropping down and doing most things in code yourself even in C# (without all of the abstraction) seems to run rings around relying heavily on the Unity systems. I guess it is 2 to 4 TIMES faster doing it this way.

 

You can program in very dos-like fashion on windows system using either wrapper like libSDL, directly via something like SetDIBitsToDevice, or by utilizing opengl.

While some overhead is there, you'll have some very dos-like experience with manual raster manipualtion. The layer is thin enough.

I'm going to disagree with that one. Layers of abstraction by themselves are not a bad thing.
Check this out:
http://doc.qt.io/qt-5/qtwidgets-graphicsview-chip-example.html
http://doc.qt.io/qt-4.8/qtdemo.html
^^^ this is a framework done right.

 

I'm out of the house atm and will check that out when I get back because I'm thinking it be something detailed more than I have time for while fiddling on my phone waiting for food.

I have no doubt at all the layers of abstraction add overhead and slow things down. Interfaces, events and so forth are great for business dev and they allow us to use the latest fancy (kind of overcomplicated engineering) patterns. But for game dev I want everything to be direct and instant. Not perform a lookup scanning a table to find the address of a method to call to deliver a message for example. Instead just call a method directly and it is done. Another good example of the performance hit from abstraction is the callbacks to C# Update method from the C++ Unity engine code. This kind of thing is just so common these days so when I started piling more and more of these "best practices" into my own game code it hit me besides making simple things more complex it is also just plain wasteful. Like walking around the house instead of walking straight in the front door. It adds up.

 

The example I linked is a Qt gui application.
It is a single view of 40000 "chips" object which you can rotate and zoom (imagine an integrated circuit schematic).
Each chip has a caption on it, few dozen "legs", and if you zoom all the way out they form a picture, because each one is differently colored. You can click those, IIRC you can move it around.

The thing is the performance is comparable to real-time game. Originally Qt had animated demo application that included all the demos and resembled flash, except it was much faster. You'll have hard time making your own application from scratch that performs better. And it is done via abstraction framework. The framework is a behemoth with at least a gigabyte of C++ code (well, probably less than a gigabyte, because it involves examples). The thing is it is easy to use and highly performant despite that.

The problems with slow downs you mention are results of a bad design. I consider interacting with C# code from C++ code a bad design. I consider unity's event system to be a bad design as well. I consider unity standard shader to be badly designed.

The problem is sometimes people just forget that it is supposed to be about expressive programming languages that perform fast on top of that. The moment where there are bad coders and the moment when there are good coders overly fascinated with a particular approach/technology, we get bloat and slow frameworks. This is the issue, in my opinion, no the idea of abstraction layers in itself. I think people just want to move too fast, and because of this many technologies forever remain unpolished mess. Which is why they create slowdown.

 

A 48 x 48 or 2,304 invader wave version -> https://dl.dropboxusercontent.com/u/19148487/SpaceInvaders/siPerformanceTest4/index.html

Not very performant on my hardware, your mileage may vary.

Note: that the Animation system (it's on each invader and bomb) seems to be the performance bottleneck at least on the profiler in editor mode.

 

If I could survive more than a couple seconds I could give you a good idea of how it works. Since I can't I'm just going to say the moving wave of death is smooth when the bullets of insanity are not raining down on me.

 

I'll have to take a real crack at optimizing Vampaders perf test sometime to see how much I can get it to push around. Maybe after I finish my current game project.

I don't have any performance concerns considering I "rolled back" all the way to the scope of early 80s arcade and 2600 console games. Of course some extra stuff such as particle FX and a bit of animation here and there in backgrounds in some games. I went all the way back to the beginning and will do a few games then move forward a generation, do a few games move forward a generation until I reach the point where I think development is becoming a pain in the ass then probably stick there.

 

Cheers Gar that was fun.

I'll think I will stick with my Unity centric way. But now I know I will have the option to move to a more managed approach when I need to boost performance or take my games across to other games engines.

Although I think with a more complex game, something with 3D and Physics, you would probably end up being more reliant upon Unity's physics engine. For instance a 3D racing game.

 

Yeah it was a fun little project all around. Just getting it out here to begin with since I've mentioned using a different approach many times on these forums and then the perf experiments were interesting too.

Well, I use this approach for everything even the 3D game I was doing until I got burn-out. It's all basically the same stuff in the end. Just some more managers and more stuff in general. Instead of just velocities, I'll have acceleration and just "forces" such as impacts, gravity and so forth that can be applied. Like just an InteractionManager could handle collision forces which would be loaded into additional properties of the object. So instead of just Velocity have an Acceleration and also Force. Of course, things need to be tweaked and such but really so much of the stuff in games doesn't need to be run through a full "proper" physics system. A bouncy bunch of cubes or rolling balls demo would probably be a good thing to knock out for that kind of thing. Maybe I will do that at some point.

 

LOL wow Im not alone.

 

You have to watch this on the new managed batched component system, it's so old school!

 

That is alot to take in but I am glad to see this pop up in a thread for discussion. I am wondering how to go about using this for a current problem in performance. I get files with sometimes 100k objects in them with metadata attached that gets extracted to each individual object. Hence i cannot combineMesh with them or lose the individuality of the metadata. However, out of those 100k objects there are 500 - 2500 unique objects and 50 - 250 unique materials. He did not speak much to rendering, which seems to be the big bottleneck here. Runtime occlusion culling may be one solution..but how would that work..no baking allowed as files are loaded at runtime. And how do you optimize rendering using the Unity Jobs system if you do have alot of duplicates of objects, which seems to be one of the parameters as I could organize these into arrays..but then what??

 

Wow i just read through this whole thread while sitting with my wife watching the World Series (not a baseball fan).

Very cool interaction here!

@GarBenjamin do you test your code or do you consider that unnecessary? Not talking about this example I just mean in general.

It sounds like you do business development by day and are not quite sold on these "best practices" even in your day job. Forgive me if my interpretation of your comments here is incorrect.

Abstraction shouldn't inherently make your code hard to follow, which seems to be a theme here.

I am a c# dev by day and just trying to find my way in game dev, so this was a really interesting discussion for me.

The first measure of quality is working software, so hats off to you guys for 2 working space invader implementations!

Thanks all for your openness.

 

It definitely sounds like you could use multi-threaded code to extract those objects and probably to check for duplication of materials or meshes. You would still probably have to pass them back to the main thread for rendering thought, as only the transform information was directly available to the Job system.(based on that video talk no mention of job driven rendering?*)

The rendering optimisation of duplicate objects could be done with the material instancing system. The example uses GPU driven animation with lots of 'cloned' instanced objects.

Now if you could use the Unity Occlusion system via the job system to dynamically occlude that would be a great speed boost. If not the examples include batch based raycasting so you could write your own occlusion system that would not draw occluded objects.

*Maybe the scriptable rendering pipeline will be able to work with the Jobs system in some way or in the future?