How much could the Microsoft Direct Storage API speed up Unity?

Direct Storage aims to take advantage of the bandwidth of modern SSD drives and is aimed at games.

It is designed to reduce CPU load and IO overhead during IO operations. (e.g. Allowing 35,000 IO requests per second on high end NVMe drives vs only hundreds on current file IO). Along with the best compression/decompression technologies to maximise bandwidth utilisation.

Could Unity take advantage of this technology within the game engine as well allowing for super fast loading and building processes?

So what are the heaviest CPU IO operations within Unity?

Could Unity fast load graphical data directly to the GPU to improve loading times?

Tomshardware Article on DirectStorage https://www.tomshardware.com/news/m...i-windows-2021-gaming-nvme-ssds-nivida-rtx-io

Microsofts blog post on Direct Storage https://devblogs.microsoft.com/directx/directstorage-is-coming-to-pc/

 

I know that the WebGL WASM build process can be extremely CPU intensive when in theory it is only compiling so maybe it would be a good candidate for DirectStorage IO boosting with reduced CPU load.


Analysis of WebGL build note the heavy CPU load areas 0,1,2,3 which could be related to heavy file access.

I did a more in depth dive here https://forum.unity.com/threads/i-cannot-believe-how-slow-webgl-builds-are-now.534041/

 

For fun I recorded when I 1) Open unity 2) Open large scene and 3) Build game

1) DIdnt see much activivity
2) Did see both cpu and disk. Disk went up to 80
3) Here I saw spikes on 100% disk usage.

So unity tops my nvme atleast

 

LOL you have a 32 core CPU!

Still it took a lot of time to run the processes and often your CPU cores/NVME were underutilised a lot of the time so there is still room for improvement.

Current IO operations are very serial so your OS has to wait for one to complete before sending another and often Unity loads lots of small files which could in theory be batched if they are all used together.

In addition as IO operations are bandwidth limited and as your system shows CPU power is plentiful on modern systems compression would allow for increased overall performance.

 

16 core 32 threads

Yeah I'm sure it can be optimized further, our game is fairly big so it's not a too shabby time

 

If Unity is already planning Direct Storage support as part of their support for the new X Box, that might make it relatively simple to also add that support to Windows builds (simple porting existing support over, as opposed to adding support for a new API from scratch). Editor support is likely far less trivial though. My speculation.

 

Good point as every build process exe would need to Utilise the DirectStorage API and there are quite a few tools that Unity may not be able to improve e.g. MS Compiler/Linker, WASM build processes and Python scripts (would Python be DirectStorage compatible).

 

DirectStorage API Works Even with PCIe Gen3 NVMe SSDs | TechPowerUp

Should work with any DX12 GPU and Gen3+ NVMe SSDs.

Some slides from the MS presentation on DirectStorage.




If anyone can find any benchmarks that look into how much DirectStorage speeds up load times that would be amazing.

 

So I redid this test on my new hardware Zen 3 vs Zen 2 and also a gen4 nvme vs gen3 nvme, also 2019 vs 2020
Huge difference

 

Is that diagram oversimplified? Because it looks like all DirectStorage does is move GPU asset decompression over to the GPU. That sounds less beneficial than I initially thought as far as its benefits for PC gaming. Typical PC gaming hardware utilization tends towards using the GPU at the highest rate possible, utilizing a small number of CPU cores very high, and leaving remaining CPU cores underutilized.

Moving more of the CPU work over to the GPU doesn't at first glance sound like a performance benefit in that scenario. It might actually reduce performance in games already GPU bottlenecked.

 

Consider this:

• Compressed Image A read from SSD moving over PCIE channels to RAM
• from RAM to CPU to decompress then
• from CPU back to RAM until fully decompressed
• then transferred over PCIE channels to GPU

Versus

• Compressed Image B read for SSD moving over PCIE channels to VRAM
• decompressed by GPU
• then back to VRAM uncompressed

It all depends on where the bottlenecks in the system are CPU bandwidth, GPU bandwidth or PCIE bandwidth. And as you can see from my simple example above going from SSD to GPU more than halves* the PCIE bandwidth used.

* Note the decompression happens on the GPU so only the compressed textures travel once over PCIE.

Also GPUs with thousands of cores are probably better suited to decompressing large image files with a good parallel compression technique.

 

While the GPU will often be the bottleneck during gameplay it's equally often just sitting there idle during a load screen. Additionally DirectStorage isn't the final answer to I/O performance but rather one of the first steps towards improving it. Once DirectStorage is in place we can start looking at advanced compression schemes like Kraken.

 

A bit more in depth on the technology and background to it.

 

Why take this data through system memory surely it would be faster to take it direct to GPU saving 2 PCIE bus trips and reduce the memory footprint of your game/app?

 

Better yet since modern graphics cards can decompress on the fly during lookups we can skip the decompression stage and simply store them on the card.

https://gamedev.stackexchange.com/a/146983

 

So, Direct Storage is now released. When will it be available in Unity?

 

It'll be a half-useable preview package in 1.5 years that never gets updated and is silently discontinued in 3 years.

 

Since there's not been any half-usable or discontinued package in all of unity's lifecycle yet, that's doubtful. Adding it with a package would be nice though.

 

I think it could also be used to speed up the editor on SSD hardware.

 

No / very unlikely. Direct Storage API is to stream and decompress primarily image files directly at GPU level and not involve the CPU. It would have very little if any significance for the editor.

 

That's part of what it does but it also takes advantage of how SSDs work compared to HDDs to allow more data to be loaded faster via parallelism and reduced CPU load.



Note: It looks like the on GPU decompression will be a feature that they will be adding later. (link)

 

but what data are you expecting to load directly to the GPU for editor purposes?

 

current Scene data, UI data.

 

You sure about that? I can name a few. AI Nav package, Unity UI, Hybrid Renderer, Input System, Localization...

I guess they are not technically discontinued...but they just keep updating the release date but no actual change to the version. I guess the standard for "half-usable" and "good enough" varies user to user but some packages are still hard to work with and haven't received any updates for like 3~4 months. Just dead quiet, no updates, no news, no roadmap...just silence. I am literally fighting the devs now to include a fix for the bug I reported (which took a good month for them to "find" the bug) and then another few weeks to "evaluate" if the fix should be in the beta...in the beta! Apparently it is fixed for the next alpha but not the current beta...

I am still waiting for them to even add this to the list of "known issues". They have left it out on 3 releases so far and I am waiting to see how long it will take. Or if it will ever be listed.

And the actual fixing of the bug...let's hope it happens this half...if it happens

 

To get back on topic I don't think this would be a Package it would need to be at the File IO level which is probably low level code in the engine/editor e.g. iostream.h

Also to make a real impact it would have to be implemented throught all IO operations that Unity and it's tool chain uses e.g. compilers/linkers/assemblers/loaders/parsers and all tools within Unity.

 

It's not usable for any of that

 

That's because the package system is new, and there are a few half usable packages.

This is basically how Unity has worked for as long as I've had to deal with it.

 

DirectStorage/HelloDirectStorage.cpp at main · microsoft/DirectStorage · GitHub

It does look more convoluted and complex but for most uses could probable be wrapped into an easy to use function/handle style of file operation.

 

To a degree it's like multithreading. You need a significantly different approach at the implementation to achieve a speedup.

 

Would be a headache indeed for naïve support. I would rather Unity continue on the path of general scene streaming without hiccups because this is something that might benefit overall than supporting xbox and windows with the latest storage hardware.

Because OK scene streaming doesn't load anywhere near as fast, but the user might still never notice it's slower than Direct Storage. So long as they're playing, that's all that matters. And that works on all machines.

Thing is Unity hasn't got there yet (hitch-free scene based streaming) in public.

 

DirectStorage/DStorageLoader.cpp at main · microsoft/DirectStorage · GitHub
They have another example that shows how to use it with a 3D scene note how they wrapped the Direct Storage API into a dedicated class.

 

None of this has anything to do with what was just said.

 

Things like direct storage, mesh shaders, raytracing, dlss, etc are fundamental aspects of games going forward. Unity was able to relatively quickly implement raytracing and dlss, so let's keep going and implement the open aspects like DirectIO, mesh shaders, .... in the next 6 months. But it's also important to not slow down on developing any other unity aspects, like the migration to .net 6, dots, the render pipelines, etc. So if you don't have enough people unity, employ more. I thought you wanted to be a competitor.

 

So basically, just windows dev then?

 

Even saying raytracing is the future is a bit off base because the vast majority of computers out there can't handle it right now and getting hardware that can is still basically a nightmare due to widespread supply chain disruptions that have been happening for years now, which is absolutely going to affect raytracing adoption.

And to say mesh shaders are the future is way off base because right now only one engine really supports them to any real degree and we've yet to see it properly battle tested.

Fundamental this ain't.

 

What the hell is that argument? Fine. Raytracing i can understand. It's not yet used everywhere and still has a little bit to go, but mesh shaders?? Saying that they aren't the future is just ignorance. Just because they have not been implemented yet by most engines (even though they're available since 2018), doesn't mean they are *not* the future. Just looking at 3DMark for an accurate benchmark on mesh shaders immediately shows how superior and down to the metal they are.

Granted, DirectIO is not yet available on linux, but that doesn't mean that there won't ever be something similar, also, gaming still happens mostly on windows devices only. Vulkan has support for raytracing, dlss is available on linux, Vulkan supports mesh shaders....

 

If only it worked that way. Adding more people doesn't magically speed up development.

Meanwhile their implementations of DirectX 12 and Vulkan leave a great deal to be desired.

A DirectStorage-like API has no reason to exist on Linux because the reason it exists has nothing to do with actual limitations in hardware. DirectStorage was created to overcome the limitations of the Windows OS.

 

3DMark isn't a game engine. Benchmark performance is not practical application. Being around since 2018 also means way less than you think because that is a relatively short period of time in this space. I work from graphics whitepapers for things that increase performance and quality dramatically in some cases that are from 2008 that have seen extremely limited implementation in practice.

You are making an argument in favour of tech you think is neat but is not actually used.

 

Fine, don't use it then, but let me use it.

 

I'm not asking them to speed up development, of course adding more people doesn't improve that, i am asking that every one of their people keeps doing what they are working on and employ a new team that works specifically on DirectIO.

 

Then implement it yourself instead of assuming it has to be baseline functionality.

 

I'm not arguing with any of you on why *not* to implement a thing. I don't care if you use it or not. Implementing something is always better than not implementing something. And these features are HUGELY needed. So get off your a***es and get to work. The coming of these features was announced a long time ago and there was plenty of time to reach out to microsoft, etc... Show me some progress!

 

This is absolutely untrue and it is actually baffling that anyone would think this. This is exactly the kind of thing that leads to tech debt and increasing support complexity that keeps existing features from being refined.

No, they aren't hugely needed. That is literally the crux of my argument here. These are neat technologies that function as little more than tech novelty at the moment.

You're acting like these things are never internally evaluated at all, which is not the case. The use case for Direct Storage is extremely limited because it requires either having specific high performance hardware (most people do not have NVMes to spare or even an Xbox Series X) and only work on specific platforms (Microsoft's)

 

This is really not how the entirety of all software development as worked so far, so I think you might need a bit more experience in development to see just why that's a bad idea.

When you load a file you have to sit around massaging that data into a usable form. So Unity has a lot more they can do before this becomes something that would actually accelerate anything.

To me, it looks like a simple feature for Xbox ports of very high end console games that typically would target high end desktop. I would be interested to see which games use it in the coming years. Games have more agility than a game engine does, as they know what they serve and can make breaking changes.

I think DOTS 1.0 will have the ability to transform data quick enough for it to be worth a look. It's a bit soon at the moment and certain things like texture data don't really benefit vs meshes. Can you make more meshes than a blockbuster AAA game, to merit this actually accelerating the game for you?

You would need more mesh data than Unity has the technology to currently effectively render. Think on that.

 

Yes and no. One situation where you wouldn't necessarily need more mesh data is if you wanted to stream the world in at high speeds.

 

We would want the data to be unique as well, since a half-decent cache system will still mean the bottleneck would be somewhere else. Again it's a problem indies have a hard time reaching for this particular feature on Windows.

 

Unity had provided an answer. Ignorant speculation and assumptions (demands) helps no one and this thread is just arguing. Closed as useless.