Question Exclude files from uploading to accelerator

Is there a way to exclude file types (or maybe even files with a size smaller than x) to be uploaded to the accelerator? For very small files it would be faster to have them processed locally (especially if the accelerator is not in the same network / accessed through VPN), while we would like to have textures, cubemaps and meshes on the accelerator. Can this be configured? E.g. exclude all *.asset files with size <=10KB

 

Thanks for the reply! I hope I can accelerate (pun intended) that process by showcasing problems that arise without that support.

I built a test project in which I want to see the impact of the accelerator when using a lot of small scriptable objects: https://github.com/JohannesDeml/OneMillionScriptableObjects
I actually mostly wanted to focus on import times (especially if the accelerator is not used locally, but accessed via vpn), but I was very surprised to notice already a big effect of the accelerator locally when creating 10,000 ScriptableObjects:

• Without Accelerator after saving all ScriptableObjects: 1,306 MB
  
• With Accelerator after saving all ScriptableObjects: 2,895 MB

I really didn't expect that much of an impact, this would be one more reason to exclude small files from the accelerator. I hope I will also find the time to do tests with import times with VPN (the thing I initially feared the most).

 

Wait are you saying that there is no logic whatsoever behind the cache? So if artifact is present in the cache, it's downloaded no matter if the CPU has already done everything and there is nothing to do? So if you happen to be behind a bad network connection, your import times can be longer than if you actually just disabled Accelerator? Surely it can't be like that.

 

@unity_Jonny correct me if I'm wrong, but from my understanding the current logic is, that if an asset does not exist in the library folder (for your current target platform), it will ask the accelerator if the file already exists tehre, and if it does it will download the processed file from the accelerator - No matter how small or quick to process that file would be locally. And I guess it will also then not try to race against the download by trying to import the file locally, but just wait for the accelerator to supply the file.

 

Yeah makes sense if you don't want to accelerate things

I had installed latest Accelerator after upgrading to latest Unity 2021 LTS with hopes of these issues would have been resolved already. Not only download and upload were incredibly slow, but also not all of the imported assets were put in the cache. Then on the second day I was hit by an issue where Unity hanged waiting for some Accelerator download to finish that never did. Only way was to kill Unity editor process.

I can't impose this to our team. Had to remove the cache server setting and just let everybody crunch their own textures as usual.

 

I did another test with importing 100_000 scriptable objects with and without the accelerator. The accelerator instance runs in our network and the client is connected via VPN with 30Mbps download speed. I ran the test on a MacBook Pro 2021 with an M1 chip and 32GB ram. Script for measuring the time

Results:
With Accelerator: 369 seconds
Without Accelerator: 194 seconds

So my initial expectation that excluding files can help very much holds true (even though I did expect event worse results tbh). @unity_Jonny did you do any progress on the topic so far?