Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only. On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live. Read our full announcement for more information and let us know if you have any questions.

Question Exclude files from uploading to accelerator

Discussion in 'Unity Accelerator' started by Johannski, Mar 20, 2023.

  1. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    830
    Is there a way to exclude file types (or maybe even files with a size smaller than x) to be uploaded to the accelerator? For very small files it would be faster to have them processed locally (especially if the accelerator is not in the same network / accessed through VPN), while we would like to have textures, cubemaps and meshes on the accelerator. Can this be configured? E.g. exclude all *.asset files with size <=10KB
     
    xavier-ateo and Xarbrough like this.
  2. unity_Jonny

    unity_Jonny

    Unity Technologies

    Joined:
    Feb 11, 2020
    Posts:
    25
    That's not currently possible I'm afraid, but it is something we have considered for a future update
     
  3. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    830
    Thanks for the reply! I hope I can accelerate (pun intended) that process by showcasing problems that arise without that support.

    I built a test project in which I want to see the impact of the accelerator when using a lot of small scriptable objects: https://github.com/JohannesDeml/OneMillionScriptableObjects
    I actually mostly wanted to focus on import times (especially if the accelerator is not used locally, but accessed via vpn), but I was very surprised to notice already a big effect of the accelerator locally when creating 10,000 ScriptableObjects:
    • Without Accelerator after saving all ScriptableObjects: 1,306 MB
    • With Accelerator after saving all ScriptableObjects: 2,895 MB

    I really didn't expect that much of an impact, this would be one more reason to exclude small files from the accelerator. I hope I will also find the time to do tests with import times with VPN (the thing I initially feared the most).
     
  4. SampsaPlaysome

    SampsaPlaysome

    Joined:
    Oct 20, 2019
    Posts:
    34
    Wait are you saying that there is no logic whatsoever behind the cache? So if artifact is present in the cache, it's downloaded no matter if the CPU has already done everything and there is nothing to do? So if you happen to be behind a bad network connection, your import times can be longer than if you actually just disabled Accelerator? Surely it can't be like that.
     
  5. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    830
    @unity_Jonny correct me if I'm wrong, but from my understanding the current logic is, that if an asset does not exist in the library folder (for your current target platform), it will ask the accelerator if the file already exists tehre, and if it does it will download the processed file from the accelerator - No matter how small or quick to process that file would be locally. And I guess it will also then not try to race against the download by trying to import the file locally, but just wait for the accelerator to supply the file.
     
  6. SampsaPlaysome

    SampsaPlaysome

    Joined:
    Oct 20, 2019
    Posts:
    34
    Yeah makes sense if you don't want to accelerate things :)

    I had installed latest Accelerator after upgrading to latest Unity 2021 LTS with hopes of these issues would have been resolved already. Not only download and upload were incredibly slow, but also not all of the imported assets were put in the cache. Then on the second day I was hit by an issue where Unity hanged waiting for some Accelerator download to finish that never did. Only way was to kill Unity editor process.

    I can't impose this to our team. Had to remove the cache server setting and just let everybody crunch their own textures as usual.
     
  7. Johannski

    Johannski

    Joined:
    Jan 25, 2014
    Posts:
    830
    I did another test with importing 100_000 scriptable objects with and without the accelerator. The accelerator instance runs in our network and the client is connected via VPN with 30Mbps download speed. I ran the test on a MacBook Pro 2021 with an M1 chip and 32GB ram. Script for measuring the time

    Results:
    With Accelerator: 369 seconds
    Without Accelerator: 194 seconds

    So my initial expectation that excluding files can help very much holds true (even though I did expect event worse results tbh). @unity_Jonny did you do any progress on the topic so far? :)
     
    xavier-ateo likes this.