Search Unity

Question Any way to inspect the contents of the accelerator server?

Discussion in 'Unity Accelerator' started by jamie_xr, Jan 9, 2023.

  1. jamie_xr

    jamie_xr

    Joined:
    Feb 28, 2020
    Posts:
    67
    Most of the issues we have around accelerator comes down to "Is that asset cached or not?" Quite often it's not and we have to trigger reimport to get it to upload.

    I'd like to know if there is an api to ping the server to see if the asset exists in cache or not, and if not then we can upload it ourselves ideally. It must exist since unity must be using this to query the cache itself. However it's not documented. Why not open this api up for everyone to use. It would be quite quick to make a debug window showing which assets are in cache and which are not.
     
    jamie_xr, Jan 9, 2023
    #1

  2. cdr9042

    cdr9042

    Joined:
    Apr 22, 2018
    Posts:
    173
    If you don't mind, can you tell me the reason for checking if it's cached or not? I've just started using Unity Accelerator. As I understand it, the process is automatic, if the client does not have the file, then it will request the server to download that file and then store it on the server, then send it to the client.
     
    cdr9042, May 10, 2023
    #2

  3. jamie_xr

    jamie_xr

    Joined:
    Feb 28, 2020
    Posts:
    67
    We have had a lot of issues with corrupted caches and other such things like packages not caching properly. Also it won't automatically upload it. You have to manually reimport an asset to get it to upload to the cache. I may want to upload any uncached assets rather than reimport the whole project. I think it would just be a useful tool for debugging these kinda issues
     
    jamie_xr, May 17, 2023
    #3

  4. unity_Jonny

    unity_Jonny

    Unity Technologies

    Joined:
    Feb 11, 2020
    Posts:
    24
    The unity-accelerator command line tool has a lot of usefull sub commands you can use to interact with a running accelerator instance.
    You can query for cached keys, clear individual keys or whole namespaces and lots more. Run the tool with no args and check the usage instructions
     
    unity_Jonny, May 24, 2023
    #4

  5. jamie_xr

    jamie_xr

    Joined:
    Feb 28, 2020
    Posts:
    67
    I've used this quite a bit just for purging corrupted caches and such.
    Is there a way to get the key of an asset inside unity? It's not as simple as it's asset guid. If so I can make some tools to check by using the cli tool to check individual assets
     
    jamie_xr, May 26, 2023
    #5

  6. cdr9042

    cdr9042

    Joined:
    Apr 22, 2018
    Posts:
    173
    what could possibly cause corrupt cache in the first place? Do you use old hard drive or wrong drive format? I used to use exFAT drive format for Unity projects and it caused a lot of errors so I reformatted the drive to Mac OS Extended and it fixed the errors.
     
    cdr9042, May 29, 2023
    #6

  7. jamie_xr

    jamie_xr

    Joined:
    Feb 28, 2020
    Posts:
    67
    Any import bug can corrupt a cache, are you telling me you've never had to reimport an asset or a whole project or something? It happens quite frequently. In recent versions of unity we have had real issues with upgrading a package version in the package manager. The packages didn't reimport properly meaning the cache didn't get populated. This is a problem on our headless build machines. If they don't import the new package they don't have correct cache unless it's in the cache server. Every time we upgraded a packages we'd need to purge the cache server and manually reimport the assets in question in order to populate it properly
     
    jamie_xr, Jun 2, 2023
    #7

  8. cdr9042

    cdr9042

    Joined:
    Apr 22, 2018
    Posts:
    173
    My coworker setup a Docker and the build process is ran on it through Gitlab CI/CD. Every time a build is initiated, the Docker load an image and then it clones the git repo or load the cached Unity project if it's run before, then fetch from git again to update the project; then it runs the Unity build command. I've never had any cache problems.
     
    cdr9042, Jun 5, 2023
    #8

  9. unity_Jonny

    unity_Jonny

    Unity Technologies

    Joined:
    Feb 11, 2020
    Posts:
    24
    You can find the artifact key uploaded to the cache from the cache's unity-accelerator.log file, then find the file corresponding to that hash in the project /Library/artifacts/ folder.
    You can then run the tool binary2text found on Windows under your Unity install folder /Editor/Data/Tools/binary2text.exe or on MacOS under Unity.app/Contents/Tools/binary2text.exe to see the text representation of the asset, or use the Editor.log file to see what asset resulted in that artifact key.

    You can use the unity-accelerator.exe binary to query if a cache key is present on the cache, and it also has commands to generate the cache key from the artifact to make this easier. As you say, its not possible to reverse the cache key to get the artifact key.
    The tool has a fairly good usage instructions describing all the available sub-commands, the ones you'll want are most likely under the 'cache' or 'tool' sub-command

    Code (Boo):
    1. > ./unity-accelerator cache --help
    2. ./unity-accelerator cache <subcommand> [args]...
    3.  
    4. The cache subcommand contains tools for working with the actual cache contents.
    5.  
    6. Options:
    7.     -? -h --help  Output this help text.
    8.     --all-help    Output this help text and the help text for all subcommands.
    9.  
    10. Subcommands:
    11.     delete            Delete an item from the cache.
    12.     delete-namespace  Delete all items under a namespace from the cache.
    13.     get               Get an item from the cache.
    14.     head              Check the cache for an item's existence.
    15.    info              Send an info request to the cache.
    16.    keys-since        Lists keys in use in the cache, within a namespace, created after a timestamp.
    17.    list-namespaces   Lists namespaces in use in the cache.
    18.    perf              Tools for quick cache performance tests.
    19.    pull              Retrieve cached data from another agent or agents.
    20.    push              Send all cached data to another agent.
    21.    put               Put an item into the cache.
    Code (Boo):
    1. > ./unity-accelerator tool --help
    2. ./unity-accelerator tool <subcommand> [args]...
    3.  
    4. Advanced debugging tools.
    5.  
    6. WARNING: These commands and their arguments may change or be removed in future releases.
    7.  
    8. Options:
    9.     -? -h --help  Output this help text.
    10.     --all-help    Output this help text and the help text for all subcommands.
    11.  
    12. Subcommands:
    13.     bbolt             Debug access to the bbolt.db (cache tracking).
    14.     blob-hash         Calculate the blob hash for content.
    15.     cdp               Post JSON data to CDP endpoint.
    16.     cert              TLS certificate commands.
    17.     delete-multi      Delete multiple items from cache.
    18.     get-multi         Get multiple items from cache.
    19.     head-multi        Check for value existence of multiple items.
    20.     load-test         Load testing coordinator.
    21.     load-test-client  Load testing client.
    22.     put-multi         Put multiple items into cache.
    23.     wrun              Run the agent (called by "unity-accelerator run" usually).
     
    unity_Jonny, Jun 8, 2023
    #9
    jamie_xr likes this.