Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

Question Accelerator v1.0.130+g5e61869 doesn't DO anything? :)

Discussion in 'Unity Accelerator' started by Moritz5thPlanet, Feb 13, 2020.

  1. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    After some fiddling with configs (despite Unity 2019.3.0f1 insisting "connection works"), I got accelerator to run on a separate Linux machine, on ports that it could actually use. I have been seeing the same behaviour with a local accelerator install unter Windows (on two separate machines, in two separate organizations), though.

    I set this up to only accelerate the V2 asset pipeline (no Collaborate shenanigans)

    Problem 1: There seems zero discernible acceleration of Unity Imports. There seems only a very low utilization of disk space on the cache server. (sometimes local installs show a bit of usage)

    Problem 2: When in Unity I select "Assets->Cache Server->Upload Library to Cache Server", unity gets "busy" (and is doing SOMETHING because when I disconnect it from the net or kill the accelerator process, it sometimes drops out of this stage) - but doesn't respond to input anymore, for hours. There seems no discernible process, and only homeopathic CPU and Network usage.

    On the accelerator side, whatever I do, all I see in http://<myserver>:<myport>api/agent-health is: {"version":"v1.0.130+g5e61869","up":"3m22.035853273s","load":0}

    Load is always zero. It appears the only actual log entries (which get flushed to disk inconveniently only if I terminate accelerator), are from the connection tests in Unity.

    Thanks in advance for your help. :)
     
    Last edited: Feb 13, 2020
  2. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    12 minutes into one of these "Uploading" sessions, this is what my cachedb looks like:

    Code (csharp):
    1. <user>@<myserver>:~/.config/unity-accelerator/cachedb$ du -h
    2. 8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6/e4
    3. 8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6/29
    4. 20K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6
    5. 12K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/5d
    6. 8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/4c
    7. 12K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/f8
    8. 36K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24
    9. 60K     ./e24b6d24ba7d7918e08be3d9b15d5fd8
    10. 92K     .
    11.  
    I must kill unity to ever get it back.

    But it DOES connect.

    Code (csharp):
    1. TCP    192.168.0.190:54410    192.168.0.196:10245    ESTABLISHED
     
    Last edited: Feb 13, 2020
  3. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Code (csharp):
    1. {"level":"info","ts":"2020-02-13T16:13:26+01:00","msg":"tool wrun start","cmd":"/opt/Unity/accelerator/unity-accelerator tool wrun --persist /home/berlin/.config/unity-accelerator","pid":4014}
    2. {"level":"info","ts":"2020-02-13T16:13:26.719+0100","msg":"sysinfo","agent_id":"neptune_id","agent_name":"neptune","component":"agent","pid":4014,"agentversion":"v1.0.130+g5e61869","goversion":"go1.13.1","os":"linux","arch":"amd64","compiler":"gc","maxprocs":8,"numcpu":8}
    3. {"level":"info","ts":"2020-02-13T16:13:27.129+0100","msg":"handling connections","agent_id":"neptune_id","agent_name":"neptune","component":"pbservice","subprocess_id":7,"url":"agentprotobuf://neptune:10245","addr":"[::]:10245","using_tls":false,"auth_required":false}
    4. {"level":"info","ts":"2020-02-13T16:13:27.129+0100","msg":"handling connections","agent_id":"neptune_id","agent_name":"neptune","component":"httpservice","subprocess_id":6,"url":"http://neptune:10000","addr":"[::]:10000"}
    5.  
    Some logs
     
  4. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    I found out that the /etc/init.d/unity-accelerator script leaks processes when you start it repeatedly by accident.

    Some of the processes even get stuck (need to be killed -9), but not all of them.

    This is usually the case when your logs show that it couldn't bind to an address.
     
  5. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    There are embedded tools in the accelerator we can use to try to debug.

    You can try to put something into the cache with:
    unity-accelerator tool put neptune:10245 namespace key <somefile>

    Like many Unix tools, no output equals success here.

    Then, you can try to get that item from the cache with:
    unity-accelerator tool get neptune:10245 namespace key

    A longer running test may be done with:
    unity-accelerator tool perf neptune:10245

    If these all run well, perhaps we can narrow the problem down to the project on the Unity Editor side of things.

    You may also find doing an HTTP request to your http://neptune:10000/metrics to be useful.
     
  6. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Thanks for these tips.

    The unity-accelerator tool works great locally on the linux server, as well as remotely on one of the machines I've been using Unity Editor on (Windows).

    I've tried PUTting and GETting a few files, works as expected.

    The perf test seems to work as well (this is on the remote machine, across the network):

    Code (CSharp):
    1.  
    2. 10000 ops of 100% put and 0% get with 131072 bytes each, 25.29s total, 395.39 r/s, 51824403.70 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
    3. 10000 ops of 100% put and 0% get with 131072 bytes each, 24.88s total, 401.87 r/s, 52674293.60 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
    4. 10000 ops of 0% put and 100% get with 131072 bytes each, 0.83s total, 12061.62 r/s, 1580940900.59 b/s, 0 hits, 10000 misses, 0 put errors, 0 get errors
    5. 10000 ops of 0% put and 100% get with 131072 bytes each, 29.74s total, 336.23 r/s, 44069834.71 b/s, 10000 hits, 0 misses, 0 put errors, 0 get errors
    6. (still running)
    This also results in some load. (does 1 mean 1 as in 1.0? the machine isn't breaking a sweat but I guess this is mostly throughput bottlenecked, I haven't hooked up the 2nd ethernet interface)
    Code (CSharp):
    1. {"version":"v1.0.130+g5e61869","up":"17m37.079119876s","load":1}
    Running the perf test locally does heat up the server :)
    Code (csharp):
    1.  
    2. 10000 ops of 100% put and 0% get with 131072 bytes each, 1.21s total, 8286.52 r/s, 1086130250.69 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
    3. 10000 ops of 100% put and 0% get with 131072 bytes each, 1.37s total, 7307.67 r/s, 957830931.29 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
    4. 10000 ops of 0% put and 100% get with 131072 bytes each, 0.24s total, 42085.17 r/s, 5516187473.76 b/s, 0 hits, 10000 misses, 0 put errors, 0 get errors
    5. 10000 ops of 0% put and 100% get with 131072 bytes each, 1.35s total, 7429.80 r/s, 973838190.52 b/s, 10000 hits, 0 misses, 0 put errors, 0 get errors
    6. 10000 ops of 25% put and 75% get with 131072 bytes each, 1.42s total, 7062.89 r/s, 925746733.94 b/s, 7500 hits, 0 misses, 0 put errors, 0 get errors
    7. 10000 ops of 50% put and 50% get with 131072 bytes each, 1.48s total, 6741.65 r/s, 883641359.84 b/s, 5000 hits, 0 misses, 0 put errors, 0 get errors
    8. 10000 ops of 75% put and 25% get with 131072 bytes each, 1.51s total, 6608.37 r/s, 866172077.60 b/s, 2500 hits, 0 misses, 0 put errors, 0 get errors
    9.  
    Code (csharp):
    1.  {"version":"v1.0.130+g5e61869","up":"21m41.849513351s","load":2}
     
    Last edited: Feb 13, 2020
  7. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Coming straight out of these tests, using Unity 2019.3.0f1, it locks up exactly the same.

    I also see this "bug" occasionally in Project settings:

    upload_2020-2-13_17-15-10.png
    upload_2020-2-13_17-14-23.png

    (global settings are of course successfully connecting in Preferences->Editor)
     
  8. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Now what's interesting is... my other project (I picked one that's very unrelated to the ones that weren't working) ... doesn't even have the Assets->Asset Server menu. It IS a project from a different organization though.
     
  9. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    You may need to upgrade your project to use the asset pipeline v2 on the other project you're testing on.

    In the /metrics data you posted, I noticed it says you have uta_agent_cache_bytes_stored 1.333615737e+09 which is about 1.25GB and uta_agent_cache_bytes_in 3.955045843e+09 which is about 3.7GB transfers into the accelerator. But I'm guessing a du -sh of your cachedb doesn't show ~1.25GB? These values are also reflected in the _protobuf_cache_ metrics, which indicates use of the asset pipeline v2.

    My boss here indicated you might check the Editor logs similarly to the thread here https://forum.unity.com/threads/are-we-doing-this-right.826530/#post-5469018 to see if the Editor is actually connecting and using the accelerator.

    You can also turn on debug logging on the accelerator side if you wish, by editing the unity-accelerator.cfg file and flipping the Debug value to true and restarting the accelerator. With debug on, every request will be logged.
     
  10. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Yes these values came from the tests, the cachedb was practically empty before.

    The project IS asset pipeline v2.

    upload_2020-2-13_17-37-34.png

    Unless there is another concurrent setting?
     
  11. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    My editor logs are practically empty. (there are two collab-accelerator related lines).

    Code (CSharp):
    1. [collab-accelerator] discovery started due to a new cloud project binding
    2. UPID Received '36a28678-5c67-4da9-b25f-24ff20d2972f'.
    3.  
    4. .....
    5.  
    6. [collab-accelerator] Set collab endpoint to https://collab.cloud.unity3d.com (collab-service)
    7. <RI> Initialized touch support.
    8.  
    When I try to connect to a non-existent server, the error shows up in the editor log.

    When I connect to my existing (and now tested...) accelerator server, only one thing appears in the logs:


    Code (CSharp):
    1. Refresh completed in 0.156207 seconds.
    2. RefreshInfo: RefreshV2(ForceSynchronousImport)
    3. RefreshProfiler: Total: 155.487ms
    4.     InvokeBeforeRefreshCallbacks: 0.522ms
    5.     ApplyChangesToAssetFolders: 0.096ms
    6.     WriteModifiedImportersToTextMetaFiles: 0.000ms
    7.     CleanLegacyArtifacts: 0.000ms
    8.     Scan: 138.931ms
    9.     OnSourceAssetsModified: 0.000ms
    10.     UnregisterDeletedAssets: 0.000ms
    11.     InitializeImportedAssetsSnapshot: 8.737ms
    12.     GetAllGuidsForCategorization: 0.000ms
    13.     CategorizeAssets: 0.000ms
    14.     ImportAndPostprocessOutOfDateAssets: 0.000ms (0.001ms without children)
    15.         ImportManagerImport: 0.000ms (0.000ms without children)
    16.             ImportInProcess: 0.000ms
    17.             ImportOutOfProcess: 0.000ms
    18.             UpdateCategorizedAssets: 0.000ms
    19.             RemoteAssetCacheGetArtifact: 0.000ms (0.000ms without children)
    20.                 RemoteAssetCacheResolve: 0.000ms
    21.                 RemoteAssetCacheDownloadFile: 0.000ms
    22.         CompileScripts: 0.000ms
    23.         PostProcessAllAssets: 0.000ms
    24.         ReloadImportedAssets: 0.000ms
    25.         VerifyAssetsAreUpToDateAndCorrect: 0.000ms
    26.         EnsureUptoDateAssetsAreRegisteredWithGuidPM: 0.000ms
    27.         InitializingProgressBar: 0.000ms
    28.         PostProcessAllAssetNotificationsAddChangedAssets: 0.000ms
    29.         OnDemandSchedulerStart: 0.000ms
    30.         RestoreLoadedAssetsState: 0.000ms
    31.     InvokeProjectHasChanged: 0.000ms
    32.     UpdateImportedAssetsSnapshot: -0.001ms
    33.     ReloadSourceAssets: 0.000ms
    34.     UnloadImportedAssets: 0.000ms
    35.     Hotreload: 0.112ms
    36.     FixTempGuids: 0.003ms
    37.     VerifyGuidPMRegistrations: 0.000ms
    38.     GatherAllCurrentPrimaryArtifactRevisions: 0.755ms
    39.     UnloadStreamsBegin: 0.086ms
    40.     LoadedImportedAssetsSnapshotReleaseGCHandles: 1.715ms
    41.     GetLoadedSourceAssetsSnapshot: 1.891ms
    42.     PersistCurrentRevisions: 0.000ms
    43.     UnloadStreamsEnd: 0.029ms
    44.     Untracked: 2.610ms
    45.  
    Unity then freezes (doesn't respond to input, but windows doesn't think the process is unresponsive)
     
    Last edited: Feb 13, 2020
  12. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Ok, creating a new project (3D with Extras template) in 2019.3.1f1 DOES utilize the server upon import.
     
  13. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    And now the import of the other project works.

    But I still need to find the "Cache Server->Upload All Assets" tool item and kill it. I suspect it's from the previous Cache Server, but I've been grepping the entire project and all packages and haven't found it yet.
     
  14. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Oh good, I was actually about to type that up -- I'd just tried a fresh project over here too to double-check what you should see in the logs on both ends for that. Menu Assets->Reimport All -- on a new tester project - should definitely hit the cache server right at the end of its reimport, just before the Editor is ready to use again. If you have debug on on the accelerator service you should see a flood of requests, mostly getRequests.

    I will see if I can find someone to help out with your existing project that is having issues -- I think I might be beyond my knowledge levels there. :)
     
  15. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    Code (CSharp):
    1. C:\Projects\tintin\tintin-unity\Library\PackageCache\com.unity.scriptablebuildpipeline@1.5.4\Editor\CacheServer\CacheServerUploaderWindow.cs:
    2.    52:                     "This will upload all assets in your Library folder to the specified Cache Server.", "Continue", "Cancel"))
    3.    81:         [MenuItem("Assets/Cache Server/Upload All Assets")]
    4.  

    The delinquent Asset Menu Entry is from this Unity package. I don't think this works with Accelerator.

    upload_2020-2-13_18-8-15.png
     
  16. Moritz5thPlanet

    Moritz5thPlanet

    Joined:
    Feb 5, 2019
    Posts:
    73
    My problem appears to be solved, thank you for showing me the "tool" subcommand in Accelerator, I'm already thinking of fun ways to abuse this :D, err, I mean, troubleshoot in the future.

    Summary:
    - Scriptable Build Pipeline provides an Asset Upload command that doesn't work.
    - Reimport/Deleting Library works acceptably (our project has maybe a 10-15 min import time)
    - The earlier Import problems were probably from multiple Accelerator installs / processes occupying the same resources or something.
     
    andreiagmu likes this.