Question Accelerator v1.0.130+g5e61869 doesn't DO anything? :)

After some fiddling with configs (despite Unity 2019.3.0f1 insisting "connection works"), I got accelerator to run on a separate Linux machine, on ports that it could actually use. I have been seeing the same behaviour with a local accelerator install unter Windows (on two separate machines, in two separate organizations), though.

I set this up to only accelerate the V2 asset pipeline (no Collaborate shenanigans)

Problem 1: There seems zero discernible acceleration of Unity Imports. There seems only a very low utilization of disk space on the cache server. (sometimes local installs show a bit of usage)

Problem 2: When in Unity I select "Assets->Cache Server->Upload Library to Cache Server", unity gets "busy" (and is doing SOMETHING because when I disconnect it from the net or kill the accelerator process, it sometimes drops out of this stage) - but doesn't respond to input anymore, for hours. There seems no discernible process, and only homeopathic CPU and Network usage.

On the accelerator side, whatever I do, all I see in http://<myserver>:<myport>api/agent-health is: {"version":"v1.0.130+g5e61869","up":"3m22.035853273s","load":0}

Load is always zero. It appears the only actual log entries (which get flushed to disk inconveniently only if I terminate accelerator), are from the connection tests in Unity.

Thanks in advance for your help.

 

12 minutes into one of these "Uploading" sessions, this is what my cachedb looks like:

Code (csharp):

1. <user>@<myserver>:~/.config/unity-accelerator/cachedb$ du -h
2. 8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6/e4
3. 8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6/29
4. 20K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/72f65e84c36782f65e12c9549d2e99d6
5. 12K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/5d
6. 8,0K    ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/4c
7. 12K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24/f8
8. 36K     ./e24b6d24ba7d7918e08be3d9b15d5fd8/c29fd141daab4fcb6fc63ac644936a24
9. 60K     ./e24b6d24ba7d7918e08be3d9b15d5fd8
10. 92K     .
11.  

I must kill unity to ever get it back.

But it DOES connect.

Code (csharp):

1. TCP    192.168.0.190:54410    192.168.0.196:10245    ESTABLISHED

 

Code (csharp):

1. {"level":"info","ts":"2020-02-13T16:13:26+01:00","msg":"tool wrun start","cmd":"/opt/Unity/accelerator/unity-accelerator tool wrun --persist /home/berlin/.config/unity-accelerator","pid":4014}
2. {"level":"info","ts":"2020-02-13T16:13:26.719+0100","msg":"sysinfo","agent_id":"neptune_id","agent_name":"neptune","component":"agent","pid":4014,"agentversion":"v1.0.130+g5e61869","goversion":"go1.13.1","os":"linux","arch":"amd64","compiler":"gc","maxprocs":8,"numcpu":8}
3. {"level":"info","ts":"2020-02-13T16:13:27.129+0100","msg":"handling connections","agent_id":"neptune_id","agent_name":"neptune","component":"pbservice","subprocess_id":7,"url":"agentprotobuf://neptune:10245","addr":"[::]:10245","using_tls":false,"auth_required":false}
4. {"level":"info","ts":"2020-02-13T16:13:27.129+0100","msg":"handling connections","agent_id":"neptune_id","agent_name":"neptune","component":"httpservice","subprocess_id":6,"url":"http://neptune:10000","addr":"[::]:10000"}
5.  

Some logs

 

I found out that the /etc/init.d/unity-accelerator script leaks processes when you start it repeatedly by accident.

Some of the processes even get stuck (need to be killed -9), but not all of them.

This is usually the case when your logs show that it couldn't bind to an address.

 

Thanks for these tips.

The unity-accelerator tool works great locally on the linux server, as well as remotely on one of the machines I've been using Unity Editor on (Windows).

I've tried PUTting and GETting a few files, works as expected.

The perf test seems to work as well (this is on the remote machine, across the network):

Code (CSharp):

1.  
2. 10000 ops of 100% put and 0% get with 131072 bytes each, 25.29s total, 395.39 r/s, 51824403.70 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
3. 10000 ops of 100% put and 0% get with 131072 bytes each, 24.88s total, 401.87 r/s, 52674293.60 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
4. 10000 ops of 0% put and 100% get with 131072 bytes each, 0.83s total, 12061.62 r/s, 1580940900.59 b/s, 0 hits, 10000 misses, 0 put errors, 0 get errors
5. 10000 ops of 0% put and 100% get with 131072 bytes each, 29.74s total, 336.23 r/s, 44069834.71 b/s, 10000 hits, 0 misses, 0 put errors, 0 get errors
6. (still running)

This also results in some load. (does 1 mean 1 as in 1.0? the machine isn't breaking a sweat but I guess this is mostly throughput bottlenecked, I haven't hooked up the 2nd ethernet interface)

Code (CSharp):

1. {"version":"v1.0.130+g5e61869","up":"17m37.079119876s","load":1}

Running the perf test locally does heat up the server

Code (csharp):

1.  
2. 10000 ops of 100% put and 0% get with 131072 bytes each, 1.21s total, 8286.52 r/s, 1086130250.69 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
3. 10000 ops of 100% put and 0% get with 131072 bytes each, 1.37s total, 7307.67 r/s, 957830931.29 b/s, 0 hits, 0 misses, 0 put errors, 0 get errors
4. 10000 ops of 0% put and 100% get with 131072 bytes each, 0.24s total, 42085.17 r/s, 5516187473.76 b/s, 0 hits, 10000 misses, 0 put errors, 0 get errors
5. 10000 ops of 0% put and 100% get with 131072 bytes each, 1.35s total, 7429.80 r/s, 973838190.52 b/s, 10000 hits, 0 misses, 0 put errors, 0 get errors
6. 10000 ops of 25% put and 75% get with 131072 bytes each, 1.42s total, 7062.89 r/s, 925746733.94 b/s, 7500 hits, 0 misses, 0 put errors, 0 get errors
7. 10000 ops of 50% put and 50% get with 131072 bytes each, 1.48s total, 6741.65 r/s, 883641359.84 b/s, 5000 hits, 0 misses, 0 put errors, 0 get errors
8. 10000 ops of 75% put and 25% get with 131072 bytes each, 1.51s total, 6608.37 r/s, 866172077.60 b/s, 2500 hits, 0 misses, 0 put errors, 0 get errors
9.  

Code (csharp):

1.  {"version":"v1.0.130+g5e61869","up":"21m41.849513351s","load":2}

 

Coming straight out of these tests, using Unity 2019.3.0f1, it locks up exactly the same.

I also see this "bug" occasionally in Project settings:




(global settings are of course successfully connecting in Preferences->Editor)

 

Now what's interesting is... my other project (I picked one that's very unrelated to the ones that weren't working) ... doesn't even have the Assets->Asset Server menu. It IS a project from a different organization though.

 

Yes these values came from the tests, the cachedb was practically empty before.

The project IS asset pipeline v2.



Unless there is another concurrent setting?

 

My editor logs are practically empty. (there are two collab-accelerator related lines).

Code (CSharp):

1. [collab-accelerator] discovery started due to a new cloud project binding
2. UPID Received '36a28678-5c67-4da9-b25f-24ff20d2972f'.
3.  
4. .....
5.  
6. [collab-accelerator] Set collab endpoint to https://collab.cloud.unity3d.com (collab-service)
7. <RI> Initialized touch support.
8.  

When I try to connect to a non-existent server, the error shows up in the editor log.

When I connect to my existing (and now tested...) accelerator server, only one thing appears in the logs:

Code (CSharp):

1. Refresh completed in 0.156207 seconds.
2. RefreshInfo: RefreshV2(ForceSynchronousImport)
3. RefreshProfiler: Total: 155.487ms
4.     InvokeBeforeRefreshCallbacks: 0.522ms
5.     ApplyChangesToAssetFolders: 0.096ms
6.     WriteModifiedImportersToTextMetaFiles: 0.000ms
7.     CleanLegacyArtifacts: 0.000ms
8.     Scan: 138.931ms
9.     OnSourceAssetsModified: 0.000ms
10.     UnregisterDeletedAssets: 0.000ms
11.     InitializeImportedAssetsSnapshot: 8.737ms
12.     GetAllGuidsForCategorization: 0.000ms
13.     CategorizeAssets: 0.000ms
14.     ImportAndPostprocessOutOfDateAssets: 0.000ms (0.001ms without children)
15.         ImportManagerImport: 0.000ms (0.000ms without children)
16.             ImportInProcess: 0.000ms
17.             ImportOutOfProcess: 0.000ms
18.             UpdateCategorizedAssets: 0.000ms
19.             RemoteAssetCacheGetArtifact: 0.000ms (0.000ms without children)
20.                 RemoteAssetCacheResolve: 0.000ms
21.                 RemoteAssetCacheDownloadFile: 0.000ms
22.         CompileScripts: 0.000ms
23.         PostProcessAllAssets: 0.000ms
24.         ReloadImportedAssets: 0.000ms
25.         VerifyAssetsAreUpToDateAndCorrect: 0.000ms
26.         EnsureUptoDateAssetsAreRegisteredWithGuidPM: 0.000ms
27.         InitializingProgressBar: 0.000ms
28.         PostProcessAllAssetNotificationsAddChangedAssets: 0.000ms
29.         OnDemandSchedulerStart: 0.000ms
30.         RestoreLoadedAssetsState: 0.000ms
31.     InvokeProjectHasChanged: 0.000ms
32.     UpdateImportedAssetsSnapshot: -0.001ms
33.     ReloadSourceAssets: 0.000ms
34.     UnloadImportedAssets: 0.000ms
35.     Hotreload: 0.112ms
36.     FixTempGuids: 0.003ms
37.     VerifyGuidPMRegistrations: 0.000ms
38.     GatherAllCurrentPrimaryArtifactRevisions: 0.755ms
39.     UnloadStreamsBegin: 0.086ms
40.     LoadedImportedAssetsSnapshotReleaseGCHandles: 1.715ms
41.     GetLoadedSourceAssetsSnapshot: 1.891ms
42.     PersistCurrentRevisions: 0.000ms
43.     UnloadStreamsEnd: 0.029ms
44.     Untracked: 2.610ms
45.  

Unity then freezes (doesn't respond to input, but windows doesn't think the process is unresponsive)

 

Ok, creating a new project (3D with Extras template) in 2019.3.1f1 DOES utilize the server upon import.

 

And now the import of the other project works.

But I still need to find the "Cache Server->Upload All Assets" tool item and kill it. I suspect it's from the previous Cache Server, but I've been grepping the entire project and all packages and haven't found it yet.

 

Code (CSharp):

1. C:\Projects\tintin\tintin-unity\Library\PackageCache\com.unity.scriptablebuildpipeline@1.5.4\Editor\CacheServer\CacheServerUploaderWindow.cs:
2.    52:                     "This will upload all assets in your Library folder to the specified Cache Server.", "Continue", "Cancel"))
3.    81:         [MenuItem("Assets/Cache Server/Upload All Assets")]
4.  

The delinquent Asset Menu Entry is from this Unity package. I don't think this works with Accelerator.

 

My problem appears to be solved, thank you for showing me the "tool" subcommand in Accelerator, I'm already thinking of fun ways to abuse this , err, I mean, troubleshoot in the future.

Summary:
- Scriptable Build Pipeline provides an Asset Upload command that doesn't work.
- Reimport/Deleting Library works acceptably (our project has maybe a 10-15 min import time)
- The earlier Import problems were probably from multiple Accelerator installs / processes occupying the same resources or something.