Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Question Many error in Accelerator log

Discussion in 'Unity Accelerator' started by bluekylin, Jun 24, 2020.

  1. bluekylin

    bluekylin

    Joined:
    Sep 3, 2012
    Posts:
    1
    Such as:
    {"level":"error","ts":"2020-06-24T11:33:14.440+0800","msg":"not enough found to evict!","agent_id":"ubuntu20_id","agent_name":"ubuntu20","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left
    to evict":3898040}

    What's wrong?
     
    bluekylin, Jun 24, 2020
    #1
    Moonlit-Games-Studio likes this.

  2. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Hi, this usually indicates a disk that is full outside of what the accelerator's cache is using. The accelerator detects the low disk space and tries to evict data in its cache to free up some space, but in this case it has no data to evict.

    By default, the accelerator will try to free disk space when there is less than 1GB free, or less than 5% free. Once triggered, it will try to evict its own cached data until there is 5GB free, or at least 10% free. You can tweak these defaults if you want in the unity-accelerator.cfg file. See the items with names starting with "Cache". The command line tool "unity-accelerator config help" has explanatory text for each config item.

    My guess is that you have the accelerator using the same disk that other programs are using, and that those other programs are using the majority of that disk, and that the disk has less that 5% free or less than 1GB free.
     
    gregoryh_unity, Jun 24, 2020
    #2
    bluekylin and Xmxm like this.

  3. Babybus_Study

    Babybus_Study

    Joined:
    Oct 31, 2016
    Posts:
    14
    I use run unitytechnologies/accelerator:latest on Docker for Mac, same problem.
    disk utilization is under 50%.
    At It goes well at first, after few minutes the cachedb folder just corrupts into 42MB. I tried many times, same problem again and again.
    and I'm using Unity 2019.4.2f1
     
    Babybus_Study, Jul 8, 2020
    #3

  4. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    This seems like a separate issue, but we should be able to track down the problem with more information. Just based on what you posted I tried a quick test here:

    Code (csharp):
    1.  
    2. # mkdir dockertest
    3. # cd dockertest
    4. # mkdir agent
    5. # docker run --rm -ti -v "${PWD}/agent:/agent" unitytechnologies/accelerator:latest
    6. {"level":"info","ts":"2020-07-08T17:13:40+0000","msg":"Setting up configuration: /agent/unity-accelerator.cfg"}
    7. ...
    8. {"level":"info","ts":"2020-07-08T17:13:41.902Z","msg":"handling protobuf connections","agent_id":"d3d27c3476c6_id","agent_name":"d3d27c3476c6","component":"pbservice","subprocess_id":6,"url":"protobuf://172.17.0.2:10080"}
    9.  
    In another terminal:

    Code (csharp):
    1.  
    2. # df -h | grep sda2
    3. /dev/sda2       125G   83G   36G  70% /
    4. # du -sh agent
    5. 48K     agent
    6. # unity-accelerator cache put 172.17.0.2:10080 ns k ~/1gb
    7. # du -sh agent
    8. 954M    agent
    9. # unity-accelerator cache head 172.17.0.2:10080 ns k
    10. 1000000000
    11.  
     
    gregoryh_unity, Jul 8, 2020
    #4

  5. Babybus_Study

    Babybus_Study

    Joined:
    Oct 31, 2016
    Posts:
    14
    I just solve my problem with turning "NoAutoUpdates" to true and delete the unity-accelerator-next file.
     
    Babybus_Study, Jul 9, 2020
    #5

  6. delebru

    delebru

    Joined:
    May 21, 2015
    Posts:
    9
    There seems to be an issue on how the accelerator calculates the available space on certain situations.

    Code (csharp):
    1. # df -h
    2. Filesystem                         Size  Used Avail Use% Mounted on
    3. rpool/data/subvol-114-disk-0        50G  670M   50G   2% /
    4. rpool/data/subvol-114-disk-1       250G  256K  250G   1% /mnt/cache
    5. /dev/mapper/vm--114--disk--0       246G  4.8G  228G   3% /mnt/cache-alt
    In our case, we run the accelerator on a Linux LXC container. If we use /mnt/cache as cache dir, the caching doesn't work and we get an infinity of "not enough found to evict!" error messages. Replace /mnt/cache for /mnt/cache-alt in the config and it all works fine.

    rpool/data/subvol-114-disk-1 is a ZVOL on a ZFS pool
    /dev/mapper/vm--114--disk--0 is a logical volume mounted directly on a single SSD's volume group

    In both situations the configuration is the exact same, and both drives were completely empty when the accelerator was first started.
     
    delebru, Aug 10, 2020
    #6

  7. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    That is quite interesting and I'm not sure what to make of it yet. We will have to do so some testing to find out if the code we have is somehow confused by zfs. As far as I know we're just using the standard OS calls to gather the size, free, used, etc. information.
     
    gregoryh_unity, Aug 10, 2020
    #7

  8. delebru

    delebru

    Joined:
    May 21, 2015
    Posts:
    9
    I'd be happy to run any tests or even set you up with access to our instance. Just let me know
     
    delebru, Aug 10, 2020
    #8

  9. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    289
    I wanted to add that I am experiencing the same issue using docker on my mac. My cache drive is an external ssd that has 60% free (roughly 600gb) formatted Mac OS Extended (Journaled) - not zfs. This issue basically makes accelerator via docker unusable in this case.

    Here is my docker-compose for reference.
    Code (csharp):
    1. version: '3'
    2. services:
    3.   unity_accelerator:
    4.     container_name: accelerator
    5.     image: unitytechnologies/accelerator
    6.     restart: unless-stopped
    7.     environment:
    8.         - DISABLE_USAGE_STATS=TRUE
    9.         - USER=username
    10.         - PASSWORD=password
    11.     volumes:
    12.         - ./data:/agent
    13.     ports:
    14.         - "8081:80"
    15.         - "10080:10080"
     
    Last edited: Aug 12, 2020
    mons00n, Aug 12, 2020
    #9
    delebru likes this.

  10. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    If any are game for it, we could check what the accelerator thinks the disk size is. It provides this info via the /metrics http endpoint. For example, on one of my test machines:

    Code (csharp):
    1.  
    2. $ curl -s http://myhost:myport/metrics | grep ^uta | grep disk
    3. uta_agent_sys_disk_bytes_free 4.2008764416e+10
    4. uta_agent_sys_disk_bytes_total 1.33685121024e+11
    5. uta_agent_sys_disk_bytes_used 8.4841476096e+10
    6. uta_agent_sys_disk_inodes_free 4.471563e+06
    7. uta_agent_sys_disk_inodes_total 8.323072e+06
    8. uta_agent_sys_disk_inodes_used 3.851509e+06
    9.  
    Those values should be coming from whatever disk is associated with the path stored in the unity-accelerator.cfg as the CacheDir value. Perhaps those disk detection routines are failing.
     
    gregoryh_unity, Aug 13, 2020
    #10

  11. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    289
    Here are the results from my server running MacOS 10.15.6. The container data is linked to an external SSD formatted as MacOS Extended (Journaled).
    Code (csharp):
    1. username@mycomputer ~ % curl -s http://127.0.0.1:8081/metrics | grep ^uta | grep disk
    2. uta_agent_sys_disk_bytes_free 1.47661913063424e+14
    3. uta_agent_sys_disk_bytes_total 2.45722406453248e+14
    4. uta_agent_sys_disk_bytes_used 9.8060493389824e+13
    5. uta_agent_sys_disk_inodes_free 4.294425207e+09
    6. uta_agent_sys_disk_inodes_total 4.294967279e+09
    7. uta_agent_sys_disk_inodes_used 542072
    For reference here is df of my system. Accelerator's cache drive is /Volumes/StorageSSD
    Code (csharp):
    1. username@mycomputer ~ % df -h
    2. Filesystem      Size   Used  Avail Capacity iused      ifree %iused  Mounted on
    3. /dev/disk1s5   466Gi   10Gi  408Gi     3%  488275 4882988645    0%   /
    4. devfs          194Ki  194Ki    0Bi   100%     670          0  100%   /dev
    5. /dev/disk1s1   466Gi   44Gi  408Gi    10%  613547 4882863373    0%   /System/Volumes/Data
    6. /dev/disk1s4   466Gi  3.0Gi  408Gi     1%       3 4883476917    0%   /private/var/vm
    7. map auto_home    0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home
    8. /dev/disk2s2   894Gi  357Gi  537Gi    40%  542154 4294425125    0%   /Volumes/StorageSSD
    9. /dev/disk1s3   466Gi  504Mi  408Gi     1%      44 4883476876    0%   /Volumes/Recovery
     
    Last edited: Aug 13, 2020
    mons00n, Aug 13, 2020
    #11

  12. delebru

    delebru

    Joined:
    May 21, 2015
    Posts:
    9
    Code (CSharp):
    1. # df -h
    2. Filesystem                         Size  Used Avail Use% Mounted on
    3. rpool/data/subvol-114-disk-1       250G  256K  250G   1% /mnt/cache
    4. /dev/mapper/nvme-vm--114--disk--0  246G   30G  204G  13% /mnt/cache-alt
    with "CacheDir" set to /mnt/cache-alt with our existing cache (LVM):
    Code (CSharp):
    1. # curl -s http://localhost/metrics | grep ^uta | grep disk
    2. uta_agent_sys_disk_bytes_free 2.18370256896e+11
    3. uta_agent_sys_disk_bytes_total 2.63148998656e+11
    4. uta_agent_sys_disk_bytes_used 3.1340191744e+10
    5. uta_agent_sys_disk_inodes_free 1.5810075e+07
    6. uta_agent_sys_disk_inodes_total 1.6384e+07
    7. uta_agent_sys_disk_inodes_used 573925

    with "CacheDir" set to /mnt/cache (ZFS):
    Code (CSharp):
    1. # curl -s http://localhost/metrics | grep ^uta | grep disk
    2. uta_agent_sys_disk_bytes_free 2.68435193856e+11
    3. uta_agent_sys_disk_bytes_total 2.68435456e+11
    4. uta_agent_sys_disk_bytes_used 262144
    5. uta_agent_sys_disk_inodes_free 5.24287616e+08
    6. uta_agent_sys_disk_inodes_total 5.24287625e+08
    7. uta_agent_sys_disk_inodes_used 9
     
    delebru, Aug 14, 2020
    #12

  13. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Wow, that's something else. It seems to think you have a ~225 TB drive, that you're using ~90 TB, and leaving ~135 TB free. So at least it's consistent within itself, but a 225 TB drive? Must be nice. ;-)

    I'll do some research and see if I can figure out how it got so confused.

    However, those values should be fine with respect to how eviction works. It definitely thinks you have enough free space, by those metrics, both by raw byte count and by percentage. So it never should have started an eviction pass.

    The default cache settings for the accelerator are:

    Code (CSharp):
    1. $ grep Cache unity-accelerator.cfg
    2.     "CacheDir": "/home/gregoryh/fresh",
    3.     "CacheDirSplit": 2,
    4.     "CacheMonitorInterval": "1m0s",
    5.     "CacheMinFreeBytes": 1073741824,
    6.     "CacheMinFreePercent": 5,
    7.     "CacheEvictUntilFreeBytes": 5368709120,
    8.     "CacheEvictUntilFreePercent": 10,
    9.     "CacheMaxUsedBytes": -1,
    10.     "CacheMaxUsedPercent": -1,
    11.     "CacheEvictUntilUsedBytes": -1,
    12.     "CacheEvictUntilUsedPercent": -1,
    13.     "CacheMaxPending": 100,
    14.     "CacheNoAccessTimes": false,
    15.  
    The CacheMinFreeBytes and Percent show when the evictor will be triggered: less than 1 GB free or less than 5%. Then it says it'll keep trying to evict until it has at least 5 GB free and 10% free. And, even with the weird numbers being reported via the metrics, all those parameters seem like they'd be fine.

    However... if it did start an eviction pass for whatever reason, I can see why it'd never stop evicting. The code would calculate the number of bytes to evict, which would be huge due to the odd metrics reporting, and so it'd never be able to finish.

    I will do more research to try to fix the metrics reporting. Also, I'll see if we can tweak the evictor with some additional sanity checks.

    If you want, you can turn Debug to true in the unity-accelerator.cfg and that will log more information that could help. For example, it should log a "starting eviction pass" log line that should have the reasons for the pass starting.

    You could also set CacheMinFreePercent to 0 and CacheEvictUntilFreePercent to 0 as well, which should disable the percentage based calculations of the evictor, leaving it with just raw byte counts to worry about.
     
    gregoryh_unity, Aug 14, 2020
    #13

  14. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Interestingly, your numbers seem right inline with what they should be. Perhaps you both are affected by the same underlying issue and the metrics are just a secondary issue that also needs fixing. Back to researching...

    Oh, but you also might tweak the config settings as I mentioned just above and see if it helps. If you turn on debug, that may help as well, at least it may help us narrow down the situation.
     
    gregoryh_unity, Aug 14, 2020
    #14

  15. delebru

    delebru

    Joined:
    May 21, 2015
    Posts:
    9
    I've gathered 3 logs, all with debug enabled and containing only the information from the service start. I was also only re-importing the "Skybox_Mat" from the 3D with extras sample project.

    - cache dir on LVM folder: https://www.dropbox.com/s/7ffwudbn3ta7o5h/unity-accelerator-debug-alt.log?dl=0
    - cache dir on ZFS folder: https://www.dropbox.com/s/30bijw53s99psnp/unity-accelerator-debug.log?dl=0
    - cache dir on ZFS folder and suggested eviction settings: https://www.dropbox.com/s/5pgftyjrodo6n72/unity-accelerator-debug-0.log?dl=0

    Hope it helps! If you want any further info just let me know :)
     
    delebru, Aug 14, 2020
    #15
    gregoryh_unity likes this.

  16. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    This was great info. It is logging that it is running unscheduled eviction passes constantly. I double-checked the code and it only does that when it has difficulty opening a new file and setting that new file's size to the expected upload size. The evictor is asked for 41,680 bytes, so it's not like that value is excessively high or anything. Unfortunately there is a bug where the underlying error isn't being logged, even under debug logging (I will fix that for a future version release) so unfortunately I couldn't tell you what it thinks is wrong at the moment.

    I just edited this post. I had suggested double-checking permissions on the cache drive, but now that I've read the code more closely, this condition would happen after creating a new file, but being unable to preallocate space for its eventual full size. I'm not sure yet what the error would be, so we'll just have to wait for more info with the improved logging.
     
    Last edited: Aug 17, 2020
    gregoryh_unity, Aug 17, 2020
    #16

  17. delebru

    delebru

    Joined:
    May 21, 2015
    Posts:
    9
    Thanks for looking into this, glad the logs helped! Please let me know when the update is available so I can run it again and get the error logs. After hitting this issue the first time I've changed the service owner to root so it shouldn't be a lack of write permissions; though, being an LXC container (privileged), I wouldn't be surprised it's lacking some extra access to preallocate the required space on certain mount types.

    Looking forward to having a solution for this so we can move our cache to a redundant storage
     
    delebru, Aug 18, 2020
    #17

  18. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    289
    sorry for the late reply @gregoryh_unity but here is an accelerator log from my machine with Debug=true trying to import a single mesh. I thought it might be some weird external drive permission issue with MacOS Catalina but even putting the cache on the local OS drive I still got this issue.

    Code (CSharp):
    1. {"level":"info","ts":"2020-08-18T20:23:14+0000","msg":"Using existing configuration: /agent/unity-accelerator.cfg"}
    2. {"level":"info","ts":"2020-08-18T20:23:14+0000","msg":"Changed set password."}
    3. {"level":"info","ts":"2020-08-18T20:23:14+0000","msg":"Executing: unity-accelerator run"}
    4. {"level":"info","ts":"2020-08-18T20:23:14Z","msg":"run start","agentversion":"v1.0.524+g96c5e18","arch":"amd64","cmd":"unity-accelerator run","compiler":"gc","goversion":"go1.14.4","maxopenfiles":1048576,"maxprocs":4,"numcpu":4,"os":"linux","pid":1,"process":"Unity Accelerator service monitor"}
    5. {"level":"info","ts":"2020-08-18T20:23:15Z","msg":"tool wrun start","cmd":"/agent/bin/unity-accelerator tool wrun","pid":66}
    6. {"level":"debug","ts":"2020-08-18T20:23:15.245Z","msg":"http ports to try","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","ports":[80,80,8080,0]}
    7. {"level":"debug","ts":"2020-08-18T20:23:15.245Z","msg":"protobuf ports to try","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","ports":[10080,10080,18080,0]}
    8. {"level":"info","ts":"2020-08-18T20:23:15.279Z","msg":"DASHBOARD AVAILABLE AT http://172.22.0.2/dashboard/","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49"}
    9. {"level":"info","ts":"2020-08-18T20:23:15.360Z","msg":"sysinfo","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"agent","pid":66,"agentversion":"v1.0.569+g2096e44","goversion":"go1.14.4","os":"linux","arch":"amd64","compiler":"gc","maxprocs":4,"numcpu":4}
    10. {"level":"info","ts":"2020-08-18T20:23:15.362Z","msg":"handling protobuf connections","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"url":"protobuf://172.22.0.2:10080"}
    11. {"level":"info","ts":"2020-08-18T20:23:15.363Z","msg":"handling http connections","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"httpservice","subprocess_id":7,"url":"http://172.22.0.2"}
    12. {"level":"debug","ts":"2020-08-18T20:23:17.601Z","msg":"new connection","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"conn_id":1,"remote":"172.22.0.1:33270"}
    13. {"level":"debug","ts":"2020-08-18T20:23:17.601Z","msg":"new connection","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"conn_id":2,"remote":"172.22.0.1:33272"}
    14. {"level":"debug","ts":"2020-08-18T20:24:01.892Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    15. {"level":"error","ts":"2020-08-18T20:24:01.893Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    16. {"level":"debug","ts":"2020-08-18T20:24:01.893Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    17. {"level":"debug","ts":"2020-08-18T20:24:01.996Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    18. {"level":"error","ts":"2020-08-18T20:24:01.996Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    19. {"level":"debug","ts":"2020-08-18T20:24:01.996Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    20. {"level":"debug","ts":"2020-08-18T20:24:02.100Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    21. {"level":"error","ts":"2020-08-18T20:24:02.101Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    22. {"level":"debug","ts":"2020-08-18T20:24:02.101Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    23. {"level":"debug","ts":"2020-08-18T20:24:02.204Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    24. {"level":"error","ts":"2020-08-18T20:24:02.205Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    25. {"level":"debug","ts":"2020-08-18T20:24:02.205Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    26. {"level":"debug","ts":"2020-08-18T20:24:02.310Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    27. {"level":"error","ts":"2020-08-18T20:24:02.310Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    28. {"level":"debug","ts":"2020-08-18T20:24:02.310Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    29. {"level":"debug","ts":"2020-08-18T20:24:02.414Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    30. {"level":"error","ts":"2020-08-18T20:24:02.415Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    31. {"level":"debug","ts":"2020-08-18T20:24:02.415Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    32. {"level":"debug","ts":"2020-08-18T20:24:02.523Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    33. {"level":"error","ts":"2020-08-18T20:24:02.523Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    34. {"level":"debug","ts":"2020-08-18T20:24:02.523Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    35. {"level":"debug","ts":"2020-08-18T20:24:02.628Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    36. {"level":"error","ts":"2020-08-18T20:24:02.629Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    37. {"level":"debug","ts":"2020-08-18T20:24:02.629Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    38. {"level":"debug","ts":"2020-08-18T20:24:02.733Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    39. {"level":"error","ts":"2020-08-18T20:24:02.733Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    40. {"level":"debug","ts":"2020-08-18T20:24:02.733Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    41. {"level":"debug","ts":"2020-08-18T20:24:02.838Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":651680}
    42. {"level":"error","ts":"2020-08-18T20:24:02.838Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":651680}
    43. {"level":"debug","ts":"2020-08-18T20:24:02.838Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":651680,"actual bytes evicted":0,"files evicted":0}
    44. {"level":"debug","ts":"2020-08-18T20:24:02.956Z","msg":"handled putRequest","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"conn_id":2,"remote":"172.22.0.1:33272","segment_id":10,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"1f1c5f61f16b10f7afaa744eecd4b6cd","size":65168,"status":"error","status_code":1,"error":"cache.create error: operation not supported"}
    45. {"level":"debug","ts":"2020-08-18T20:24:02.964Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    46. {"level":"error","ts":"2020-08-18T20:24:02.965Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    47. {"level":"debug","ts":"2020-08-18T20:24:02.965Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    48. {"level":"debug","ts":"2020-08-18T20:24:03.072Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    49. {"level":"error","ts":"2020-08-18T20:24:03.073Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    50. {"level":"debug","ts":"2020-08-18T20:24:03.073Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    51. {"level":"debug","ts":"2020-08-18T20:24:03.177Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    52. {"level":"error","ts":"2020-08-18T20:24:03.178Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    53. {"level":"debug","ts":"2020-08-18T20:24:03.178Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    54. {"level":"debug","ts":"2020-08-18T20:24:03.287Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    55. {"level":"error","ts":"2020-08-18T20:24:03.288Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    56. {"level":"debug","ts":"2020-08-18T20:24:03.288Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    57. {"level":"debug","ts":"2020-08-18T20:24:03.392Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    58. {"level":"error","ts":"2020-08-18T20:24:03.393Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    59. {"level":"debug","ts":"2020-08-18T20:24:03.393Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    60. {"level":"debug","ts":"2020-08-18T20:24:03.497Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    61. {"level":"error","ts":"2020-08-18T20:24:03.498Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    62. {"level":"debug","ts":"2020-08-18T20:24:03.498Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    63. {"level":"debug","ts":"2020-08-18T20:24:03.603Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    64. {"level":"error","ts":"2020-08-18T20:24:03.604Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    65. {"level":"debug","ts":"2020-08-18T20:24:03.604Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    66. {"level":"debug","ts":"2020-08-18T20:24:03.708Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    67. {"level":"error","ts":"2020-08-18T20:24:03.708Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    68. {"level":"debug","ts":"2020-08-18T20:24:03.708Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    69. {"level":"debug","ts":"2020-08-18T20:24:03.814Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    70. {"level":"error","ts":"2020-08-18T20:24:03.815Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    71. {"level":"debug","ts":"2020-08-18T20:24:03.815Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    72. {"level":"debug","ts":"2020-08-18T20:24:03.925Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":150340}
    73. {"level":"error","ts":"2020-08-18T20:24:03.926Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":150340}
    74. {"level":"debug","ts":"2020-08-18T20:24:03.926Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":150340,"actual bytes evicted":0,"files evicted":0}
    75. {"level":"debug","ts":"2020-08-18T20:24:04.028Z","msg":"handled putRequest","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"conn_id":2,"remote":"172.22.0.1:33272","segment_id":12,"namespace":"78b2fdc254b0a4f8a4693a81bb96598a","key":"844f4cf5fece3b3d76d40fea7282e604","size":15034,"status":"error","status_code":1,"error":"cache.create error: operation not supported"}
    76. {"level":"debug","ts":"2020-08-18T20:24:04.050Z","msg":"handled getRequest","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"conn_id":2,"remote":"172.22.0.1:33272","segment_id":13,"namespace":"81e94844d19a16919208533e08183531","key":"c490e58784ecb60677414c05d9e8ce3b","status":"key not found","status_code":3,"size":0}
    77. {"level":"debug","ts":"2020-08-18T20:24:04.059Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    78. {"level":"error","ts":"2020-08-18T20:24:04.059Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    79. {"level":"debug","ts":"2020-08-18T20:24:04.059Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    80. {"level":"debug","ts":"2020-08-18T20:24:04.165Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    81. {"level":"error","ts":"2020-08-18T20:24:04.166Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    82. {"level":"debug","ts":"2020-08-18T20:24:04.166Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    83. {"level":"debug","ts":"2020-08-18T20:24:04.271Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    84. {"level":"error","ts":"2020-08-18T20:24:04.272Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    85. {"level":"debug","ts":"2020-08-18T20:24:04.272Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    86. {"level":"debug","ts":"2020-08-18T20:24:04.376Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    87. {"level":"error","ts":"2020-08-18T20:24:04.376Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    88. {"level":"debug","ts":"2020-08-18T20:24:04.376Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    89. {"level":"debug","ts":"2020-08-18T20:24:04.483Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    90. {"level":"error","ts":"2020-08-18T20:24:04.483Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    91. {"level":"debug","ts":"2020-08-18T20:24:04.483Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    92. {"level":"debug","ts":"2020-08-18T20:24:04.592Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    93. {"level":"error","ts":"2020-08-18T20:24:04.592Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    94. {"level":"debug","ts":"2020-08-18T20:24:04.592Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    95. {"level":"debug","ts":"2020-08-18T20:24:04.696Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    96. {"level":"error","ts":"2020-08-18T20:24:04.697Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    97. {"level":"debug","ts":"2020-08-18T20:24:04.697Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    98. {"level":"debug","ts":"2020-08-18T20:24:04.803Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    99. {"level":"error","ts":"2020-08-18T20:24:04.804Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    100. {"level":"debug","ts":"2020-08-18T20:24:04.804Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    101. {"level":"debug","ts":"2020-08-18T20:24:04.910Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    102. {"level":"error","ts":"2020-08-18T20:24:04.910Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    103. {"level":"debug","ts":"2020-08-18T20:24:04.910Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    104. {"level":"debug","ts":"2020-08-18T20:24:05.015Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes to evict":8240}
    105. {"level":"error","ts":"2020-08-18T20:24:05.015Z","msg":"not enough found to evict!","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":8240}
    106. {"level":"debug","ts":"2020-08-18T20:24:05.016Z","msg":"eviction pass complete","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"cache","subprocess_id":2,"desired bytes to evict":8240,"actual bytes evicted":0,"files evicted":0}
    107. {"level":"debug","ts":"2020-08-18T20:24:05.116Z","msg":"handled putRequest","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","component":"pbservice","subprocess_id":5,"conn_id":2,"remote":"172.22.0.1:33272","segment_id":14,"namespace":"81e94844d19a16919208533e08183531","key":"c490e58784ecb60677414c05d9e8ce3b","size":824,"status":"error","status_code":1,"error":"cache.create error: operation not supported"}
    108. {"level":"debug","ts":"2020-08-18T20:24:13.470Z","msg":"meta out request","agent_id":"f95e63033e49_id","agent_name":"f95e63033e49","host":"accelerator.cloud.unity3d.com","scheme":"https","method":"GET","target":"/api/v1/accelerator/jwt_public_key","remote":"","status":200,"elapsed_header_ns":115363574,"elapsed_ns":115396876,"request_header":{"Authorization":"NO-SIR","Content-Length":"0","Content-Type":"application/json","User-Agent":"Unity Accelerator v1.0.569+g2096e44","X-Request-Id":"53da0ee6-7fce-4480-bb4b-ea710ee609b2","X-Unity-Accelerator":"v1.0.569+g2096e44"},"response_header":{"Alt-Svc":"clear","Content-Length":"382","Content-Security-Policy":"frame-ancestors 'self' unity3d.com *.unity3d.com unity.com *.unity.com","Content-Type":"application/json; charset=utf-8","Date":"Tue, 18 Aug 2020 20:24:13 GMT","Referrer-Policy":"origin-when-cross-origin","Strict-Transport-Security":"max-age=31536000; includeSubDomains","Via":"1.1 google","X-Content-Type-Options":"nosniff","X-Frame-Options":"SAMEORIGIN","X-Request-Id":"53da0ee6-7fce-4480-bb4b-ea710ee609b2","X-Unity-Accelerator":"v1.0.569+g2096e44","X-Xss-Protection":"1; mode=block"},"bytes_in":382}
     
    mons00n, Aug 18, 2020
    #18
    Moonlit-Games-Studio likes this.

  19. Moonlit-Games-Studio

    Moonlit-Games-Studio

    Joined:
    Feb 11, 2014
    Posts:
    26
    Same here, running latest accelerator on Docker+ZFS, set permissions to 777 on all dirs (persistent/cache/agent - not sure yet which is which).
    Still getting errors:


    Code (CSharp):
    1. accelerator    | {"level":"debug","ts":"2020-08-20T08:28:31.374Z","msg":"incoming requests need space; starting unscheduled eviction pass","agent_id":"#######_id","agent_name":"#######","component":"cache","subprocess_id":2,"bytes to evict":7820}
    2. accelerator    | {"level":"error","ts":"2020-08-20T08:28:31.374Z","msg":"not enough found to evict!","agent_id":"#######_id","agent_name":"#######","component":"cache","subprocess_id":2,"bytes evicted":0,"files evicted":0,"bytes left to evict":7820}
    3. accelerator    | {"level":"debug","ts":"2020-08-20T08:28:31.374Z","msg":"eviction pass complete","agent_id":"#######_id","agent_name":"#######","component":"cache","subprocess_id":2,"desired bytes to evict":7820,"actual bytes evicted":0,"files evicted":0}
    4. accelerator    | {"level":"debug","ts":"2020-08-20T08:28:31.474Z","msg":"handled putRequest","agent_id":"#######_id","agent_name":"#######","component":"pbservice","subprocess_id":6,"conn_id":2,"remote":"#######:#####","segment_id":348,"namespace":"81e94844d19a16919208533e08183531","key":"a1c7b394975a8eaa7d6f66a30495818f","size":782,"status":"error","status_code":1,"error":"cache.create error: operation not supported"}
    Apparently, setting unity-accelerator.cfg free cache values to 0 didn't help, receiving same error. Config values:
    Code (CSharp):
    1.     "CacheDirSplit": 2,
    2.     "CacheMonitorInterval": "1m0s",
    3.     "CacheMinFreeBytes": 0,
    4.     "CacheMinFreePercent": 0,
    5.     "CacheEvictUntilFreeBytes": 0,
    6.     "CacheEvictUntilFreePercent": 0,
    7.     "CacheMaxUsedBytes": -1,
    8.     "CacheMaxUsedPercent": -1,
    9.     "CacheEvictUntilUsedBytes": -1,
    10.     "CacheEvictUntilUsedPercent": -1,
     
    Last edited: Aug 20, 2020
    Moonlit-Games-Studio, Aug 20, 2020
    #19

  20. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Okay, with today's release, v1.0.591+g23eefb5, there is more logging around these unscheduled eviction triggers. Hopefully that will help in finding the underlying issue.

    Also, possibly fixing the issue, we added a check at startup to see if preallocating space for a temporary file succeeds or fails. If this startup check fails, it will just skip preallocating space with uploads as the underlying file system probably doesn't support it. If this check fails, it will be logged as a warning with the message "not enabling preallocations; test case failed" along with the underlying error text.
     
    gregoryh_unity, Aug 20, 2020
    #20
    Moonlit-Games-Studio likes this.

  21. Moonlit-Games-Studio

    Moonlit-Games-Studio

    Joined:
    Feb 11, 2014
    Posts:
    26
    If this is the test then it's failing at the startup:
    Code (CSharp):
    1. accelerator    | {"level":"warn","ts":"2020-08-24T09:19:24.144Z","msg":"not enabling preallocations; test case failed","agent_id":"#########_id","agent_name":"#########","component":"cache","subprocess_id":2,"path":"/persist/cachedbpending/.test-preallocate.279036665","error":"operation not supported"}
    EDIT:
    Beside the preallocation warning, it seems that current docker version is uploading and downloading data for clients :)
     
    Last edited: Aug 24, 2020
    Moonlit-Games-Studio, Aug 24, 2020
    #21

  22. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,328
    Can you please remove the percentage based disk space check ?
    And only check for available space

    because for example I have an 8 terra-byte disk and 5 percent empty space is still plenty of space
     
    Lars-Steenhoff, Aug 24, 2020
    #22

  23. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Great news, thank you! So it does seem that somewhere in the docker => filesystem mechanism preallocation operations (fallocate and the like) are not supported. TIL. I think that part of the issues is good to go now. Thank you for your help, and patience.
     
    gregoryh_unity, Aug 24, 2020
    #23

  24. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    This sounds like a great idea; I'll talk it over with the team and see if we should change the default CacheMinFreePercent to 0 and CacheEvictUntilFreePercent to 0 as well. I don't think the check needs to be removed completely, some may prefer it, just disabled by default.
     
    gregoryh_unity, Aug 24, 2020
    #24
    Lars-Steenhoff likes this.

  25. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,328
    Screenshot 2020-08-24 at 18.11.22.png

    Yes thats what I suggest to disabled by default, and a user setting to be changed later.
    At the moment I cannot even install accelerator because it checks for disk-space percentage when I try to run it.

    I have a 1 TB SDD in my laptop and have 90 Gigabyte free space, but it's not enough for accelerator
     
    Lars-Steenhoff, Aug 24, 2020
    #25
    gregoryh_unity likes this.

  26. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Ah! Thank you for that, I'll be sure that's part of the change.
     
    gregoryh_unity, Aug 24, 2020
    #26
    Lars-Steenhoff likes this.