Search Unity

Bug Report - Unity Accelerator, SnapRAID, MergerFS

Discussion in 'Unity Accelerator' started by SideSwipe9th, Jan 18, 2021.

  1. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    Issue: Unity Accelerator lockup on startup when installed onto home NAS using SnapRAID and MergerFS.

    What happened?:

    When you install Unity Accelerator onto a mount point created/managed by MergerFS, Unity Accelerator silently fails to start. According to the log files this is because it is unable to access the bbolt database at the configured file location. After encountering this issue, the Unity Accelerator service appears to be running, however neither the webui nor command line tools are able to access it. When encountering this error the unity-accelerator.log file will contain the following:

    Code (CSharp):
    1. {"level":"info","ts":"2021-01-18T19:00:34Z","msg":"tool wrun start","cmd":"/mnt/storage/UnityAccelerator/unity-accelerator tool wrun --persist /mnt/storage/UnityAccelerator/cache","pid":9066}
    2. {"level":"info","ts":"2021-01-18T19:00:35.046Z","msg":"DASHBOARD AVAILABLE AT http://192.168.1.5/dashboard/","agent_id":"hoderi_id","agent_name":"hoderi"}
    3. {"level":"error","ts":"2021-01-18T19:00:35Z","msg":"tool wrun exited","error":"wrun execute returned errors: [cannot open bbolt database: no such device]","errs":"[cannot open bbolt database: no such device]"}
    4. {"level":"info","ts":"2021-01-18T19:02:17Z","msg":"tool wrun start","cmd":"/mnt/storage/UnityAccelerator/unity-accelerator tool wrun --persist /mnt/storage/UnityAccelerator/cache","pid":9209}
    5. {"level":"info","ts":"2021-01-18T19:02:17.282Z","msg":"DASHBOARD AVAILABLE AT http://192.168.1.5/dashboard/","agent_id":"hoderi_id","agent_name":"hoderi"}
    6. {"level":"error","ts":"2021-01-18T19:02:17Z","msg":"tool wrun exited","error":"wrun execute returned errors: [cannot open bbolt database: no such device]","errs":"[cannot open bbolt database: no such device]"}
    7. {"level":"info","ts":"2021-01-18T19:03:17Z","msg":"tool wrun start","cmd":"/mnt/storage/UnityAccelerator/unity-accelerator tool wrun --persist /mnt/storage/UnityAccelerator/cache","pid":9434}
    8. {"level":"info","ts":"2021-01-18T19:03:18.152Z","msg":"DASHBOARD AVAILABLE AT http://192.168.1.5/dashboard/","agent_id":"hoderi_id","agent_name":"hoderi"}
    9. {"level":"error","ts":"2021-01-18T19:03:18Z","msg":"tool wrun exited","error":"wrun execute returned errors: [cannot open bbolt database: no such device]","errs":"[cannot open bbolt database: no such device]"}
    10. {"level":"info","ts":"2021-01-18T19:04:17Z","msg":"tool wrun start","cmd":"/mnt/storage/UnityAccelerator/unity-accelerator tool wrun --persist /mnt/storage/UnityAccelerator/cache","pid":9535}
    11. {"level":"info","ts":"2021-01-18T19:04:17.312Z","msg":"DASHBOARD AVAILABLE AT http://192.168.1.5:10070/dashboard/","agent_id":"hoderi_id","agent_name":"hoderi"}
    Whenever I'm encountering this error the following files are stored at the following locations:
    • bbolt.db - /mnt/storage/UnityAccelerator/cache/cachedb/bbolt.db
    • unity-accelerator.cfg - /mnt/storage/UnityAccelerator/cache/unity-accelerator.cfg
    • unity-accelerator.log - /mnt/storage/UnityAccelerator/cache/log/unity-accelerator.log
    • unity-accelerator (executable) - /mnt/storage/UnityAccelerator

    While the unity-accelerator executable is able to execute, and write to both its configuration and log files, it seems unable to open/write to the bbolt.db database. This issue exists on multiple versions of Unity Accelerator, I have tested 1.0.909, 1.0.524, and 1.0.144.

    Moving Unity Accelerator off the SnapRAID/MergerFS mountpoint, eg to /home/user/UnityAccelerator and editing the configuration and start scripts to the new install location allows Unity Accelerator to start as expected.

    Steps to reproduce:

    Detailed configuration options for SnapRAID and MergerFS at, my setup follows mostly the defaults here except for more drives: https://forums.serverbuilds.net/t/setting-up-media-server-using-ubuntu-and-snapraid/239

    Install Ubuntu 18.04 or 20.04
    Install SnapRAID, and configure with at least two data drives and one parity drive (eg /mnt/data1 /mnt/data2 /mnt/parity1).
    Install MergerFS, and configure it to merge the data drives into a single mount point (eg /mnt/storage)
    Enable the MergerFS mount point either through fstab or `sudo mount -a`
    Install UnityAccelerator to a directory on the MergerFS mount point (eg /mnt/storage/UnityAccelerator)
    Attempt to start Unity Accelerator
    Attempt to access Unity Accelerator through web UI or command line
    Unity Accelerator will silently fail, though the service is still reported as alive by systemd
    Unity Accelerator logs will state that it is unable to open the bbolt database at the configured location.

    I've just filed a case number with this, 1307279. I'm posting this here encase anyone else is encountering the issue. I will happily unearth any other logs needed to debug this :)
     
    SideSwipe9th, Jan 18, 2021
    #1

  2. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Interesting information, thank you!

    I'm pretty sure this all has to do with fuse mounted filesystems (mergerfs) and memory mapped files not being supported together, at least not in direct io mode, but I am guessing a bit here since I have not tested this directly myself.

    The accelerator agent uses bbolt https://github.com/etcd-io/bbolt#caveats--limitations to track what it has cached to disk and few other bookkeeping details, and that is what uses memory-mapped io. I don't know of any workarounds at this point, other than avoiding fuse file systems. :/
     
    gregoryh_unity, Jan 19, 2021
    #2
    SideSwipe9th likes this.

  3. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    I think you're right with that thought, as soon as I moved the accelerator to another drive in the system it started working as expected.

    After re-reading the mergerfs documentation, would I be right in saying that Accelerator uses mmap to open the bbolt database and possibly other files relating to the cache? If that's the case, a workaround I could test would be to have two mergerfs mountpoints, one with cache.files set to partial, full, or auto-full, and one with it set to off. Then it'd be a matter of configuring Accelerator to start from the mountpoint that supports mmap.

    As this is my home server primarily for media use, if I wanted to run Plex I would need to have potentially the same double mount setup anyway.
     
    SideSwipe9th, Jan 20, 2021
    #3

  4. gregoryh_unity

    gregoryh_unity

    Unity Technologies

    Joined:
    Oct 1, 2018
    Posts:
    50
    Yes, the Accelerator Agent definitely uses mmap file io via that bbolt database. I'm not aware of any other cases of mmap in the accelerator. You can keep cache files and log files, etc. in separate directories. The CacheDir config setting is what determines where the bbolt database file resides -- and unfortunately for this use case, it also defines where all the cached files themselves go. You might be able to symlink to have the bbolt db and the cached files on separate mounts, but I don't think anybody has tested that, or if it's even worth it in your case. Might be easiest just to ensure wherever CacheDir points, supports mmap.
     
    gregoryh_unity, Jan 20, 2021
    #4
    SideSwipe9th likes this.

  5. SideSwipe9th

    SideSwipe9th

    Joined:
    Jan 10, 2019
    Posts:
    46
    Awesome!

    I was able to test setting up another mountpoint for my mergerfs file system, with file caching enabled (cache.files=auto-full,dropcacheonclose=true). Once I copied across the Accelerator instance back into the array and updated the config files and startup script to point to mmap compatible mount point, everything worked as intended. Performance was a bit slower, but that's to be expected as the SnapRAID array is a bunch of hard drives, and the boot drive that I moved Accelerator to is an NVMe SSD .

    As with my other post, could this information, about Unity Accelerator requiring FUSE mountpoints to be configured for compatibility with mmap be added to the documentation? And as this is not a bug per say, but a configuration issue that I've now been able to resolve I guess we can also close the ticket?
     
    SideSwipe9th, Jan 20, 2021
    #5