Search Unity

Where to host big packages with GIT LFS?

Discussion in 'Package Manager' started by tteneder, Sep 13, 2019.

  1. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    175
    Hi,

    I created 2-3 open source Unity packages and host them on github, so they can be installed into projects via GIT URL.
    They contain some rather big files, that's why I use GIT LFS (large file support) to keep the repository efficient.
    After a short time I got messages from Github that they disabled GIT LFS though, because the repo is too big and creates too much data traffic.

    That's the biggest package:
    https://github.com/atteneder/DracoUnity

    In my particular case, the huge files are native libraries for all kinds of platforms (iOS, Android, mac, windows, ...). The static iOS library Xcode produces alone is a crazy 131 MB!

    My first idea was to set up some CI, remove the build files from the repository and do a custom editor script, that downloads the libraries somehow.

    But I can imagine, that other packages with lots of data (textures, etc.) struggle with this as well.

    Is there a better, free hosting option than Github?
    I'm aware, that I could chip in some currency to get it running, but spending my time on OSS projects is enough imho.

    Thanks for your input,
    Andreas
     
    tteneder, Sep 13, 2019
    #1

  2. manu73

    manu73

    Unity Technologies

    Joined:
    Aug 22, 2014
    Posts:
    68
    manu73, Sep 13, 2019
    #2
    akuno and tteneder like this.

  3. Grizmu

    Grizmu

    Joined:
    Aug 27, 2013
    Posts:
    131
    Seconding that GitLab's free edition is good for LFS project hosting. Biggest project so far that I've worked on, which had 20 GBs, was hosted on GitLab. Just have in mind that in free edition your single LFS repository can't be bigger than 10 GBs, what can be quickly eaten by raw asset formats.
     
    Grizmu, Sep 13, 2019
    #3
    MNNoxMortem and tteneder like this.

  4. MNNoxMortem

    MNNoxMortem

    Joined:
    Sep 11, 2016
    Posts:
    723
    For small scale (up to 10GB) i also can recommend Gitlab.

    However past that you usually use a different solution (altough here on the forum is at least 1 dev who said they run a 100GB unity project via git *shudder*). There are some VCS with different handling for binaries/large files.

    However, after that you usually go for a download server. It's actually the same as for databases. You try to avoid storing the actual thing in there, but store it just on some space and just keep the link in code and download on demand. If security is not a problem - but it should always be a concern, and if it just is traffic inflicted - then just host in on any virtual or dedicated server option which is rather cheap.

    On really large scale: that is what cloud storage like S3 is build for.
     
    MNNoxMortem, Sep 14, 2019
    #4
    Grizmu likes this.

  5. tteneder

    tteneder

    Unity Technologies

    Joined:
    Feb 22, 2011
    Posts:
    175
    I managed to significantly reduce the size by not building the iOS library with Xcode, but with Makefiles (all through CMake). The file went from 131MB to 15MB. Xcode what are you doin' ? LOL

    Also I moved it to gitlab (since I cannot even push the now smaller files without paying):
    https://gitlab.com/atteneder/DracoUnity

    Thanks for your inputs!
     
    tteneder, Sep 17, 2019
    #5
    MNNoxMortem and samuelb_unity like this.

  6. matniedoba3d

    matniedoba3d

    Joined:
    Aug 10, 2021
    Posts:
    7
    matniedoba3d, Jun 3, 2022
    #6
    tnaseem, tteneder, Favo-Yang and 1 other person like this.