Search Unity

Where to host big packages with GIT LFS?

Discussion in 'Package Manager' started by atti, Sep 13, 2019.

  1. atti

    atti

    Joined:
    Feb 22, 2011
    Posts:
    80
    Hi,

    I created 2-3 open source Unity packages and host them on github, so they can be installed into projects via GIT URL.
    They contain some rather big files, that's why I use GIT LFS (large file support) to keep the repository efficient.
    After a short time I got messages from Github that they disabled GIT LFS though, because the repo is too big and creates too much data traffic.

    That's the biggest package:
    https://github.com/atteneder/DracoUnity

    In my particular case, the huge files are native libraries for all kinds of platforms (iOS, Android, mac, windows, ...). The static iOS library Xcode produces alone is a crazy 131 MB!

    My first idea was to set up some CI, remove the build files from the repository and do a custom editor script, that downloads the libraries somehow.

    But I can imagine, that other packages with lots of data (textures, etc.) struggle with this as well.

    Is there a better, free hosting option than Github?
    I'm aware, that I could chip in some currency to get it running, but spending my time on OSS projects is enough imho.

    Thanks for your input,
    Andreas
     
  2. manu73

    manu73

    Unity Technologies

    Joined:
    Aug 22, 2014
    Posts:
    40
    atti likes this.
  3. Griz

    Griz

    Joined:
    Aug 27, 2013
    Posts:
    108
    Seconding that GitLab's free edition is good for LFS project hosting. Biggest project so far that I've worked on, which had 20 GBs, was hosted on GitLab. Just have in mind that in free edition your single LFS repository can't be bigger than 10 GBs, what can be quickly eaten by raw asset formats.
     
    MNNoxMortem and atti like this.
  4. MNNoxMortem

    MNNoxMortem

    Joined:
    Sep 11, 2016
    Posts:
    366
    For small scale (up to 10GB) i also can recommend Gitlab.

    However past that you usually use a different solution (altough here on the forum is at least 1 dev who said they run a 100GB unity project via git *shudder*). There are some VCS with different handling for binaries/large files.

    However, after that you usually go for a download server. It's actually the same as for databases. You try to avoid storing the actual thing in there, but store it just on some space and just keep the link in code and download on demand. If security is not a problem - but it should always be a concern, and if it just is traffic inflicted - then just host in on any virtual or dedicated server option which is rather cheap.

    On really large scale: that is what cloud storage like S3 is build for.
     
    Griz likes this.
  5. atti

    atti

    Joined:
    Feb 22, 2011
    Posts:
    80
    I managed to significantly reduce the size by not building the iOS library with Xcode, but with Makefiles (all through CMake). The file went from 131MB to 15MB. Xcode what are you doin' ? LOL

    Also I moved it to gitlab (since I cannot even push the now smaller files without paying):
    https://gitlab.com/atteneder/DracoUnity

    Thanks for your inputs!
     
    MNNoxMortem and samuelb_unity like this.