Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice
  3. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Life with Unity + git + LFS: a rant and a call for help.

Discussion in 'General Discussion' started by bitinn, Mar 30, 2019.

  1. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    734
    Hi all,


    I don't know about you, but I personally find making git and git LFS works properly with Unity increasingly difficult today.

    The concept of git and LFS are simple:

    - You put text-based files in git
    - You put static binary files in LFS
    - You exclude other stuffs like generated files (because they are auto-generated on compile, on build or at runtime, so no need to track them.)

    That's the end of version control with git, easy!

    But to do so effectively, you need:

    - Know what files are text-based and what files are binary, this is usually done by looking at file extensions.
    - Know where these files are, so you can include or exclude them by paths.
    - Know how to merge these files properly, either with git, or some third-party merge tool like UnityYAMLMerge.

    It used to be relatively easy, you can Google for a gitignore, and guarantee it works as intended.

    Not today, let me explain:

    - .asset/.unity files can be text or binary or both:
    - a simple scriptable object? text
    - lighting data or terrain data or nav mesh? binary
    - text mesh pro font asset? both (because you can store binary data in YAML)
    - a unity scene with pro builder mesh in it? both (because these mesh are encoded as binary data)

    - things are generated without you knowing:
    - Addressables put generated content under both /Assets/AddressableAssetsData/ and /Assets/StreamingAssets/ (I understand why it does it, but then it's up to me to tell git and LFS to exclude these generated content, or track them but face frequent changes)

    - merge tool is no longer at fixed location:
    - With unity hub, we no longer install Unity at fixed location, hence unityyamlmerge location can change too, good luck updating it each time.


    So my git repo is never quite right, and there are probably other issues that I don't notice.

    I guess that's why many game dev hates git.

    /rant over, please share your version control strategy?
     
  2. AndersMalmgren

    AndersMalmgren

    Joined:
    Aug 31, 2014
    Posts:
    5,403
    We only use LFS for textures. Lightmaps we treat as compile time data (even though it takes hours to compile). All other binary data we check into git. It have worked pretty well over the years.

    edit: Forgot .fbx btw
     
  3. Ryiah

    Ryiah

    Joined:
    Oct 11, 2012
    Posts:
    16,624
    Our main programmer hosts a SVN (Subversion) repository. Everything we need to commit goes into that one repo.
     
  4. bitinn

    bitinn

    Joined:
    Aug 20, 2016
    Posts:
    734
    I should probably explain why we would like to setup git this way:

    - git is by-design distributed, which mean everybody owns a copy of the "entire history" of the repo.
    - this sounds very nice, until you realize everything ever committed into the repo, even on change and deletion, is still in the history.
    - this is why committing large binary files are so evil, it will force everyone to download every version of these files.

    Thus LFS and my semi- control freak rant.

    I think things like NavMesh asset or Addressables' bin (which stores the player versioning data for content update build), are specially evil as they can change quite often.

    And of course there are cases where you know a file is plain text, but because it's large and static, you commit to LFS anyway. (like a .obj file)

    Anyway, I thought I should post my gitattributes and gitignore so everyone can take a look, it's by no means perfect or anything, just the things I have currently done to make my repo history lighter.

    https://gist.github.com/bitinn/4a82d7dbdaff62163031d318c57cd62d

    (Spoiler: it's not easy.)
     
    W_M_Reszke likes this.
  5. AndreasO

    AndreasO

    Joined:
    Sep 25, 2012
    Posts:
    90
    Unity should use extensions to denote whether it's a binary file or not. Right now it's the responsibility of the VCS to either know or guess this by some crude file name patterns. This is not reliable and before I even noticed this issue I lost every single terrin data I thought I had saved and versioned in my git repo using LFS...
     
  6. WallaceT_MFM

    WallaceT_MFM

    Joined:
    Sep 25, 2017
    Posts:
    364
    We're a bit lazy with our LFS. We put essentially everything that isn't a text format or a meta file into LFS, and then some very large text files get stuffed in there too. The problem with our approach is that it uses a lot of LFS space and makes the git repo larger than it needs to be. It isn't a problem for us, but it might be for someone else if you have limited hard drive space or something. Our decision was based on the idea that harddrives are cheaper than labor costs.
    Here's a fancy one-liner you can run in git-bash that I use to help determine what I should be putting into LFS. Note that this takes a while to run on large repos.
    { git ls-files && git lfs ls-files | cut -d' ' -f3-; } | sort | uniq -u | tr '\n' '\0' | du -b --files0-from=- | sort -g -r > sortOut.txt


    You'll want to cd to your git repo before running that. It will then output a file called sortOut.txt that will have a list of all the things that are in your git repo that are not already in lfs, sorted in descending order of size. I then threw the file extensions/paths of the largest files into our .gitattributes file until the largest things left were .cs files and meta files.
     
    bitinn likes this.
  7. Metron

    Metron

    Joined:
    Aug 24, 2009
    Posts:
    1,060
    Use SVN... There's no benefit in using git in game development.
     
    MothDoctor likes this.
  8. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    3,229
    Unless you actually care about workflow and branching and things teams generally care about. Git alongside SVN/Perforce is fairly common and has been for quite a while. Optimizing workflows by using the best tool for the job. It's only tiny teams that see benefits from an all in one solution, and even there it's debatable IMO. It's more tiny teams don't have the experience to know how to set things up well.
     
  9. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    13,515
    What benefits does SVN have over Git for this particular use case? Isn't it going to run into exactly the same problems with not knowing binary vs. text and storage of large binary files?
     
    MadeFromPolygons likes this.
  10. frosted

    frosted

    Joined:
    Jan 17, 2014
    Posts:
    3,822
    I was considering a switch from GIT to SVN for next project. I used to use SVN a long time ago and always found it pretty straight forward. GIT has some nice features though, I really appreciate git stash for example.

    Anyone have a real practical comparison? I've heard SVN handles binary more efficiently in general.
     
  11. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    13,515
    Right, I see. It's because SVN only keeps the current version of checked out files on local machines.
     
  12. frosted

    frosted

    Joined:
    Jan 17, 2014
    Posts:
    3,822
    Oh, so thats the advantage for binary - just no local change log?
     
  13. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    13,515
    As far as my brief Googling turned up, yeah. As far as I know (I haven't tried it myself because it hasn't become a problem) that's solved by Git LFS, which splits large binaries into separate storage with different rules.
     
  14. Metron

    Metron

    Joined:
    Aug 24, 2009
    Posts:
    1,060
    SVN has no local repository. Binary files are handled without any problems.

    The LFS is a system added to git because they ran into problems with binary files. I don't see any real benefit of having a local repository with history. The same can easily be achieved by using branches. In fact, I actually see local repositories as a problem: if the developer commits in his local repository but doesn't push to the master branch, all work is lost if his HD fails.
     
    MothDoctor likes this.
  15. sandstedt

    sandstedt

    Joined:
    May 24, 2015
    Posts:
    27
    Having a local copy of the whole history is GOLD in my option. Super easy to go back in time and see what have changed and checkout that directly without an internet connection.

    And not to mentioning the security to have the entire GIT project backed up on every persons computer, in case the GIT server would crash for some reason.
     
  16. AndersMalmgren

    AndersMalmgren

    Joined:
    Aug 31, 2014
    Posts:
    5,403
    There is a reason SVN have lost the race to git, distributed versioning is just so much better, more so in enterprise sized teams
     
  17. Thom_Denick

    Thom_Denick

    Joined:
    May 19, 2014
    Posts:
    13
    Wow someone in 2019 advocating for SVN. You never know what you'll run into on the Unity forums!
     
  18. Metron

    Metron

    Joined:
    Aug 24, 2009
    Posts:
    1,060
    Wow... Someone in 2020 debunking an old thread... Without making any point why my arguments are invalid...
     
    ErikH2000 and MadeFromPolygons like this.
  19. benthroop

    benthroop

    Joined:
    Jan 5, 2007
    Posts:
    191
    If you like branching and using binaries without pain check out PlasticSCM. It really works!
     
  20. KokkuHub

    KokkuHub

    Joined:
    Feb 15, 2018
    Posts:
    705
    Except when you use LFS Git is no longer really distributed. Also, non coders can barely adapt to the update/commit flow, in my experience getting them to follow the commit/pull/push flow (in that exact order, don't you dare pull before committing) is a struggle.
     
    MothDoctor likes this.
  21. Ziplock9000

    Ziplock9000

    Joined:
    Jan 26, 2016
    Posts:
    277
    Unfortunately, it's more expensive than several other enterprise solutions out there for the storage, which is the main statistic for indies. Also the price scaling is much worse too.
     
  22. benthroop

    benthroop

    Joined:
    Jan 5, 2007
    Posts:
    191
    It's been a few months and I actually agree with you now. Shortly after I wrote my message, Unity acquired PlasticSCM. In doing so they changed their plans to push almost all users into their cloud offering. The hosted version is over $20/user per month, which is out of range for most every small shop.

    It's one thing to use a proprietary source control software. It's another to be proprietary AND have your stuff locked in the cloud. So, I'm moving away from Plastic. But the only reason I can even do that is because Github + LFS has been pretty dang good. I would probably look at Plastic again for a really large project, but for things under 100GB, I think Github + LFS works.

    I've been a user and fan of Plastic for several years, and shipped games with it. It's great software at its core. I hope that Unity does something great with it that moves it beyond the current state.
     
  23. MDADigital

    MDADigital

    Joined:
    Apr 18, 2020
    Posts:
    1,908
    Azure devops + LFS works perfect for us. I love distributed versioning.
     
unityunity