Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Support Git LFS.

Discussion in 'Unity Build Automation' started by techmage, Dec 30, 2015.

  1. techmage

    techmage

    Joined:
    Oct 31, 2009
    Posts:
    2,133
    https://git-lfs.github.com

    I think it's really important to get support for this in place. Both Github and Gitlab support it now. I've been hosting all my development on Github with it. So I can't use Cloud Build without it.

    It's also super easy to install....
     
  2. unitychrism

    unitychrism

    Joined:
    Sep 16, 2015
    Posts:
    122
    Hi there,

    This is something that's definitely on our radar, and while there is currently no ETA you are correct in that it's pretty straightforward to install. Once we have built support for LFS into our infrastructure, I'll be sure to post an update here to alert you and everyone to its availability. In the meantime, for anyone else looking for LFS support, please post a vote at our feedback site as it helps our team gauge demand for new feature requests from the community:

    https://feedback.unity3d.com/suggestions/git-lfs-large-file-support-for-unity-cloud-build

    Stay tuned!

    Regards,
    -Chris
     
    mfindlater likes this.
  3. RJS

    RJS

    Joined:
    Jun 27, 2013
    Posts:
    2
    This issue is clearly the most voted suggestion that the Cloud Build team has in the feedback system. Given this and your suggestion that it's straightforward to implement, why is there no ETA or roadmap item?
     
  4. Ash-Blue

    Ash-Blue

    Joined:
    Aug 18, 2013
    Posts:
    102
    Our team can't use Cloud Build without Git LFS. Would absolutely love for you guys to add this feature.
     
  5. GalxyzStudios

    GalxyzStudios

    Joined:
    Jul 15, 2015
    Posts:
    31
    Any updates on this? Relatively simple to implement and a lot of us here are waiting for this, sounds like it should be a feature pushed out on a very high priority. Any timeline?
     
  6. unitychrism

    unitychrism

    Joined:
    Sep 16, 2015
    Posts:
    122
    Hey all,

    We've been actively working on Git LFS support, as it's the highest rated item for Cloud Build on the Feedback site. Some early experiments are showing promising results, so right now it's possible that your LFS-enabled project will work. If you do give it a try, please reach out and let us know how it goes. Keep in mind that LFS is not technically supported right now, so this is targeted more at existing users of LFS that are comfortable with it.

    Some basic guidelines:
    • You must have SSH access to your projects at an account level (GitLab doesn't currently support SSH authentication with LFS).
    • Deployment keys are not supported on GitHub.
     
    Ash-Blue likes this.
  7. bryantdrewjones

    bryantdrewjones

    Joined:
    Aug 29, 2011
    Posts:
    147
    Hi @unitychrism :)

    Just reporting in to confirm that whatever early experiments you're doing are working for our project! :) Thanks for adding in LFS support so quickly! :)
     
  8. unitychrism

    unitychrism

    Joined:
    Sep 16, 2015
    Posts:
    122
    Yes, positive feedback! :) Glad to hear it, and would appreciate feedback from any others that configure their LFS project in Cloud Build. There should be more information available in the near future, which I will also link to here.
     
  9. bryantdrewjones

    bryantdrewjones

    Joined:
    Aug 29, 2011
    Posts:
    147
    Hello again @unitychrism :)

    We got an LFS-specific error in our latest Cloud Build. It's complaining about a file that it had no trouble with in the past, which is weird. Here's the relevant portion of the log:

    The Cloud Build system took just over 50 minutes before hitting that error. I'm attempting a new build right now just to see if it can recover on its own. I wonder if it just doesn't like how many assets we have stored via LFS?

    Edit: Just for reference, here's our .gitattributes file:

     
  10. bryantdrewjones

    bryantdrewjones

    Joined:
    Aug 29, 2011
    Posts:
    147
    Second attempt at a build failed, as well :( But on a different file this time.

    Is there anything we can do to workaround this? :(
     
  11. unitychrism

    unitychrism

    Joined:
    Sep 16, 2015
    Posts:
    122
    Hi there,

    It looks like you were able to build successfully after these errors yesterday. As part of an update next week, there should be an improvement in LFS logging that will allow you to view the specific error being encountered. If this continues to happen sporadically, please let me know via direct message.
     
  12. bryantdrewjones

    bryantdrewjones

    Joined:
    Aug 29, 2011
    Posts:
    147
    Yes, third attempt (after waiting about half a day) was a success :) Maybe the Cloud Build system was just having a bad night :) Thanks for the reply!
     
  13. SureSight

    SureSight

    Joined:
    Aug 2, 2013
    Posts:
    65
    I'm considering enabling LFS for our current project to take advantage of Unity Cloud Build.
    Is this officially supported yet?
     
    GalxyzStudios likes this.
  14. GalxyzStudios

    GalxyzStudios

    Joined:
    Jul 15, 2015
    Posts:
    31
    When will this be implemented officially? We have to move over to LFS as soon as possible.
     
    SureSight likes this.
  15. Afropenguinn

    Afropenguinn

    Joined:
    May 15, 2013
    Posts:
    305
    Any updates on this for those of us using GitLab?
     
  16. DFT-Games

    DFT-Games

    Joined:
    Jun 24, 2010
    Posts:
    454
    Is that even possible to write a game without git lfs??? Repos are quite big (my current over 35GB) so it's unclear how should we use a continuous integration that does not support that???

    And you hope that I'm gonna pay a subscription?
     
    SureSight likes this.
  17. Shawn-Halwes

    Shawn-Halwes

    Joined:
    Jul 17, 2013
    Posts:
    51
    Did this ever get supported for GitHub?
     
  18. Shawn-Halwes

    Shawn-Halwes

    Joined:
    Jul 17, 2013
    Posts:
    51
    NM it appears to be working, just getting some LFS related errors...
     
  19. Shawn-Halwes

    Shawn-Halwes

    Joined:
    Jul 17, 2013
    Posts:
    51
    OK new question. Have you guys looked into updating Git and LFS?

    According to my error log you are still using Git v2.6.6 and LFS v1.1.1.

    There have been some LFS specific improvements with a Git v2.8.x and LFS v1.2 combination. Using it locally for a few weeks now.

    Here's an article from a SourceTree dev about some of the speed improvements:
    https://developer.atlassian.com/blog/2016/04/git-lfs-12-clone-faster/

    Using the new "git lfs clone" command would probably be a speed and stability improvement for Cloud Build.
     
  20. K0ZMiK

    K0ZMiK

    Joined:
    Jul 5, 2016
    Posts:
    1
    Hello,

    Am I doing something wrong or Unity Cloud build and LFS is not yet operational / is experimental?

    I have a Github repo with LFS, and I have have setup SSH keys.
    Previously I used Bitbucket git successully but without LFS, until I hit their 2GB hardlimit...
    Now, I tried to move to github for their LFS support, but I am stuck here...

    I get a bunch of errors and a failed build

    [Unity] FSBTool ERROR: The format of the source file is invalid, see output for details.
    [Unity] ImportFBX Errors:

    [Unity] Updating Assets/max/Materials/BACKPACK2NORMAL.PSD - GUID: 02f73260494e6f94eb3054045a56269a...
    [Unity] Could not create texture from Assets/max/Materials/BACKPACK2NORMAL.PSD: File could not be read


    Can give more details upon request....

    regards
     
  21. darkForester

    darkForester

    Joined:
    Sep 19, 2013
    Posts:
    80
    Greetings @unitychrism,

    We are happy to see that Git LFS is being integrated. Here are some LFS related issues that are keeping our builds from finishing / starting.

    LFS Smudge Filter Bug


    1315: fatal: Assets/Audio/Games/InOtherWords/Level 4/club.iowclue.en.wav: smudge filter lfs failed
    1316: git-lfs/1.1.1 (GitHub; darwin 386; go 1.5.3; git 7de0397)
    1317: git version 2.6.6


    It doesn't always fail on the same file however, for instance on a different build target it fails on:


    1031: fatal: Assets/Audio/Games/Glyphs/Short Recipe/glyphs.shortrecipe.s5.audio.2.en.wav: smudge filter lfs failed
    1032: git-lfs/1.1.1 (GitHub; darwin 386; go 1.5.3; git 7de0397)
    1033: git version 2.6.6


    I have cloned this repository successfully on other devices, so I know that the files exist. It may be a matter of updating the LFS version.


    Thanks,
     
  22. gonzorob

    gonzorob

    Joined:
    Dec 1, 2011
    Posts:
    96
    Also getting regular smudge filter errors. It's a known problem with LFS solved, partly, by the resume feature in LFS 1.2+.

    As Unity Cloud doesn't have a 'resume failed' feature I'm not certain what the solution is.
     
    darkForester likes this.
  23. DerWoDaSo

    DerWoDaSo

    Joined:
    May 25, 2009
    Posts:
    131
    Same thing here... trying to get data from our selfhosted Bitbucket Server.

    But I also get a certificate error, I guess because we usa a selfsigned SSL certificate. I thought that for a connection via SSH that certificate would not be used, but seems like LFS objects are transfered via HTTP(S).

     
  24. DerWoDaSo

    DerWoDaSo

    Joined:
    May 25, 2009
    Posts:
    131
    Ok, forget my previous post. As the error stated the IP was not included in the certificate.

    My solution which works now: using a subdomain and Let's Encrypt to generate the SSL certificate and everything seems to work fine.
     
  25. unitychrism

    unitychrism

    Joined:
    Sep 16, 2015
    Posts:
    122
    @gonzorob, thank you for pointing out the resume functionality in LFS 1.2. I'll look into us updating our version of Git LFS to provide that functionality, as it likely will help with smudge filter errors you'll see if there is any sort of hiccup in the LFS pull.

    @DerWoDaSo, I certainly appreciate you posting about your solution using Let's Encrypt to generate a valid SSL certificate for your system. To provide some context, while Git LFS communicates via HTTPS the only way we can authenticate for you against the system is to use LFS' inbuilt authentication via SSH since we do not handle user/pass information for Git repositories.

    Will come back to this thread when I have more information, thanks for your patience.
     
  26. unitychrism

    unitychrism

    Joined:
    Sep 16, 2015
    Posts:
    122
    Git LFS 1.2.1 is now available in Cloud Build and I am running through some test builds myself. Please respond to me directly with a Cloud Build build attempt link if you encounter issues while building with LFS.
     
  27. Afropenguinn

    Afropenguinn

    Joined:
    May 15, 2013
    Posts:
    305
    I sent you a private message with my failed log.
     
  28. DerWoDaSo

    DerWoDaSo

    Joined:
    May 25, 2009
    Posts:
    131
    @Afropenguinn GitLab CE 8.2 is out now, which should support SSH authentication for LFS files... I saw your post in their forums. I am currently updating, hope it will work.
     
  29. Afropenguinn

    Afropenguinn

    Joined:
    May 15, 2013
    Posts:
    305
    It does work. I made a post about it here on the Unity forums :)
     
  30. u3dxt

    u3dxt

    Joined:
    Jul 6, 2013
    Posts:
    317
    Git LFS 1.2.1 is still failing for me. I was able to clone the project on another computer successfully.

    24: fatal: Assets/Forest Bat/Animations/Forest Bat@attackbite.FBX: smudge filter lfs failed
    25: git-lfs/1.2.1 (GitHub; darwin amd64; go 1.6.2; git 79bedeb)
    26: git version 2.6.6

    Anyone able to do cloud build with LFS successfully? What is your setup like?
     
  31. maxkofford

    maxkofford

    Joined:
    Sep 29, 2016
    Posts:
    3
    Our project is also getting the "smudge filter lfs failed" errors when we try to build using the Unity Cloud Build.

    1833: Error downloading object: ModularCityAlleyPack/Assets/Walls/Meshes/Wall_D_Brick_Window_J_2x.FBX (cfae39034d62a1f719ec9901be67bf67b85a960aff945019ba55588abdbed13d)
    1834: Errors logged to /BUILD_PATH/strata.vr_studio.default-windows-desktop-64-bit/.git/modules/ModularCityAlleyPack/lfs/objects/logs/20161119T175014.075029395.log
    1835: fatal: ModularCityAlleyPack/Assets/Walls/Meshes/Wall_D_Brick_Window_J_2x.FBX: smudge filter lfs failed

    It might be related to how large our project is (8+ GB) that we tend to run into these errors nearly every build on the Unity Cloud Build. I didn't noticed any similar errors in our local clones/builds but that might just be because we don't have to re-clone our repository for every build.
     
  32. tom_bb

    tom_bb

    Joined:
    Sep 21, 2016
    Posts:
    8
    We get a "smudge filter" failure when new files have been added via Git LFS, it usually needs several buildings to get it through to be successfuly, each one that fails falls over on a different LFS file (further on from the previous fail)

    We are using BitBucket cloud with LFS enabled and we find even on our PC's that it can be quite slow to download sometimes, so it might be a timeout issue perhaps?
     
  33. zvinless

    zvinless

    Joined:
    Oct 6, 2014
    Posts:
    18
    It could just be a matter of updating LFS on the build servers. I get the same smudge filter issues on older versions of LFS, but haven't had problems with 1.5+.
     
  34. Devil_Inside

    Devil_Inside

    Joined:
    Nov 19, 2012
    Posts:
    1,117
    Getting this all the time. Any solution?
     
  35. Devil_Inside

    Devil_Inside

    Joined:
    Nov 19, 2012
    Posts:
    1,117
    chrome_2016-12-17_20-59-31.png
    This is kind of annoying...
     
  36. dwa_engineering

    dwa_engineering

    Joined:
    May 13, 2015
    Posts:
    1
    Getting the "smudge filter lfs failed" error. haven't had a successful build yet. :(

    Any suggestions on how to fix this?


    cloudbuild.PNG
     
  37. Valthaer

    Valthaer

    Joined:
    Mar 21, 2015
    Posts:
    6
    Any news about this? I´m using Visual Studio Team Services with GIT to host my project and Unity Cloud Build is reporting a " smudge filter lfs failed" error.

    Thanks!
     
  38. Guirao

    Guirao

    Joined:
    Nov 24, 2012
    Posts:
    68
    Same,,, upload_2017-1-18_9-32-58.png
     
  39. katiecanning

    katiecanning

    Joined:
    Aug 25, 2016
    Posts:
    2
    Would really appreciate an update on this.
     
  40. Anatolii_Landyshev

    Anatolii_Landyshev

    Joined:
    Aug 10, 2012
    Posts:
    31
    I'm having same Smudge error when building for iOS.
    For Android it builds fine.
     
  41. SophiaC

    SophiaC

    Joined:
    Sep 6, 2016
    Posts:
    238
  42. SteSig

    SteSig

    Joined:
    Oct 26, 2016
    Posts:
    2
    Hi is there anywhere a summary of what hosts are now supported with lfs? i'm looking and looking and getting mixed feedback ... i'm currently failing with Bitbucket lfs on Cloudbuild
     
  43. SophiaC

    SophiaC

    Joined:
    Sep 6, 2016
    Posts:
    238
    PM me a link to your project

    - Sophia
     
  44. yamamori-akihiro

    yamamori-akihiro

    Joined:
    Apr 11, 2016
    Posts:
    1
    We also have a problem involving git lfs clone.
    We are occurring `dial tcp: i/o timeout` error while downloading 3000~5000 files.
    We have 10000+ lfs-managed files and we've never succeeded git clone.

    The failing files differ each time we build.
    The error messages are always `dial tcp: i/o timeout` for every build.


    Is it helpful to PM our project name for someone Unity Technologies?
     

    Attached Files:

    Last edited: Mar 3, 2017
  45. SophiaC

    SophiaC

    Joined:
    Sep 6, 2016
    Posts:
    238
    Krillbite-Studio likes this.
  46. PavanJakhu

    PavanJakhu

    Joined:
    Apr 28, 2017
    Posts:
    2
    I'm currently having the same issue as a couple of the other people above me it seems. Here is the Full Log of my build in Unity Cloud Build:


    I was wondering if I could get some support?
     
  47. SophiaC

    SophiaC

    Joined:
    Sep 6, 2016
    Posts:
    238
    I've responded to your PM :)
     
  48. pfleetwood22

    pfleetwood22

    Joined:
    Jun 6, 2016
    Posts:
    28
    It seems my builds are not attempting to pull LFS assets, leading to build failures about not being able to load textures or find namespaces that are implemented in dlls (both of which are file types we manage with LFS). Can I get some support?

    Update: Retried again today (same commit SHA) and it has pulled 1.5 GB instead of 300 MB. So it appears like it is at least getting all the code again.
     
    Last edited: May 11, 2017
  49. SophiaC

    SophiaC

    Joined:
    Sep 6, 2016
    Posts:
    238
    PM me :)
     
  50. antifuzz

    antifuzz

    Joined:
    Feb 22, 2013
    Posts:
    99
    I'm also getting the same "Error downloading object" with LFS. Have sent you a pm, @SophiaC .