Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Does Unity Cloud Build shallow clone Git repositories?

Discussion in 'Unity Build Automation' started by OlafZ, Mar 1, 2018.

  1. OlafZ

    OlafZ

    Joined:
    Aug 3, 2016
    Posts:
    17
    How does Unity Cloud Build clone a specific branch?
    Does it do a shallow clone (--depth 1)?

    I have not been able to find an answer to these questions in the cloud build logs.

    For a little background: within our automated workflow, we regularly create new cloud build configurations. This means that cloud build has not yet had a chance to cache anything. And we are seeing Git clone performance with our cloud builds that differs wildly, from less than a minute to several minutes. Occasionally we even hit the timeout limit, causing builds to fail on Git cloning.
    And it appears to do a full git clone instead of a shallow clone of only the relevant branch. For our repository, the difference between a shallow clone and a full clone is a difference between a 50MB transfer or a 700+MB transfer, roughly. So shallow cloning could benefit us a lot.
     
    L4ZZAR8 and chentmin like this.
  2. chentmin

    chentmin

    Joined:
    Jan 1, 2016
    Posts:
    2
    In our case, a shallow clone is about 1GB, comparing to 7GB in full.
     
  3. dannyd

    dannyd

    Unity Technologies

    Joined:
    Jun 3, 2014
    Posts:
    785
    Not currently supported, but something we are investigating allowing opting into. The primary reason UCB doesn't use/support a shallow clone is because it would break getting the changesets between builds (which I realize isn't a concern for everyone).
     
  4. chentmin

    chentmin

    Joined:
    Jan 1, 2016
    Posts:
    2
    For Github, it has api to get information about commits. https://developer.github.com/v3/repos/commits/

    Maybe at least for Github, UCB can use api to get the changesets and just use shallow clones. It won't break anything but makes builds faster and uses [a lot] less network transfer, which saves unity a lot of money.
     
    L4ZZAR8 likes this.
  5. OlafZ

    OlafZ

    Joined:
    Aug 3, 2016
    Posts:
    17
    If shallow cloning was enabled (or opt in) on Cloud Build configurations that have automatic building disabled, that would indeed save quite a bit of bandwidth and improve build performance. That would also be a pretty safe change if I understand the workflow correctly.
     
  6. jani_alatyppo

    jani_alatyppo

    Joined:
    Dec 23, 2015
    Posts:
    1
    Also adding support for --shallow-submodules would benefit us quite a bit. Note that it needs uploadpack.allowReachableSHA1InWant=true to be set on server config. So it has to be opt-in setting.
    Losing change list is no big deal.
     
  7. Haze-Games

    Haze-Games

    Joined:
    Mar 1, 2015
    Posts:
    186
    +1
    This would be great :)
     
  8. ValeryNikulina

    ValeryNikulina

    Unity Technologies

    Joined:
    Oct 26, 2020
    Posts:
    138
    Hi everyone! I am happy to let you know that shallow clone functionality is now available for Cloud Build via the API. This has not yet been added to the front end.
     
    Haze-Games likes this.
  9. Haze-Games

    Haze-Games

    Joined:
    Mar 1, 2015
    Posts:
    186
    Hi there @ValeryNikulina !

    Awesome, that's excellent news! This will greatly reduce LFS bandwidth usage for us and speed up builds significantly.
    However, since it's available, but not yet in the UI, how can we start a build using a shallow clone? We've never worked with the API. Is it worth it trying to get it working, or better to wait for support in the UCB settings? Do you have an ETA of when this can be used through target settings or such?

    EDIT: I just saw there is an explanation at the top of the page you sent. Will try this out!

    Thank you!
    Charles
     
  10. Haze-Games

    Haze-Games

    Joined:
    Mar 1, 2015
    Posts:
    186
    Hi again,

    OK, this is beautiful, thank you so much! And for 2 reasons:

    - Clone step went from 45 mins to 11 mins, so much faster than previously
    - Since the past 2 weeks, we've been having a "git lfs smudge filter error" that has been crippling many of our builds. Using a shallow clone solved this, and avoids this error (I'm guessing there is a new UCB bug related to cloning full LFS history, that recently appeared, as this issue magically appeared without any changes on our side).

    So, big thank you to you and your team for fixing 2 of our issues at once with this new addition :)

    PS: Probably shallow clone should be the default, with a setting to adjust it per-project or per-target, but definitely pulling entire repo history + LFS history does not seem like a good default setting to me.
     
  11. Danny327

    Danny327

    Unity Technologies

    Joined:
    Jan 4, 2021
    Posts:
    78
    Hi,
    I want to inform you that we will be able to assess the possibility of having this feature in the front-end by default by the end of the year after we launch the new pricing plan for storage. Hope this helps.
     
    Haze-Games likes this.
  12. Benjamin-Gooding

    Benjamin-Gooding

    Unity Technologies

    Joined:
    Apr 12, 2022
    Posts:
    291
    Shallow clones should now be the default behavior when using Git SSH or Git OAuth. This configuration isn't available in the UI yet as Danny mentioned. Instead of enabling shallow clone, the API will need to be used to disable shallow clone if full history is needed.

    One of the side effects of using Shallow Clones by default, is that we will no longer be able to properly populate the Changes tab in build details.
     
  13. psyhama

    psyhama

    Joined:
    Sep 4, 2018
    Posts:
    26
    @benjamingooding

    Hi, the change broke the `Build with commit-id specified` feature, which I use very frequently. Please fix.

    upload_2022-9-7_14-46-53.png

    log:
    [2022-09-07T05:31:59.187Z] - 6.4.0.2.4.2 - INFO: OAUTH: Checking out https://github.com/***** with depth of 1
    [2022-09-07T05:32:01.516Z] - 6.4.0.2.4.2 - INFO: GIT: Checking out via URI...
    [2022-09-07T05:35:00.839Z] - 6.4.0.2.4.2 - INFO: Cloned in 179 seconds.
    [2022-09-07T05:35:00.839Z] - 6.4.0.2.4.2 - INFO: GIT: Requested revision: 6851ecbcc7dbcefae91cb7d27c94a2102a28a60e
    [2022-09-07T05:35:01.014Z] - 6.4.0.2.4.2 - ERROR: GIT: Error during clone: /cygdrive/c/tools/cygwin/bin/gitc.exe '--git-dir=BUILD_PATH/p/.git' '--work-tree=BUILD_PATH/p' reset '--hard' '6851ecbcc7dbcefae91cb7d27c94a2102a28a60e' 2>&1:fatal: Could not parse object '6851ecbcc7dbcefae91cb7d27c94a2102a28a60e'.
    [2022-09-07T05:35:01.014Z] - 6.4.0.2.4.2 - ERROR: OAUTH: Error during checkout: /cygdrive/c/tools/cygwin/bin/gitc.exe '--git-dir=BUILD_PATH/p/.git' '--work-tree=BUILD_PATH/p' reset '--hard' '6851ecbcc7dbcefae91cb7d27c94a2102a28a60e' 2>&1:fatal: Could not parse object '6851ecbcc7dbcefae91cb7d27c94a2102a28a60e'.
    ! checkout failed. /cygdrive/c/tools/cygwin/bin/gitc.exe '--git-dir=BUILD_PATH/p/.git' '--work-tree=BUILD_PATH/p' reset '--hard' '6851ecbcc7dbcefae91cb7d27c94a2102a28a60e' 2>&1:fatal: Could not parse object '6851ecbcc7dbcefae91cb7d27c94a2102a28a60e'.
     
  14. dawidk

    dawidk

    Joined:
    Jul 27, 2012
    Posts:
    7
    @benjamingooding is the api documentation updated with info how to disable it?
     
  15. Benjamin-Gooding

    Benjamin-Gooding

    Unity Technologies

    Joined:
    Apr 12, 2022
    Posts:
    291
    We are going to back this out in the next 10-15 minutes and return to the default behavior of a full clone while we wait for the UI team to implement support for specifying shallow cloning when configuring a source control.
     
  16. dawidk

    dawidk

    Joined:
    Jul 27, 2012
    Posts:
    7
    ok cool. I will advise to do as opt-in in the future, gather some feedback and move on from there. if you have fairly complex git project with tagging and submodules, shallow cloning is not what you want.
     
  17. ruslan_seleznov

    ruslan_seleznov

    Joined:
    May 15, 2019
    Posts:
    5
  18. Benjamin-Gooding

    Benjamin-Gooding

    Unity Technologies

    Joined:
    Apr 12, 2022
    Posts:
    291
    I just get back from paternity leave (hence the delayed response).

    You can use the Rest APIs for cloud build to enable shallow cloning across all of your build targets. Individual build target support for shallow cloning is in the road map for next year. Right now, we have an open bug for fixing specifying a commit hash when using shallow clone. Unfortunately it is more complex than we would like due to the version of git running on our windows machines. We will have to do a full checkout and then checkout the commit hash which wouldn't be a true shallow commit checkout.

    As we work on upgrading the version of git across our systems we will eventually be able to do a true shallow checkout of a specific commit. I'm hoping to have the commit hash checkout using non-shallow clone when shallow clone is enabled to unblock users that have configured this before the end of this calendar year.