Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Git support on Package Manager

Discussion in 'Package Manager' started by rz_0lento, Oct 24, 2018.

  1. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    2018.3.0b7 added following:
    but docs haven't been updated for this yet, would it be possible to get example what to manually put into manifest to ref specific git branch?
     
    dzamani likes this.
  2. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
  3. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
    Hi @rizu,

    Glad you have found the pattern :). As stated in the release notes, this is an experimental feature. We are targeting production ready quality for 2019.1. Super glad that you are trying this out though. Please, let us know if you find any issues.

    And here is a minimalist documentation if you would like to expand your testing further and maybe for others who are reading this thread.

    Requirements

    To use Git packages in a project, Git must be installed on the user machine. The Git executable path should be listed in the PATH system environment variable.

    Project manifest lock attribute

    This feature introduces a new lock attribute in the project manifest. This attribute is populated by the Package Manager to render the package configuration resolution deterministic.

    Supported URL schemes/protocols

    The Package Manager supports the same URL schemes as Git does. In fact, the URL is passed to the Git executable (minus the revision part). If Git ever adds support for more protocol/schemes, they will automatically be supported by the Package Manager.

    List of supported schemes
    • git:
    • ssh:
    • https:
    • http:
    • file: (only URL-like path are supported for now. i.e. file://absolute/path/repo.git. System path will be supported later. This could allow using relative path for example file:../../repo.git)
    We also support SCP style pattern:

    git@github.com:user/repo.git


    Revisions

    You can select a specific revision to be cloned by the Package Manager. The revision is prefixed by the # character. The revision should be added after .git extension:

     https://repository/path.git#revision


    The revision can be any commit-ish - an expression that eventually resolves to a commit hash. The most common types of commit-ish are commit hashes, tags, and branches.

    Limitations and good to know:
    • The Package Manager UI was not updated to support this new package source (ETA 2019.1). I haven't seen any more issue though.
    • Big repositories can take a long time to clone. We don't cache the repository so if you change revision, the repository will be cloned again.
    • The only way to update a Git package to the latest version is to remove the lock attribute from the project manifest or manually update the revision suffix.
    • Since the Package Manager uses the Git executable installed on your machine. Any global or system configuration or environment variable will be picked up. These settings may affect the behaviour of the system.
    • Some package managers offer a shorthand version of their supported URL schemes. For example, npm support short URLs like these: <github user>/<repository>, gitlab:<gitlab user>/<repository>. We don't support these syntaxes for the moment.

    Hope this will help you test the feature.

    Regards,

    Pascal
     
  4. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    @okcompute_unity, thanks, that was very helpful. Any chance to get similar brief intro for scoped registries (private registries)? :D
     
  5. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
    Sure! Just start a thread for it :p
     
    rz_0lento likes this.
  6. falkj17

    falkj17

    Joined:
    Sep 11, 2017
    Posts:
    9
    @okcompute_unity thanks for that information! That is really useful!!
    Is there any chance that the package manager will support finding packages in subdirectories of a git repository? Let's say I have one repository that contains multiple packages in various subfolders of that repo. I want to target a specific one, would that be possible?
     
    Sonic1584 likes this.
  7. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
    @falkj17 Yes, this is something on our roadmap. But we suggest putting your package at the root of the repository if possible. This will be easier for your users to find the package. For instance, users will have to know and specify the name of your package if it is deep in the repo. While if the package is at the root, only the URL will be required, the name will be optional.
     
    EirikWahl likes this.
  8. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
  9. falkj17

    falkj17

    Joined:
    Sep 11, 2017
    Posts:
    9
    Great, thanks! Any idea on how long it will take to add that functionality? ;)

    I completely agree that for visibility it would be better to have it in the root, but we would like to group them by different themes and have all relevant packages in one repo, if possible. Having every single package in a different repo also makes it more difficult to develop multiple packages in parallel
     
  10. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    IMO for development alone it's better to keep the repos separated, but I could see use for being able to keep different packages in single repo as well, like if you keep the git as custom repository that just holds various open source assets as packages (I already have two separate repos for this purpose atm).

    Also if you have like say, custom SRP fork right now, the repo contains many packages already and it would be handy if people could just ref to them directly.

    Right now you can workaround this by just putting different packages into different branches/releases but it's kinda hacky IMO.
     
  11. dzamani

    dzamani

    Joined:
    Feb 25, 2014
    Posts:
    122
    Hi, quick question about this:

    If there isn't any revision specified, is the package manager able to detect new version of the git package ?

    In other dependency manager similar to this, you can for example specify an exact version match and it will go fetch that one that's what specifying the revision looks like at the moment. But is there a way with git repos (I know I can do it with the scoped registries) to fetch and check if there is an update of the repo (based on the package.json) ?

    Thanks again for this feature!
     
  12. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    if you don't specify the revision, I think it will just use the default branch configured for the git repo, and to update it later on refer to:
    Package specific lock attribute is generated at the end of manifest once the package manager processes the git clone/update, it basically stores the hash from git repo and revision. If you clear it, it should update to latest.
     
  13. dzamani

    dzamani

    Joined:
    Feb 25, 2014
    Posts:
    122
    That's what I was asking, dependency managers like Cocoapods will check if it's the latest commit if you did not specify a revision.

    So what I'm actually asking is : Is there any plan to handle this usecase where the package manager will check if he has the latest commit or not on a specific branch?
     
  14. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    559
    I tried this and it didn't work.
    I managed to make it work by specifying the url with the ssh scheme:
    ssh://git@bitbucket.org/myorg/my-repo.git#1.0.0

    git: scheme didn't work. also note the slash "/" between host and org instead of colon ":"
    http(s) scheme is not working as it complains about missing credentials (and I don't want to specify my password in the url (that will get committed)) (note: my repos are all private)
     
    Claytonious likes this.
  15. Rotary-Heart

    Rotary-Heart

    Joined:
    Dec 18, 2012
    Posts:
    810
    Is there anyway to add a dependency between git packages? Current dependency seems to only accept name and version:
    Code (CSharp):
    1.     "dependencies": {
    2.         "packagename": "1.1.1-preview"
    3.     }
    If a git link is added it will throw an error since it doesn't recognize it as a version number:
    Code (CSharp):
    1.     "dependencies": {
    2.         "packagename": "https://gitprovider.com/MyPackage.git"
    3.     }
    Other than this, pretty amazing how easily is to setup the git links, great job!
     
    Gunvald and sp-jordi-vilalta like this.
  16. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    I renamed the thread for clarity as this has gone well past the syntax of the manifest entry :)

    I'd guess it's the same deal as with direct folder refs atm (no dependency support). I do admit it would be handy but how should that be setup so it would work both locally and from git repos?
     
    Rotary-Heart likes this.
  17. Rotary-Heart

    Rotary-Heart

    Joined:
    Dec 18, 2012
    Posts:
    810
    I see, I missed that dependencies didn't work with local folders either.

    I would guess that the same way you setup them on your project manifest. All it really needs is to point to the location where the other package.json is located and process it.
     
  18. andybak

    andybak

    Joined:
    Jan 14, 2017
    Posts:
    563
    I think you're missing a major use case for this. There is a LOT of great stuff already on Github and it usually ends up cluttering up Asset folders because there's no other way to bring it in to a project.

    If you supported packages in subdirectories we could:

    1. Fork the repo we want to use
    2. Add a package.json to the correct directory.

    Most Unity repos are entire projects. Forking and adding a package.json to the root dir won't work in most cases. Forking and rearranging the directory structure makes it impossible to track upstream changes.

    I've experimented with submodules, sparse checkouts and similar to get this working but they are extremely fragile.
     
  19. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    If you share whole project IMO best place for custom packages is just to place them into Packages folder for two reasons: you don't have to modify manifest at all, Unity picks them from there as packages automatically and it also keeps your Assets folder clean as they are not there bloating it.

    I'd love to be able to point into that git repos Packages/<packagename> path directly from the manifest as this would let people share whole projects and also at the same time let others pick the package only from the project.
     
  20. andybak

    andybak

    Joined:
    Jan 14, 2017
    Posts:
    563
    Unless I'm misreading you - you're making a slightly orthogonal point (which I agree with)?

    Just checking I haven't misunderstood.
     
  21. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    559
    I also understand that you are talking 2 different thing (I agree on both):
    - @rizu says, if you own a repo with a Unity project, you should put your stuff inside Packages/<your.package.name>/ and leave Assets/ empty (or with samples only)
    - @andybak says, existing github repos in the wild currently have stuff in Assets/, and he needs to fork them, add a package.json in their "root" (and asmdefs probably) then point to it.

    given that (especially wrt. existing stuff), I vote for being able to point UPM to a subfolder of a git repo.

    ps: question for unity devs: what would be the best practice for developing a git-hosted package from now on (until we get the ability to push them privately to your registry/asset store)?
    (1) repo should contain everything needed to develop the package, without being a submodule
    (2) I want to be able to pull it from package manager, either by using a "git:" dependency pointing to a tag, or by releasing it to a registry (main one or scoped) and requesting with the version
     
    leni8ec likes this.
  22. zeh

    zeh

    Joined:
    Feb 2, 2013
    Posts:
    28
    Another point to being able to point to a folder (say,
    https://YourGitHost.com/UserName/RepoName.git/folderName
    ): you don't want everything on the root of a public repository. That'd make things a bit messy. For public repositories, it's better to just have a
    README.md
    , a
    LICENSE.md
    , a
    CONTRIBUTE.md
    , and any other documentation on the root, and the actual package on a subfolder so people don't need to download useless files into a project.

    The Unity WakaTime plugin at https://github.com/vladfaust/unity-wakatime is currently following this syntax. But it means it can't be included in the manifest as a Git URL because the actual package is inside a subfolder.
     
    jashan and erwand like this.
  23. JesOb

    JesOb

    Joined:
    Sep 3, 2012
    Posts:
    1,096
    Hi @okcompute_unity

    Is there plans to support not only git repo but mercurial either ?
     
    alexzzzz likes this.
  24. pedro_unity

    pedro_unity

    Joined:
    Jan 16, 2017
    Posts:
    172
    Hello all,

    @Jes28 : There are, currently, no plans to directly support Mercurial at this point. I'll bubble up your request to the team nonetheless.

    @andybak : You're right, currently package.json needs to be at the root of the package to be consumed by UPM. I've taken note of your request and we will follow up internally.

    Cheers,
    Pedro
    Unity STE
     
  25. JesOb

    JesOb

    Joined:
    Sep 3, 2012
    Posts:
    1,096
    Thanks for fast response :)

    May be UT can just open some api, so we can add mercurial support ourselves :)
     
  26. stopiccot_tds

    stopiccot_tds

    Joined:
    Oct 1, 2016
    Posts:
    111
    @andybak @zeh I agree, without possibility to reference a subfolder as a package git referencing is almost useless(
     
  27. pedro_unity

    pedro_unity

    Joined:
    Jan 16, 2017
    Posts:
    172
    @Jes28 As your signature states, "Nothing's Impossible" :)

    Cheers,
    Pedro
     
    SnailSimulator likes this.
  28. Foriero

    Foriero

    Joined:
    Jan 24, 2012
    Posts:
    584
    Hi, will the new feature be able to add only specific subfolder. We have spine runtime that is containing code for other engines and we just need to include folder that is intended to be used with unity. Zeh mentioned it with https://YourGitHost.com/UserName/RepoName.git/folderName

    Thank you, Marek.
     
  29. Foriero

    Foriero

    Joined:
    Jan 24, 2012
    Posts:
    584
    btw may I ask you what tool internally you use to check out the git repository? I mean unity has to have internally command line tool to do so right?
     
  30. print_helloworld

    print_helloworld

    Joined:
    Nov 14, 2016
    Posts:
    231
    they use the git that is registered in your environment path
     
  31. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
    Hi @Foriero,

    Git repository sub-folder is not supported for the moment. Your package has to be located at the root of the Git repository otherwise the Package Manager won't find it. We are considering to implement sub-folder support in the future. No ETA though.

    Out of curiosity, anyone knows of a package manager supporting sub-folders? I can't find one. I would be really interested to see the chosen syntax. I know the popular ones don't support it (e.g. npm or pip).

    Regards,

    Pascal
     
  32. Foriero

    Foriero

    Joined:
    Jan 24, 2012
    Posts:
    584
    We use svn to checkout subfolders on github.
     
  33. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
    @Foriero, Interesting. Did not know it was possible to partially download a Git repository through SVN!. Sadly, this does not fit well with our implementation and future features that we have planned. Thank you for sharing though :)
     
  34. rawna

    rawna

    Joined:
    Aug 13, 2015
    Posts:
    35
    @okcompute_unity

    Not sure if this exactly matches what you're looking for, but in Paket (a .NET dependency manager) you can specify a directory in the repository which contains NuGet packages. It uses the following syntax:-

    git https://github.com/forki/nupkgtest.git master Packages: /source/


    you can read about it here:-

    https://fsprojects.github.io/Paket/git-dependencies.html#Using-Git-repositories-as-NuGet-source

    I also would like to add my voice to this request, for the above mentioned reasons
     
  35. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
  36. Rock360

    Rock360

    Joined:
    Oct 26, 2016
    Posts:
    8
    How do I change the document url, change log url and license url?
     
  37. spaceemotion

    spaceemotion

    Joined:
    Sep 29, 2015
    Posts:
    95
    Hey, I'm having trouble with git trying to clone a repo using my password-protected SSH-key. I am running an ssh-agent in the background though, so I don't know what I might need to do so unity picks it up.... Many thanks!
     
  38. karolwieczorek9

    karolwieczorek9

    Joined:
    Jun 8, 2015
    Posts:
    6
    Be sure to use package naming convention. Primarily: "Package names are written in all lower case" since otherwise it will not work. At least it not worked for me.
     
  39. spaceemotion

    spaceemotion

    Joined:
    Sep 29, 2015
    Posts:
    95
    I actually did that. The error I'm getting is not a "not found" but rather an "authentication error".
     
  40. pedro_unity

    pedro_unity

    Joined:
    Jan 16, 2017
    Posts:
    172
    Hello @spaceemotion,

    Was this solved?

    Package Manager is basically passing along GIT error messages through UPM when something goes amiss. If you're getting an error message that says "authentication error", it might be something to do with the SSH key location/availability/agent or credentials not being properly entered.

    Cheers,
    Pedro
    Unity STE
     
  41. spaceemotion

    spaceemotion

    Joined:
    Sep 29, 2015
    Posts:
    95
    @pedro_unity Hey, thanks for the reply.

    I am running an ssh-agent in the background and the PID env-variables are being set properly. For some reason Unity does not seem to pick up on this though.

    This is in my .bashrc file, I'm starting it by opening the git bash on windows (Havent tested this on my mac yet):

    Code (bash):
    1.  
    2. env=~/.ssh/agent.env
    3.  
    4. agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; }
    5.  
    6. agent_start () {
    7.     (umask 077; ssh-agent >| "$env")
    8.     . "$env" >| /dev/null ; }
    9.  
    10. agent_load_env
    11.  
    12. # agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2= agent not running
    13. agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?)
    14.  
    15. if [ ! "$SSH_AUTH_SOCK" ] || [ $agent_run_state = 2 ]; then
    16.     agent_start
    17.     ssh-add
    18. elif [ "$SSH_AUTH_SOCK" ] && [ $agent_run_state = 1 ]; then
    19.     ssh-add
    20. fi
    21.  
    22. unset env
    23.  
    The contents of that .env file after authenticating is:

    Code (bash):
    1.  
    2. SSH_AUTH_SOCK=/tmp/ssh-x7j1vRXCkHpc/agent.17496; export SSH_AUTH_SOCK;
    3. SSH_AGENT_PID=10644; export SSH_AGENT_PID;
    4. echo Agent pid 10644;
    5.  
     
  42. pedro_unity

    pedro_unity

    Joined:
    Jan 16, 2017
    Posts:
    172
    Hi @spaceemotion,

    That's weird, let me dig a bit more and get back to you.

    Cheers,
    Pedro
    Unity STE
     
  43. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    545
    Hi @spaceemotion,

    While your script may successfully start the ssh-agent in the background, the exported environment variables are only available to the bash shell running the script and to any child process it launches from that point on; environment variables from parent processes, unrelated processes, or previously created child processes cannot be changed by your script. Further, even if the script is located in .bashrc, it only gets executed in bash shells. Launching programs (both in Windows and macOS) does not use a shell, so the script is not executed. On the other hand, if you open a bash shell first, then "manually" launch the Unity Editor from that shell, it should work.

    In other words, if you use the script by running it once, then you launch Unity by double-clicking its icon or using the Hub, the Unity Editor process won't have these variables set so ultimately the Git process invoked by Unity won't have them either.

    That said, I see a few possible workarounds.
    - You can create a bash script to launch the Unity executable or the Hub (caveat: the Hub must not already be running, otherwise that will just activate the existing process and it won't have the newly exported environment variables)
    - Or since you're on Windows, you might try to invoke SETX to permanently set the environment variables in the Windows registry. However you need to ensure the Hub process is not already running, or that it gets restarted if it's already running. This solution would work best if you can ensure that the script runs once on startup and that the Hub is not started with Windows.

    The last suggestion would have the benefit of seamlessly integrating the ssh-agent so it works as you expect it to.
     
    Last edited: Jan 9, 2019
    spaceemotion likes this.
  44. spaceemotion

    spaceemotion

    Joined:
    Sep 29, 2015
    Posts:
    95
    @maximeb_unity hey thanks for the suggestion. The second one might just what i needed to also fix other ssh-agent related issues I've had with git clients (Tower, specifically). Will try this out asap! Cheers!
     
    maximeb_unity likes this.
  45. spaceemotion

    spaceemotion

    Joined:
    Sep 29, 2015
    Posts:
    95
    @maximeb_unity

    Good news! I got the whole thing to work by appending the following to my .bashrc:

    Code (CSharp):
    1. # Expose env variables to native windows
    2. setx SSH_AGENT_PID "$SSH_AGENT_PID"
    3. setx SSH_AUTH_SOCK "$SSH_AUTH_SOCK"
    (Tip: you can pipe the success messages to /dev/null to not have this pop up on each terminal load)

    I then added the following line to the package.json:

    Code (CSharp):
    1. "com.company.package": "ssh://git@gitlab.com/company/package.git"
    The important bit about gitlab is that their URLs tend to look like this:
    gitlab.com:company
    which gives you a hostname resolve error, so be sure to convert that
    :
    into a
    /
    .

    Also: Unity Collaborate ignored the whole package folder which is great, since it previously didn't when we just had the package with it's
    .git
    folder in there.

    Cheers!
     
  46. maximeb_unity

    maximeb_unity

    Unity Technologies

    Joined:
    Mar 20, 2018
    Posts:
    545
    Hi @spaceemotion,

    I'm glad it worked, thanks for letting us know!

    Yeah, Gitlab provides SSH "URLs" in the SCP-like format, which is not quite a URL. We don't support that format (yet) but we intend to because it will be easier - that format is commonly used in the Git world.
     
    Deleted User and spaceemotion like this.
  47. Rock360

    Rock360

    Joined:
    Oct 26, 2016
    Posts:
    8
  48. harko12

    harko12

    Joined:
    Jul 6, 2012
    Posts:
    10
    I think I have set everything up right. The git.exe path is in my System PATH variable. But still every time I try to start up the project, I get:

    An error occurred while resolving packages:
    Project has invalid dependencies:
    com.harkogames.sportfun: No 'git' executable was found. Please install Git on your system and restart Unity

    A re-import of the project may be required to fix the issue or a manual modification of E:/Unity Projects/MFE/Packages/manifest.json file.

    Is there any way I can further troubleshoot why it isn't finding the git.exe ?
     
  49. okcompute_unity

    okcompute_unity

    Joined:
    Jan 16, 2017
    Posts:
    756
    Hi @harko12 ,

    If you open a terminal, and you type
    git
    , does it work?

    Also, make sure you close/restart the Hub after installing Git.

    Pascal
     
  50. harko12

    harko12

    Joined:
    Jul 6, 2012
    Posts:
    10
    I ended up starting unity in Administrator mode, and then it was able to find git. So, maybe it just needed the restart. Thanks!
     
    okcompute_unity likes this.