Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice

Unity Using GitHub Packages Registry with Unity Package Manager

Discussion in 'Package Manager' started by okcompute_unity, Apr 5, 2020.

  1. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    748
    Hi,

    Thanks to forum user @JakHussain, there is now a known method to use GitHub Packages Registry with Unity Package Manager. See the original post where @JakHussain shared his clever finding.

    WARNING: Using GitHub Packages registries with Unity Package Manager comes with a couple of limitations (Goto the end of the post for the list). Please be aware that we are posting this only to share community knowledge. Unity doesn't officially support GitHub Packages Registry. Use it at your own risk.

    There are a couple of steps needed to make it work.

    1 - You need to configure the
    publishConfig
    setting as described in GitHub Packages Registry documentation in a special way. You need to suffix the pakcage scope:

    Code (JavaScript):
    1.   "publishConfig": { "registry": "https://npm.pkg.github.com/@USER" },
    2.  
    Where USER is the Github organization or user (i.e.the scope). Basically, we are cheating on how NPM works and _inject_ the scope in the registry URL instead of prefixing the package name. This is one half of the trick. Unity Package Manager doesn't accept the
    @
    and
    /
    characters in package names. By moving the scope reference in the registry URL we don't need these characters in the package name anymore!

    Add the
    publisConfig
    configuration in your package manifest (package.json)

    2 - Set the name of your package without an NPM scope

    This is the other part of the magic. As stated in step #1, make sure you don't prefix you package name with an NPM scope (@User)

    Example:
    Code (JavaScript):
    1.   "name": "com.my-company.my-package",
    Not:
    Code (JavaScript):
    1.   "name": "@USER/com.my-company.my-package",
    WARNING: If you have previously published a package with an NPM scope in the name, it won't work. Your package should not have an MPM scope for all existing versions. It won't work otherwise.

    3 - Publish your package

    For that step, you can refer to GitHub documentation.

    4- Configure the scope registry in your Unity project manifest

    Code (CSharp):
    1.   "scopedRegistries": [
    2.     {
    3.       "name": "Company Packages",
    4.       "url": "https://npm.pkg.github.com/@USER",
    5.       "scopes": [
    6.         "com.my-company"
    7.       ]
    8.     }
    9.   ],
    10.  
    The URL must match the one configured in your publishConfig above in step #1.

    5 - Setup Npm Auth configuration

    WARNING: GitHub Packages registry always requires authentication. For private and public packages.

    For this one, I'll refer you to another post I've made explaining how.

    6 - Add your package in your project manifest

    GitHub Package Registry packages won't show up in the Package Manager UI All section. You have to install the package manually in your project manifest.

    Known limitations
    * The main limitation is discoverability. Even if your Github Packages registry-based scoped registry is correctly configured, packages won't show up in the Package Manager UI's All section. GitHub doesn't support any of the NPM search API endpoints (/-/all or /-/v1/search). You will have to resort to GitHub UI to search packages and manually add them to your project manifest.
    * Even if the GitHub Registry package is public, your users will need to provide authentication.

    As you can see, using GitHub Packages Registry with Unity Package Manager is not trivial. But, if you really need it, you can make it work!

    Hope this post will be useful to some of you. Again, thank you to @JakHussain for discovering how it can be done!

    Regards,

    Pascal and the Unity Package Manager team.
     
    asus4, _slash_, NagaChiang and 6 others like this.
  2. Thaina

    Thaina

    Joined:
    Jul 13, 2012
    Posts:
    608
    This is sad. We should have option to use

    ```
    "publishConfig": { "registry": "https://npm.pkg.github.com" },

    // with

    "name": "USER/com.my-company.my-package", // without @
    ```

    Instead. This would be easier to orchestrate cross domain dependency. I may publish project that utilize project of other person or other organization easily. It would critically expand the usage of package manager over everything in one go with just one line of registry
     
  3. JakHussain

    JakHussain

    Joined:
    Oct 20, 2016
    Posts:
    318
    @Thaina that's down to how githubs package registry works rather than how unitys package manager works unfortunately. So far my experience with GitHub Packages has been pretty underwhelming since they really don't make it easy for you to use. Fingers crossed for the future though!
     
    hyagogow, Thaina and okcompute_unity like this.
  4. RobertAcksel

    RobertAcksel

    Joined:
    Sep 20, 2018
    Posts:
    3
    Why this? This excludes GitLab for use since GitLab requires scopes to be present and rejects packages without scopes.
    Seems that your suggested cheat does not work with Gitlab. And cheating should never be necessary (strictly against cheating, makes life harder for all of us).

    'Packages must be scoped in the root namespace of the project.'
    from https://docs.gitlab.com/ee/user/packages/npm_registry/#package-naming-convention
     
    Last edited: May 6, 2020
    watsonsong likes this.
  5. JakHussain

    JakHussain

    Joined:
    Oct 20, 2016
    Posts:
    318
    I would chalk this up to githubs package registry forcing our hand in cheating in this way. As far as I can tell, unitys package manager isn't the bottle neck in how packages are read and pulled in.
     
  6. JesperSmith

    JesperSmith

    Joined:
    Mar 23, 2018
    Posts:
    17
    Thanks for this guide. I tested it and it seems to work. We have the need to share with outside resources and this will help, although a pain to setup.

    We are using some custom tooling to manage/publish NPM packages, and this could be added to it.

    Is there a future where you think packages with the naming scheme @Scope/package-name will be supported? It seems that this is enforced by multiple NPM repository hosts. An option could be to ignore @Scope/ in the name except when pulling from the NPM repo, so the full package name will become "@vendor/tld.vendor.package-name"
     
    watsonsong and Favo-Yang like this.
  7. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    748
    Hi @JesperSmith,

    As you pointed out, the Gitthub Packages Registry is not easy to use with the Package Manager. Since it lacks discoverability API, we will not put more effort to try to be compatible with this registry. Maybe you can reach out to GitHub and ask them to support UPM registries format (including documentation, adapted API, etc.)?

    Regards,

    Pascal
     
    vzlomvl and wcoastsands like this.
  8. JesperSmith

    JesperSmith

    Joined:
    Mar 23, 2018
    Posts:
    17
    Fair enough. I'll see if we can make some extra tooling on our side to make it work well enough for our purposes.
     
  9. RobertAcksel

    RobertAcksel

    Joined:
    Sep 20, 2018
    Posts:
    3
    Unity would not need to support Github natively. We would even be capable of developing an extension on our own to discover packages on any registry which makes this somehow possible via an API. I don't belive GitHub will go implementing UPM registries format.

    Instead we are asking to support the NPM Scoping syntax which we will not be able to support via an extension on our own (except of hacking around it which would end up in writing our own package manager)
    This is because NPM registries like the one of GitLab (not GitHub...) requires a scope for packages and this is why unity editor can not import those packages from GitLab.

    (Maybe there is a better thread for this or there should be one)
     
    Last edited: May 14, 2020
    Zamaroht likes this.
  10. Favo-Yang

    Favo-Yang

    Joined:
    Apr 4, 2011
    Posts:
    456
    I think the issue been discussed a few times. Supporting /-/all/ discovery endpoint is a better-to-have thing. What really blocks this is the scopeRegistry syntax rejects "@" letter. Unity decides to use dot notation as the scope, but other NPM based registry defaults to @Username syntax. It didn't sound like a tech limitation but a design decision that whether @Username conflicts with dot notation. Maybe there're other reasons behind, but for me, below are all valid scopes.
    Sure, there're other registries that won't force you to use @Username as scope for the private package. But it's kinda convention here for NPM, GitHub and GitLab. They're all big players on the market.
     
  11. sbinter_levelex

    sbinter_levelex

    Joined:
    Jul 5, 2018
    Posts:
    9
    Using 2019.3.7f1 we have been unable to replicate this behavior. UPM fails to pull in the manually specified package due to an inability to find it. Using npm install with the same token pointed towards the same URL works perfectly.

    This forum thread is only a little less than 2 months old, so it seems a bit strange that the behavior would already have diverged.
     
  12. TactileGames

    TactileGames

    Joined:
    Oct 6, 2014
    Posts:
    7
    This workaround seems to have been broken by a recent change by Github:

    https://github.community/t/unable-t...t-name-to-publish-url-as-of-07-10-2020/136643

    They no longer allow publishing of packages without the scope in the name.

    @okcompute_unity - do you know of any way that we can work around this and continue to use the Github Package Registry with Unity 2019.4?

    Alternatively, can you provide any suggestions for an alternative private package manager, that does not require us to host our own package manager (i.e. we are not really interested in self hosting verdaccio)?

    Best regards,
    Morten
     
    tactileunity50 likes this.
  13. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    748

    Hi @TactileGames,

    I'm sorry you were affected by this change from Github. Some discussions are happening right now between Unity and Github. I cannot promise anything but fingers crossed!

    In the meantime, a self-hosting solution that I can recommend is JFrog's Artifactory (https://jfrog.com/artifactory/). Easy setup and they offer a free tier that can be interesting for a small team/studio.

    Regards,

    Pascal
     
    JakHussain likes this.
  14. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    748
    I was told this morning by Github that they reverted the change they made:
    https://github.community/t/unable-t...sh-url-as-of-07-10-2020/136643/11?u=jcansdale

    So the method described in this post should work again. Note that I was advised this is temporary but it should work for a while. The next major version of their system may not support this anymore.

    Reminder: Unity doesn't officially support the Github Packages registry. Github did not implement all the npm API UPM relies on for the *scoped registries* feature.

    Regards,

    Pascal
     
  15. ddsim

    ddsim

    Joined:
    Oct 6, 2016
    Posts:
    6
    @okcompute_unity
    Excellent that it works again.

    What is the reason that the Unity Package Manager doesn't allow package names beginning with an `@` scope?
     
  16. bsymon

    bsymon

    Joined:
    Dec 11, 2016
    Posts:
    18
    @okcompute_unity Thank you for the guide ! It works, I succeed to publish my package on GitHub and download it on a project with the scoped registry.

    There is just one issue on the Package Manager. As you said, GitHub doesn't support for searching, but the Package Manager will try anyway and it "crash" it a bit.

    fail.png

    I'm on the "All packages" section, but once it failed to search on GitHub, it won't load any others packages ... It could be nice to just display a warning or fail silently.

    For information, i'm on 2019.4.5f1
     
  17. ImmotionPeter

    ImmotionPeter

    Joined:
    Sep 15, 2020
    Posts:
    21
    I've gotten as far as steps 4/5/6 - not sure which bit is failing. Package is hosted on a private GitHub organization repo.

    Unity gives me.
    I've got the .upmconfig.toml setup in Users/MyLogin/ with
    The project manifest.json is like so:
     
  18. bigcheese_

    bigcheese_

    Joined:
    Sep 6, 2011
    Posts:
    31
    Has anyone tried to have package dependencies and if you need to do the following with the scope @Owner/com.test.dependency ?
     
  19. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    13
    Will the Unity Package Manager support npm’s scope notation in future versions? We use GitLab, and it can host npm packages, but only when using exactly that ("@Scope/package.name") naming convention. So after getting my package published on our GitLab server, I found out that all my work was for nothing because the UPM is not fully npm compatible.
    Should have read this thread first I guess. But seeing that GitHub also tried moving into the scoped naming requirement, are you going to fix this?
     
  20. UnityMaru

    UnityMaru

    Community Engagement Manager Unity Technologies

    Joined:
    Mar 16, 2016
    Posts:
    1,163
    In our view, this is not something that needs to be fixed, but a new feature request that we don't plan to implement at this time. We don't plan to support npm's scope notation in the package names, a package with the name `@foo/bar` is not valid.
     
  21. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    425
    I think the problem is not the scope notation supporting, but the request to support gitlab or github registry.
     
    Zamaroht, Thaina and prisonerjohn like this.
  22. prisonerjohn

    prisonerjohn

    Joined:
    Jul 31, 2013
    Posts:
    27
    Agreed. We want to use Gitlab / Github repos as we don't need to run or host separate servers for it. That seems like a reasonable request, especially as these are the two largest code hosting services used today. If they use `@foo/bar` for package names then maybe Unity should look into supporting that nomenclature.
     
  23. bsymon

    bsymon

    Joined:
    Dec 11, 2016
    Posts:
    18
    I agree, should be supported. I use GitHub for personnal stuff and at work we use a self-hosted GitLab. Would save us tons of time.
     
    Zamaroht and Enderlook like this.
  24. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    425
    If there is no official solution, can add an extension point that let's the community support the gitlab?
     
    Zamaroht likes this.
  25. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    425
  26. Adrian

    Adrian

    Joined:
    Apr 5, 2008
    Posts:
    686
  27. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    425
    I see an article in GitLab about `Store all of your packages in one GitLab project`:
    https://docs.gitlab.com/ee/user/packages/workflows/project_registry.html

    I think it worth to try it in the new GitLab version.
     
  28. frank-ijsfontein

    frank-ijsfontein

    Joined:
    Sep 11, 2015
    Posts:
    13
    Has anyone tried this already on GitLab version 13.9? We're still on 13.4 and would like to get someone to update our server, but only if that actually helps solve this issue. From the docs it's not clear if this is actually implemented yet.
     
  29. watsonsong

    watsonsong

    Joined:
    May 13, 2015
    Posts:
    425
    It delay to 13.10 (2021-03-22)。I am waiting it release and test directly on the gitlab.com.
     
    frank-ijsfontein likes this.
  30. IroxGames

    IroxGames

    Joined:
    Feb 8, 2020
    Posts:
    10
    HOW TO GITLAB PACKAGES

    TO PUBLISH A PACKAGE

    Create a .npmrc file in your package folder with this content:

    registry=https://gitlab.com/api/v4/packages/npm/
    //gitlab.com/api/v4/projects/YOURPROJECTID/packages/npm/:_authToken=YOURDEPLOYTOKEN


    In your package.json add this:

    "publishConfig": {
    "registry": "https://gitlab.com/api/v4/projects/YOURPROJECTID/packages/npm/"
    }


    you can find the project id beneath your project name at the gitlab project page

    TO DOWNLOAD THE PACKAGE

    Create a File at Users/Your Username/.upmconfig.toml
    [npmAuth."https://gitlab.com/api/v4/projects/YOURPROJECTID/packages/npm/"]
    _authToken = "YOURAUTHTOKEN"
    alwaysAuth = true


    Unity package manifest
    "scopedRegistries": [
    {
    "name": "PACKAGE NAME",
    "url": "https://gitlab.com/api/v4/projects/YOURPROJECTID/packages/npm/",
    "scopes": [
    "com.YOURCOMPANYNAME"
    ]
    }
    ],


    Sry for the caps sections, seems like it's not possible to format this bold.
    insert your data in the caps lock fields

    Not supported right now
    Discovery feature in the Unity Package Manager
     
  31. Zamaroht

    Zamaroht

    Joined:
    Nov 4, 2013
    Posts:
    18
    I can confirm that this solution works with Gitlab!
    However it would be great to also support discovery, since the Package Manager UI won't list all the available packages from the Gitlab registry.

    As some people noted before, it would be really amazing if Unity cared to integrate correctly with either GitHub and/or Gitlab registries. These are two very popular code platforms that offer amazing integration and automation with the rest of the repository, making it as easy as merging to a master branch to automatically create a new release version with automatic description as shown here: https://docs.gitlab.com/ee/ci/examples/semantic-release.html

    The correct way to do it right now to have full Unity support requires us to host a npm server ourselves, with all the infrastructure and maintenance overhead that implies for small or lone developers. Maybe the Unity team is not aware of how important this can be to small developers' workflows.
     
  32. Zamaroht

    Zamaroht

    Joined:
    Nov 4, 2013
    Posts:
    18
    Another issue I'm noticing with this approach, is that for some reason the Package Manager UI is not showing me the package names or descriptions, even though they are configured in the package.jsons.

    Edit:

    I just gave a try to AWS CodeArtifact (https://aws.amazon.com/codeartifact/), which allows to host a private npm registry for free if using < 2GB/month bandwidth.
    It's working better than the GItLab private repository. It shows me correctly the package title, description and author in the Package Manager UI. The discoverability doesn't work though, so it only lists installed packages from the registry.
     
    Last edited: Apr 25, 2021
  33. korsour

    korsour

    Joined:
    Apr 27, 2016
    Posts:
    2
    Still having an issue with GitHub, their hotfix with allowing not using scopes if the name contains dots, seems to be removed.

    Trying to publish a package with the name “com.myCompany.core”.
    Getting error “405 Method Not Allowed - PUT https://npm.pkg.github.com/myName/com.myCompany.core

    If I change a name of a package to “@myName/com.myCompany.core” - it’s publishing like a charm. But Unity 2021 (as all the previous) can’t work with scope.

    Is there any workaround to finally proceed with the GitHub registry?
     
  34. ImmotionPeter

    ImmotionPeter

    Joined:
    Sep 15, 2020
    Posts:
    21
  35. Nigey

    Nigey

    Joined:
    Sep 29, 2013
    Posts:
    1,118
    Been following that, thanks for posting it! I'm getting the 405 error though following the example:
    I'm able to publish the package to GitHub, so I know the bearer token is functioning. Furthermore when copying the get request in Unity Editor and trying it in Postman you get the same result, but making a get request for the package directly:
    You get the correct response, returning the package.json file for it.

    Thoughts?
     
    ImmotionPeter likes this.
  36. ImmotionPeter

    ImmotionPeter

    Joined:
    Sep 15, 2020
    Posts:
    21
    The last section of the blog is about those errors, I ignore them. With those errors there, I have our github hosted packages in the package manager and refreshing for updates, which can be imported without issue.

    To get them to show the first time I have to:
    - add the scoped registery
    - manually add the package in the manifest.json
     
  37. BlackclawsK

    BlackclawsK

    Joined:
    Jan 9, 2019
    Posts:
    24
    I'm also a bit flabbergasted that the unity package manager does not support @Scope notation which has been the defacto standard way to annotate npm packages for quite a while now. It took me quite some time to realize that this was actually the issue as there is also no error presented by the package manager when you add an @Scope.

    It would be really interesting to know what exactly the blockers are for this, as its being called by Unity Devs, "new feature". It appears to me that package resolution containing an @xyz in the beginning would be just as easy to regex/search for than reverse dns notation.

    We are also using gitlab internally and not being able to use the instance level endpoint for packages is an issue as we now have to add a ton of scoped registries instead.

    Edit: Just to clarify, this is not even about search functionality which as I understand needs a /all endpoint which Gitlab does not provide, but about basic package resolution when they are manually added to manifest.json or via the package manager add by name feature

    Edit 2: My guess regarding the @ is that its already being used as denoting the package version in the PackageCache folder, but in that case you could just default to the last @ in order to fix that issue
     
    Last edited: Jul 9, 2021
unityunity