Search Unity

  1. Unity 2018.3 is now released.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. We've updated our Terms of Service. Please read our blog post from Unity CTO and Co-Founder Joachim Ante here
    Dismiss Notice
  4. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  5. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
2018.3 released Bundle up and save Updated Terms of Service Provide feedback Live virtual classes

Setup for scoped registries (private registries)

Discussion in 'Package Manager' started by rizu, Oct 25, 2018.

  1. rizu

    rizu

    Joined:
    Oct 8, 2013
    Posts:
    678
    2018.3.0b7 added following:
    We already got informative post about how to use git from package manager here:
    https://forum.unity.com/threads/syntax-for-git-paths-on-package-manager.573673/#post-3819487

    Any chance to we could also get some brief introduction on how to setup scoped registries (private registries)?
     
    rizu, Oct 25, 2018
    #1

  2. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Hello @rizu,

    Sure. Here it is! :)

    Introduction
    The concept of scopes is borrowed from npm (https://docs.npmjs.com/misc/scope), it allows a subset of packages to be hosted on a registry other than the default.

    Configuration (manifest.json)
    Project manifests (manifest.json) can now contain any number of scoped registry configurations in the scopedRegistries attribute. This new configuration does not supersede the registry attribute or the UPM_REGISTRY environment variable; those will still be used as the default when a package matches none of the configured scopes.

    The scopes attribute of each configuration is an array of strings that can be mapped to a package name in two ways:
    1. As a namespace. This type of configuration assumes that packages follow the Reverse Domain Name Notation (https://en.wikipedia.org/wiki/Reverse_domain_name_notation). In other words, a scope such as com.unity is the equivalent of com.unity.*
    2. As an exact package name match.
    When deciding in which registry a given package should be searched, Package Manager will settle on the scope that is the closest match to the requested package name.

    The snippet below demonstrates how scope selection works:

    Code (JavaScript):
    1. {
    2.   "scopedRegistries": [
    3.     {
    4.       "name": "Main",
    5.       "url": "https://my.company.com/registry",
    6.       "scopes": [
    7.         "com.my-company", "com.my-company.tools.packageX"
    8.       ]
    9.     },
    10.     {
    11.       "name": "Tools",
    12.       "url": "https://my.company.com/tools-registry",
    13.       "scopes": [
    14.         "com.my-company.tools"
    15.       ]
    16.     }
    17.   ],
    18.   "dependencies": {
    19.     "com.unity.cinemachine": "1.0.0",
    20.     "com.unity.2d.common": "1.0.0",
    21.     "com.unity.2d.animation": "1.0.0",
    22.     "com.my-company.package1": "1.0.0"
    23.   }
    24. }
    With the above configuration:
    1. com.my-company.packageA
      . will be fetched from Main because it is in the com.my-company.* namespace.
    2. com.other-namespace.packageX
      will be fetched from Main because its name is an exact match. It also matches com.my-company.tools.* (Tools) but that is less specific.
    3. com.my-company.tools.animation
      will be fetched from Tools because it is in the com.my-company.tools.* namespace. It also matches com.my-company.* (Main) but that is less specific.
    4. com.other-company.packageA
      will be fetched from the default registry because it matches none of the configured scopes.
    Why "scoped" registries?
    It's to ensure determinism when deciding from which registry a package should be fetched. By using scopes, a package will always be mapped to one and only one registry, guaranteeing a consistent result regardless of network conditions.

    Supported Registries type
    Unity Package Manager supports npm protocol based registries. You can use any off the shelves npm registries server and it should work.

    These are the registries that we know should work
    Search all package limitation

    Some npm registry server does not support the 
    /all
    route to search all packages. Displaying scoped registry packages as part of the "All packages" list relies on scoped registries to support the old npm API protocol, which has an HTTP endpoint that returns the metadata of all published packages (eg: https://registry.my-company.com/-/all). When a registry does not support the old protocol, packages from that registry will simply not be displayed in the UI. However, this limitation does not apply to package resolution, so packages from scoped registries can still be manually added to the project manifest.

    Limitations
    • In the Package Manager window, the "All packages" tab will now display and allow you to install packages found in scoped registries. But, they will be mixed with the default Unity registry. There is no differentiation for now. In future releases of the UI, the source and the scope of each package will be made clearly visible.
    Thanks for trying out this feature. Let us know if you have any issues. Look for a production quality release in 2019.1.

    Regards,

    Pascal
     
    Last edited: Dec 14, 2018
    okcompute_unity, Oct 25, 2018
    #2
    pedro_unity, M_R, dzamani and 1 other person like this.

  3. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    311
    Hi,
    how does it work with registries/packages that requires auth? it picks up whatever 
    npm login
    you do in the terminal? or requires env variables / cli startup?

    also do you plan to integrate with the unity account management (so it picks the credentials I am logged in unity with)?
     
    M_R, Oct 25, 2018
    #3

  4. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Hi @M_R ,

    The Package Manager does not support authentication for now. Your repository needs to be public. One thing you can do is limit access to people inside your LAN.

    If you have a use case that is common, we could look into adding this feature. Can you expand a little bit more on why you would need auth on a local/private registry?

    Regards,

    Pascal
     
    okcompute_unity, Oct 25, 2018
    #4

  5. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    311
    to prevent unwanted access to the packages, the same way we have auth required on bitbucket (ssh, it works btw).
    also we need access at least from:
    - our office
    - our homes (or anywhere we may be, i.e. if we need a urgent hotfix in the weekend)
    - cloud build

    the best pattern would be to link it with our organization for Unity account, so anyone (plus cloud build) can have access without modifying either env or manifest.json.

    for the main registry, I can access private stuff by modifying the manifest, but it feels hacky (and doesn't support multiple users without gitignoring it):
    Code (CSharp):
    1. {
    2.     "registry":"https://blablablabla",
    3.     "always-auth":true,
    4.     "email":"hardcoded@email.com"
    5.     "_auth":"<hardcoded basic auth>"
    6.     "dependencies":{...}
    7. }
     
    M_R, Oct 26, 2018
    #5
    Ricky-Tonic likes this.

  6. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Thank you @M_R for the feedback! Will transfer your request to our product manager to validate priority.

    Regards,

    Pascal
     
    okcompute_unity, Oct 26, 2018
    #6

  7. Adrian

    Adrian

    Joined:
    Apr 5, 2008
    Posts:
    309
    Easiest would be if Unity supported the standard npm authentication.

    i.e. a user would log in with npm adduser REGISTRY, which updates the .npmrc file in the user's home directory and Unity would pick up those credentials.

    This would mean existing private npm registry setups could be used with Unity without having a separate authentication process.
     
    Adrian, Nov 1, 2018
    #7
    M_R likes this.

  8. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Hi @Adrian ,

    I took note of your suggestion.

    Thanks,

    Pascal
     
    okcompute_unity, Nov 1, 2018
    #8

  9. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    311
    as @Adrian says, plus the ability for 
    npm adduser ...
    in the cloud build
     
    M_R, Nov 2, 2018
    #9

  10. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    @M_R I just wanted to give you a follow-up on this feature request. We analyzed it, and it will require a bit of work to provide a good user experience (persist credentials, env vars, login window, etc.). The complexity is a bit greater than I would have anticipated. I hope we can implement this feature sometime in 2019. We have a couple of higher priority issues to address in our backlog before attacking this one. As for your request to support Unity auth on scoped registries, it is not something we have in our roadmap. Sorry.

    @Adrian Even though Package Manager is compatible with *npm* web API, it is not the same technology underneath (e.g. we don't share the same configuration scheme for scoped registries)

    Regards,

    Pascal
     
    okcompute_unity, Dec 4, 2018
    #10

  11. dzamani

    dzamani

    Joined:
    Feb 25, 2014
    Posts:
    48
    Hi @okcompute_unity,

    I was wondering if you could ask about another way to do auth with packages. The idea would be to have a callback everytime the package manager try to access scoped registries. That callback would be called for each scoped registry and should return a boolean saying if it has access or not to it.

    Something like this:
    Code (CSharp):
    1. bool HasAccessToRegistry(string registryUrl)
    2. {
    3.     var authToken = FetchToken();
    4.     if (registryUrl == "someurl" && authToken.IsValid())
    5.         return true;
    6.     return false;
    7. }
    The FetchToken would be something the user will develop.

    Now I know the management of the registry isn't done at the package level but somewhere in the engine (if I'm not wrong) so having that callback may not be that easy based on your architecture but that would be perfect for us to do what we want about these registries.

    Auth is the main reason why we can't have public registries since we can't share our code but having LAN only packages is preventing us from remote working (well not absolute prevention but still it's very painful to grab each package manually from their git repo when you are at home).

    Anyway if you could ask about the feasibility of this solution, that would be great, thanks!
     
    dzamani, Dec 7, 2018
    #11

  12. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Hi @dzamani ,

    Thank you for your suggestion. We have a well-defined plan to implement this feature. This is only a matter of prioritization. If you have an enterprise support account, you can contact your assigned account manager to express the value for your company to make this feature available sooner.

    Regards,

    Pascal
     
    okcompute_unity, Dec 7, 2018
    #12
    dzamani likes this.

  13. nhold

    nhold

    Joined:
    May 2, 2017
    Posts:
    50
    Will there\could there be support for just a git repository representing a registry? I do this method with my custom package manager and it makes it really easy for anyone to make one.
     
    nhold, Dec 19, 2018
    #13

  14. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    @nhold Can you expand a bit more on your setup? Do you have examples? Or a public repository that you can share?

    Thank you!

    Pascal
     
    okcompute_unity, Dec 19, 2018
    #14

  15. nhold

    nhold

    Joined:
    May 2, 2017
    Posts:
    50
    Obviously my setup will be a lot simpler but you can see the repo for the package manager itself here:

    https://github.com/nhold/ubPackageManager

    Essentially the idea is a 'registry' is just a git repo with lots of `Package Definition Files` which is defined as:

    Code (JavaScript):
    1. {
    2.     "name": "ubGridArray",
    3.     "versions": [{
    4.         "version": "1",
    5.         "branch": "version-1"
    6.     }],
    7.     "description": "1D array as 2D array.",
    8.     "location": "https://someurl/ubgridarray.git",
    9.     "parentDir": "Bifrost",
    10.     "childDir": "ubgridarray/Assets/Plugins/ubGridArray"
    11.      "dependencies": [
    12.         "ubConfig"
    13.     ]
    14. }
    So to read from the registry you just pull\clone it and read the json files to populate the data.
     
    nhold, Dec 21, 2018
    #15

  16. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Interesting. We have no plan to support such a feature. I just don't want to give you false hope. But will see. If hosting out of the shelve npm registries turns out to be too hard/complex for people, we could consider this alternative approach.

    Btw, were you aware that the Unity Package Manager supports Git URLs project dependencies? https://forum.unity.com/threads/git-support-on-package-manager.573673/

    Thanks a lot for sharing. This gives me another perspective :)

    Regards,

    Pascal
     
    okcompute_unity, Dec 21, 2018
    #16
    nhold likes this.

  17. dtaTrifork

    dtaTrifork

    Joined:
    Aug 21, 2017
    Posts:
    3
    What are the rules about version matching within scoped registries?

    Traditionally we would base rules on the NPM protocol found here: https://docs.npmjs.com/files/package.json

    Suppose you have the following packages:
    - A in versions 1.0.3, and 1.2.0
    - B depends on A version 1.0.3
    - C depends on A version 1.2.0

    Is this something we can expect the package manager to be able to handle and correctly isolate? As far as my testing illustrates the package manager will default to the highest versioned dependency.

    Finally, the dependency on package A seems to be a hidden package within the `Library/PackageCache` along with the remaining packages. But said dependency will not be illustrated within unity.
     
    dtaTrifork, Jan 15, 2019
    #17

  18. okcompute_unity

    okcompute_unity

    Unity Technologies

    Joined:
    Jan 16, 2017
    Posts:
    289
    Hi @dtaTrifork ,

    The package manager conflict resolution algorithm will always choose the highest version number requested unless the package is a root dependency (i.e. you explicitly added the package to your project). We are working on documenting the conflict resolution algorithm. Coming soon.

    The Package Manager UI does not show transitive dependencies. This is something we have on our roadmap.

    Regards,

    Pascal
     
    okcompute_unity, Jan 15, 2019
    #18
    technicat likes this.