Setup for scoped registries (private registries)

2018.3.0b7 added following:

We already got informative post about how to use git from package manager here:
https://forum.unity.com/threads/syntax-for-git-paths-on-package-manager.573673/#post-3819487

Any chance to we could also get some brief introduction on how to setup scoped registries (private registries)?

 

Hi,
how does it work with registries/packages that requires auth? it picks up whatever

npm login

you do in the terminal? or requires env variables / cli startup?

also do you plan to integrate with the unity account management (so it picks the credentials I am logged in unity with)?

 

to prevent unwanted access to the packages, the same way we have auth required on bitbucket (ssh, it works btw).
also we need access at least from:
- our office
- our homes (or anywhere we may be, i.e. if we need a urgent hotfix in the weekend)
- cloud build

the best pattern would be to link it with our organization for Unity account, so anyone (plus cloud build) can have access without modifying either env or manifest.json.

for the main registry, I can access private stuff by modifying the manifest, but it feels hacky (and doesn't support multiple users without gitignoring it):

Code (CSharp):

1. {
2.     "registry":"https://blablablabla",
3.     "always-auth":true,
4.     "email":"hardcoded@email.com"
5.     "_auth":"<hardcoded basic auth>"
6.     "dependencies":{...}
7. }

 

Easiest would be if Unity supported the standard npm authentication.

i.e. a user would log in with npm adduser REGISTRY, which updates the .npmrc file in the user's home directory and Unity would pick up those credentials.

This would mean existing private npm registry setups could be used with Unity without having a separate authentication process.

 

as @Adrian says, plus the ability for

npm adduser ...

in the cloud build

 

Hi @okcompute_unity,

I was wondering if you could ask about another way to do auth with packages. The idea would be to have a callback everytime the package manager try to access scoped registries. That callback would be called for each scoped registry and should return a boolean saying if it has access or not to it.

Something like this:

Code (CSharp):

1. bool HasAccessToRegistry(string registryUrl)
2. {
3.     var authToken = FetchToken();
4.     if (registryUrl == "someurl" && authToken.IsValid())
5.         return true;
6.     return false;
7. }

The FetchToken would be something the user will develop.

Now I know the management of the registry isn't done at the package level but somewhere in the engine (if I'm not wrong) so having that callback may not be that easy based on your architecture but that would be perfect for us to do what we want about these registries.

Auth is the main reason why we can't have public registries since we can't share our code but having LAN only packages is preventing us from remote working (well not absolute prevention but still it's very painful to grab each package manually from their git repo when you are at home).

Anyway if you could ask about the feasibility of this solution, that would be great, thanks!

 

Will there\could there be support for just a git repository representing a registry? I do this method with my custom package manager and it makes it really easy for anyone to make one.

 

Obviously my setup will be a lot simpler but you can see the repo for the package manager itself here:

https://github.com/nhold/ubPackageManager

Essentially the idea is a 'registry' is just a git repo with lots of `Package Definition Files` which is defined as:

Code (JavaScript):

1. {
2.     "name": "ubGridArray",
3.     "versions": [{
4.         "version": "1",
5.         "branch": "version-1"
6.     }],
7.     "description": "1D array as 2D array.",
8.     "location": "https://someurl/ubgridarray.git",
9.     "parentDir": "Bifrost",
10.     "childDir": "ubgridarray/Assets/Plugins/ubGridArray"
11.      "dependencies": [
12.         "ubConfig"
13.     ]
14. }

So to read from the registry you just pull\clone it and read the json files to populate the data.

 

What are the rules about version matching within scoped registries?

Traditionally we would base rules on the NPM protocol found here: https://docs.npmjs.com/files/package.json

Suppose you have the following packages:
- A in versions 1.0.3, and 1.2.0
- B depends on A version 1.0.3
- C depends on A version 1.2.0

Is this something we can expect the package manager to be able to handle and correctly isolate? As far as my testing illustrates the package manager will default to the highest versioned dependency.

Finally, the dependency on package A seems to be a hidden package within the `Library/PackageCache` along with the remaining packages. But said dependency will not be illustrated within unity.

 

Hi,
I am setting up Unity Package Manager to work with a private npm registry.
I am getting an error:

An error occurred while resolving packages:
Project has invalid dependencies:
mypackage: self signed certificate in certificate chain

I can pull the package from the registry using NPM fine, how can I specify the cafile for the Unity package manager?

 

Hi @okcompute_unity,

I would love to see standard .npmrc authentication support as well. That would allow our organization to connect to an Azure Artifacts NPM feed, which can only be connected to with credentials.

 

@okcompute_unity @benoitv_unity I have a question about versioning format.

Suppose I have two packages:

Package A: 1.0.0
Package B: 1.0.0

"Package B" has a dependency on "Package A" on version "1.0.0".

1. I set in my project "manifest.json" a dependency on "Package B -> 1.0.0".
2. I make an update to my "Package A" and set a "2.0.0" version.
3. I add to my project "manifest.json" a dependency on "Package A -> 2.0.0".
4. What dependency is getting my "Package B" now? "1.0.0" or "2.0.0"?

Also I want to know if it's possible to have dependencies on this format: https://docs.npmjs.com/files/package.json#dependencies

I tested ">=1.0.0" and I got the following error: Version is invalid. Expected a 'semver' compatible version

 

No problem.

Ok, thanks.

It would be nice to document that as it's a confusing concept.

Ok, thanks.

 

I've setup a local NPM registery to try this out, but have been running into some local iteration workflow issues.
Say I have `com.company.test` published at `0.0.1`, and a unity test project that has this as a dependency in its manifest using the scoped registery.
I now want to make some changes to my package, and see these changes in my unity test project. This requires me to up the version in the package.json, the manifest json, and republish the package. Is there an easier way to do this?
I could use `file:` to directly use the package, but this would require me to change it back to a version if I ever want to test my project and package in CI, or publish it for the public.

So advise is greatly appreciated.

 

I will give the symlinking method a try and see how this goes. How does the unity team work with this for packages currently developed?

If the manifest contains a package defined by a version (not a

file:

) but exists in the local

Packages

folder, does it use the local copy instead?

 

I've been struggling to get Artifactory to work with the Unity package manager. I can create npm repos and successfully publish to them with npm. However there doesn't appear to be any artifactory-recommended URL that works in the scopedRegistries section of the manifest.json. I also noticed that Artifactory doesn't appear to support "/all". Could you explain how you have configured Artifactory internally at Unity and what is the recommended URL to use in the manifest.json? We're running Artifactory Pro 6.8.7 and the most current (as of this writing) version of Unity.

 

Thanks for the reply. Any chance you'll be changing the Package Manager to better support Artifactory and listing packages? We use artifactory but want to have a good user experience when adding/updating packages, so for now we will deploy verdaccio which I've been testing with. But not what you would call optimal.

Another question/problem. When I update my package in verdaccio, the Package Manager correctly shows that a new version is available. However, when I update, it doesn't always recompile the scripts. It seems to do it if I've not updated for some time (like first update in the morning), but if I do it too frequently, it just says it have updated but the scripts are not compiled. My npm package includes meta files with guids and timeCreated fields (which are updated appropriately). Any help would be greatly appreciated.

 

Yes, we're using npm 6.7.0. We should be able to use the older version, thanks for the tip! Also, thanks for the +1 for the package manager artifactory feature!

 

About Scoped Packages,
Following documentation from NPM the scoped package has to have a name in this way:
https://docs.npmjs.com/creating-and-publishing-an-org-scoped-package
So in package.json

"name": "@com.vos.test/com.vos.test.packagethree"

But Unity is not able to load Scoped packages in this way even I link them locally. Why is that?
Error message:

Code (CSharp):

1. An error occurred while resolving packages:
2.   Project has invalid dependencies:
3.     com.vos.test.packagethree: Package [com.vos.test.packagethree@1.0.0] cannot be found

 

@okcompute_unity Ok, I see. Thank you for the fast answer.
It is working if I do not use npm scope registries.
But functionality with npm `npmjs` standard registry should be granted?

 

I will put this here if anyone has the same issue as we had:

In versions < 2018.3, we were using the following parameters to authenticate:

Code (csharp):

1.  
2. "registry": "http://the-registry-url/",
3. "email": "the-email",
4. "always-auth": true,
5. "_auth":"the-auth"
6.  

This does not work anymore in newer version *but* using

Code (csharp):

1. registry": "http://user:password@the-registry-url/"

*does* work.

 

Still no date for support for password protected private registries ? I have to use the import from folder feature for now to manually import my packages, a little bit bothersome although it is a huge up upgrade compared to no dependency management system, happy to see Unity also think about developers as well.
Also excited about the extra CLI commands that will come during the year, those features made me reconsider quitting unity, please keep up the good work !

 

I'm curious if this approach still works for Unity 2018.4 or newer. My team would like to stay on the 2018.4 LTS and we were looking at having credentials embedded via the http://user:password@the-registry-url/ approach, however I'm finding it doesn't seem to consistently pass the credentials through. It'll correctly list the packages, but fail to actually pull it down (if it's not already cached on disk). This is using Verdaccio as the back-end, I can see in the logs that the user-password is only forwarded on some of the requests.

Since we have several offices globally it would be preferable if we can have a public package registry e.g. setup with Verdaccio but it absolutely needs authentication. I've tried to go as far as patching the managed package manager DLLs at runtime to pass in an 'Authorisation' HTTP header for JWT tokens but unfortunately the requests are done via a native binary which isn't something we can easily patch in memory.

We're really quite keen to use the package manager but we may have to look at using a Nuget alternative due to this limitation.

Edit:

Currently investigating using some sort of proxy via AWS to handle authentication, don't want to give up yet as I personally do really like the idea of the built-in package manager!

 

+1

edit: i did some tests with a local registry (verdaccio), and these are the requests it makes:

when opening the window:

Code (CSharp):

1. http <-- 200, user: <user>(127.0.0.1), req: 'GET /-/all', bytes: 0/526
2. http <-- 200, user: <user>(127.0.0.1), req: 'GET /com.xxx.test', bytes: 0/2497
3. http <-- 200, user: <user>(127.0.0.1), req: 'GET /com.xxx.test2', bytes: 0/1116
4. http <-- 200, user: <user>(127.0.0.1), req: 'GET /com.xxx.test', bytes: 0/2497
5. http <-- 200, user: <user>(127.0.0.1), req: 'GET /com.xxx.test2', bytes: 0/1116
6. http <-- 401, user: null(127.0.0.1), req: 'HEAD /com.xxx.test/-/com.xxx.test-3.0.0.tgz', error: authorization required to access package com.xxx.test

when trying to upgrade a package:

Code (csharp):

1. http <-- 401, user: null(127.0.0.1), req: 'GET /com.xxx.test/-/com.whatwapp.test-3.0.0.tgz', error: authorization required to access package com.xxx.test

the HEAD and GET request for the package archive are not authenticated

(I had version 1.0.0 cached, so it didn't trigger any server requests)

I hope at least this can be fixed soon

 

I can't get the packages on my registry to show up on the UI. But I have managed to manually validate the json response for /-/all using curl. May I know what tags/keys that the UI uses to filter packages?

 

@okcompute_unity, I just want to add my voice to the calls for auth support. I've gone the verdaccio route setting up a scoped registry, and we're really excited to start setting up our own internal packages. We just can't have them public.

 

+1

Same problem here.

 

Hi! I've a verdaccio running in docker locally just to test this but I guess I'm missing something.
I have this in my manifest.json

Code (CSharp):

1. "scopedRegistries": [
2.     {
3.       "name": "MyModules",
4.       "url": "http://0.0.0.0:4873/",
5.       "scopes": [
6.         "@etermax"
7.       ]
8.     }
9.   ]

And I've a published package called @etermax/protoz-core and I cannot see it in the package manager.
If I enter to http://0.0.0.0:4873/ I can see the package and doing http://0.0.0.0:4873/-/all can see all my packages (in this case only one)