Custom Package with Git Dependencies

Now this is stale again. Are there any progress at all? Could you just make works about this feature be more transparent?

 

This is the second quite big road block in 2 days. And as far as I can see now i have to set up an NPM server and probably pay for hosting just so my Untility package can be downloaded, when somebody uses the main package.
I get that unity is focusing on bug fixes... but in my eye this is a big bug.

 

GH and GL both have package registries so you can serve NPM packages from them. I have used GL for standalone packages in the manifest but not as dependencies yet.

 

Hi,

any news?

 

This issue is a big deal to package creators.
I'm hesitating on using third-party dependency resolvers, since they rely on Unity's internal api which can change any moment.

 

Why is this still a thing in 2021? a package manager that cannot resolve dependencies? can we git ssh key per domain too please? what a pain for such a basic features.

 

+1

The prospect of pulling out some commonly used utilities into packages seemed attractive! Just uploading each of them to a separate git repo and have the dependency of one utility on the other resolved automatically.

For me that's the main use case..

The only other simple solution is to just create one monolithic "utilities" package? Makes it a lot less attractive to go down the upm road. I guess I'll just keep copying files from one project to the next.

 

Any news on this one?

 

Since it looks like we are not gonna have UPM handling git dependencies, the better option at the moment is using Scoped Registries, where you would create NPM packages out of your Unity Package and upload it to a NPM service and add the registry URL to your list of Scoped Registries on the Unity Package Manager Settings.

I did test some services and the better option for having a private Registry was JFrog in my opinion.

I'm writing a document with my Pros and Cons when using them (Verdaccio, JFrog, Gitlab, AWS CodeArtifact, NPM). And when I finish it I will share it here.

 

There is still the problem of not being able to declare the scope of dependencies in the

package.json

itself.

 

There is no problem at all when using Scoped Registries.
You can declare all your dependencies in the package.json exactly how Unity does with their packages. If they are found in any registry added to your PackageManager settings (Unity registry is done automatically), all of them will be resolved automatically.

That is how I do it:

package.json

Code (json):

1. {
2.   "name": "com.mycompanyname.mypackagename",
3.   "version": "1.0.12",
4.   "displayName": "My Package Name",
5.   "description": "My package description",
6.   "keywords": [
7.     "mypackagename"
8.   ],
9.   "author": {
10.     "name": "Packages with the same author are grouped on UPM window"
11.   },
12.   "dependencies": {
13.     "com.unity.localization": "1.0.1",
14.     "com.unity.addressables": "1.19.6",
15.     "com.mycompanyname.myutilities": "1.0.7",
16.     "com.othercompanyname.singleton": "1.0.1"
17.   }
18. }

Packages/manifest.json

Code (json):

1. {
2.   "scopedRegistries": [
3.     {
4.       "name": "My JFrog Registry",
5.       "url": "https://mycompany.jfrog.io/artifactory/api/npm/unity-npm-local",
6.       "scopes": [
7.         "com.mycompanyname",
8.         "com.othercompanyname"
9.       ]
10.     }
11.   ]
12. }

 

The problem is that if your scoped registry package depends on a second scoped registry, both scopes will have to be defined at the project level.

Would love to be proven wrong.

 

Are you talking about the

"name": "@ scope/packagename"

issue?
If it is not about that specific issue (which I still don't fully understand), I believe it is correct to add every scoped registry since Unity needs to know from where it should download the dependencies.

btw, I'm not an expert at NPM scoped registries, I have started looking into it this year just to make it work on Unity and it really works perfectly with JFrog for my needs.

 

@sandolkakos Are you still working on your pros/cons document, and is JFrog still your preferred solution?

I just started working on migrating some code to UPM packages stored in private GitHub repositories, and I would like to establish a private scoped registry that will serve packages from those private repositories.

I have minimal experience working with NPM scoped registries and no experience setting them up. Any information you share would be greatly appreciated.

 

I've read in a few posts that seem to suggest JFrog (like the GitHub Packages registry) doesn't allow searching the registry for available packages, but that it does resolve dependencies fine. Is that the case? It would be great to have the Unity Package Manager UI show all packages available on the private registry.

 

Can't speak for JFrog Artifactory but I've been running a Verdaccio instance and that has worked well. Package info, listing and searching works on the Unity side and it's relatively straightforward to install (I have it running in a docker container). I'm using Git hooks to automatically publish packages whenever I push a new tag.
Verdaccio does seem to be geared more towards single-instance smaller setups and Artifactory more towards being part of a bigger infrastructure.

 

@Michael-Ryan
I was formatting the text and also trying to find a place to post it. So I used Gatsby + GitHub Pages.

That is a compilation of my findings regarding the use of the Unity Package Manager + package dependencies:
https://sandolkakos.github.io/my-blog/upm-studies/

 

@sandolkakos thanks for the summary, it is very useful!
I managed to get public npmjs packages to be visible in UPM though.
For me, the problem was that one of the scope names was too long and it was not discoverable (e.g. com.supersuperlongname.name1.name2). After shortening this name it worked

 

Hey @ekyah411, nice to know that you like my post.

Hmm, I haven't thought about long names. Could you try my packages with the settings you use:
com.sandolkakosdigital.myunitypackage01
com.sandolkakosdigital.myunitypackage02

they are hosted here:
https://www.npmjs.com/settings/sandolkakos/packages

 

Wowww, I can confirm it.
I just created another package with a shorter company name "com.sandolkakos.myunitypackage03" and it is now showing on the UPM window.
Thanks for the hint @ekyah411, I will include it in my blog post.

 

Bump...?
I'm trying to create incremental packages that I can reuse in my projects. Is there some easy and obvious solution that I'm missing instead of making a package for each piece of functionality?

 

You can create packages with the structure you decide. But I strongly recommend following that layout:
- https://docs.unity3d.com/Manual/cus-layout.html

We have a lot of ways of creating/importing a Package using the Unity Package Manager, but maybe as a starting point for working with packages, you could do one of those (sorted by difficulty):

1. Create a new folder inside your `UnityProject/Packages` with a package.json
- you will see the folder inside your Unity Editor, as a child of the Packages folder.
- do what you want inside it
- commit it normally with your game code (same project and not shared)

2. Create a git repository with a package.json at the root of the repo
- clone it to a specific folder of your PC
- import that folder using the Unity Package Manager > Add package from disk...
- do what you want inside it
- commit and push the content to your specific remote repo
- In the project you just want to reuse:
- import the git package using the Unity Package Manager > Add package from git URL...

3. When you start to understand more about how the packages are handled, you will for sure want to learn how to use the Scoped Registries to store your packages.

 

How about your ETA? Permission granted to get this feature in Unity 2022

 

Any news on resolving git dependencies per package? I’d love to be able to write packages of code for my projects that depend on each other and then install one and the package manager automatically install the other packages from github that the installed package depends on.

 

It is totally possible. For that you need to store your packages in an NPM service and make use of the Unity Scoped Registries to add the URL where your packages are stored, so that the Unity Package Manager will be able to download them and resolve dependencies as well.

I wrote a blog text about how I have learned thst and maybe it can give you an idea about how it works:

• https://sandolkakos.github.io/my-blog/upm-studies/

 

They are Git URLs, which means he doesn't have a registry, and he is using Git URLs instead. Not a registry, but Git URLs. I repeat: Git repos as Git URLs, not an NPM-like registry.

Unity doesn't have native support for this, and currently, the only way to make it work is by using something like this: https://github.com/mob-sakai/GitDependencyResolverForUnity

I don't know why you are doing it, but please stop dodging the issue.

 

ohhh, totally my bad. I was reading that on my cell phone and totally missed the "git" word in joshrs926 question. Okay, git dependencies are not possible, sorry for my misleading.

I'm just sharing a knowledge that I didn't have before and was bringing so much pain to my brain every time I wanted to keep on the same track with git URLs.

Should I wait for Unity to redesign their whole UPM system (if they would do that someday) or should I try anything else to have a good workflow in my projects?
Well, if I have the Scoped Registries to help me with what I need, I will of course make use of it instead of trying to use half of the functionalities I would have by using git URLs.

I believe there are more people in the same situation I was in before, so talking about it really helps those guys. But if one does not want it, that is fine. There are lots of people that would like to develop games in Unity using Python as the programming language, but since Unity uses C#, what can those guys do about it?

 

Not in this thread because this thread from start to now only related to
Custom Package with Git Dependencies

 

My point from more than two years ago still stands – don't plan on this feature arriving soon, find a good workflow with what's available now. As @sandolkakos has outlined.

 

we still hope, this + custom dependency for specific samples within the package.

 

2023 is about to be started, yet no bugfix would be more important for me than custom package git dependancies in UPM. Is there gonna ever again be any update on this?

Even deny this feature development so I know I should abandon my hopes?

 

Unity Are you there? Are you listening? For whom are you creating your product? Isn't that supposed to be us? Please add this feature. The Scoped registries are not equal to what we are asking here. Assume I'm developing two different projects on git, one depending on the other one, for each small change in the first repo, you want me to make a tar file and deploy it to a npm scoped registry? while that could be as easy as committing and pushing?
It's about FOUR years from the day you have confirmed that what we are asking does make sense. Are you there? If yeah for who?

 

BUMP! Unity pls fix this already

 

When I recently came back to Unity and GamdeDev in general I saw the potential to Custom Packages but was also quickly disappointed by how they couldn't easily resolve dependencies. I ended up making a very simple tool to work around the issue. Essentially you just clone all of your packages to a single local folder and then you can point this tool to it and use that to quickly install which packages you need into a project either via file copy or symlinks depending on needs. It's been quite handy for myself lately even in its limited capacity. Perhaps someone else will find use for it. Here is a link to the source code.

https://github.com/Slugronaut/UnityPackageInstaller

It was a rush job so don't expect anything fancy. It's also Windows only as I wrote it using WPF. It currently can resolve simple dependencies between custom packages that are found within the folder but doesn't pull anything from the internet. Perhaps some day I'll consider adding that functionality but atm I don't have time. Maybe it will help someone in the meantime.

 

@maximeb_unity soooo we got any news?

 

So for anyone wondering, I do have a hacky way of making dependencies work...

Code (CSharp):

1.  
2. public class Setup
3.     {
4.         [InitializeOnLoadMethod]
5.         public static async void InstallDependencies()
6.         {
7.             var value = Client.List(false, true);
8.  
9.             while (!value.IsCompleted)
10.                 await Task.Delay(100);
11.  
12.             foreach (var item in value.Result)
13.             {
14.                 if (item.name == "com.username.package")
15.                     return;
16.             }
17.  
18.             Debug.LogWarning("The dependency 'com.username.package' is not installed! Installing...");
19.             Client.Add("https://github.com/gitusername/gitpackage.git");
20.         }
21.  

It literally checks every time whenever there was a compilation triggered and searches for if the required dependency is in the project. If not it installs it forcibly...

Just put it inside the Editor assembly of your package that requires the dependencies!

 

Four years later, and we're still asking for this.

 

I have recently stumbled upon this repo - https://github.com/needle-tools/hybrid-packages

Which seems it could help. It does in my use case.

 

Thank you for the suggestion, but it seems like this approach involves .unitypackage files, which is exactly what I'm trying to shy away from. I would like to use UPM, and for it to solve some (private) git dependencies in packages.

 

It's been so long and I finally gave in and made my own tool to handle the git dependencies. I wish Unity would soon release native support for this and make this tool redundant.

Here's a link to the repository: https://github.com/hibzzgames/Hibzz.DependencyResolver

Basically, package developers can define dependencies under a new field in the package.json file called git-dependencies and list the dependencies. Like this:

Code (JSON):

1. "git-dependencies": [
2.     "https://github.com/username/dependency1.git",
3.     "https://github.com/username/dependency2.git"
4.   ]

And the tool will automatically download the dependencies (after prompting the user for security reasons). Let me know if you guys find it useful.

 

It looks like a nice approach, I've been working on a similar solution 2 years ago.
I'm posting more to give you a suggestion... for people that use to execute the builds in Batch Mode, you will probably need to put a check like `if (Application.batchMode)` to bypass the confirmation dialog.

 

In the meantime I would like to present my own solution

https://gist.github.com/Thaina/eec5752b25f7bfd3737f7dd9ed2fa53c

This is a script to just curate all installed package, get all dependencies to see if any was missing, then present the wizard windows to let user reorder what and which should be resolved first

Require package com.unity.editorcoroutines. And have not tested for anything more than 1 dependency as of yet

 

Found this thread and was hoping there might be some good news as the end of it. Seeing how unity has not replied in quite a while. This is probably dead.

They complain about trying to solve every possibly use case. We don't need that. Just tell us how it has to be and let us be able to have a git package, get a dependency from another git package. By tag or commit, and if it doesn't follow their guidelines then it wont work.

I just setup like 10 git repos for packages hoping I could have one include the other but no. Each package is a single thing, and would have a tag for every version if any.
Sure it might not work for everyone but it seems like a simple and common use case.
Just make that one work at least.

But now I am going to go back to having my own package server and hopefully that will work.
But that makes it harder to share since, git did the hosting for me. Now i have to get access for the packager server to have access to others.

 

Until dependency URLs for packages are implemented, I've found a workaround that doesn't involve a custom package manager:

provided your base package is coming from git, you can setup your dependencies as git submodules of your base package.

for example, if you have a package A that depends on package B, you can go into terminal for package A and type

git add submodule [path to repo for package B]

, and that will add B as a submodule for A, and fortunately Unity Package Manager resolves submodules when it imports packages from git.

 

Wait UPM resolves git submodules?! That thought never crossed me... Why would UPM allow submodules, but not actual git repos???

You are a genius, if this actually works!

 

Maybe I'm ignorant, but aren't submodules supposed to be located inside an existing repository folder? And that would make a package look like this:

Code (CSharp):

1. package.json
2. Runtime/..
3. Editor/..
4. MyDependencies/Submodule1/..
5. MyDependencies/Submodule2/..
6. MyDependencies/Submodule3/..

So if two different packages use the same dependency submodule, they would duplicate the source?

 

Yes, I think so, and that's the reason I would not use it

 

yeah, that's a downside of this method, but if only one of your packages uses that submodule then things are happy.

 

It would probably be good to mention the minimum Unity version supported. Unity 2019.4 doesn't have package manager events.