Feedback Package manager automated testing and other issues

First off, I have to say I love package manager and I really like the direction Unity is going towards!
I'm really looking forward to using more and more packages and tests.

I got started by reading this great article posted here in the Unity forums https://forum.unity.com/threads/publish-unity-package-with-npmjs.882010/
So thanks for these forums and StansAssets too!

Also thanks for https://openupm.com/ for providing a free registry with amazing tools.

When I dug up a bit I found out that there is an entire world of Unity Open Source out there!
Nice to see so many people committing their time and effort to the community.

So inspired from that, I have recently been open sourcing some of the utilities used by our company as Unity packages.

Here's some (MIT licensed, do whatever you want with them):
https://github.com/3d-group/unity-package-template
https://github.com/3d-group/unity-simple-notifications

I came across some issues and suggestions along the way, below are listed some that I can remember.

Note: I'm using Unity 2019.4.1f1 (LTS) for package developing. If there are fixes for these in 2020 please do let me know

1. Automated testing for packages proved to be a bit painful

What I wanted was for our version control (Github) to run tests automatically every time push event occurs.
Like so:


I knew about Github actions so that's where I started.

I used awesome Unity CI project I found to run tests with github actions.
While this project made it a lot easier, there was still some issues.

Unity requires a project to run tests.

Also Unity project requires a license. Which requires an activation file.
Luckily Unity CI project had an automated activation file action included. However, license has to be retrieved manually from Unity's website. EDIT: I checked and they have action for retrieving license too, awesome!

So I created Samples/ExampleUnityProject (which is just an empty project).
Then I imported my package via git url.
Then I ran the activation file action manually.
Got the license.
Stored the license as a secret.
Then I could run tests with Unity CI in that sub folder.
However, this seems a bit silly since I have to include an empty Unity project with every package just for running tests

Of course this can be solved by Unity CI too and there is already an issue about it: https://github.com/webbertakken/unity-test-runner/issues/71 But I would really appreciate some official solution provided by Unity in the future!

Suggestion: Make it possible to run tests - and perhaps only run tests - without Unity license. Perhaps easy solution would be a separate version of unity which only included TestRunner capabilities? Or require license only if doing something else than running tests?

Some products have solved similar problem, eg. testing databases by creating in-memory version that simulates database. It won't have 100% same features but is great for testing.

Suggestion: I'm aware of the Unity's Cloud Build system, but it seemed a bit too much hassle to run tests for just packages there (still requires the empty project). It would be awesome if there was an option / automatic check to run tests for packages!

2. npm org scope isn't supported

While searching the forums I found out that this has been discussed before.

3. Is there a reason why unity package files require .meta files?
This seemed a bit counter-intuitive at first.
I get that for assets and for assembly definitions this can be useful but for eg. READMEs, changelogs etc it seemed a bit too excessive. Perhaps these could be generated automatically by Unity if not existed?

When making template repository for Unity Packages I ran into issues because I cannot use .meta files as templates because they need to have different GUIDs.
Generating GUIDs for these meta files proved to be challenging hence (I think) Unity uses its own GUID format, not the standard.
I found some code that generated these but this seemed to be to much hassle so I just added into instructions to copy paste the package into some Unity project's asset folder once instead.

4. Fluent assertions missing
I'm used to writing tests with NUnit and FluentAssertions. However, using nuget packages with Unity packages proved to be somewhat difficult. There was some solutions, but they all seemed a bit sketchy.
eg. Downloading an utility to use nugets and then depend on that.
Or including the nuget with every package...


==================================================================

If any of you fine readers out there happen to know a better solution to any of these problems, or see issues with these approaches please let me know!

 

Hello there. Apologies for the delay in reply. I'll reply to each point individually.

Re: 1.

This is already being done to have some "headless/batch mode-only" license types, but we don't have any ETA on this or the full details to share.

Packages are meant to be used (and then tested) within a Unity project and for that a bit of boilerpate is needed as they figured out. Maybe a better workflow could be achieved if Unity Cloud Build UX was improved and streamlined for package development as well, but again, no news on if this is possible righ now.


Re 2:

Yes, we don't support npm scopes


Re 3.

Package assets are imported in the asset database. Every asset has a meta file and every meta file has a GUID, which is a unique identifier that is generated randomly. This identifier will be used by Unity to create references and dependencies between files, and they are required by all files.

Unity doesn't use its own GUIDS, there are some reserved GUIDs, but most of them are computed and it is a 32-digit hexadecimal number, as long as the identifier is unique for the project, generating the id should be fine, but leaving the job to Unity while coping the folder is perfetcly fine.


Re: 4. Fluent assertions missing

It's a feature request, but currently deemed as not a high priority. Using nuget as suggested is a solution. It would be nice to know how other ways users approach this though

 

No problem about and thank you for answering!

Re: 1.

Good to know that this is given some thought, looking forward for it if it comes out some day!

Re: 2.

Hmm, unfortunate. I'd really like to hear as of why.
When I was first trying to add my own npm packages to Unity I was following npm documentation online and used scopes. I ran into following error in Unity:

An error occurred while resolving packages:​

Project has invalid dependencies:
A re-import of the project may be required to fix the issue or a manual modification of manifest.json file.
This got me confused because I thought that something was wrong with either my url, or with my package when in reality it was only the package name that caused this error.
It would be awesome if Unity would just say here eg. "We don't support npm scopes". This could also be written in Unity Custom Packages documentation. This would have saved me hours of productive work time.
If at all possible. please consider adding this to the backlog

Re: 3.

Ah ok I see, I just tried to generate them automatically with uuidgen tool in bash and ran into error

The .meta file ... does not have a valid GUID and its corresponding Asset file will be ignored. If this file is not malformed, please add a GUID, or delete the .meta file and it will be recreated correctly​

What I found out that the tool generates GUID with dashes eg. b567eb0c-0935-4b3d-bc18-c856b1bace91
and Unity doesn't really appreciate those dashes
I don't know about GUIDs much but dashes seems to be the standard though...eg. Powershell [guid]::NewGuid() generates GUIDs with dashes too.
Is there any reason why Unity doesn't allow dashes here?
Considering this is just supposed to be a unique identifier I have to wonder why generating GUID with standard tools (tools you find first when you google) throws an ambiguous error. I first got spooked by that and thought that there was some special logic behind generating GUIDs

But when I remove dashes it works fine!
I just wish there was better error / handling eg. if dashes are encountered they are removed by Unity or error clearly states that "Perhaps try removing dashes?"

Re: 4.

Good to know that this is already in backlog! I hope you'll find time to do this.
I'd really appreciate official nuget support in following years to come!

 

Hey there! I've not forgotten about you - just awaiting feedback so I'll be in touch once I get one.

 

Right, so - regarding the error reporting. The team would definitely welcome a bug report on this to look into. Can I ask if you've already done this? If so, i can make sure they are aware

 

Unfortunately, no. I'd very much appreciate if you could do that.

 

You'd need to file the bug report yourself, then I can tell the team via the number you're provided.

The bug report process is here: https://unity3d.com/unity/qa/bug-reporting

 

Hi, I took time and created following tickets:

• (Case 1297937) Malicious error reporting with npm scopes
• (Case 1297938) Malicious error reporting on GUIDs

 

Thank you Let me know if you don't get a response in a week or two.