Official Feedback Wanted: Unity Downloader - Powering the Editor as a continuous integration dependency

That was a nice blog post and it gave us some hint on your work and how your workflow works internally with CI. That's always nice to ear that kind of things.

For me, this tools could be useful to retrieve specific version when we build our apps with CI. My main and sadly unsoluble issue is to retrieve a specific version of Unity for a specific project. If it's not installed on the building machine it's currently not possible (without some effort) to retrieve it. I was waiting for Unity Hub to release some specific CLI version and in this way allowing me to retrieve specific version but your Unity Downloader tools seems to do (more or less that kind of stuff)

Cause basically what I'm doing currently to retrieve and start the good version is to let my Gitlab-runner start the job, check the project on the branch where the job start. In this project I check the "ProjectVersion.txt"'s content generated by Unity for this one and start the good Unity Editor accordingly.

Maybe it could be used separatly or maybe you could "plug" it behind your Unity Hub stuff to allow any CI devs to retrieve specific Unity's version.

 

This is exactly what we're doing as well. Any time we update the project to a newer Unity version we have to remember to also install the new version on the GitLab Runner, otherwise the build will fail. Would be nice to finally automate this completely.

I like the idea of being able to just specify a project and let the downloader find and download the correct version. For this feature it would be good to have a couple of modes:

• Just download the determined Unity version
• Download + open project
• Download + open project with specified launch arguments
• Download + return path to Unity.exe or install folder

I also think that it would be nice to integrate this into Unity Hub in some way. When the project is updated or I check out a different branch that uses a different Unity version and I try to open the project without having that version of Unity installed locally, Unity Hub should offer to download that version of Unity, instead of running the script conversion tool.

 

Unity hub restricts editor versions to a small subset, so when I wanted to get the same project behavior of some old version I had no choice other than download the closest editor version and hope there will be not so much differences and I can handle them in one pass. Otherwise, I could easily upgrade my project with a small steps.

 

Our workflow is basically as follows:

• our build system config contains mapping of our builds to unity version (SuperGame1 ios version -> unity 5.5f1, SuperGame 1 windows version -> unity 5.5p5, MegaGame2 any version -> unity 2019.1f1)
• before build it checks if unity version is installed and if not downloads it
• we've been using u3d with great success. and before our custom scripts.
• This works for both mac and windows machines

This system has some downsides:

• There is no way to add packages. If we suddenly decided to add xbox platform we need to delete unity from boxes and then u3d will download it from the start. This is quite seldom but still an annoyance.
• There is no cache right now - every box has to download it by itself. This is seldom too.
• There is no way to add Android build tools using u3d right now. This is very annoying, because this is actually is a great feature, and when using unity >=2019 we don't need to manage multiple android sdk/ndk on machines

 

The Hub only shows the latest version of each major release i.e. latest 2018.3.x, latest 2019.1.x etc. But on the same window, they provide a link to the archive where you can download all version.

 

This seems like it would be best served as an add on to the Unity Hub, and Unity Cloud Builds. Focus should be on improving your existing CI tools like Unity Cloud Builds, which to an extent has this feature sort of implemented as is. One thing that's a pain (that others mentioned) is downloading a version of Unity that is buried within the Download Archive, and not directly available through the Hub, a simple CLI would be great, or better yet a proper search filter within the Hub. I enjoyed the blog post, and hope to see this integrated into existing tools instead of being yet another dependency I have to install.

 

Concerning my part, the cache is definietly useful because we have only one building machine and testting each profile to build our app take something like 10-20minutes (12 profiles to recompile each time someone is committing something).
We want to avoid adding more time to this pipeline already quite long (can't buy others machine to do this for now). Just because we use few Unity version, we keep them on the machine forever (almost) and we only reset the project content because it's logical to have something clean when the runner start.

Anyway the runner clear automatically each element which is not pushed on a branch (like Library folder obvisouly) that's why I can say the project is clean when a job start.

Thanks again for your time and your return based on ours.

 

I rarely post here but the blog post actually intrigued me enough to do it (albeit for the first time)

The CLI is tremendously needed if not just for the editor download, since it would be able to run the downloading and the compilation of the Unity Editor in a cloud environment under our control, but also because it could aid the total CI/CD process in total.

As it stands now, the CLI aspect of downloading the editor would be fine to bundle with Hub for the user experience part of it (kind of shipping both the GUI and an "CLI" version of the gui in one is a good solution for most users)
That is how some applications do it, I remember Butler (the itch.io CLI for shipping builds) - used to be bundled with their Game Store application.

However I see a greater potential of this tool not only to download the Editor, but also resolve the Packages from within a project we try to open. (Even maybe preheat the GUI editor ?)

If Unity are serious about Packages being the way forward, we should also be able to "move" the Unity Asset Store *.unitypackage (packages) - over to this system.
Therefore resolving any Unity Asset Store dependency enabling us to do all of our builds within a CI/CD environment.

It would be a HUGE enabler for me because I would be able to ship my builds just from within my repository without ever having to boot up my PC to initiate a build - simply just push to my master/trunk branch and the CI/CD pipeline would be able to push it for me.

 

Asset Store Tools developers would benefit immensely from the ability to test installs and Editor Unit Tests as new versions are released.

In fact, all Asset Store assets could benefit from this to some regard. Some visual reporting mechanism could be used for art assets, provided a system could be devised to include a manifest describing scenes to test (and take framebuffer captures of for reporting/comparison).

I would highly recommend posting to the Asset Store Publishers forum as well as we represent a segment of your user-base that actually shares quite a bit in common with your own internal development processes and requirements.

 

We faced the same issue. So we created a tool in Ruby we called uvm which in the beginning allowed to simple set symlinks to the “active editor” on MacOS

I rewrote this tool in rust last year and reverse engineered Unity Hubs install logic.
https://github.com/Larusso/unity-version-manager

I created a small web service to fetch the latest release of the editor and store it in a github repo.
https://github.com/Larusso/unity-versions-service

Last but not least I wrote a JNI wrapper to call the tool/lib from Java/gradle

https://github.com/wooga/unity-version-manager-jni/pulls

https://github.com/wooga/atlas-unity-version-manager

Most of it is still under development but these tools run our CI system for over a year now.

 

First, thanks for sharing that tool (whenever it will be available)!

I was thinking but would it be possible to have some authentication or something in order to also download consoles components ?
Since I guess you don't need auth at Unity we don't have any way to automatically download those component at the moment. It would be the perfect time to think about a way to make it possible.

About the tool itself, not much haven't already been asked but I will ask for the simplest thing which is a way to download any Unity with a command line while also being able to check which versions and which components are already installed. Also (maybe it's already the case) but it should work on Windows and MacOS.

Thanks!

 

Sounds like a great tool.
Over the years I got to work on different projects, where we needed to test our content in different Unity versions.

Games:
To check if we can upgrade the projects to new editor versions.
Then we decided if the hassle of upgrading them is worth it.

Plugins:
To check if plugins that we developed can be used in new editor versions, and if we need to do adjustments for them to work.

So a way for us to check if there's a new editor, install it, and test a project could be great

 

Thanks for the post, it's great to hear about the problems you guys face internally and how you guys are addressing it. It is also extremely relatable.

Our company does CI in the cloud with stateless VMs so every time we run a build we have to install unity.

Currently, we are pulling down the installation files and just installing them via the command line. This whole process is automated (and changing the version is simply updating a variable in our CI pipeline) except for one step - the need for having to prepare those installation files in the first place. This means someone actually has to go and download the installation files and store them in a place for ease of access.

So far this is not too bad, downloading and installing Unity and the editors takes us about ~50seconds in total and the manual step of having to upload the installer files are only ever required if a project requires a new version of unity and we are looking to automate this soon.

However that being said, it'd still be great if Unity can offer a first-party tool that will do this. My concern would probably be the speed of the download, as long as it is easy to cache that it'd probably be something we would like to adopt.

One last issue we have which may be off-topic, but is Unity looking at offering build licenses for companies? That is one of the biggest issue to us scaling our CI system.

 

This would be wonderful to have. Looking forward!

 

This tool would be extremely useful for our CI in Azure DevOps. We are currently downloading the packages seperately, but this would ease a lot of headache for us. Especially since it seems that this tool can automatically detect the Unity version to download based on the project. I'm super excited to use it!

 

Yes. Not only for CI, but just for downloading Unity versions/packages in general. If we could cache the downloads on LAN we would save a bunch of time - a single download doesn't take that long, but they add up. We'd also save you guys some upload bandwidth!

I find it's a bit of a pity that you have internal tools that can automagically find and install the correct version of Unity, but the Hub's just able to say "that's the wrong version, pal!". Ideally, we should be able to click a Unity project which we don't have the correct version for installed, and then have it ask to download and install that version.

 

@Baste I'm agree with your post but I think they wouldn't make it available until they were sure it was enough stable even for "random" dev like us
Just because they always want to have stuff easy to use for everyone.
I know that feeling concerning the Hub sadly and the lack of CLI but everything will land when it will be mature enough I think

 

Would local editor caching be useful?
Yes, and you already have a tool perfect for facilitating this: Unity Hub

A local install cache of automatically mirrored to build-machines, perfect! (master-slaves) just setup the master and boom all slaves will match in a few minutes.
In production, you should really try to reduce the use of different Unity versions, it's easier for everyone if they know what to expect from a unity version. So removing versions from master, should also remove them from slaves, this way you can force slow and steady upgrading. If at some point you really need a specific older version, add it back in again.

Also instead of an installer, it might be more useful to just unzip the engine parts (and if needed a mini-installer to do the setup part if any is still needed). Allow Hub to detect the build master as well, easier time grabbing files already available in the network and it's exactly the version that you needed, bonus!

If you then happen to expose a cli tool that can download a specific version (with certain addons) that might or might not be missing and complete instantly if it is already there, awesome!

 

If I can make a suggestion (and while I may be the first person to ask for this, I won't be the last), please create a linux docker image of every version of the Unity Editor that we can easily grab through a hosted repo, where the specific editor version is tagged.

If you did that, it would literally be a single "docker pull unityeditor:version", followed by "docker run "... with command line arguments after it. For build machines, this solves the whole problem with zero effort. All anyone would have to do is change the tag version that they are on, and the rest just works. As it stands, I have to grab the most recent linux version and dockerize it myself, which is a lot more effort, and really, Unity should be able to do this trivially and at the same time as their publish cycle, for the community to use. A Linux build has always kind of lagged behind, but they announced in May that by 2019.3 it would be fully supported as an editor platform, not just a headless cli.

I know people use VMs for stuff like this, but sharing VMs is infeasible for so many logical reasons. Sharing applications as a docker image, though, is lightweight and easy. Rather than reinventing the wheel with unity version caching, just use (non-game) industry standards. They are pretty good.

Thanks!

 

It's great that you're investing time in these kinds of tools, and even collecting feedback from your user base regarding e.g. CLI requirements!

We feel the pain of the current muckyness for CI/CD with different versions and modules. We've dev'd a Python tool that runs before each build, verifying that both Editor and platform is installed. If not, automatically download (from your archive) and install.

The downside with our internal tool is that my colleagues need to specify both version, e.g.

2017.3.1f1

and the changeset

fc1d3344e6ea

from

https://netstorage.unity3d.com/unity/fc1d3344e6ea/UnityDownloadAssistant-2017.3.1f1.exe

and which platforms they want, e.g.

android

windows

il2cpp

in a separate .yaml file.

In my head, the platforms/modules would/should be part of ProjectSettings (or the packages manifest.json), which preferably Hub (and this CLI tool) would read, download and install from.

On that note, I don't see the explicit reason for this tool being separate from the Hub. The Hub currently lacks any CLI interface (AFAICT, apart from the URL handler) and cannot therefore install any modules/platforms without mouse interaction. I think it makes sense to put this functionality there rather than a separate tool, since that could then almost become a one stop shop for CI/CD, dealing with licenses, Unity ID etc.

I would also like to flag for the modules behind auth, e.g. PSVR and whatnot; We'd "need that" for this to be a viable tech switch. I'm sure we're not alone in that either.

As you wrote in the blog post, your internal motivation for this type of tool was mostly automation and testing, e.g. bisecting Editor versions. For the consumer base (us), I imagine we'd be using this mostly for CI/CD, so allowing the Android SDK and maybe even NDK to be a download/install too would be great. Dreaming here, obviously...

We're building ~6 projects continually, each to on average 4 separate SKUs (e.g. different SDKs/platforms) and some of them are a bit more "exotic", e.g. MagicLeap and PSVR, so anything to simplify my sysops/devops days is appreciated Licenses is definately part of my hopes for improvement, but that's obviously not for this thread.

 

This is something I would start using day one if you released it publicly. I'm trying to set up a build pipeline with GitHub Actions, but dealing with manual activation has been the biggest headache.

 

I gave up trying to set up CI/CD with Unity because of those activation problems. My bet is that Unity makes it harder to activate by command line so people subscribe to their cloud builds.

 

To be more on-topic: The best thing that Unity could possibly do for a better CI experience is to completely remove the licensing process, at least for CI use; maybe to satisfy your legal team there could be some special CI build that can only be run headless (and is therefore useless for actual editing), but is otherwise fully functional?

As of right now, I have to manually activate Unity every time I set up a CI pipeline or upgrade Unity, which really undermines the whole "automation" thing. And I don't know when the license has to be renewed, either.

This Docker image is a good start, but only for Linux hosts; get rid of the licensing process for CI, and it will make everyone's life easier.

If you're going to have a CI-only build of Unity, it wouldn't hurt to explore ways to make it smaller, so it'll download faster. Or even provide official Docker images, if you're feeling bold.

I'm working on setting something up with GitHub Actions. Once I finish, I'll blog about it and link the post in this thread.

EDIT: I gave up. Unity's licensing process killed it for me. Details here.

 

I am also very interested in this topic.
For me, automated tests and builds is one of the core topics for developing good software with a minimum of quality standards.

So I'm currently struggling to setup a CI test and build workflow with Azure. Gladly there exists some help with the unity azure pipelines tasks:
https://dinomite-studios.github.io/unity-azure-pipelines-tasks/
It actually uses Unity Hub to check project version and downloads automatically the correct editor version. Each build...

Still, if you release your own tool for this, I would also like to try this!

 

We have a build environment, but downloading and installing Unity is not an automated process.
It's only done once, and we upgrade Unity only once in a very long time. why makes this so useful then? sure it's nice to be able to install Unity automatically, but the installation itself (via the Hub) takes around 5 seconds of manual labor.

 

You are right. Every build downloading and installing unity editor is not very performant.
Best would be to cache the installation and only update the cache if the version changed.

 

@henrikpAtUnity how is this going? It's still a thing?

In the past years I've used:

u3d: https://github.com/DragonBox/u3d

Issues with 2019+ editors detecting installed modules.

Unity Setup Powershell: https://github.com/microsoft/unitysetup.powershell

Doesn't detect installed modules.
Can't install 2019+ Android modules.

Unity Hub cli: really unstable, half developed...

https://forum.unity.com/threads/problems-with-unity-hub-cli.908960/

https://forum.unity.com/threads/ins...ssing-options-when-installing-modules.780467/

https://forum.unity.com/threads/stu...re-first-unity-is-already-in-list-etc.903851/

 

Any news on this @henrikpAtUnity??

Hub cli it's really really buggy...

 

I have been giving feedback for a long time and in several threads and I often mention your colleagues.

The problem is the same as always, messages get lost without reply.

You have the example right here, my message is from September 2020, the bump from May and you have replied now. That is, a year waiting for a response so that the answer is that I keep giving feedback that you do not read or answer.... It's frustrating.

In fact this thread you started it with a lot of interest creating even a blog post but the famous Unity Downloader was left in oblivion.

And about what we use now. Right now we are installing by hand in all CI agents the versions we need (with the loss of time it means) because there is no reliable tool ... u3d, Unity PowerShell, etc .... They have the problem that they don't handle well with new versions and modules like android's sdk, ndk, etc... The hub cli as I have said many times is not a real cli and has many bugs and little or no documentation except for the --help parameter.

 

Thanks for the answer but I honestly don't understand it.

You put the documentation of the Unity 5.6 editor (from almost 5 years) cli, not the hub which is what we are talking about. With the Unity editor cli it has never been possible to install another Unity (it would be absurd) it allows to launch projects, publish them, etc...

And the CI setup I don't know why it is relevant, what we need is a real and reliable cli in the hub.

And above all, a place to write feedback and be read and responded to by Unity in an active way.

 

The tool you describe is what I would like. In fact it is the tool that would have to be distributed separately or included in the hub.This way it would not be two separate developments (what you have described and the actual hub cli).

The only thing is that at the moment you only mention Windows and Mac and we also develop on Android, iOS, WebGL, UWP... So it should allow you to install any module available in that version of the editor and not be limited to a specific one.

 

Have you considered forgetting about Unity Hub for CI downloads, and instead maintaining repos for each platform's most popular package installer? (Chocolatey for Windows, Homebrew for macOS, apt-get or yum for Linux depending on the distro)