Search Unity

Feedback Feedback: Unity Downloader - Powering the Editor as a continuous integration dependency

Discussion in 'Testing & Automation' started by henrikpAtUnity, Jul 15, 2019.

  1. henrikpAtUnity

    henrikpAtUnity

    Unity Technologies

    Joined:
    Jan 6, 2017
    Posts:
    36
    We would love to hear your feedback about the blog post we wrote about one of our internal tools that make it possible to grab Unity versions through CI:

    Unity Downloader: Powering the Editor as a continuous integration dependency

    What did you think about the blog post? Does this type of tool relate to problems you currently have as a Unity user? How could we make your CI work more fluent and enjoyable?
     
  2. aurelien-morel-ubiant

    aurelien-morel-ubiant

    Joined:
    Sep 27, 2017
    Posts:
    231
    That was a nice blog post and it gave us some hint on your work and how your workflow works internally with CI. That's always nice to ear that kind of things.

    For me, this tools could be useful to retrieve specific version when we build our apps with CI. My main and sadly unsoluble issue is to retrieve a specific version of Unity for a specific project. If it's not installed on the building machine it's currently not possible (without some effort) to retrieve it. I was waiting for Unity Hub to release some specific CLI version and in this way allowing me to retrieve specific version but your Unity Downloader tools seems to do (more or less that kind of stuff)

    Cause basically what I'm doing currently to retrieve and start the good version is to let my Gitlab-runner start the job, check the project on the branch where the job start. In this project I check the "ProjectVersion.txt"'s content generated by Unity for this one and start the good Unity Editor accordingly.

    Maybe it could be used separatly or maybe you could "plug" it behind your Unity Hub stuff to allow any CI devs to retrieve specific Unity's version.
     
    henrikpAtUnity likes this.
  3. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    42
    This is exactly what we're doing as well. Any time we update the project to a newer Unity version we have to remember to also install the new version on the GitLab Runner, otherwise the build will fail. Would be nice to finally automate this completely.

    I like the idea of being able to just specify a project and let the downloader find and download the correct version. For this feature it would be good to have a couple of modes:
    • Just download the determined Unity version
    • Download + open project
    • Download + open project with specified launch arguments
    • Download + return path to Unity.exe or install folder
    I also think that it would be nice to integrate this into Unity Hub in some way. When the project is updated or I check out a different branch that uses a different Unity version and I try to open the project without having that version of Unity installed locally, Unity Hub should offer to download that version of Unity, instead of running the script conversion tool.
     
  4. Skyblade

    Skyblade

    Joined:
    Nov 19, 2013
    Posts:
    73
    Unity hub restricts editor versions to a small subset, so when I wanted to get the same project behavior of some old version I had no choice other than download the closest editor version and hope there will be not so much differences and I can handle them in one pass. Otherwise, I could easily upgrade my project with a small steps.
     
    henrikpAtUnity likes this.
  5. henrikpAtUnity

    henrikpAtUnity

    Unity Technologies

    Joined:
    Jan 6, 2017
    Posts:
    36
    Thank you for your feedback.
    This is for sure one use case that I suspected people might need tooling for.
    The current unity downloader does have support for specifying an existing Unity project, and as you mentioned, takes a look in the ProjectVersion.txt and downloads that exact editor it was imported with.
    So I would like us to be able to deliver something that would solve this for our end users as well.

    Edit: Posted a previous reply with my other user account so deleted and recreated it. Sorry about that :D
     
  6. michael-v

    michael-v

    Joined:
    Jun 21, 2013
    Posts:
    19
    Our workflow is basically as follows:

    • our build system config contains mapping of our builds to unity version (SuperGame1 ios version -> unity 5.5f1, SuperGame 1 windows version -> unity 5.5p5, MegaGame2 any version -> unity 2019.1f1)
    • before build it checks if unity version is installed and if not downloads it
    • we've been using u3d with great success. and before our custom scripts.
    • This works for both mac and windows machines
    This system has some downsides:
    • There is no way to add packages. If we suddenly decided to add xbox platform we need to delete unity from boxes and then u3d will download it from the start. This is quite seldom but still an annoyance.
    • There is no cache right now - every box has to download it by itself. This is seldom too.
    • There is no way to add Android build tools using u3d right now. This is very annoying, because this is actually is a great feature, and when using unity >=2019 we don't need to manage multiple android sdk/ndk on machines
     
    henrikpAtUnity likes this.
  7. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    42
    The Hub only shows the latest version of each major release i.e. latest 2018.3.x, latest 2019.1.x etc. But on the same window, they provide a link to the archive where you can download all version.
     
    henrikpAtUnity likes this.
  8. Time_Flys

    Time_Flys

    Joined:
    Feb 12, 2015
    Posts:
    16
    This seems like it would be best served as an add on to the Unity Hub, and Unity Cloud Builds. Focus should be on improving your existing CI tools like Unity Cloud Builds, which to an extent has this feature sort of implemented as is. One thing that's a pain (that others mentioned) is downloading a version of Unity that is buried within the Download Archive, and not directly available through the Hub, a simple CLI would be great, or better yet a proper search filter within the Hub. I enjoyed the blog post, and hope to see this integrated into existing tools instead of being yet another dependency I have to install.
     
    henrikpAtUnity likes this.
  9. henrikpAtUnity

    henrikpAtUnity

    Unity Technologies

    Joined:
    Jan 6, 2017
    Posts:
    36
    Thanks for all the feedback so far. It is really useful.
    This solution came from an urgent need to get something in place, since packages were already upon us a year ago and we needed a single solution internally for delivering editors our different teams had for their CI.
    It was very much a "Let's see if this is useful for us" on my part which is why I spent my 20% time on it.
    Almost a year later it turns out it was very successful and so I wanted to reach out and see with you, our users, if similar functionality would be of use to you as well.
    Based on reactions so far that seems to be the case.
    I have gone into this with very few assumptions for what a public solution for this would look like.
    If it makes the most sense to add this type of functionality to the Hub, then I will spend my time towards that.

    So a question:
    Would local editor caching be useful?
    What I mean by that is for example a server sitting somewhere in a closet where all editor versions previously downloaded sit and will be served when someone at the location selects to install it.
    Which is basically what we do with the current downloader. We have a cache sitting on premise where our build VMs are located so we do not have pre-installed editors on the machines. We just download and sets them up at the start of every CI job.
     
  10. aurelien-morel-ubiant

    aurelien-morel-ubiant

    Joined:
    Sep 27, 2017
    Posts:
    231
    Concerning my part, the cache is definietly useful because we have only one building machine and testting each profile to build our app take something like 10-20minutes (12 profiles to recompile each time someone is committing something).
    We want to avoid adding more time to this pipeline already quite long (can't buy others machine to do this for now). Just because we use few Unity version, we keep them on the machine forever (almost) and we only reset the project content because it's logical to have something clean when the runner start.

    Anyway the runner clear automatically each element which is not pushed on a branch (like Library folder obvisouly) that's why I can say the project is clean when a job start.

    Thanks again for your time and your return based on ours.
     
    Stormy102 and henrikpAtUnity like this.
  11. dechowmedia

    dechowmedia

    Joined:
    Dec 2, 2013
    Posts:
    1
    I rarely post here but the blog post actually intrigued me enough to do it (albeit for the first time)

    The CLI is tremendously needed if not just for the editor download, since it would be able to run the downloading and the compilation of the Unity Editor in a cloud environment under our control, but also because it could aid the total CI/CD process in total.

    As it stands now, the CLI aspect of downloading the editor would be fine to bundle with Hub for the user experience part of it (kind of shipping both the GUI and an "CLI" version of the gui in one is a good solution for most users)
    That is how some applications do it, I remember Butler (the itch.io CLI for shipping builds) - used to be bundled with their Game Store application.

    However I see a greater potential of this tool not only to download the Editor, but also resolve the Packages from within a project we try to open. (Even maybe preheat the GUI editor ?)

    If Unity are serious about Packages being the way forward, we should also be able to "move" the Unity Asset Store *.unitypackage (packages) - over to this system.
    Therefore resolving any Unity Asset Store dependency enabling us to do all of our builds within a CI/CD environment.

    It would be a HUGE enabler for me because I would be able to ship my builds just from within my repository without ever having to boot up my PC to initiate a build - simply just push to my master/trunk branch and the CI/CD pipeline would be able to push it for me.
     
    henrikpAtUnity likes this.
  12. SonicBloomEric

    SonicBloomEric

    Joined:
    Sep 11, 2014
    Posts:
    611
    Asset Store Tools developers would benefit immensely from the ability to test installs and Editor Unit Tests as new versions are released.

    In fact, all Asset Store assets could benefit from this to some regard. Some visual reporting mechanism could be used for art assets, provided a system could be devised to include a manifest describing scenes to test (and take framebuffer captures of for reporting/comparison).

    I would highly recommend posting to the Asset Store Publishers forum as well as we represent a segment of your user-base that actually shares quite a bit in common with your own internal development processes and requirements.
     
    henrikpAtUnity and MukaSchultze like this.
  13. Larusso

    Larusso

    Joined:
    Sep 28, 2015
    Posts:
    3
    We faced the same issue. So we created a tool in Ruby we called uvm which in the beginning allowed to simple set symlinks to the “active editor” on MacOS

    I rewrote this tool in rust last year and reverse engineered Unity Hubs install logic.
    https://github.com/Larusso/unity-version-manager

    I created a small web service to fetch the latest release of the editor and store it in a github repo.
    https://github.com/Larusso/unity-versions-service

    Last but not least I wrote a JNI wrapper to call the tool/lib from Java/gradle

    https://github.com/wooga/unity-version-manager-jni/pulls

    https://github.com/wooga/atlas-unity-version-manager

    Most of it is still under development but these tools run our CI system for over a year now.
     
    henrikpAtUnity likes this.
  14. henrikpAtUnity

    henrikpAtUnity

    Unity Technologies

    Joined:
    Jan 6, 2017
    Posts:
    36
    That should be done now, thanks for the tip :)

    And in general:
    Thanks for all the great feedback everyone, keep it coming, I can take it.
    It is of immense value to us to know your thoughts and concerns on this subject.
    So many of you sharing your own solutions is also very appreciated.
     
    SonicBloomEric likes this.
  15. dzamani

    dzamani

    Joined:
    Feb 25, 2014
    Posts:
    89
    First, thanks for sharing that tool (whenever it will be available)!

    I was thinking but would it be possible to have some authentication or something in order to also download consoles components ?
    Since I guess you don't need auth at Unity we don't have any way to automatically download those component at the moment. It would be the perfect time to think about a way to make it possible.

    About the tool itself, not much haven't already been asked but I will ask for the simplest thing which is a way to download any Unity with a command line while also being able to check which versions and which components are already installed. Also (maybe it's already the case) but it should work on Windows and MacOS.

    Thanks!
     
    henrikpAtUnity likes this.
  16. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    287
    Sounds like a great tool.
    Over the years I got to work on different projects, where we needed to test our content in different Unity versions.

    Games:
    To check if we can upgrade the projects to new editor versions.
    Then we decided if the hassle of upgrading them is worth it.

    Plugins:
    To check if plugins that we developed can be used in new editor versions, and if we need to do adjustments for them to work.

    So a way for us to check if there's a new editor, install it, and test a project could be great
     
    henrikpAtUnity likes this.
  17. cicdguy

    cicdguy

    Joined:
    Apr 16, 2019
    Posts:
    1
    Thanks for the post, it's great to hear about the problems you guys face internally and how you guys are addressing it. It is also extremely relatable.

    Our company does CI in the cloud with stateless VMs so every time we run a build we have to install unity.

    Currently, we are pulling down the installation files and just installing them via the command line. This whole process is automated (and changing the version is simply updating a variable in our CI pipeline) except for one step - the need for having to prepare those installation files in the first place. This means someone actually has to go and download the installation files and store them in a place for ease of access.

    So far this is not too bad, downloading and installing Unity and the editors takes us about ~50seconds in total and the manual step of having to upload the installer files are only ever required if a project requires a new version of unity and we are looking to automate this soon.

    However that being said, it'd still be great if Unity can offer a first-party tool that will do this. My concern would probably be the speed of the download, as long as it is easy to cache that it'd probably be something we would like to adopt.

    One last issue we have which may be off-topic, but is Unity looking at offering build licenses for companies? That is one of the biggest issue to us scaling our CI system.
     
    Last edited: Jul 16, 2019
    henrikpAtUnity likes this.
  18. yossi_horowitz_artie

    yossi_horowitz_artie

    Joined:
    Jan 30, 2019
    Posts:
    56
    This would be wonderful to have. Looking forward!
     
    henrikpAtUnity likes this.
  19. FrozenLazer

    FrozenLazer

    Joined:
    Jan 10, 2018
    Posts:
    8
    This tool would be extremely useful for our CI in Azure DevOps. We are currently downloading the packages seperately, but this would ease a lot of headache for us. Especially since it seems that this tool can automatically detect the Unity version to download based on the project. I'm super excited to use it! :)
     
    henrikpAtUnity likes this.
  20. henrikpAtUnity

    henrikpAtUnity

    Unity Technologies

    Joined:
    Jan 6, 2017
    Posts:
    36
    Thanks for the detailed explanation.
    Regarding licensing, we are currently working on a solution that would help with this. I don't have more details than that for the time being :)
     
  21. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    4,224
    Yes. Not only for CI, but just for downloading Unity versions/packages in general. If we could cache the downloads on LAN we would save a bunch of time - a single download doesn't take that long, but they add up. We'd also save you guys some upload bandwidth!

    I find it's a bit of a pity that you have internal tools that can automagically find and install the correct version of Unity, but the Hub's just able to say "that's the wrong version, pal!". Ideally, we should be able to click a Unity project which we don't have the correct version for installed, and then have it ask to download and install that version.
     
  22. aurelien-morel-ubiant

    aurelien-morel-ubiant

    Joined:
    Sep 27, 2017
    Posts:
    231
    @Baste I'm agree with your post but I think they wouldn't make it available until they were sure it was enough stable even for "random" dev like us :D
    Just because they always want to have stuff easy to use for everyone.
    I know that feeling concerning the Hub sadly and the lack of CLI but everything will land when it will be mature enough I think
     
    henrikpAtUnity likes this.
  23. Freezy

    Freezy

    Joined:
    Jul 15, 2012
    Posts:
    232
    Would local editor caching be useful?
    Yes, and you already have a tool perfect for facilitating this: Unity Hub

    A local install cache of automatically mirrored to build-machines, perfect! (master-slaves) just setup the master and boom all slaves will match in a few minutes.
    In production, you should really try to reduce the use of different Unity versions, it's easier for everyone if they know what to expect from a unity version. So removing versions from master, should also remove them from slaves, this way you can force slow and steady upgrading. If at some point you really need a specific older version, add it back in again.

    Also instead of an installer, it might be more useful to just unzip the engine parts (and if needed a mini-installer to do the setup part if any is still needed). Allow Hub to detect the build master as well, easier time grabbing files already available in the network and it's exactly the version that you needed, bonus!

    If you then happen to expose a cli tool that can download a specific version (with certain addons) that might or might not be missing and complete instantly if it is already there, awesome!
     
    henrikpAtUnity likes this.
  24. jhughes2112

    jhughes2112

    Joined:
    Nov 20, 2014
    Posts:
    73
    If I can make a suggestion (and while I may be the first person to ask for this, I won't be the last), please create a linux docker image of every version of the Unity Editor that we can easily grab through a hosted repo, where the specific editor version is tagged.

    If you did that, it would literally be a single "docker pull unityeditor:version", followed by "docker run "... with command line arguments after it. For build machines, this solves the whole problem with zero effort. All anyone would have to do is change the tag version that they are on, and the rest just works. As it stands, I have to grab the most recent linux version and dockerize it myself, which is a lot more effort, and really, Unity should be able to do this trivially and at the same time as their publish cycle, for the community to use. A Linux build has always kind of lagged behind, but they announced in May that by 2019.3 it would be fully supported as an editor platform, not just a headless cli.

    I know people use VMs for stuff like this, but sharing VMs is infeasible for so many logical reasons. Sharing applications as a docker image, though, is lightweight and easy. Rather than reinventing the wheel with unity version caching, just use (non-game) industry standards. They are pretty good.

    Thanks!
     
    henrikpAtUnity likes this.
  25. daniel-hagstrom

    daniel-hagstrom

    Joined:
    Apr 11, 2016
    Posts:
    5
    It's great that you're investing time in these kinds of tools, and even collecting feedback from your user base regarding e.g. CLI requirements!

    We feel the pain of the current muckyness for CI/CD with different versions and modules. We've dev'd a Python tool that runs before each build, verifying that both Editor and platform is installed. If not, automatically download (from your archive) and install.

    The downside with our internal tool is that my colleagues need to specify both version, e.g.
    2017.3.1f1
    and the changeset
    fc1d3344e6ea
    from
    https://netstorage.unity3d.com/unity/fc1d3344e6ea/UnityDownloadAssistant-2017.3.1f1.exe
    and which platforms they want, e.g.
    android
    windows
    il2cpp
    in a separate .yaml file.

    In my head, the platforms/modules would/should be part of ProjectSettings (or the packages manifest.json), which preferably Hub (and this CLI tool) would read, download and install from.

    On that note, I don't see the explicit reason for this tool being separate from the Hub. The Hub currently lacks any CLI interface (AFAICT, apart from the URL handler) and cannot therefore install any modules/platforms without mouse interaction. I think it makes sense to put this functionality there rather than a separate tool, since that could then almost become a one stop shop for CI/CD, dealing with licenses, Unity ID etc.

    I would also like to flag for the modules behind auth, e.g. PSVR and whatnot; We'd "need that" for this to be a viable tech switch. :) I'm sure we're not alone in that either.

    As you wrote in the blog post, your internal motivation for this type of tool was mostly automation and testing, e.g. bisecting Editor versions. For the consumer base (us), I imagine we'd be using this mostly for CI/CD, so allowing the Android SDK and maybe even NDK to be a download/install too would be great. Dreaming here, obviously...

    We're building ~6 projects continually, each to on average 4 separate SKUs (e.g. different SDKs/platforms) and some of them are a bit more "exotic", e.g. MagicLeap and PSVR, so anything to simplify my sysops/devops days is appreciated :) Licenses is definately part of my hopes for improvement, but that's obviously not for this thread.
     
  26. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    This is something I would start using day one if you released it publicly. I'm trying to set up a build pipeline with GitHub Actions, but dealing with manual activation has been the biggest headache.
     
    henrikpAtUnity likes this.
  27. MukaSchultze

    MukaSchultze

    Joined:
    Feb 12, 2014
    Posts:
    83
    I gave up trying to set up CI/CD with Unity because of those activation problems. My bet is that Unity makes it harder to activate by command line so people subscribe to their cloud builds.
     
  28. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    To be more on-topic: The best thing that Unity could possibly do for a better CI experience is to completely remove the licensing process, at least for CI use; maybe to satisfy your legal team there could be some special CI build that can only be run headless (and is therefore useless for actual editing), but is otherwise fully functional?

    As of right now, I have to manually activate Unity every time I set up a CI pipeline or upgrade Unity, which really undermines the whole "automation" thing. And I don't know when the license has to be renewed, either.

    This Docker image is a good start, but only for Linux hosts; get rid of the licensing process for CI, and it will make everyone's life easier.

    If you're going to have a CI-only build of Unity, it wouldn't hurt to explore ways to make it smaller, so it'll download faster. Or even provide official Docker images, if you're feeling bold.

    I'm working on setting something up with GitHub Actions. Once I finish, I'll blog about it and link the post in this thread.

    EDIT: I gave up. Unity's licensing process killed it for me. Details here.
     
    Last edited: Aug 31, 2019