Search Unity

Feedback My disappointing experience using Unity in a CI pipeline

Discussion in 'Testing & Automation' started by JesseSTG, Aug 31, 2019.

  1. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    This past week I've been trying to set up a CI pipeline for my game. I'm currently using the Personal edition of Unity 2019.2.2, but will upgrade to Unity Plus if I need to. I want my pipeline to automatically do these three things, so I don't have to do them by hand:
    • To ensure that my tests pass when I make changes to my game
    • To ensure that my game builds on all the platforms I support
    • To deploy to all platforms and storefronts I support
    I looked into Unity Cloud Build, but unfortunately that's not an option for me, for these reasons:
    Instead, I decided to try out the GitHub Actions beta for these reasons:
    • I already have a paid GitHub plan, so I get 3000 minutes included per month
    • It supports Windows, macOS, and Linux, so I can build for any supported Unity platform on the same pipeline
    • If I want to add some build step external to Unity (like importing files or generating code), I can do that at my leisure
    GitHub Actions has been a great experience for me; it's doing everything I want it to and doesn't get in my way with technical limits. Seriously, give it a try...with a project unrelated to Unity. Here's why.

    Unity recently published some improvements to their test framework, and has expressed a renewed interest in CI. These new efforts led me to believe that I could set something up that would at least fit my requirements, even if there were some warts along the way. I was mistaken.

    Unity's licensing mechanism is 100% getting in the way of me building the CI pipeline that I want. Currently, a Unity license must be obtained manually, either in the editor or through the process described here and here. For manual use (i.e. when actually designing a game), this is fine; I had no issue installing and licensing Unity for my development machine.

    The problem is that you still have to do this when installing Unity on a CI runner. Most CI services don't make promises about a runner's identifying details (network address, hostname, serial numbers, other OS identifiers). As a result, a Unity license for a CI runner could become useless at any time without warning. In fact, that's exactly what's been happening with my own pipeline. This undermines every effort the Unity developers make to offer a smooth automation experience. What's the point of automated builds, tests, or deployments if I have to manually license Unity every time I run my CI pipeline?

    I don't have a problem with downloading and installing Unity on the CI runners, nor do I with actually invoking it on the command line. It's rougher around the edges than I'd prefer, but at least it fits my needs. All I want is to be able to use Unity for automated builds, test, and deployments in a CI runner without any manual intervention; the best way I can see that is to remove or totally automate the licensing process for CI pipelines.

    Or, if your legal team would get nervous, perhaps consider special CI-only builds of Unity that don't need to be licensed, but can only be run headless? Such a build couldn't be used for designing content, and thus would not be worth pirating.

    I love Unity and it's generally a bliss to work with, but in this case I'm incredibly disappointed. I know the dev team can do better.
     
  2. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,184
    What was the actual problem you ran into? Are you trying to preinstall the license inside a Docker container, or something?

    We are working on a new licensing system that will make this sort of situation much smoother, but it should still be possible to get something working today, albeit with a bit of a bumpy ride.
     
    JesseSTG likes this.
  3. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    Hi, Richard. Thank you for your reply, especially on a Saturday night.

    No. This is what I did for each operating system;
    1. Spin up a runner
    2. Install Unity (or, in the case of Linux runners, pull this Docker image) as part of the script
    3. Request a license as described here (section "Create activation file and import license by command"), and save the .alf file
    4. Add the generated .alf file as an artifact, then manually get a license as described here (The runner is finished by this point)
    5. Manually add the contents of the .ulf license file as a secret to my GitHub configuration (not the repository, to a config item on the website)
    6. Spin up a new runner of the same OS (I can't continue using the first runner because of the manual intervention needed)
    7. Install Unity again
    8. Activate the license as described here (section "Create activation file and import license by command")
    9. Run the tests or build the game (depending on the workflow that's running)
    10. If applicable, deploy the game
    If I already have a license then steps 1-5 are ignored. They're still run, but I haven't figured out how to conditionally disable them yet.

    The problem is:
    • This works well enough on Github Actions' Linux runners, I guess because of the Docker image I'm using. But I have no idea when the license will be made useless under my feet (because of changes to the runner's environment)
    • I believe that at a minimum I have to do this every time I upgrade Unity (which I do frequently, even for patch releases)
    • This doesn't work at all on the macOS or Windows runners, because whatever information Unity uses to request a license doesn't seem to be constant as I expected. As a result, I have not managed to make a single successful macOS or Windows build on GitHub Actions. When I try to activate Unity on these runners I will get errors that look like this:
    Code (csharp):
    1. /Applications/Unity/Unity.app/Contents/MacOS/Unity -quit -batchmode -username <redacted> -password <redacted> -manualLicenseFile /Users/runner/.local/share/unity3d/Unity/Unity_lic.ulf -nographics -silent-crashes -disable-assembly-updater -logFile -
    2.  
    3. Initiating legacy licensing module
    4. DisplayProgressbar: Unity license
    5.  
    6. LICENSE SYSTEM [2019831 16:49:13] Next license update check is after 2019-09-01T16:20:51
    7.  
    8.  
    9. LICENSE SYSTEM [2019831 16:49:13] VMuD6VlWKP6I != VMPkeFQqLH/F
    10.  
    11. [UnityConnectServicesConfig] config is NOT valid, switching to default
    12. User <redacted> logged in successfully
    13. Cancelling DisplayDialog: Failed to activate/update license. Timeout occured while trying to update license. Please try again later or contact support@unity3d.com
    14. This should not be called in batch mode.
    15. (Filename: /Users/builduser/buildslave/unity/build/Editor/Platform/OSX/EditorUtility.mm Line: 288)
    16.  
    17.  
    It looks about the same on the Windows runners, but I can post more detailed information or logs if you'd like.

    And, of course, these extra steps eat up my runner minutes.


    Color me intrigued. If it's at all possible, can you tell me more about how it'll work, and what kind of timeline you have in mind?
     
    Last edited: Sep 1, 2019
  4. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,184
    Right, the .alf file contains machine binding information that you should not be relying on from run to run.

    Ohhhhh, you're on Personal. Right, so I think usually when people are doing this they're on Plus or Pro, which means they have a serial number and can just pass it to the Editor with -serial (and then -returnlicense when done), as per the documentation, in a kind of ghetto license-leasing setup. If you want you could contact your regional salesperson and ask for a trial license for Plus, in order to test it out. Unfortunately we don't have a good solution for Personal right now AFAIK :(

    Well, there are technical and business decisions that haven't been made yet, so don't take this as any kind of guarantee or promise... but I think we're headed towards a future where you can just sign into your Unity account, and then an appropriate license is acquired for you automatically, without the license being bound to your specific machine. We've been beta-testing it with a handful of enterprise customers for the past six months or so, and that's gone well, but there's a lot of work still to do before a public release - so no timeline to share just yet.
     
    JesseSTG likes this.
  5. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    Huh. Wasn't aware of this. So Plus/Pro don't use the
    .alf
    /
    .ulf
    system?

    Forgot to mention; another big plus of GitHub Actions is that I can run many jobs at the same time. As of right now, my attempt at a pipeline runs ten jobs concurrently; four to activate Unity (one for each build target), four to build and deploy the game (one for each build target), and two to run the tests (play mode and edit mode).

    I could try this out, but the FAQ tells me that I can only use two machines per license. That's a problem, because I have Unity Personal installed on three computers.* Even if I uninstalled Unity on two of them (I can't because I use them for development), I wouldn't be able to run more than one job at a time.

    *Two unique machines, one of which has two operating systems installed.

    Fair enough. Thank you for the honesty; if this is already in the oven then I suppose all you can do is keep cooking it. Do you at least have an idea of the order of magnitude you're working with? (Weeks? Months? Years?)
     
    Last edited: Sep 1, 2019
  6. Yacuzo

    Yacuzo

    Joined:
    Sep 22, 2016
    Posts:
    8
    +1 for this issue in general.
    Also +1 for a completely headless unity executable and headless installer. This would make it possible to run on more efficient headless OSes and make it possible to automate the installation of new unity versions on the runners. The last one is important if you update often.

    Edit: It would be a dream if it could be run on Windows Nano. Windows Server is so heavy.
     
    JesseSTG likes this.
  7. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,184
    All versions of Unity use .ulf licensing (for now), but usually - like when you activate Unity interactively - the file is automatically generated, uploaded, and the .ulf is downloaded and installed. The problem is that we have no way to trigger that mechanism from the command line without providing a serial number, which you don't have for Personal (it's generated automatically during activation).


    Indeed, if you want to run Unity on four build nodes, you would need at least two licenses. For Personal that's not a problem because you can just generate as many as you want, but not so for Plus/Pro...

    I think we're talking 1-2 years to roll out the complete solution. You may see individual pieces start to appear much earlier than that though - the client-side pieces of new system are already included in 2019.3, for example.
     
    JesseSTG likes this.
  8. JesseSTG

    JesseSTG

    Joined:
    Jan 10, 2019
    Posts:
    11
    Is this inherent to the licensing process? There's no way you could offer serials for Personal?

    My budget is best described as "shoestring". I can't justify three (builders + test runners) Plus licenses right now.

    Can you tell us more about these client-side pieces that made it to 2019.3?