Search Unity

Over night build times increased by a factor of 5. Why?

Discussion in 'Unity Build Automation' started by HaakonL, Jan 28, 2019.

  1. HaakonL

    HaakonL

    Joined:
    Mar 13, 2014
    Posts:
    123
    Before Unity moved their services to the google platform, we saw build times for our current project quite consistently at 1.5 hours.

    After the move to google, our builds started taking around 40 minutes, and it was a real pleasure to see this boost in speed.

    Then suddenly, some 5 days ago, the builds started taking between 2,5 and 4 hours. So even worse than before the google move. Now this is not acceptable when we used to have 1.5 hours at worst, and lately less than an hour.

    What happened, Unity? Tuned down the servers to save some $$, or did you do anything else with the cloud build pipeline lately that causes this? We want it fixed asap, it takes so long with the builds now it is actually a problem business wise.

    We've been building with 2018.2.19 and .20 for quite some time, and we haven't changed anything in our setup that should cause this. iOS builds.
     
    HaakonL, Jan 28, 2019
    #1

  2. HaakonL

    HaakonL

    Joined:
    Mar 13, 2014
    Posts:
    123
    Just got a reply from the Unity Cloud Build team, and it is a known issue:

    "We are aware of a performance issue causing builds to be a lot slower than usual which has been the case since last Wednesday. We are putting in place a fix for this which should be resolved by the end of the day today."
     
    HaakonL, Jan 28, 2019
    #2
    ollieblanks likes this.

  3. ollieblanks

    ollieblanks

    Unity Technologies

    Joined:
    Aug 21, 2017
    Posts:
    460
    Hi @HaakonL,

    Thanks for reporting this, and for quoting me above. :)

    A fix has now been put in place, meaning that builds should return to usual build speeds. This has caused heavier than normal build queues meaning that queue times may also be slightly longer than expected, but this should now naturally return to normal shortly.

    Sorry for the inconvenience this has caused. Please let me know if you are still experiencing issues and I will be happy to investigate.
     
    ollieblanks, Jan 28, 2019
    #3

  4. ollieblanks

    ollieblanks

    Unity Technologies

    Joined:
    Aug 21, 2017
    Posts:
    460
    Unfortunately, I have prematurely misinformed you. There has been another development to this situation and build times will not have returned to normal just yet. We have just found another bug which may have been introduced by a security patch.

    The team are working on resolving this right now. Please bear with us and I will update you once this has been resolved.
     
    ollieblanks, Jan 28, 2019
    #4

  5. PhilMcJ

    PhilMcJ

    Unity Technologies

    Joined:
    May 29, 2014
    Posts:
    187
    A quick update with a little more info: the root cause is that a minor update rolled out last week behaved fine last week and over the weekend - and then under heavier loads today (Monday) has proved to cause some significant slowdowns in build times when the system is under heavy load.

    Those changes are being rolled back now, and new builds should be gradually getting faster over the next several hours. The team are pretty sure the specific issue has been identified for the future (meaning it can now be tested for) - in the meantime, thanks for everyone's patience as this has been diagnosed and dealt with today.
     
    PhilMcJ, Jan 28, 2019
    #5
    Deleted User likes this.

  6. GuillaumeZAHRA

    GuillaumeZAHRA

    Joined:
    Jan 5, 2013
    Posts:
    53
    Is this issue resolved ?

    Maybe it's not related, but my build are inconsistent.

    On the same codebase, one take 10 minutes, then the next one never figured to finish, somewhat cancelled by your services (even if they write "by a user"), after many hours.

    Oddly, i have a "Finished exporting player successfully" message somewhere but everything hang on a "Refresh: detecting if any assets need to be imported or removed" forever.
     
    GuillaumeZAHRA, Feb 11, 2019
    #6