Search Unity

  1. Unity 2019.1 beta is now available.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. We're looking for insight from anyone who has experience with game testing to help us better Unity. Take our survey here. If chosen to participate you'll be entered into a sweepstake to win an Amazon gift card.
    Dismiss Notice
  4. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  5. Unity 2018.3 is now released.
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice

Over night build times increased by a factor of 5. Why?

Discussion in 'Unity Cloud Build' started by HaakonL, Jan 28, 2019.

  1. HaakonL

    HaakonL

    Joined:
    Mar 13, 2014
    Posts:
    96
    Before Unity moved their services to the google platform, we saw build times for our current project quite consistently at 1.5 hours.

    After the move to google, our builds started taking around 40 minutes, and it was a real pleasure to see this boost in speed.

    Then suddenly, some 5 days ago, the builds started taking between 2,5 and 4 hours. So even worse than before the google move. Now this is not acceptable when we used to have 1.5 hours at worst, and lately less than an hour.

    What happened, Unity? Tuned down the servers to save some $$, or did you do anything else with the cloud build pipeline lately that causes this? We want it fixed asap, it takes so long with the builds now it is actually a problem business wise.

    We've been building with 2018.2.19 and .20 for quite some time, and we haven't changed anything in our setup that should cause this. iOS builds.
     
  2. HaakonL

    HaakonL

    Joined:
    Mar 13, 2014
    Posts:
    96
    Just got a reply from the Unity Cloud Build team, and it is a known issue:

    "We are aware of a performance issue causing builds to be a lot slower than usual which has been the case since last Wednesday. We are putting in place a fix for this which should be resolved by the end of the day today."
     
    ollieblanks likes this.
  3. ollieblanks

    ollieblanks

    Unity Technologies

    Joined:
    Aug 21, 2017
    Posts:
    125
    Hi @HaakonL,

    Thanks for reporting this, and for quoting me above. :)

    A fix has now been put in place, meaning that builds should return to usual build speeds. This has caused heavier than normal build queues meaning that queue times may also be slightly longer than expected, but this should now naturally return to normal shortly.

    Sorry for the inconvenience this has caused. Please let me know if you are still experiencing issues and I will be happy to investigate.
     
  4. ollieblanks

    ollieblanks

    Unity Technologies

    Joined:
    Aug 21, 2017
    Posts:
    125
    Unfortunately, I have prematurely misinformed you. There has been another development to this situation and build times will not have returned to normal just yet. We have just found another bug which may have been introduced by a security patch.

    The team are working on resolving this right now. Please bear with us and I will update you once this has been resolved.
     
  5. PhilMcJ

    PhilMcJ

    Unity Technologies

    Joined:
    May 29, 2014
    Posts:
    183
    A quick update with a little more info: the root cause is that a minor update rolled out last week behaved fine last week and over the weekend - and then under heavier loads today (Monday) has proved to cause some significant slowdowns in build times when the system is under heavy load.

    Those changes are being rolled back now, and new builds should be gradually getting faster over the next several hours. The team are pretty sure the specific issue has been identified for the future (meaning it can now be tested for) - in the meantime, thanks for everyone's patience as this has been diagnosed and dealt with today.
     
    technicat likes this.
  6. Daddoon

    Daddoon

    Joined:
    Jan 5, 2013
    Posts:
    51
    Is this issue resolved ?

    Maybe it's not related, but my build are inconsistent.

    On the same codebase, one take 10 minutes, then the next one never figured to finish, somewhat cancelled by your services (even if they write "by a user"), after many hours.

    Oddly, i have a "Finished exporting player successfully" message somewhere but everything hang on a "Refresh: detecting if any assets need to be imported or removed" forever.