Unity 5.3 Performance issues

Interesting to know you can just stop work and put your feet up knowing they are solving specific bugs. In reality thats not how it works.. More like wait for months, and hope at best. While productivity handbrakes.

So even if an engines looking rock solid best not to use it anyway because theres a chance that a feature you haven't used yet could be showstopping buggy. (Mecanim, Enlighten spring to mind)

If Unity tested their software in house rather than in the marketplace for these bugs that would make our lives a lot easier and preserve Unitys reputation.

 

What do you constitute as a "stable" version though? The problem is that it's usually the case that this new patch fixes something that got broken in the last patch. So you can really go down the list all the way to Unity 3.5. Most of the time, the issues are either not shown in the "Known Issues" section or just haven't been encountered yet.

It's definitely a good idea to keep your last working editor in place as a safeguard. Steps like that should always be taken to ensure you don't muck up your work. But that isn't entirely on the user. Unity doesn't get to just raise that flag unload all the responsibility. There needs to be accountability on both sides.

It kind of reminds me when Diablo 3 first launched and thousands of accounts got hacked. Blizzard's copy-paste response was "You should have bought our Blizzard Authenticator, this is entirely your fault."

Sure, okay, I probably could have done more to prevent this, but don't pretend like better server security wouldn't make a difference either.

 

We had PBR in Unity not good, a thread in the forums allowed to have Allegorithmic guys attention and work with Unity to finally bring GGX. I also think some times the forum is the better way to bring crucial problems to lights and have more chances to get them fixed.

Unity can't find all combinations, because users code or custom values and adjustements make it infinite variations and cases, still i think it could help to have a complete game demo containing all features to test frame rate stability and game visuals and behaviour when adding new features.

 

You might want to conduct a survey explicitly on perceived product quality. You might want to widen the scope or increase the number of customers you address. If you did, you'd have those numbers you're asking for and without which you claim the devs and suits won't listen. The way you describe things sounds a bit defeatist, to be honest. "Can't do that, can't do that, we've already done that, tech is in denial and nobody is listening to us... So please continue to do our QA for us: submit bug reports and vote for features in the usual way, folks. You have to prove to us that you're not just a vocal minority."

 

If it is split up, they can most likely automate the testing and use their testing framework for it:
http://blogs.unity3d.com/2015/10/07/unified-test-runner-test-analytics/

Yes, the test coverage is not (yet) good enough. Having an automated solution makes in general more sense in my opinion. But I agree with everyone who is currently saying that more manual testing is needed, at least for functionality that was recently changed and as long as there are no automated tests for that.

 

Unity say they are working hard for solving bugs but I'm afraid the priority actually is not stability and performance but features race with competitors.

 

Well, it seems that, so far, automated testing just doesn't cut it. Maybe it will do with most cases, but some bugs slip through. And I wouldn't even have a problem with that. As already stated numerous times before, there will always be bugs. But if there is such a wide audience having easily reproducable and crucial problems, I am starting to wonder what's going on.

Also, this is one of my games on 5.3.1 (so far, I haven't had the time to install p1):


I tried to delete OpenGL 4.1, but it will stay there in the top bar and some "Overhead" attacks my profiler hard. While for me it is not as bad as for the others here, it sure is a problem causing occasional hiccups that haven't been there before.

 

This is exactly the kind of problem that can always happen. I haven't seen this problem being frequently mentioned for iOS. It could be that this only occurs for certain configurations and is not easy to spot with automatic testing. I certainly don't know what the actual problem is, but starting a new thread just for that would make a lot of sense in my opinion and also submitting a bug report.

 

The "Overhead" issue was fixed for me in P1, so I recommend taking the time to download and install it. Unfortunately, it doesn't fix every performance issue, as P1 is still about 40% slower than normal OpenGL 2 for me. However, that seriously beats the 98% slowdown I was seeing.
You can run in OpenGL 2 mode by adding a command line switch (-force-opengl). You'll need to launch unity from terminal in order to do that, though.

 

Well, I disagree for it not being easy to spot with automated testing. This happened on every iMac that I tested with (Yosemite and El Capitan), and it happened with every demo scene that ships with Unity. I would say they had seriously dropped the ball on testing this version in the rush to meet their ship date. Note that I am not blaming the testers -- bugs aren't their fault after all. Their process is broken, but I think they know this.

 

I mentioned that this kind of issue can happen only for certain configurations. And because of that it can always slip through, even with automated testing in place. I clearly stated too that I don't know the actual problem and as such whether it only happens for certain configurations.
In this particular case it should not have slipped through, that's for sure!

 

So I'm a troll for what exactly? This thread is about poor performance on unity 5.3, I personally have wasted weeks because of it so thought it appropriate to chip in. I didn't waste my time using the useless feedback system, I instead wasted half a day to create a repro project to submit as a bug. Then I had a whinge on here for my wasted effort. If that makes me a troll, then so be it. For the record I've struggled to get my games back up to 60 fps ever since porting them from 4.x to 5.x, time that would have been far better spent actually making games.

 

I agree, some of the bugs are so easy to spot. Take the one on the Mac where clicking on anything in the hierarchy window wouldn't always unfold the object you clicked on. For unity to have discovered that one prior to release all they would have had to do was click on an object in the hierarchy on a Mac. Hardly rocket science and no need for a repro project, any project would have done it. It only leads to the conclusion that hardly any testing on Mac releases are done.

 

@Jaimi
Thanks for mentioning! I will probably update as soon as I can.
Concerning OpenGL: But I deleted OpenGL 4.1 in the player settings, though. Isn't that supposed to do the trick, too?Or is this something different?

 

I believe that it will only take the editor setting from the player settings if you are developing on the same platform you are building for. In your case, you're building for IOS, I believe, so you'll need to force the editor to use the old OpenGL. I'm hoping that P2 will fix the most troubling OpenGL 4 issues.

 

Why are we talking about bugs? performance people performance!!!

 

Because apparently performance issues are a bug and need bug reports to get any chance of being fixed. Thousands of posts on here are apparently not enough. I've had to now disable metal and es3 to get anywhere near the performance I used to get.

 

It is certainly a lot of work. I'm myself sceptical as to the chances of such strategy being taken by Unity (I called it strategy because of how massive such a project is), but I think it is worth the suggestion and consideration. The biggest beneficiaries of such a change are Unity.

 

It is called regression testing. Blender does this, last time I checked, by using one huge test scene that contains a bit of everything. That one can't be automated and require human operator (good luck detecting lighting aritifacts, flickering and jittering with software), but is doable. Given the amount of bugs added in, say, 5.2 I have impression that unity team does not test for regressions. Not in this way, in any case.

 

I once saw some slides where they had set up an automated system that makes screenshots of testscenes and compares them to archived screenshots. Comparison can be done with a difference (at least that's what it is called in photoshop) blend mode and checking for pixels above a certain brightness, and then a mail gets send out for a human to double check if this was intended or not.

 

Well said and this is exactly what I been trying to get people understand. Automated tests cannot always cover everything and they constantly need to keep updating the tests as Unity develops. Manual/human operator is always going to be required on some level.

Example: There was a issue with Unity splash screens/launch screens a month back, where I could not release games until it was fixed, it took 2-3 weeks to fully fix this really simple bug/ issue. All it required was someone to see with own eyes iOS splashscreen upside down and not correct. But this will require a human to do. What got to me is they insisted on filing bug reports for someone that requires no repo. Why cannot QA make own internal bug report is beyond me .

 

I do understand, but really Unity need have own internal projects and not require customer projects.

Example:

5.3 Beta tests, showed that the APK size in Android was increased from 5.2 version. One customer spoted this, as he obviously has a care for the build size. Unity could have easily had a test written for this as its really a big deal for Mobile developers. If customer did not spot this it would have made it into the 5.3 final release.

Example 2:

Same thing with FPS, if internal project checked GLES2/3 performance tests on internal projects, there would have seen a decrease on Android mobile and investigated. Again this test is really important considering there has been so many issues with this in the past.

Internal projects no way can be so perfectly coded for there tests to pass each time. So there is some breakdown and asking for customers to send projects and some special cases where not good enough answer to be honest.

Mobile developers demand better performance for obvious reasons, these days people run powerful desktops so some issues not going to be seen or felt by these customers, whereas mobile its clear to see.

 

Just out of curiosity couldn't you have flipped the images for your splash screens upside down before building? Or are you referring to the official splash screens too? On a side note this is one of those examples where having the source ourselves would be beneficial.

 

The issue has been fixed now by Unity, so not going to bring up the technical side to it as there lengthy chat on iOS section of the forum.

 

Why isn't it possible to automatically test that? Unity can create a reference screenshot when the application is started. Whenever an update is made, another screenshot is taken and compared to the reference screenshot. Setting up the reference screenshot is certainly manual work, but besides that it can be automated.
It is obvious that Unity doesn't or didn't have automated tests for that.

 

I see. Thanks for your help! I don't know why, but I am kind of uncomfortable with using the Terminal, but I will try once I fail using p1, which I will update to very soon, hopefully.

Well, performance problems are based on bugs, right? I mean, it worked before and hardware hasn't changed for the worse, so it must be bugs causing worse performance.

 

I guess it might make more sense for Unity to indicate a month for release rather than a specific date, because when automated testing throws up issues* like this, they get perhaps a couple more weeks to sort it out before ship... not sure what everyone thinks?

I don't mind having ballpark release dates if it means quality goes up a bit and it saves a mad rush for patches?

* when they add more test cases

 

Im not sure if this much time is needed.
The final version of 5.3 was compiled long before it was made available to the public.
I base this on: http://autoupdate-revision.unity3d.com/revisions.plist
So technically there would have been enough time to test (I know they know bugs existed and missed cutoff point for fix) hence the big update in the first Patch, but its not clear why 5.3 was released to public with bad stability in some crucial areas, I think this was the main issue and start of the thread, if they have missed the issue after testing then yes something went wrong.

 

dont over estimate the lazy bastards

 

//opinion.
While this is technically true, there's large slew of scenarios where you really need to put result of hundreds millions of years of evolution - your brain - to work, instead of trying to make brainless machine do your job. Machine can only do what it was programmed to do, meaning human tested might spot something extra, that's not part of test case.

Until we get computer to a level where it UNDERSTANDS what it is supposed to be looking for, you'll need human testers.

 

Because machine doesn't know what it is looking for.

The place for unit tests is for machine-to-machine interaction. To ensure protocol always work as expected, to ensure that mathematical functions return valid value.

All things UI related, all things that are played via computers speakers and displayed on the monitors are aimed towards humans. For example, if something happens between intervals where system takes screenshot, system won't catch that. OF course, trying to stuff UI usability testing into unit test framework is an amazingly interesting task on which your programmers and engineers can, without a doubt, waste a lot of company time, which will cost money all the while thinking they're doing something useful. It is better and cheaper just to put human on the job.

 

I am not talking about unit testing. I have only described what Unity is already doing anyways for certain areas to ensure consistent results among different versions. The initial version is approved by a human and the follow ups can be automatically compared to that.

 

I wonder where the point is, where the disadvantage of having to test every feature on a ton of supported plattforms outweighs the advantage of having it all in one package. If it helps performance and stability I would be fine with Unity being split up in one version for mobile, smart-TVs and web, and one for consoles, desktop OSes and web. Web would make sense on both because it's important to be able to show stuff on the forum. Just my two cents, I don't know if this actually is the main issue.

 

Developing for multiple platforms becomes a lot harder if we have to use different Unity versions. Indeed, Unity has spent a lot of time to merge all that into just one software package. The amount of required testing is not really different whether there is one or multiple versions for the different build targets.

 

sounds like a good idea

 

+1

 

Very interesting that this idea got immediate support. Could anyone explain how that improves the stability and performance?

 

You could have a different release version for each supported platform. So they could focus on testing platform after platform. I don't know if that's going to change anything, though, because eventually they will have to get through all the platforms anyway.
Also I see a major drawback: You would have to install dozens of gigabytes if you want to support a number of platforms.

 

Unity is adding more test cases to their build testing suite, and they are discussing it now or very soon how to improve it all, as far as I understood from various staff posts on the forum. I can't remember where, but I am certain staff will be responding to the thread and everyone's concerns

 

You mean previous discussions ?

 

I mean splitting Unity up into different versions.

 

I didn't thaught on Unity versions, this would become unmaintainable very quickly.
I just meaned having a complete mobile project containing all features to make tests, and the same game project with highter features for desktop testing. It was more like different versions of a complete game project to test performance when a feature is added and stabiltiy and to be able to check that anything is broken at first look.

 

That's something you don't want to do. It will quickly turn into maintenance nightmare, and ideally you'll want one codebase for all.

Just regression testing framework would've been enough.

 

The splitting up was not my suggestion.

Unity has already a regression testing framework in place. It is simply not (yet) used extensively enough.

 

Unity needs regression testing and integration testing. They also need to test things on as many platforms as possible. We simply shouldn't be experiencing crashes when doing basic things like changing scenes, on any platform. We shouldn't be experiencing graphics regressions making whole platforms (such as the Mac) unusable. Not even with minor releases. This is especially true for Pro users, who have paid/are paying for the product.

Testing isn't rocket science. I've spent the last four or five years doing TDD in a fully automated C.I. pipeline featuring extensive tests on multiple platforms and clients. It can be very complex, but it isn't rocket science. Unity as a company knows this (or they would be out of step with the software industry as a whole).

The QA video mentions that the Unity QA tests have only about 60 per cent coverage. This needs to be improved dramatically. It's also evident that developers only test stuff in their own silo. This needs to change, and the silos need to be broken down.

While there still is time.

 

I hope they add more automated graphics tests like this on both low and high end PCs etc. because they don't seem to be catching some issues like shadow maps in 5.x - we have bad shadow map issues on lower-end PC hardware when pushing the light count despite ample VRAM. This would be easy to catch with screenshot tests, though would require the right stress tests in the first place of course which may be the biggest issue (it may be with shadow maps they just have too many ways to break, but if so more documentation on limits or in-editor warnings would help).
Anyway slightly off topic - more focus on bugs, less on features please

 

Well I definitely want features like instancing which improve performance...

 

That Red Green Show is actually funny...especially the black & white sight gag cartoon style segments they do from time to time.

 

I just updated to 5.3.1p1 and this actually made things worse. Now, Camera.Render is going insane, too. Even in the Editor there are major hiccups. And it's supposed to run on iOS without any performance problems, which was the case in 5.2.
Here are some screenshots:







EDIT: The FPS in the Profiler are estimated depending on the platform that is selected in the Editor, right? Please, let me be right.
Also, sometimes Camera.Render even goes up to 410ms.

 

Pretty sure the profiler doesn't make any "estimates". Performance in the Editor can be pretty different from performance in a build, and it is not a linear factor across all usecases either. If you care about iOS performance you need to test a build on a device, there's no way around that.

I'm on 5.3.0f4 and don't have any big problems, but I haven't done much with it either. For the time being I'm not updating till I encounter grave problems with this version.