Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Feedback The process for submitting bugs is too cumbersome for us to justify submitting many bugs we find.

Discussion in 'General Discussion' started by hoesterey, Jun 6, 2022.

  1. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    The current process that requires an example project in order to submit bugs is causing many bugs to go unreported and this leads to multiple community members needing to spend time identifying and working around the same bugs.

    Unity Goals

    1) I would assume your goal is to get feedback from the community on how to make the Unity better
    2) I'd assume you do not want to spend engineering time on non-issues and user errors.

    My Goals
    1) solve issues that effect my teams products or workflows
    2) to evaluate the feasibility of using new pre-release features for my company
    3) be a good community member that contributes back to the community.

    The problem
    Creating test projects / or uploading ours is time consuming and problematic. We have tons of proprietary work flows and even stripping a project out of our custom dev environment is hugely problematic

    Our Reaction
    We only submit the issue if it is a showstopper or we have extra non-product critical time to contribute to the community. If a bug isn't a show stopper for us we often don't submit it as it will be closed no-repro without a test project even if we provide code examples that demonstrate how to make it happen. Even if we can tell you how to make it happen by copying our code into one of your own demos.

    Effect on the community
    The community members will spend time finding, identifying, and working around the same issues we found previously.

    Example
    Case 1419941 is a great example of an issue that is in pre-release. The issue was created in an attempt to be a "good community member". The code sample and explanation I provided should be enough to evaluate the issue. This is an issue that multiple users have reported:
    https://forum.unity.com/threads/you...-the-onunloadeventcompleted-callback.1267208/

    NOTE: in this case the multiplayer team has a separate process to evaluate things on "git" and this was resolved in that way. However this would not have been resolved through the normal process and as it is not effecting one of our products and we can not justify spending the time required to create an example project to help you repro the issue. In fact I didn't know this would be evaluated through the git process and have failed to get this looked at with unity's official process without attaching an example project.

    Suggestion
    Evaluate the likelihood of an issues and have a team who has the technical chops to evaluate the problems without always putting it on your community to do the work of creating an example project for you. I understand you probably get countless bogus issues. That you often cannot repro an issue without a test project. The current process however just leaves us not submitting bugs.

    When simple code samples are provided with a clear description of an issue that should be enough Case 1419941 is a great example of something unity should be able to evaluate without the community creating an example product for them.

    Creating example projects wouldn't be an issue if we were finding a bug a month, but sadly the cadence is much higher and whenever we do a major upgrade (version 2021-2022) we find multiple bugs daily. We simply don't have the time.

    I'd very much like to see unity move to a single official process that was like the multiplayer teams response in git where the issues are evaluated without example projects.

    My Experience with Unity:
    My team spends a huge amount of time working around bugs within unity. Our last cross platform project shipped on four different versions of Unity. Different versions for each platform as each version had different bugs we could not work around. We completely stopped sending in bugs as we neared ship, we would have needed a dedicated head to create all the test projects.


    My Background:
    1) I am a studio head that manages a internal game studio and two external game studios.
    2) I have a background in publishing/firefighting where my job was to learn and evaluate tools, pipelines, and processes. I still spend time personally evaluating new Unity features for my studios.
    3) I have used Unity to ship AAA product
    4) I have shipped product with unity on every major platform.

    Hopefully these thoughts help improve the process.
    Thanks.
     
    Last edited: Jun 6, 2022
  2. frosted

    frosted

    Joined:
    Jan 17, 2014
    Posts:
    3,998
    My suspicion is actually that the cumbersome nature of bug submission is somewhat by design.

    Unity is in a weird place where they're providing software tools for developers but the majority of their user base is most likely still not professional or experienced w software dev. They probably need some degree of barrier to bug reporting to prevent being buried by amateur programmers with syntax errors.

    I recently went through some bug submission stuff with jetbrainz and it was a really exposed process that included direct communication w their staff. The bug got fixed, fast. It was super impressive. But I don't think you could do the same kind of thing w unity's userbase.
     
    Last edited: Jun 6, 2022
  3. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    Yes. I'd suggest they have a different process for people with pro licensees vs. those with free. I think that would filter a huge portion of the less experienced community.
     
    MadeFromPolygons likes this.
  4. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,746
    They do have *some* things differently for paying customers, but just because you cough up some money they won't be able to magically reproduce your problem if you don't show them how, usually.
     
  5. BaBiAGameStudio

    BaBiAGameStudio

    Joined:
    Mar 31, 2020
    Posts:
    292
    This is basically how bugs have been logged at all companies I have worked at for the last 30+ years. If someone wants a bug to be properly investigated they need to provide something that allows it to be replicated quickly by the developer(s).

    Otherwise what can happen is that the developers spend loads of time building projects to test the issue only to find that it works fine for them - simply because their way of building stuff does not fit the edge-case situation that the client is seeing.

    Some places where I have worked will simply reject submitted bugs if there is no supplied project or replication steps to follow as without these it is just a waste of time for the developers.
     
    ontrigger, NotaNaN, elcionap and 4 others like this.
  6. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    9,611
    This is the correct attitude by the way. But only the real showstoppers.

    If you can work around the bug by, say, dropping a Unity feature completely and implementing your own solution, then it's not a show stopper.
     
  7. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    I didn't say they we wouldn't show them how to repro the issue and example code, I said it wasn't worth our time to create a test project.
     
  8. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    I cannot tell if you are being sarcastic.
     
  9. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    I did not suggest we wouldn't provide anything. If you look at this thread as an example we provided a code sample and a repro. This wasn't enough to submit the issue through official channels:
    https://forum.unity.com/threads/you...-the-onunloadeventcompleted-callback.1267208/

    Any developer worth his salt could look at that code sample and do a quick evaluation. (and a unity developer did with the git process). While the multiplayer team has a unique process allowing users to submit bugs without repro projects through git and do investigate issues like I posted this process is not shared across all of unity.
     
    MadeFromPolygons likes this.
  10. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    8,746
    Yeah, the multiplayer team doesn't have their package in production. The usage of their packages is limited since they are not production ready. Hence the multiplayer team shouldn't used as the baseline what's viable and what's not.
     
  11. Lukas_Andriejunas

    Lukas_Andriejunas

    Unity Technologies

    Joined:
    Aug 19, 2016
    Posts:
    16
    Thank you for sharing this. Your feedback is very important for us. In case of 1419941, we could have done a better job investigating this and we discussed with the team how to handle such and similar cases better in the future.

    Having said that, I would like to mark that a significant majority of the reports require a project. This greatly improves the process of investigating the issue, makes the bug fixing process more efficient, and helps us to ensure that the verification process is correct and the PR has actually fixed the problem.
     
  12. timsibiski

    timsibiski

    Unity Technologies

    Joined:
    Jun 3, 2021
    Posts:
    13
    I want to second this. At least 8 out of 10 tickets that I encounter will ultimately need a project to reproduce properly, and then make a fix for. (This excludes tickets for issues that are already known and waiting for a fix. If I counted them, it would be more like half the tickets I work with.)

    But providing access to a project immediately will dramatically affect how quickly we can investigate and give you a meaningful response. This is because, if we don't get a project, we still generally need to try to reproduce the issue without it. That is a chunk of time spent reproducing the issue from scratch, which is rarely fruitful (because many released Unity issues involve multiple combined/complex situations to trigger an issue). And that time could be going to getting a fix in, or finding a workaround for you. Which isn't to say that we would complain about that. We are happy to do what it takes to find a fix whether you provide a project or not!
     
    NotaNaN, Ryiah and hippocoder like this.
  13. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,722
    Yep, I don't understand how a dev can't see why unity needs a repro project. That's the sort of thing that's required by any programmer. It should be self explanatory.

    Otherwise it's basically asking a doctor to diagnose your problems without ever seeing you.

    If they only had you and not thousands of others to look at. If your company is doing well selling that many titles, then Unity has dedicated paid support to speed things up.
     
  14. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    They didn't need a repro project if you read my entire post. They looked at the code sample I provided and identified the issue. I'm in complete agreement that sometimes a repro project makes sense. other times it does not. I don't think a one time fits all rules is ever viable.

    We have paid for unity support both when I worked at Microsoft on HoloLens and Recore. At my current company we use unity as a visual editor tool rather then a engine we don't need support as we don't use the runtime.

    My main point here is that when I evaluate the engine (because we are considering using the runtime) and find issues the community isn't benefiting from my findings. That was also true when I ran an indie company as we just didn't have time.
     
    angrypenguin likes this.
  15. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    I totally get it and I can imagine. I'm sure you waste countless hours when the problem is on the user side. Keep in mind what I'm talking about isn't issues we care about a response for. If we wanted a response we'd create you a test project. I'm talking about all the issues we find that cost us time to identify, but where the workaround takes us less time then uploading a test project. Then we just don't bother and thus there are probably countless people hitting the same issues.

    For what its worth I'd be much more apt to share test projects if we didn't have a custom dev environment. The problem is you don't have our development environment, and that is so hard to get you setup with. So we have to break out the code into a separate project, and that can take quite some time.

    Thanks for listening I hope my feedback is received as someone with the best intentions who is trying to help.
     
    angrypenguin and timsibiski like this.
  16. hoesterey

    hoesterey

    Joined:
    Mar 19, 2010
    Posts:
    659
    Agreed that most issues need a test project. In the end I would just love a way to "link" issues we find to features in a way that made it easier for the community to benefit (even if an engineer didn't evaluate). Something like Unity's issue tracker but linked directly to the documentation :)
     
  17. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    14,959
    Having done projects with both, I've not found this to be the case unless it has changed at some point.

    Edit: I misinterpreted that. It's a suggestion of how it could be, as opposed to how it is.
     
  18. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    14,959
    Many times I don't, and Unity still gets back to me on at least some of them. I read a staffer somewhere saying that they'd rather get useful info than nothing, even if we can't attach a minimal repro, so thats what I've gone by. Sometimes I attach a video showing what breaks and when, and of course a clear written report along with the logs.

    Sometimes the response has been to ask for a project, and that at least cuts down my effort to the cases which Unity is interested in looking at. In those cases I sometimes make a minimal, separate repro, and other times just send them my tens-of-gigs project with a list of repro steps.

    So my main thought is that even if you don't have time to hit Unity's ideal bug report level, if something is better than nothing and you've got quality data, send it anyway. The team does seem to pay attention.

    I can understand why they don't advertise that and would probably even discourage it in general, of course.
     
  19. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    9,611
    I am not.

    It's often a lot less effort to re-implement a Unity feature yourself (and frankly, you'll probably do a better job than Unity) than to try and make Unity fix the bugs for it in a timeframe that falls within your game's cycle (and I'm saying this, someone who is working on the same game for almost a decade).

    So only attempt it for real show stoppers where there is absolutely nothing you can do about.
     
    Last edited: Jun 19, 2022
  20. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    9,611
    Our time preparing a project with a minimal test case with super clear instructions is also rarely fruitful, because a bug fix rarely comes in time for it to be useful, if at all, so we are at an impasse.

    And from my point of view, Unity bugs are your problem, not ours. So you waste your time.
     
  21. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    14,959
    As we have to eat the impact of the bugs on our work, they're very much our problem.
     
    MadeFromPolygons likes this.