Search Unity

  1. Calling all beginners! Join the FPS Beginners Mods Challenge until December 13.
    Dismiss Notice
  2. It's Cyber Week at the Asset Store!
    Dismiss Notice

Feedback Question regarding my understanding of what Beta is for

Discussion in '2019.2 Beta' started by fherbst, Jun 11, 2019.

  1. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    408
    Hello Unity team,

    me and others on the forum have found that recently, the process for working with the beta seems to converge on variations of the following:

    I used 2019.1 beta to start a new project where I wanted to use those features, knowing that the project would not end before 2019.1 released (and knowing that some porting might be necessary to get there).
    After reporting numerous bugs through the 2019.1 beta cycle (some of which got fixed), 2019.1 finally released - and introduced crazy regressions not present in the beta versions.

    Reporting bugs on them got me the feedback to update to 2019.2a since the regression fixes weren't expected to be backported. I grudingly did, now knowing that my project might have to release with 2019.2 beta.
    Beta came, some bugs were fixed, new regressions introduced (notably, objects just downright disappearing from builds due to their variants being stripped - here, here, case 1159877 - , or not being able to switch between LWRP profiles at build time - case 1157408, case 1157420). As a good citizen, I reported them, assuming that such regressions would be nearly immediately fixed (since it would in my opinion be impossible to release 2019.2 with those bugs in it). However, reply from QA was that I should upgrade to 2019.3a since the fixes wouldn't be backported. Back to the beginning!

    This has lead to:
    • reporting bugs on 2019.1b that weren't fixed for 2019.1r and made it impossible to use the release version since LWRP is unusable besides very simple cases
    • reporting bugs on 2019.2b that won't be fixed for 2019.2r and make it impossible to use the release version since LWRP is unusable even in very simple cases
    • being stuck again with an (obviously) unstable alpha version with whoknowswhat of new issues
    What can I do to prevent that cycle from going on indefinitely? Is this supposed to work like that? I would really love if someone from Unity could clarify. Most of my issues/bug reports revolve around just using the LWRP (that is, as you keep pointing out, out of preview). And I'm getting really frustrated. I like testing early versions, I do report numerous bugs, but it's getting so frustrating and time-consuming - and useless! - that I'm not sure how long I can do that.
     
    Last edited: Jun 12, 2019
    iamarugin, konsic, Immu and 1 other person like this.
  2. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    6,411
    It is completely normal, that things are getting added and removed during alpha and beta versions.

    Some are only experimental. Other intermediate steps, which further require evolution. Which doesn't mean, it will be same in next release ofcourse.

    I also like beta testing. But, while sometimes this may be a bit frustrating, when things get added and removed, I am completely fine with Unity work flow. In the end, we can try and test things early and help shaping dev direction. Which wasn't possible in the past. But I don't expect that things get fixed for my currently owned version. However I know, I will need download later version at some point, while also expecting that I will need refactors / update prototypes accordingly.

    That of course assuming, I understood your question.
     
    Cromfeli likes this.
  3. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    408
    No, I'm not talking about adding/removing features (which is fine), but reporting bugs during beta which are deliberately not fixed for release but instead in the next alpha, while impacting a large userbase. Refactoring with a newer beta or release version is a given, no problem with that.

    In my impression the idea is that alpha/beta testers take the risk of bugs / inconsistencies and gain the benefit of making sure the release works for their usecase.
     
  4. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,409
    Its always dangerous to assume that bugs which are a complete showstopper for you are actually considered serious enough that they would not dare release a final version with those bugs still in.

    Having said that, I am concerned about some of their decisions not to backport certain things. And I was surprised by how quickly their focus seemed to switch to 2019.3.

    And all manner of things that I am more than happy to put up with when they relate to packages that are still in preview (eg HDRP) really should not apply to LWRP. But I dont really see much difference between how things were handled when LWRP was still in preview, and how they are handled since it 'came out of preview'.

    I'll put up with it for now but I hope 2020 is more settled.
     
    fherbst likes this.
  5. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,409
    A couple more thoughts:

    I have limited personal experience of being trapped by bugs and release cycles because I have been very lucky over the years, and tend to use only certain aspects of Unity on a few platforms over prolonged periods. I probably have to go back to Unity 5 to think of a time where I felt a little hung out to dry for a prolonged period. And I can usually afford to lurk on 'the bleeding edge' which means github SRPs, Unity betas and these days alphas, although I am more wary of those.

    I would be curious to know if the public availability of alphas these days has made any difference to backporting decisions. If it has, then I would ask Unity to think again about this. It is possible that it has made no difference, and its just our perception of these things that is now altered by the fact we can see alphas, their release notes, and receive advice from Unity to try the alpha rather than wait for a later release of the previous version that may also have the fix.

    On a related note, I feel that it is now quite clear that some of the promise and potential of the 'modular Unity approach', and the scriptable render pipelines, is held back by how much is still reliant on c++ engine changes. I think this is quite a large topic that hasnt had as much attention as I probably thought it would get. Its a tricky one, I always knew it would be some kind of issue without an easy fix, but perhaps I was overly optimistic about how quickly the situation would start to lessen in significance.
     
    ekakiya likes this.
  6. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    408
    I also had very good experiences with this in the past years. It seems the modularization plus the release of SRPs plus the alpha releases results in basically no (publicly available) version fully working together.

    Example: it took a year until ARFoundation got LWRP support; now it only works with a very very specific LWRP version that is not supported anymore (4.x); which results in all examples being broken. Both ARFoundation and LWRP are out of preview, which doesn't mean that they work together. In my opinion, the modularization has led to less testing, not more, because individual components "work in themselves" but aren't tested against other parts anymore - which gets exponentially more difficult as more parts are package-ized.
    Previously, when everything was monolithic "Unity", I was always very confident that different subsystems work together properly. I learned the hard way that this is not the case anymore.
     
    zakiazigazi likes this.
  7. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,409
    The thing is that I'm still a huge fan of the modular approach and the render pipelines, and very much the public access to beta, alphas and github versions of packages in various states of development.

    And I still believe it will get better over time. Some things will improve naturally as packages and systems mature and stabilise. Other improvements do require concerted efforts by Unity.

    Right now I see it as a giddy mix of immature packages, scheduling & release differences between different packages, c++ engine side issues, and the amount of resources Unity allocates towards maintaining certain packages, assets etc. And I'm way more forgiving of issues that arise as a result, when the packages in question are still in preview. There are situations with HDRP I will applaud, such as Fontainebleau getting updated with newer HDRP versions, and ones that lead to facepalms, like Unity flagship Icon assetstore packs for HDRP not currently supporting 2019.1+ versions of HDRP, even the latest one that was release only days ago.
     
  8. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,920
    Alpha is a good place to ask for change or contest Unity's decisions while breaking things and being broken. It's for people who care enough about something to get that dirty.

    Beta is mostly for people too impatient. It's a mutual relationship: the impatient get their toys but Unity gets their bug reports too. It's doubtful much will change with a feature at this point.

    Release is what Unity calls beta when it's too late to keep working on it, and hope the hard working unsung heroes backport things... hehe
     
    GameDevCouple_I likes this.
  9. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,409
    I should say that my comment about switching focus to 2019.3 was not intended to be a far reaching accusation.

    For example I do not doubt the amount of heavy focus that the HDRP team are putting into 2019.2 branch of HDRP.
     
    hippocoder likes this.
  10. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    4,634
    I think when you actually do a project on a version line, you should think about the LTS versions. If you start on the 2019 lineup, expect that you will have to finish your project on the 2019.4LTS.
    I know, it's not officially announced and all, but as far as their backporting strategy goes I found this the most viable.
    If you don't want to put up with the bug reports and all, stick to the last LTS, which means 2018 in your case.

    I know, it's frustrating, sometimes it is for me as well (also started my project in 2019, I'm moving between alpha and beta lineup, depending which bugs/problems bother me most at any given moment), but on the other hand, they have to break up the monolithic C++ side as well, considerably. If they were choosing to backport everything, that sometimes would mean to modifying certain aspects of the C++ side as well. And sometimes the best solutions aren't compatible with each other between versions. It is sad and all, but still a thing to consider.

    Obviously, everyone appreciates if you report your found bugs but also no one will blame you if it would hinder your release so you would move to a stable version and drop the 2019 cycle for now.

    PS: if a Unity dev drops in here, I'd like to ask to modify your process and please, announce not just the version you're fixing the bugs in, but straight away announce the fix backportability too among affected versions. It is crucial to us living on the bleeding edge to know what to expect.
     
    Last edited: Jun 12, 2019
    elbows likes this.
  11. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    408
    @Lurking-Ninja , point is, obviously I want to release with a stable version! That's why I start with a beta of something that is released in the foreseeable future if I can benefit from the new features. I'd like to get some feedback not only from the community, but specifically from the Unity team...

    Here's a quote from the "What to expect from a beta" site:

    As with any beta program, you’ll have early access to new features and will be able to assist in the final steps of their development. That means you’re likely to experience Unity as less stable than a final version.

    The beta phase begins once all scheduled major features have been included and a quality baseline has been established. Several beta versions will be made available during this phase and quality will improve with each iteration.​

    That's what I would expect. But 2019.1 release was less usable than 2019.1 beta, and 2019.2 is on the same path, if Unity developers / QA keep pushing me to alpha versions for major bugs / flaws.
     
  12. elbows

    elbows

    Joined:
    Nov 28, 2009
    Posts:
    2,409
    Oh, thats not quite the same as what we were talking about earlier, and is rarely my experience. In what way is the release worse than the beta?
     
  13. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    408
    Hm, there were regressions from 2019.1.0b10 to 2019.1.0f1 (and ongoing) that prevented us from using that, and Unity just said "use 2019.2a, it's not gonna be fixed in 2019.1".

    Here's a more detailled chronology in case you're interested:
    • we were using 2019.1 beta for a while, enyoing new features while reporting occasional bugs
    • 2019.1 was released, both with fixes we desperately needed and new regressions that prevented us from using it (unable to sign Android builds, particle system crashes on launching an Android VR app, LWRP VR build issues and flipped images, all not happening in the beta versions)
    • Unity support told me to move to 2019.2a since the fixes were there and not planned for backporting (in fact, most of these are still broken today in 2019.1)
    • Grudgingly moved to the alpha which "just worked" but was alpha anyways
    • Found new bugs, reported them, 2019.2b came with new regressions (LWRP objects just downright disappearing in builds, shader stripping being so aggressive that you can't switch pipeline assets at runtime at all, shader variants not properly being activated changing when loading scenes additively, environment lighting being broken randomly in Android VR builds)
    • Reply of Unity support was to move to 2019.3a since some of the issues are fixed there with no backporting plans. That's where I got kinda angry because the pattern repeats and I don't understand the idea behind beta anymore.
     
  14. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,920
    Don't forget packages can change the perceived stability, so make sure the issue is with Unity not the packages as well.

    This sounds like a very specific issue with AR, so it might well just be isolated to that. It does not help you, I know but sometimes these things will happen if you for example, start developing with a hololens experimental build which might or might not be something Unity can ship.

    The only real way to be immune from this is simply... don't use betas. I don't, for our main project - only release version upgrades, though I do download alphas and betas at least once a week to keep tabs, post bugs on my bug account and so on (so I don't get tripped up later).

    But yeah does sound like you specifically got caught out in a bad way, it's never usually a case of release being worse than beta... hopefully.
     
  15. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    699
    On the bright and off-topic side, that's not hugely difficult to fix, particularly because modules are shipped with source. We use ARKit with LWRP, which is a combination that has never even seen a public working release.
     
  16. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    699
    With features I would be inclined to agree. However, these sound like actual bugs that slipped into final release after being previously fixed, affecting core systems. That bit is not normal at all.
     
    fherbst likes this.