A Guide to Being an Effective Beta Tester

So you’d like to be a Unity beta tester. First of all - thank you!

Our QA staff works hard to make sure that our releases are stable, but there is no way we could do it without members of our developer community providing feedback on upcoming builds. We’re very happy that you want to help out!

To help you help us (help you), we have created this guide to being the best beta tester you can be - how to upload your project, how to write a bug report, and more. Following these steps will allow you to provide us with the most important information so we can fix problems that affect you and the development of your projects.

Your feedback on the beta is invaluable. We analyze every beta bug report rated 4 and 5, and do our best to look at those with lower ratings (if you’re not sure what that means, take a look at this blog post). When we have validated the bug and have a fix, we will schedule it for an upcoming beta release. We are currently unable to provide details on which fix will be in which beta, but know that it will be as soon as possible!

If you have further questions about the beta, visit the beta forum.


Part One - Installation and getting started

1. Download the latest beta build here. This page also contains information on the new features to test and stress.
   
   
2. Run the Installer, choose your options as usual, and be sure to install the beta in a new directory. It’s ok to have multiple versions of Unity side by side.
   
   
3. Make a copy of the project you plan to use for testing the beta, then open it with the latest beta version.
   
   • Make sure you create a copy in case the beta has regressions that might affect your project.
     
     
   • Note that when Unity opens a project, it automatically migrates the project to the version of Unity you’re using. So working on a copy of your project also saves you time to avoid having to re-import your project when you go back your current stable version.
4. Develop as usual and/or test new features and updates. If you think you found a bug, follow the next steps described in parts two, three, & four of this guide.

Part Two - The Beta Forum

So… you found a bug. First things first: go to the Forum

The Unity beta forum is a great community resource. You can see what other people have reported, get suggested workarounds for issues, and keep up to date with fixes. It’s also a good way to get in touch with someone at Unity if you have additional questions or information.

If you don’t find anything related to your bug, then it’s time to submit a bug report. Once you’ve submitted a bug, it’s also a good idea to come back to the forum and post a description of the bug you’ve submitted, so if others have the same issue they can add more context or provide their workaround.

Make sure to include your Case number (provided in your confirmation email) in the post so that our team can identify the bug report you submitted - it’s the first thing they will ask if you leave it out.

Another great resource is the public Issue Tracker, where you can vote and comment on bugs reported by other users. This helps our team prioritize which bugs to tackle first.


Part Three - Documenting and Reporting Your Bug

Bug reporting can seem a bit intimidating but it’s really not so tough, and it is vital to ensuring stability. Follow these simple steps to write a good bug report that our engineers will be able to easily understand and act on.

Open the Bug Reporter

While running Unity, go to Help → Report a Bug in the menu, or you can find it installed next to the editor in the program folder. It will also launch automatically if you experience a crash.



Provide Basic Information

In the “What is the problem related to” field, select whichever option aligns best with the bug you are reporting. Since you’re reporting a bug in the beta, it will usually be “A problem with the Editor” or “Crash Bug.”

In “How often does it happen,” you will need to indicate if this is a problem that you have only experienced once, sometimes, or every time you take the steps that led you to encounter it.

Provide your email address in case our team needs to contact you for more information.

Indicate whether the bug should be shared with the community through the public Issue Tracker. Only the text written in the “Title” and “Describe the problem” fields will be made publicly available, and sharing your bug helps the community. Other users will be able to comment, vote (which aids with prioritization for fixes), and see when a fix is available. If you choose not to share your bug publicly, all information will be kept confidential.



Identify the bug

In the most concise terms, how would you describe the bug? Keep it short and specific, like:

Clicking the Build to PC button causes Unity to crash.


Categorize the bug and write the title

If you had to categorize the bug, what would pick? UI? Asset import? Scripting? Specific Platform? We generally put editor crashes in their own category, so that’s what you would pick in this scenario.

Okay, now add that and your bug description to create the title in the following format:

[Category] description

In this scenario, your bug title would look like this:

[Crash] Clicking the Build to PC button causes Unity to crash.



Provide the Steps to Reproduce

The Unity QA and Development teams need all the help you can offer in diagnosing and fixing an issue. Depending on the information they receive, they may not be able to identify the root issue, or they may get misled and fix something else that isn’t your bug. So, it’s in your interest to provide as much information as possible up front to make sure your issue definitively gets addressed.The easiest way to do this is usually to backtrack through the steps you took prior to encountering the bug. So, what was the very first thing you did before you saw the bug?

Clicked the Build to PC button.

So that is the last step in the Steps to Reproduce. What did you do right before that?

Opened up the Build Settings.

Keep doing this as far back as you can remember, ideally to when you first opened up Unity. The more information you can provide, the easier it will be to reproduce and fix. If you can’t remember everything, see if you can reproduce the bug on your own. If that works, provide those steps. For instance, if you’re not sure exactly what you did before the bug occurred, but you can reproduce by just opening Unity, opening Build Settings, and clicking the Build to PC button, your steps would be:

1. Opened Unity
2. Opened the Build Settings
3. Clicked the Build to PC button
4. Unity crashes

Please note that you do not need to provide the steps in written form - for instance, you can submit the steps via video capture of your screen. What is most important is that it clearly illustrates the steps so that our engineers can recreate the bug.

Add the Expected Results

What did you think would happen before you encountered the bug?

For it to Build.

There you go! Use the following template to add your expected results:

Expected Results: summary

So to review, your bug report should currently look like this:

[Crash] Clicking the Build to PC button causes Unity to crash.

Steps to Reproduce:

1. Opened Unity
   
   
2. Opened the Build Settings
   
   
3. Clicked the Build to PC button
   
   
4. Unity crashes

Expected Results: Clicking the Build to PC button will continue and make a build.



Attach Your Project Folder

The Bug Reporter will automatically include your entire project folder in the bug report. Unless your project is already very small, it is recommended to remove the attached project folder and provide a new one that is smaller and more focused. This allows our QA and Development teams to isolate the issue more efficiently and provide a fix much faster.

To get the minimal project that reproduces your issue, you should try exporting the scene in which you encounter the bug, Then import it into a new project and see if the bug still occurs. If it does, upload the new, smaller project. If it doesn’t, you can keep trying with larger versions of the project.

The smallest project that recreates the issue is ideal, but large projects are definitely better than nothing, so please do include your whole project if you’re unable to narrow it down.



Perfect! You’ve put together an informative and concise bug report that our team can use to find and fix the issue. Just one last step:

Submit Your Bug Report

Hit “Send” to submit your bug report.

When your bug is submitted, you will be sent a confirmation email containing the Case number, which you’ll need to hold on to. The email will include a link to a web page with the current status of the bug. You can check back on that page anytime for an update.

Thanks to forum user @Peter77 for the following tip on keeping track of your bug reports and project folders:

1. Anytime you submit a bug report, create a .zip file of the project that you attached.
   
   
2. Keep a .txt file of the bug report itself and keep it in the Assets directory so that you know what error the project shows and how you can reproduce it.
   
   
3. After you submit the bug report, grab the case number provided in the confirmation email and prefix your .zip file with it.

This way, you can quickly find the project you attached to a report once Unity sends you an email to notify you that your bug report has been closed.

Upon receiving the notification from Unity regarding your bug report being closed, you can check to see if the bug has been fixed by finding the appropriate project and opening in the latest Unity beta. Follow the steps to reproduce, and you can easily determine if the bug is gone.

While this is a bit of work on your end, it means a) you are more likely to have a high-quality bug report, which means your bug is more likely to be reproduced and fixed, and b) you can easily ensure that the bug has been fixed in a future beta version.

Part Four: Follow Up

It is always ideal for our staff to be able to get in touch with you in case they have questions. Of course, they have your email address and can get in touch that way - be sure to respond promptly to any emails you receive from the team to ensure a faster bug fix.

If you didn’t see your bug mentioned in the forums, you can create a post. Remember that others may have experienced the same problem, so posting any workarounds you find is a fast way to make friends in the beta community!


That’s it!

Thank you for taking the time to learn how to be an effective beta tester. If you have questions about any of the above, please post them below.

 

That's all good as we all love a more stable version of Unity. Here's my contribution that can help you move faster toward the same goal:

A Guide to Being an Effective Engine Developer

1. Build the engine
2. Test it with several actual projects built with the latest stable version
3. Fix bugs that prevents these projects from importing, playing or building
4. Release first public beta

I know it can be hard at first, but once you master this guide, you'll release more stable betas and allow more people to submit bugs

As of Beta 17, we still can't build and play Kona. We therefore may be sitting on top of an iceberg of other bugs, hidden behind, you know, being able to test our game.

 

We appreciate your advice, and in fact already follow what you suggest.
(Though, we also weigh leaving some bugs unfixed to get other fixes out to the public)

We are usually running through numerous projects weekly, but even with all that testing it's still a fraction of the tens of thousands we know are out there and pushing the engine in different ways.

Have you posted us the bug reports? Could you refresh my memory so we can check on them?

 

I tried to subscribe to pro support and never got an answer, so I still wonder what projects you're using to run your weekly tests. I simply hope titles like Firewatch, TLD, Mordheim and other complex 3D projects are part of your test plan. Ditto for popular Asset Store packages. If they are not, something isn't right.

Then case 786090 took 18 beta releases to be fixed (removing image effects from a camera can look simple, but when you have a game with an actual ambiance and character emotional matrix binded to image effects, that's quite a hack to get around this limitation).

Then there is this crash which can't be reproduced in a smaller project hence the need for a platform where we can sync our projects for you to test (we are a studio of 6, spending a whole day shrinking 32 gb project and pinpointing the exact cause of the issue is simply out of question. Plus last time I tried to send more than 20 gb, one night wasn't enough to upload).

Then there is this build error, which is a "known issue", but still prevent many projects to build.

Then all the other crashes I encountered and simply decided to rollback to 5.3 because having a chance to win $100 isn't enough for me to spend several hours playing the QA guy and finding repros. Hence the need for Unity to allow pro devs to sync their projects so you can test them on your end.

I'd gladly sync our project weekly so you can test it but there is no official way to do so.

 

QA communications can improve with the following:

1. When fogbugz sends the email with the case number, include the details (for those keeping them).
2. When you email a customer back, auto-quote the full thread of exchange, and include the fogbugz link at the bottom. Most of the time QA gets back to me 2-3 month after I logged a bug so I don't remember the details plus delete all the fogbugs email as soon as they arrive.
3. Send image or video with the repro steps you did rather than "I can't repro" - I end up asking for this anyway and delete emails that don't.

 

Useful post! Definitely picked up some things to increase the likelihood of my incidents turning into bugs.

Noticed a small typo in the post at: "If you had to categorize the bug, what would pick?"

Keep up the good work!

 

I totally feel you, man.. I am trying to get Unity to be able to access the source of our 50GB game for bug repros. I've been to two Unites collecting tons of business cards, spoken and tweeted with numerous people and devs but up to now, there must be some magic (or rusty) structure inside Unity that prevents it from just downloading our GIT repo..

Or maybe they already got enough "big games"..?

The really sad thing for me is, that most of my bug reports are not problems for us anymore. We already hacked/patched/worked around the problem. I want Unity to be able to reproduce bugs. Many repos are just "launch the editor, press X, crash" within our big project. It just takes way too long to strip the problem down and there is really no incentive for me to do this if usually the easier "fix" for us is to remove some function calls.

I am still dreaming of a way to just upload our GIT, write the hash in a bug report and they can reproduce a bug (or at least check whether its a duplicate) as easy as we can do it in our intern QA..

 

Aren't you basically describing Cloud build functionality here?

 

Regarding what Scholz mentioned about pasting git repo instead of uploading project,
Is there any thought on the unity team to allowing this for bug reports? It would speed up the process drastically to be able to just paste a url to git repo rather than uploading our entire project to you every time

 

Yes, that is an option to consider too. I was happily surprised that Unity finally dropped the 10GB limit on cloud build making it possible to (mis?)use this service to just upload data - even if the build can never succeed because of lots of our build customization options.

Question to Unity QA: If I would specify a reference to a project enabled in cloud build in a bug report, could you clone the GIT sources that this behind this project?

 

Unity should test on more games and not simple test cases.

 

I've reported several windows store bugs with the 5.4 betas and the bug uploader always reports a successful upload however, I have never received a confirmation email for any of these bugs. How should i follow up on these bugs? These bugs were submitted weeks to months ago and yes I have checked my spam folder to make sure they didn't end up there.

 

I'd love to be a better beta tester, and would need some help doing so. We have an active game with nearly half a million users. There have been a few reports about crashes on Linux recently, we have gotten 1,2,3 crash reports from the same user. The third one doesnt tell us much, the first two do:

1. *** Error in `/media/data/Steam/SteamApps/common/Verdun/Verdun.x86_64': double free or corruption (!prev): 0x00007fc3241a4220 ***
2. *** Error in `/media/data/Steam/SteamApps/common/Verdun/Verdun.x86_64': free(): invalid size: 0x00007fd5244a73b0 ***

We have far more reports like this, that really dont tell us much and point to errors outside our control. For example: http://pastebin.com/3KnZUVJe

I dont think this is caused by user code, and i cannot reproduce it myself. How do i report this to Unity?

 

That Pastebin at the end looks a lot like a Steam overlay crash? I'd suggest taking up with them first.

re #1 and #2, agreed that it seems like the errors are outside your control since it does seem to be native land. Though, as #3 points out, it's not always what we think - so I'd suggest crash dumps would be the first line of debugging to see if we can find root cause?

 

Thanks for your reply! We've already exhausted our premium support on things like this, but its still very hard to solve. We'll try to get a hold of a few crash dumps and make tickets i guess.

ps: "1,2,3" in my post are also links to other pastebins btw, i did hid them quite well!

http://pastebin.com/ph5Q102X
http://pastebin.com/bTLtxjgu
http://pastebin.com/xvMV9mMv