Survey

What you really think of Unity (it will take 1 minute): https://www.surveylab.com/pageTag/SurveyCampaign/cId/8d6dfd6c7d04ef63e7a3de37f71d42bf53057f008/

 

Up!

 

Why should any of us take your survey?

 

It is totally optional to participate. It might be intresting to see what is the score of Unity. At the moment there are 29 responses.

 

Score of Unity in what context? All your survey questions don't offer any meaningful data points up for anything but Unity. It all exists in a vacuum.

 

The score is in SUS.
In the study [1] adjective ratings were given to the scores and they are quite reliable to rate an individual system.

 

A SUS score is meaningless in this context because it does not identify the problem sources, which you need when dealing with complex systems. It would be useful if there were comparisons to other engine scores with everything weighted, especially based on where the data was coming from. For instance, what do you think the score would look like if you posted this on the GameMaker or UE4 forums, both of which have an overlap of people who have used Unity, instead of here?

This is another boring TAKE MY SURVEY post but none of these surveys actually do anything.

 

No.

 

Agreed. That said a significant portion of the surveys posted here are just students needing a survey filled out for some undisclosed reason so when I realized that it was a relatively short survey I went ahead and filled it out. It's not like I had anything better to do at the moment.

 

The problem I see is that you're not capturing any info about the person answering. If I say the engine is not confusing, is it's because I have long years of experience with it? Or am I a noob and the tech really is intuitive? You have no way to know.

 

Not sure you understand the purpose of this survey. So I will try to explain.
The purpose of this survey is to determine Unity overall usability rating. I am not identifying problem sources based on this survey.
I understand all who are not intrested in participating and many thanks to all who did!

 

That is a good point.
From: https://measuringu.com/five-second-tests/

When I compare the SUS scores for just these first-time users the same pattern does hold. The difference between the five second users and full-time users was larger (8%) but still not statistically significant (p >.11).

Overall - it does give a meaningful result regarding usability.

 

From whose standpoint? A new developer will likely give completely different results than a professional, and that's not even talking about my own warped viewpoint which is the result of already knowing how to work with game engines and knowing C# before I started learning Unity.

 

Well actually that is not true. I hope you have some sources to prove it.

 

My source is these very forums. My fifteen thousand plus posts isn't the result of just goofing off after all. I've helped many people over the years with problems and I have seen the difficulty new developers have had picking it up. A difficulty that has not changed in a meaningful way thanks in part to Unity's constantly screwing around with their learning resources.

 

Eh, I don't think it's worth arguing. We have fair warning that his method is flawed and we weren't even asked in the first place, so good luck with your survey, friend.

 

Expert opinions are very valuable. But can you predict the answers of the users you have helped about the perceived usability?
From this study: https://measuringu.com/five-second-tests/
Interestingly enough, the perception of website usability from the 5 second condition was statistically indistinguishable from the no time limit condition. The observed difference in average SUS scores was less than 3 points (4%).

 

Yeah this is pretty clearly just a survey that solely exists because the person who made it thinks surveys are neat.

Why?

 

Are you trying to make the statement that the assistance provided on these forums is the equivalent of five seconds of video? Because the assistance provided on these forums is often far beyond what a few minutes of video is capable of portraying let alone a few seconds. We don't typically just answer and move on. We discuss it.

 

No. I never claimed that. What led you to this conclusion?

 

If that wasn't the point of your post then why did you quote the study at me?

 

Oh okay. The link was to show that the factor of user skill level will not affect the results so much. I did not want to question the help that is given in this forum in any way.

 

I just spent a few minutes reading over the blog post and attempting to read the sources. Unfortunately while it's possible to read the blog post, it's completely impossible to read the sources it makes use of.

The first source that makes the statement that a few seconds is indistinguishable to no limit is returning the typical "404 Not Found" error, and the second source that impressions happen within milliseconds is not responding. The landing page optimization article is redirecting to the front page of the website for it indicating that it too no longer exists.

Everything beyond that is either actual tests (hidden behind an account registration system that I can't be bothered to do) or simply links to their own offerings which come across as advertising and not actually beneficial to the blog post.

It's a nearly ten year old article so you could make the statement that it's understandable, but if this is going to be used to some degree as a way to prove anything it needs to be maintained properly and clearly the owner has no interest in doing so at all as one of their own images is no longer functioning.

 

(Edit: Updated with context.)

Sorry, the original version of this post probably seemed more abrupt and grumpy than intended.

Surveys seem like an easy way to get info, but unless they're well designed they can easily just be misleading. In particular, you need to write questions in a way that makes sure they mean the same thing to the people answering them as they do to you.

Taking one of your questions, about whether or not we "would like to use Unity frequently":
- Saying "would" implies that we do not already use it frequently. How do we answer if that's not the case?
- "Like" says that this is a question of preference. So are you asking in comparison to other things we use, or in isolation, or..?
- What is "frequently"? 5 days a week? Hobby stuff on weekends? Once a month? When I need to stage a picture with 3D assets on occasion at work..?

You say that the purpose of the survey is to get a usability rating, but that's not actually a purpose. What is the rating for? What practical benefit does it have, and to who? Ideally if you're doing a survey it's to get information to solve some kind of problem, so analyse each question you're asking to see if the answers will be useful to help you solve that problem.

This is about websites. You're asking for feedback on a game engine. These are used incredibly differently and have very different purposes and motivations, so I very much doubt that info about one can be applied to the other.

 

The survey set up is not perfect (especially regarding not taking into account user experience). But you can be sure that the results are not interpreted in an arbitrary way. Thanks to all for participating in the survey and in this discussion!

 

Check out my post above. I heavily edited it.

The issue with having room for interpretation isn't just with the researcher, it's also with the people answering the questions. You have no idea how I interpreted your questions and therefore my answers are meaningless to you.

Is this a part of an assignment?

 

I suggest you have a look at: https://hell.meiert.org/core/pdf/sus.pdf - Paragraph SUS - the System Usability Scale

These questions are not my invention. They are carefully selected and not random. At the end of the day it does not matter how an individual interprets them - what matters is how most of the people will interpret them.

 

SUS itself is for all kinds of systems/products - and it is reliable. Still, previous experience should be asked to get reliable results.

 

SUS is effectively useless in this case as has been explained already.

 

So what are you comparing the results against to get meaning from them?

 

Yes you are right just because u have spent most time in 'general forum' and linking articles, it still requires careful knowledge and composing of messages to help those asking help. You should still be proud of your accomplishments as am sure many other acknowledge.

 

Does it occur to you that experience is less relevant to website usability than it might be to utilizing a game engine (what basically amounts to a profession)?

 

It seems to be not.

Source: https://measuringu.com/prior-exposure/

 

None of these things are even as remotely complex as a game engine.

 

Whoa there.

five second user in a game engine is not a thing. It takes 2 to 3 months to attain proficiency and BECOME a user.

This kind of stat makes sense for webpages, but absolutely does not apply to specialist software like a game engine.

I can. Anyone starting a game engine first will be horribly confused and overwhelmed. Majority of those people will drop off. Those who remain will work for 2..3 months or so to obtain proficiency, from those a small percentage will ascend to the level of professioanl and get employed.

What you're trying to do is equivalent to determining usability rating of a space station based on 5 second impression of a random layman off the streets who has never finished astronaut training.

You're applying wrong methodology.

 

Here's why it is a wrong methodology.

"Five second impression" only matters when your target audience is "everybody", and not "specialists".
For example, if you're making a bakery website, a grocery store website, then everybody buys those goods, competition is high, hence you want the impression as positive as possible, meaning five second metric matters.

Let's say that instead of grocery store, you're now dealing with company that makes those things:

Those kind of vehicles cost many millions of dollars, and most people won't be buying one of them. So, your five second metric no longer matters, as your audience is no longer "everybody" but "your business partners". There are few places which buy those cars, and you'd be probably monitoring them specifically, and tailor your internet portal for those needs.

let's up the game a bit. You're now in charge of a factory that makes nuclear warheads and reactor rods. In this case you have one business partner, your activity will be regulated by government, and very few people will be allowed to use your service. Five second metric is now irrelevant, and information portal - if you have it - will be tailored to match needs of the partners.

-------

In case of game engine company, one in a thousand newcomers will be capable of using the engine, one in tens thousands (or even less) will generate subscription fees, and maybe one in million will generate significant profit for the company. With the engine being specialist tool, five second impression doesn't matter, what is important is long-term usability and acquisition of the engine-related knowledge and skills. This won't be done in five seconds. So the idea would be to quickly weed off people that will be incapable (while being polite with them),retain low-key profitable users (in case they suddenly up their game), and focus on heavyweight money bringing.

Those people will have months of their time invested, if not years, and for them 5 seconds since start will be a distant memory.

 

If you are intrested you can look at the studies in my previous post. Same thing with a lot more experience than 5 seconds.
Overall - I see no sources guys. Some people have studied the field of perceived usability whole lifetime. All I see are subjective opinions - this is not reliable. You can lock the thread - this is going nowhere...

 

It's going nowhere because you keep literally refusing to listen to anyone even though everyone has pointed out how none of what you have discussed here has any actual relevance and this survey offers nothing. Instead, you've been extremely condescending to everyone who disagrees with you.

 

You keep spamming the same link in all your posts, and ignore all criticism. The pdf you linked is not a study, but a proposal, and the sources it cites are from 1980s and 1990s. Also one of the references is Terry Pratchett book.

I question validity of your source, utility of your metric, existence of the study cited in the article, and existence of the person to whom said article is attributed, and relevance of the method after the time that passed since the publishing of that link (9 years ago).

Anyone can make a blog post, claim to have a PhD, claim to have commited the study, and then use that to promote their customer experience company. In addition, as it has been repeatedly said, SuS has dubious utility.

The sources are us, which are unity users, with years of experience dealing with software, and a lot of experience dealing with questions left by other users.

-------

While I understand "I want to believe" sort of desire, a recommended idea is to use scientific method and occam's razor, and not put your faith into things.

 

Few more details.

Once upon a time I've come across a "study" of (wait for it)... "Russians undermining american democracy through slandering Last Jedi on Twitter". That's an actual thing that exists and was published on some of university, and it is as insane as it sound. Yet, it is technically a study. And got published. And probably even funded

Spoiler

Long story short they picked a bunch of tweets they allegedly were left by Russians, trained neural network to recognize tweets they'd think to be left by Russians, run them on list of criticism tweets targeting LastJedi, got a list of allegedly Russian tweets that allegedly target american democracy, cleverly dropped "allegedly" along the way and started treating assumption as a proven fact, then added ten pages of rambling about nothing and called it a study.

We live in informational age, and a lot of untrustorthy information exists. You're very fond of this methodology, but for me that post rings a lot of alarm bells.

For example, it is unclear whether the person writing it is real. Fake reviews are a thing, fake testimonials are a thing too.
It is unclear whether the study existed in the first place. 5 percent increase is a very convenient figure, and I'm not seeing a link to external sources in there.
The study listed did not list familiarity with other pieces of software of this kind. As there will be a difference between someone who uses a painting program for the first time in their life, and someone who used a painting program before.
And aditionally, you used "someone spend their life studying" as an argument. That's a logical fallacy, appeal to authority. Just because they spent their entire life studying that does mean they learned anything useful. Just because they're an authority, that does not mean all their statements and opinions are valid.

A general idea when running into a study is to check if it is real, whether person involved is real, whether they have relevant degree and not "PhD in Demon Summoning", whether their study was actually conducted, and whether the work has been peer reviewed.

That's unless you're dealing with a field like Mathematics where you can verify their findings yourself without building a Hadron Collider.

Dismissing criticism is definitely not a way to go. You're supposed to have counterarguments instead.

 

Random thoughts from reading the thread.

For the avoidance of doubt, all eighteen thousand of my posts have been goofing off. In fact I should probably be working right now.

Now I want to buy one. I'll be looking for a new job in a few months. Maybe its time to go into mining.

If an 8% difference is not statistically significant, you've got to reconsider your methodology. You are going to need a pretty large spread between different products in order to find a significant difference between them. What percent difference is required to give p < 0.05? By the nature of your survey, you aren't going to be getting order of magnitude differences.

I haven't done perceived usability. On the other hand I have spent most of my career in statistics. And a considerable chunk of that has involved sampling and surveys. I completed your survey. In my professional opinion, the results will be largely meaningless.

Out of idle curiosity I googled SUS. Not much comes up. It looks like a tool that's been pushed by the small group that developed it, but isn't really taken seriously anywhere else.