Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Full Game testing automation based on ML technology.

Discussion in 'General Discussion' started by ArshakKroyan, Apr 7, 2023.

  1. ArshakKroyan

    ArshakKroyan

    Joined:
    Mar 4, 2015
    Posts:
    32
    Is there any tool which will help easily automate the game testing?

    The ML based AI should play the game like humans (click on Play button, navigate through menus etc.).
    I only know about Unity ML-Agent, but it seems they are only training game agents, but not the entire game interaction.
     
  2. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    10,019
    If you want something that interacts with a game like humans do, you need humans.
     
    neginfinity likes this.
  3. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,484
    Like Murgilod said, you need humans for this one.
     
  4. ArshakKroyan

    ArshakKroyan

    Joined:
    Mar 4, 2015
    Posts:
    32
    Ok, thanks for the reply.

    What about simple testing?

    Most of the time a simple "Go through levels" may be enough finding some errors?
     
  5. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,484
    //this is an opinion.

    You will still need human cognition. Basically, the only thing you could theoretically do is implement simple pathfinding algorithm and battle AI and have it clear the level. That is possible. It is kinda pointless thing to do unless you're developing a super-idler, but it can be done.

    However, let's say in your game half of level is missing, music is not playing, every sound effect is replaced with "ribbit", the picture is upside down, all text is in suwahili, all shaders failed to compile, the only thing player can see is pink wireframe and dragons are flying backwards.

    The AI is not going to spot ANY of this.

    At the moment machine learning tasks kinda do not do cognition. They can produce input based on output, and they can classify. And you need to feed your game into human sensory input.
     
    ArshakKroyan likes this.
  6. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    15,602
    I agree that "machine learning" probably isn't the best approach to this for now, but I think people are getting hung up on shooting that down, rather than addressing the question. ;)

    There are many. However, to know what might be useful to you we'd need to know a bit about what you're trying to test. "Do my graphics work?" is very different to "are levels of the right duration?" is very different to "is my networking performance up to par?" is very different to "where does my game crash?" is very different to "is this even fun?" etc. etc.

    As the others have said, for some of that you will just need humans. For others, automated tools may very well be of use, but it's likely to be different tools for different types of testing.

    I know that it's a buzzword at the moment, but don't get hung up on "ML". It's just one type of AI, and plenty of useful automation tools aren't based on any kind of AI at all.

    Mighty Games gave a talk about going the opposite way - instead of learning to "play" the game, they tested with armies of super simple bots essentially giving random inputs. They then ran as many instances of this as they could, hooked up to code to detect the errors they were looking for - crashes, non-progression points, stuff like that. Very useful for some types of games, but clearly there are others where it won't work so well.

    Going even simpler, recorded inputs being played back can also be really useful in many scenarios. E.g. feed them into a "unit testing" type pipeline to be run after builds. E.g. give a recording to someone trying to fix a bug, which will speed up the fix / test process. E.g. have an automated system use them to "play" levels and generate screenshots, which can then be checked for a bunch of stuff @neginfinity mentioned in a few seconds each.

    None of these are full solutions, but each of them can save a heck of a lot of human time.

    I know people who did this, too, and it certainly didn't seem "pointless". Obviously it's not useful for all types of testing, but it is very useful for some - balancing, finding ideal solutions, finding non-progress points, crashes, performance recording, and others. For areas where it can be effective, it helps fewer humans test more cases, and it helps test fixes more quickly when they're made. It even helps in cases where it's less effective, by freeing up people or time from where it is effective.

    Probably most importantly, it scales up far more effectively than putting more people in front of computers does.

    And that's without even considering multiplayer, where you'd have to be insane to test without a decent "bot" solution.
     
  7. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,484
    That was an opinion. I have that opinion because machine mind does not have human thinking, but at the same time can do some things with ridiculous precision. So by relying on bot testing you may end up designing your game for bots and not for people.

    For example, let's say your level has a bug that allows player to progress into area they're not supposed to enter using some sort of ridiculous combination of abilities. A bot most likely won't find it.

    Or, let's say your level is impassable and cannot be cleared, because there's a hole in the middle of it and the only walkable surface is a thin rope hanging in air. A bot might be able to clear the rope as walkable surface. A human will fall off.

    Stuff like t hat.

    Aside from that, sure, there's some utility for bot testing and it is not completely useless. It is just there are a lot of tasks where it will fall flat on its face.
     
  8. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    1,633
    This may not be directly helpful here but I can assure you, this concept of AI-driven testers is certainly a thing (even before the break throughs of last year).
    In the field of automotive driver assistant systems there are artificial human drivers (generated via machine learning usually from drone footage) which are used in simulations to test the actual driver assistant algorithm. We use that because it produces way more intricate behavior than e.g. classic traffic algorithms like IDM.

    Something like this for games or software in general would be quite interesting. A lot of cool stuff is gonna come out in the next couple of years.
     
    Last edited: Apr 10, 2023
  9. angrypenguin

    angrypenguin

    Joined:
    Dec 29, 2011
    Posts:
    15,602
    I think you're getting testing and designing mixed up. The test result doesn't tell you what to change. It just tells you whether or not a thing happened, and ideally some details about that thing. A human still needs to look at that and decide what to do with it, that's what "design" is.

    If someone is smart enough that they're not only designing and creating games, but are also starting to design and create automated tests for them, then I trust that they're not suddenly going to turn stupid when they get results out of those tests.

    Edit: From one of your examples... if I'm looking at a test which tells me successful ways to pass a level, and there's only one, I'm going to dig into that a little - that's already suspicious. When I look at it, if I see that the only path involves balancing across a decorative rope or whatever, then that's going to get handballed to a level designer as an issue. All of that is likely to take me a fraction of the time required to try playing the level myself just once.
     
    Antypodish likes this.
  10. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,484
    There's significant overlap here with games, but I'd say this is not the same thing, because the goals are different if not opposite of each other.

    The purpose of a auto driver development is to develop a real world pathfinding car relying on imprecise sensory data. It is a hard task, but it is narrowly defined one. The purpose is not to design AI Schumaher, load him into Boston Dynamic Atlas and let it have a career in F1 racing, while maintaining work-life balance, good income level, get married and raise a family.

    The point of in-game testing should be idenitfying situations where an intelligent biological system will encounter undesirable scenarios in a simulated world. The point of auto driver development is to develop a non-biological system that will perform tasks full of those intelligent biological systems.

    The goals are opposite. At least that's how I see it.
     
  11. BIGTIMEMASTER

    BIGTIMEMASTER

    Joined:
    Jun 1, 2017
    Posts:
    5,181
    would be nice to treat the OP with some respect and not derail thread for once
     
  12. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,484
    Heed your own advice, and use report button for things you disagree with instead of arguing in the thread and derailing it. The OP was given advice, and can receive more of it, if they choose to ask followup questions. So far in the thread there's a list of things that can cause problems from me, and list of practical attempts to use it by angrypenguin. If I discussion on the topic continues, something new could be born with it, and that could be of use for the OP (assuming they haven't left), or for people that will find this thread in the future. Things like this should be quite obvious without an explanation.
     
  13. BIGTIMEMASTER

    BIGTIMEMASTER

    Joined:
    Jun 1, 2017
    Posts:
    5,181
    discussion flourishes if you let more than one person talk. A daily post limit might help with that.
     
  14. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    10,019
    What are "errors" though? Because the thing is that we aren't building games for machines to play, we're building them for people to play. People and machines approach games entirely differently. If you want to make sure buttons work to open doors? Sure, a simple testing kit could do that, you wouldn't even need ML for that. If you want to cover edge cases though? ML won't help much for that either. That's something you should test yourself. If you want to test how those systems are interacted with? You need players for that.
     
    neginfinity likes this.
  15. neginfinity

    neginfinity

    Joined:
    Jan 27, 2013
    Posts:
    13,484
    This made me think about genetic cars.
    https://rednuht.org/genetic_cars_2/

    I wonder if designing a bot trying to clear a game this way would make sense, though.
     
  16. zombiegorilla

    zombiegorilla

    Moderator

    Joined:
    May 8, 2012
    Posts:
    9,025
    Heh... something often occurs in the industry at every level.
    Indeed. Level/map testing can be (and is) done with simple automated testing that runs through all possibilities. ML/AI isn't really needed for that. We always do a ton user testing and that involves a way more than getting simple values or answers to specific questions. The players are fully recorded and we can observe how they react and how they play. Its something one human observes another human doing and evaluates why. I just can't see how ML fits into that process in any way that improves it. (at least until AIs become the target audience).

    It seems there is a desire to apply AI to virtually everything. And there are some uses, in some cases, the effort involved to get data that may useful isn't really worth it. Most of game testing fits in to that category. The many types of testing have been done for decades and the processes to do them are pretty well worked out. AI won't improve them (by improved I mean increased value/accuracy). That isn't to say there may not be some niche application for fixed rule set type games, but those are going to be rare, and likely aren't going to yield useful actionable data.
     
  17. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    591
    Here are a bunch of options for UI Automated testing in Unity: https://forum.unity.com/threads/wha...-automation-and-unit-testing-in-unity.682720/

    The only one that offers any "out of the box" way "play through a level" is the Unity automated testing package which has a "GameCrawler" component that randomly interacts with the game and reports on errors or when it gets stuck. But that package is shelved for now, and the GameCrawler was in its infancy, so customization is likely needed.

    I know that the automated testing package and Trilleon systems have "record" logic for generating C# scripts from recording what you do in the editor.
     
    DragonCoder likes this.
  18. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    10,019
    GameCrawler was so much in its infancy, even, that you could get away with using an action planner to the same general results, no ML required.
     
  19. avijeet26

    avijeet26

    Joined:
    Apr 13, 2023
    Posts:
    1
    Hey
    SikuliX can be used to simulate user interactions, such as clicking on buttons, entering text, and navigating menus, just like a human player would. It also supports scripting in multiple programming languages, including Python, Java, and Ruby.

    Another tool that you can use is Appium, an open-source tool for automating mobile application testing. It supports both Android and iOS platforms and can automate UI interactions such as button clicks, gestures, and swipes.
     
    DragonCoder likes this.
  20. tsibiski

    tsibiski

    Joined:
    Jul 11, 2016
    Posts:
    591
    Appium only works with XML and HTML based systems. To use Appium with a video game, you will need to find a library that allows it to have commands translated into a game, or write that translation framework yourself.

    Many existing automation frameworks merely use Appium to load apps onto mobile devices and accept/decline device level alerts. I would definitely recommend using one of the existing frameworks from the link I mentioned above, as developing your own framework to make Appium work could take months, and you'd be reinventing the wheel.