Mutation Testing Unity Specific Programming Features

Hey everyone! I'm currently doing my master's thesis on developing and analyzing a tool for mutation testing Unity-games by mutating syntax of Unity specific programming features in the game's code.

Quick summary of mutation testing: Mutation testing is used for evaluating quality of current software tests. Bugs are simulated by making small modifications to program code. Then test sets are run with the modified code and if the test set does not fail we can deduce that our test set is inadequate.

So for example we could change the lines containing

Code (CSharp):

1. if (obj.CompareTag("Enemy"))

to

Code (CSharp):

1. if (obj.CompareTag("Player"))

and see if it makes our tests fail, which it should. This would be one mutation operator.


Now I'm wondering what mutation operators I should use for a tool like this and if there might be any well reasoned or even "scientific" ways to decide them. Ideally mutation operators should:

1. Be simple changes to a single line of code
2. (Simulate real errors made by real programmers)

So far the best I've got is pretty much

1. https://docs.unity3d.com/Manual/ScriptingImportantClasses.html this page which describes important scripting classes which could be used as a reason to make mutations to these classes
   
2. https://thesis.cust.edu.pk/UploadedFiles/Thesis Final.pdf#page=49 this thesis with much of same goal as mine which also defines Unity specific mutation operators. It isn't really explained why exactly these operators were chosen however.

I've been trying to find data on what kind of bugs are most common in Unity code or which Unity programming features are most commonly used but I don't think that really exists. So if anyone here has any suggestions how I could decide my mutation operators based on something other than "I feel like this kind of programming errors might happen" or "I feel like this is a much used Unity programming feature" I would love to hear them!

 

Hmm, I'm not sure if I get the point here and what you mean by being tricky to identify. Might just be my understanding of the subject or English language though .
I can definitely see that this kind of errors are quite specific to the codebase. But I'd think using wrong but existing tags instead of simply typo'd ones is a more common error and also more difficult to debug. So I think having tests – and thus mutation operators – for it could be useful.

Then again I'm not sure what would be the best way to select the mutated tag, I was thinking of just taking some tag from TagManager.asset that isn't the original. I can see how that might simulate real errors quite badly in some cases, like replacing "Player" with "UI" but still I think it's better than nothing. Do you think there are problems with this kind of approach? Or maybe it just isn't very useful? I'm definitely still not an expert at Unity or mutation testing so I may be thinking this wrong.

I agree and this is definitely something I've already planned with GameObject.Find and some other methods that take string as parameter.

This is a good point and I hadn't really thought of that. I have planned for example substituting Start/Awake but doing same with methods that have similarish names seems like a good idea too.

I have indeed thought this more in terms of simulating "thinking errors" than "typing errors" but I'll have to think about that approach again. Thanks for the ideas!