Search Unity

Question Using assets for testing

Discussion in 'Testing & Automation' started by liortal, Aug 3, 2020.

  1. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    I have 2 different use cases where i want to set up tests that actually run on different assets from our project.
    These assets might be scriptable objects, prefabs, scenes and whatnot.

    Use case 1 - using "real" assets from our project. For example - prefabs really used in the game at runtime

    I'd like to have special content validation tests that are exercised against those prefabs / scenes / assets.
    Also, the tests should be run whenever those are modified or saved.

    How do i actually tie the test to the actual prefab at a certain path / with a certain name? looking for a good system to do that.

    Use case 2 - preparing "specialized assets" just for the sake of testing.

    For example, lets say i have some component that processes textures. I'd like to feed it with certain texture assets to see that it processes them successfully, so i can create a texture asset and load it into this component.

    Where should these assets be saved? is there any simple way to link them to my tests? or is a simple "known" folder location in the project sufficient for this case?


    Any tips from your experience will be great :)
     
    liortal, Aug 3, 2020
    #1

  2. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,660
    Are you really, really sure about that? What counts as 'when modified' - every individual property change in the Inspector? What do you want to have happen when the tests fail?

    Well, I guess there's two aspects to this: you want to generate a test case for every prefab, and then you want to run particular test cases when particular prefabs are modified. The simplest way to do the former is to just have a hardcoded list of [TestCase] or [TestFixture] attributes on your tests that provide the paths... what would be the problem with that? How often do you expect the set of prefabs - or at least the set of folders of prefabs - to change?

    I think the 'known folder' approach is likely to be fine, why not? You might want to adopt a convention for them, like saying that these test assets will always live in a 'TestAssets' subfolder alongside the test scripts; then you could use something like this to figure out where your tests actually are in the project and automatically locate the assets that live next to them, or something like that.
     
    superpig, Aug 3, 2020
    #2
    liortal likes this.

  3. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    forget "modified". When saving to disk. You think it's also too low resolution ?

    i guess it's not perfect but something we can start with.
     
    liortal, Aug 6, 2020
    #3

  4. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,660
    I'm not sure about resolution - especially when e.g. we have 'Auto Save' when editing prefabs these days - but I'm still mostly skeptical about workflow. Suppose that I, as a designer on your team, save a prefab that I've made some changes to, and you run the tests, and a test fails. What do you envision should happen? How is it reported to me? Does it block me from saving, block me from committing my work, something like that? Or what?
     
    superpig, Aug 6, 2020
    #4

  5. liortal

    liortal

    Joined:
    Oct 17, 2012
    Posts:
    3,562
    well, our current workflow uses the AssetModificationProcessor to hook into the actual saving process. In that stage i can run dedicated tests for that prefab and fail its saving with some error message (e.g: Validation tests failed, check xxxxx)
     
    liortal, Aug 6, 2020
    #5

  6. superpig

    superpig

    Drink more water! Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,660
    Hmm, ok, so you would actually block the save? Interesting...

    Then yeah, using TestCase/TestFixture to generate the test points, then selecting and running those test points from the AssetModificationProcessor callback, ought to do the trick. @Warnecke and I demoed something similar to this in our talk at Unite CPH.
     
    superpig, Aug 6, 2020
    #6