There’s no fire without Smoke Tests

Tickets to my next automation workshop are on sale!

It’s a familiar tale: you’ve set up an expensive and time-consuming playtest, and just wanted to squeeze in one more feature. You cook the build, give it a quick sanity check, and send it off. The playtest starts well, but as players start to hit level 4, they encounter bugs, error spam and fatal crashes. Your final feature caused a knock-on effect deep into the game, and now your playtest is ruined.

Wouldn’t it be great if there was a type of automated test that could quickly validate the whole project? It might not find specific issues, but if there’s a fire, it’ll spot the smoke, prompting you to investigate more.

Flirtation by Andrew Yee, CC BY 2.0

What are Smoke Tests?

A smoke test is an automated test that “explores as much of the game runtime as is reasonable, looking for obvious problems”. It’s not far off the type of test people imagine when first learning about test automation: a system that plays the game from start to finish looking for issues. You might be lucky in that your game already has bots or some similar player-facing feature which can be repurposed for a full playthrough, but you can still have a very effective smoke test suite without it.

In fact an effective smoke test can be as simple as loading each level in turn, waiting, and returning to the main menu. This may not exercise every gameplay system, but a passing test still gives you a lot of confidence about the game: at least the level exists, at least it loads without problems.

What do we mean by “problems”? A great place to start is checking the game’s log for exceptions, errors and warnings. Just by doing that you can catch problems such as:

• Level 5 missing a critical gameplay object
• Initialisation order bugs between systems on level 8
• Incorrect assumptions about world state in level 12
• …and a whole load of validation that the game engine will already be doing for you to make sure the core of the game is running correctly

When a smoke test fails, often all you’re really learning is that there’s a fire somewhere, but not what that fire is. We’ll normally have to do some investigating to see what caused the failure, but it’s still an incredibly useful safety net.

And it’s extremely effective for the amount of effort it takes. By the end of this blog post you’ll see how small a real smoke test suite can be – considering the value you get out of them, they’re maybe the tests with the best bang-for-buck out there.

Creating Smoke Tests (In Unity)

As an example, we’ll implement a smoke test in Unity. I’m going to assume you’re already familiar with Unity Test Framework and NUnit, although the general principles here apply to any game engine or test framework.

Let’s build the smoke tests in Unity’s Karting demo. I like using this because it’s not designed to be automated, but we can still do a lot of stuff very quickly. For those not familiar, below is a user playing the game through the main menu, past the race countdown, and starting to race. It doesn’t seem unreasonable to simulate that in our smoke tests.

We’ll break this down into multiple individual tests:

• Can we boot to the main menu without problems?
• Can we load the track without problems?
• Does the race start without problems?

We’ll create a LevelSmokeTests.cs in an appropriate PlayMode folder, but for now it just has a blank first test:

using System.Collections;
using UnityEngine;
using UnityEngine.TestTools;

public class LevelSmokeTestsFixture
{
    [UnityTest]
    public IEnumerator MainMenu_Booted_NoErrors()
    {
        yield return new WaitForSeconds(10); // just to get it compiling
    }
}

(We’re using the GIVEN_WHEN_THEN naming convention here)

You’ll notice when you run this test, that the game boots to an empty level and does nothing, rather than booting to our main menu:

This is how all playmode tests behave. It’s up to you to get us to the next step. In our case, we want to load scene 0 in our build list and wait for the menu to load. We’ll test for problems the same way every time – subscribing to Application.logMessageRecieved, and checking for error type. We’ll set a flag to true if we see a problematic log. This lets us also fail the test if we see any warnings — I strongly believe warnings should be treated as seriously as errors, but that’s another blog post.

public class LevelSmokeTestsFixture
{
    [OneTimeSetUp]
    public void LevelSmokeTestsOneTimeSetup()
    {
        Application.logMessageReceived += OnLog;
    }

    [SetUp]
    public void SetUp()
    {
        m_isTestFailed = false;
    }
    
    private void OnLog(string condition, string stacktrace, LogType type)
    {
        if (type is LogType.Assert or LogType.Error or
                    LogType.Exception or LogType.Warning)
        {
            m_isTestFailed = true;
        }
    }
    private bool m_isTestFailed;
    
    [UnityTest]
    public IEnumerator MainMenu_Booted_NoErrors()
    {
        SceneManager.LoadScene(0);
        yield return null;
        Assert.That(m_isTestFailed, Is.False);
    }
}

Note that this is not a complicated project, so we can rely on yield return null to fully get us to the main menu, but if you use a loading screen, you might need to wait until there’s some obvious signal that the game is booted – maybe the presence of a specific gameobject or singleton.

We can now run the test and maybe catch the main menu appear in the Game window for a brief moment before the test exits. Congratulations, that’s a decent smoke test! Just having this running on every build will catch a huge category of bugs. You could stop here and it would be fine, but let’s move a little further.

Testing our Test

I said we were done with that test, but there’s actually an important step to remember: testing the test. It’s very easy to write a test that passes but isn’t actually testing what you think it is. Whenever you write a new test, you should always see it fail for a predictable reason first. To do that, we’ll just add a Debug.LogError to somewhere in the boot code – I chose an Awake function in LoadSceneButton – and run the test to see it fail.

Now we know the test works, we can commit it to source control and move on.

Avoiding UI Testing In Smoke Tests

Our next step is to hit the big Play button in the main menu. It’s tempting at this point to force a mouse click or something – after all, that’s most representative of the player’s experience, and if the Play button is broken, we’d want to know in a smoke test, right?

I always recommend people new to testing avoid the UI as much as possible. If we did a mouse click here, or even something slightly more intelligent like find the gameobject by name or tag and force a click event, then suddenly the way this test will be most likely to fail is because someone moved the button. Or renamed the button. Or changed the hierarchy.

Those aren’t true test failures, in testing lingo they would be “false positives” – and in general false positives undermine trust in the automation. It’s easy to see a failure and assume it’s just a UI change again. Not only that, but any failure caused by changed UI means updating the test, which is easy to deprioritise. Pretty soon the boot test has been failing for weeks, and it might as well not exist.

On top of that, if your Play button is genuinely broken, QA will definitely notice it very quickly. We’re not helping QA to focus here.

So while there are techniques to do this more robustly, it’s best when starting out to avoid the UI layer entirely. Find the thing that the button pokes and call that from our test. How your level loads is much less likely to change over the lifetime of a project than the UI layout is.

In our case, the button calls SceneManager.LoadSceneAsync directly, so that’s what we’ll call in our test:

[UnityTest]
public IEnumerator MainMenu_AndLoadsLevel_NoErrors()
{
    yield return SceneManager.LoadSceneAsync("MainScene");
    yield return null;
    Assert.That(m_isTestFailed, Is.False);
}

However if we run this and pay close attention, we see that this new test runs before our boot test! That’s not right. In fact, NUnit does not define an execution order for tests (although in practice it’s alphabetical), because it’s normally a good idea to not have dependencies between tests. Dependencies can lead to fragile and hard-to-maintain tests, but in the case of most smoke tests, we need them. We’re looking to move through the game in stages, and it’s important to force an ordering here.

Luckily, the [Order] attribute exists just for that. Below I’ve added it, and also renamed the tests slightly so the alphabetical display in the editor reflects our ordering:

[UnityTest,Order(00)]
public IEnumerator T00_MainMenu_Booted_NoErrors()
{
    SceneManager.LoadScene(0);
    yield return null;
    Assert.That(m_isTestFailed, Is.False);
}

[UnityTest,Order(10)]
public IEnumerator T10_MainMenu_AndLoadsLevel_NoErrors()
{
    yield return SceneManager.LoadSceneAsync("MainScene");
    yield return null;
    Assert.That(m_isTestFailed, Is.False);
}

Now when you run both tests, you might be able to catch them execute in order. You’ll notice the order is increasing in chunks of 10, so if there’s a future need, it’s not too disruptive to add tests in-between.

Now this test is complete, don’t forget to test your test and add an error somewhere in the game code for it to catch.

Waiting For Gameplay

Our last test is to wait for race start, and again check for problems in the Log. This is the first time we’ve had to make a gameplay-side code change, because TimeManager did not expose IsRaceStarted. It’s not normally a good idea to make gameplay-side changes just for tests, but a read-only property like this seems like it could be useful for future gameplay code, so it’s forgivable.

[UnityTest, Order(20)]
public IEnumerator T20_Race_Started_NoErrors()
{
    var timeManager = Object.FindFirstObjectByType<TimeManager>();
    yield return new WaitUntil(() => timeManager.IsRaceStarted);
    yield return new WaitForSeconds(0.5f);
    Assert.That(m_isTestFailed, Is.False);
}

It’s perfectly valid to stop here, or to extend the smoke tests further over time.

The final successful test run might look something like this:

Dealing With Smoke Test Failures

While it’s great that these tests are giving us a safety net, they’re not perfect. They can be slow compared to other kinds of automated test, and they are not always very informative. “There was an error while loading level 3” is useful but only the start of a journey.

That’s why it’s good to ask yourself when fixing smoke test failures if there’s a way to construct a faster, more informative automated test. A fire test to go with the smoke test. That way, if this particular problem happens again, you’ll be told more quickly and in more informative terms. That could take the form of an asset test, unit test, actor test or something else.

This can be a great way to create pragmatic useful tests for your project over time. If you follow this pattern, you’ll initially see a lot of smoke test failures, but as your project progresses, you’ll see less and less and fire test failures will take over.

More Like This

I hope you see that this was not complicated to add, to a project that wasn’t designed for automation, and consisted of basically two three-line functions and a four-line function. Adding smoke tests to your project should not be a big investment.

If you’d like to learn more about smoke tests and other test types, I often work with studios to help them get started with test automation through training and workshops. I also sometimes host public-facing workshops that you can buy individual tickets for. Find out more details at automateyour.games.