You may be familiar with Unity’s Test Runner window, where you can execute tests and see results. This is the user-facing part of the Unity Test Framework, which is a very extensible system for running tests of any kind. At Roll7 I recently set up the test runner to automatically run simple smoketests on every level of our (unannounced) game, and made jenkins report on failures. In this post I’ll outline how I did the former, and in part two I’ll cover the later.
(I’m going to assume you have passing knowledge of how to write tests for the test runner)
(a lot of this post is based on this interesting talk about UTF from its creators at Unite 2019)
The UTF is built upon NUnit, a .net testing framework. That’s what provides all those [TestFixture] and [Test] attributes. One feature of NUnit that UTF also supports is [TestFixtureSource]. This attribute allows you to make a sort of “meta testfixture”, a template for how to make test fixtures for specific resources. If you’re familiar with parameterized tests, it’s like that but on a fixture level.
We’re going to make a TestFixtureSource provider that finds all level scenes in our project, and then the TestFixutreSource itself that loads a specific level and runs some generic smoke tests on it. The end result is that adding a new level will automatically add an entry for it to the play mode tests list.
There’s a few options for different source providers (see the NUnit docs for more), but we’re going to make an IEnumerable that finds all our level scenes. The results of this IEnumerable are what gets passed to our constructor – you could use any type here.
class AllRequiredLevelsProvider : IEnumerable<string>
{
IEnumerator<string> IEnumerable<string>.GetEnumerator()
{
var allLevelGUIDs = AssetDatabase.FindAssets("t:Scene", new[] {"Assets/Scenes/Levels"} );
foreach(var levelGUID in allLevelGUIDs)
{
var levelPath = AssetDatabase.GUIDToAssetPath(levelGUID);
yield return levelPath;
}
}
public IEnumerator GetEnumerator() => (this as IEnumerable<string>).GetEnumerator();
}
Our TestFixture looks like a regular fixture, except also with the source attribute linking to our provider. Its constructor takes a string that defines which level to load.
[TestFixtureSource(typeof(AllRequiredLevelsProvider))]
public class LevelSmokeTests
{
private string m_levelToSmoke;
public LevelSmokeTests(string levelToSmoke)
{
m_levelToSmoke = levelToSmoke;
}
Now our fixture knows which level to test, but not how to load it. TestFixtures have a [SetUp] attribute which runs before each test, but loading the level fresh for each test would be slow and wasteful. Instead let’s use [OneTimeSetup] (đź‘€ at the inconsistent capitalisation) and to load and unload our level for each fixture. This depends somewhat on your game implementation, but for now let’s go with UnityEngine.SceneManagement:
// class LevelSmokeTests {
[OneTimeSetUp]
public void LoadScene()
{
SceneManager.LoadScene(m_levelToSmoke);
}
Finally, we need some tests that would work on any level we throw at it. The simplest approach is probably to just watch the console for errors as we load in, sit in the level, and then as we load out. Any console errors at any of these stages should fail the test.
UTF provides LogAsset to validate the output of the log, but at this time it only lets you prescribe what should appear. We don’t care about Debug.Log() output, but want to know if there was anything worse than that. Particularly, in our case, we’d like to fail for warnings as well as errors. Too many “benign” warnings can hide serious issues! So, here’s a little utility class called LogSeverityTracker, that helps check for clean consoles. Check the comments for usage.
Our tests can use the [Order] attribute to ensure they happen in sequence:
// class LevelSmokeTests {
[Test, Order(1)]
public void LoadsCleanly()
{
m_logTracker.AssertCleanLog();
}
[UnityTest, Order(2)]
public IEnumerator RunsCleanly()
{
// wait some arbitrary time
yield return new WaitForSeconds(5);
m_logTracker.AssertCleanLog();
}
[UnityTest, Order(3)]
public IEnumerator UnloadsCleanly()
{
// how you unload is game-dependent
yield return SceneManager.LoadSceneAsync("mainmenu");
m_logTracker.AssertCleanLog();
}
Now we’re at the point where you can hit Run All in the Test Runner and see each of your levels load in turn, wait a while, then unload. You’ll get failed tests for console warnings or errors, and newly-added levels will get automatically-generated test fixtures.
More tests are undoubtedly more useful than less. Depending on the complexity and setup of your game, the next steps might be to get the player to dumbly walk around for a little bit. You can get a surprising amount of info from a dumb walk!
In part 2, I’ll outline how I added all this to jenkins. It’s not egregiously hard, but it can be a bit cryptic at times.
Andrew Fray 