Running unity tests on a build server

(Read part 1, about creating parametric level smoke tests, here)

Now we have some useful tests, we want our Jenkins install to run them frequently. That way, we really quick feedback on failures. Here’s how we do this at Roll7.

Running tests from the command line

Let’s assume you already have a server to build your game. I won’t cover how to set up a unity Jenkins build server here, except to say that the Unity3d plugin is very helpful when you have lots of unity versions. (Although we couldn’t get the log parameter to be picked up unless we specified the path fully in our command line arguments!)

The command line to run a Unity Test Framework job is as follows:

-batchmode -projectPath "." -runTests -testResults playmodetests.xml -testPlatform playmode -logFile ${WORKSPACE}\playtestlog.txt

You can read more about command line params here, but let’s break that down:

• -batchmode tells unity not to open a GUI, which is vital for a build server. We don’t want it to hang on a dialog! You can also use Application.isBatchMode to test for this flag in your code.
• -projectPath "." just tells unity to load the project in our working directory
• -runTests starts a Unity Test Framework job as soon as the editor loads. It’ll run some specified tests, spit out some output, and make sure test failures cause a non-zero return code.
• -testPlatform playmode tells the UTF to run our playmode tests, which are the ones we care about for this blog post. You can also use editmode.
• -testResults playmodetests.xml states where to spit out a report of the run, which will include failures as well as logs. The report is formatted as an nunit test result XML. Jenkins has a plugin that can fail a job based on this file, and present a nice reporting UI.
• -logFile ${WORKSPACE}\playtest.txt specifies where to write the editor log – by default it won’t stream into the console. The ${WORKSPACE} is a jenkins environment variable, and we found specifying it was the only way to get the unity3d plugin to find the log.

…And that’s enough to do a test run.

Playmode tests in batch

This page of the docs mentions that the WaitForEndOfFrame coroutine is a bad idea in batch mode. This is because “systems like animation, physics and timeline might not work correctly in the Editor”.

There’s not much more detail than this!

In practice, we’ve found any playmode tests that depend on movement, positioning or animation fails pretty reliably. We get around this by explicitly marking tests we know can run on the server with the [Category("BuildServer")] attribute. We can then use the -testCategory "BuildServer" parameter to only run these tests.

This is pretty limiting! But there’s still plenty of value in just making sure your levels, enemies and weapons work without firing errors or warnings.

In the near future we’ll be experimenting with an ec2 instance that has a GPU, to let us run without batchmode, and also allow us to run playmode tests on finished builds more easily.

Gotchas and practicalities

• Currently, we find a full playmode test run is exceedingly slow, and we don’t yet know why. What takes a minute or two locally takes tens of minutes on the ec2 instance. It’s not a small instance, either! So we’re only scheduling a test run for our twice-daily steam builds, instead of for every push.
• WaitForEndOfFrame doesn’t work in batch mode, so beware of using that in your tests, or anything your tests depend on.
• The vagaries of unity’s compile step mean that some sometimes you can get out-of-date OnValidate calls running on new data as you open the editor. Maybe this is fixable with properly defined assemblies, but we hackily get around it by doing a dummy build right at the start of the jenkins job. It goes as far as setting the correct platform, waits for all the compiles, and quits. Compile errors still cause failures, which is good.
• If you want editmode and playmode tests in the same job, just run the editor twice. We do this with two different testResults xmls, and we can use a wildcard in the nunit jenkins plugin to pick up both.
• To test your command line in dos, use the start command: start /wait "" "path to unity.exe" -batchmode ... The extra empty spaces are important if your unity path has spaces in it too. To see the last command’s return code in dos, use echo %ERRORLEVEL%.
• These are running playmode tests in the editor on the build server. We haven’t yet got around to making playmode tests work in a build. That might end up as a follow-up post!