How to symbolicate iOS crash dumps from Unity Cloud Build games

Disclaimer: this advice is provided without warranty, works on my machine, etc. Use it at your own risk.

We’ve been using Unity Cloud Build at Spry Fox for a soon-to-be-announced project. Since we’re a remote company, it’s great to be able to push and know that both Android and iOS builds will be soon available to everyone in the team with your new feature. It’s saving a lot of time every day.

This morning I pushed our new IAP back-end, which of course involved some low-level stuff on both platforms. Everything worked on local builds, but when I tried the iOS build from Unity Cloud, it crashed on start-up. Getting the callstack of the crash turned into a bit of an odyssey. I thought I’d document it here.

By default, Unity will not generate Xcode projects that produce useful debug info. We can fix this by adjusting the Xcode project after generation, but that’s no good for Unity Cloud Build where we have no control.

So, one time only, you should add this file https://gist.github.com/tenpn/f8da1b7df7352a1d50ff to your project (EDIT: in an editor subfolder; thanks Terry Paton). That will do the post-processing for you and for Unity Cloud Build. (There are conflicting reports as to if this increases build size, but you definitely want it while developing. It might still be worthwhile leaving it in for release, because it will be extremely helpful in chasing down crashes in the wild.)

Wait for Unity Cloud Build to produce a new build containing that change, install it, and repro the crash on your device. Then follow these steps:

  1. Go to the cloud build project for your game, and find the dSYM and ipa downloads for the build you installed. At the moment, you click Install on the build in the history list, then Share, then both Download to Desktop for the ipa and Download dSYM.
  2. Unzip the dSYM somewhere safe. I put it in a new folder named after the Cloud build number we are investigating. The location should be somewhere Spotlight can index.
  3. Rename the ipa extension to zip, and unzip the .app inside to the same folder as the dSYM.
  4. Attach your iOS device via USB to your OSX machine
  5. In Xcode (these steps are for Xcode 6.1.1), go to Windows->Devices
  6. Select your device in the left pane, and click View Device Logs
  7. There should be a crash dump for your game. Locate it by the process name and crash time columns and click on it.
  8. The right window should now have the crash dump with a named callstack. If not, right-click on the crash dump in the left pane and select Re-Symbolicate Log.

Hopefully this gets you enough info to start debugging your app.

Shameless plug: DecisionFlex is a new unity plugin to let your AI make human-like and emergent decisions.

DecisionFlex

DecisionFlex considerations

For a while I’ve been working on an AI plugin for Unity3D called DecisionFlex. It’s a decision-making tool for your games, based on Utility Theory. It’s great for when you have multiple actions to choose between, and lots of factors to take into consideration. For instance:

  • A soldier choosing its next target, based on target range, type and the soldier’s ammo count
  • A Sims-like human choosing to eat, drink, work or exercise based on hunger, thirst, wealth and fitness
  • A bot choosing to prioritise picking up a health pack, based on the bot’s HP and distance to the nearest pack
  • Any time you might find yourself writing highly nested or cascading if-then statements

DecisionFlex editor-based and data-driven, so you can construct new decisions and actions, and tweak considerations, without diving into code. The code to hook your game up to DecisionFlex is minimal and you don’t need to understand complex equations.

DecisionFlex isn’t quite ready to ship yet, but I’m ready to start talking about it. You can find more information and a web demo here:

http://www.tenpn.com/decisionflex.html

Innovating In Your Pants

The company I work for, Spry Fox, is known for its innovative games like Triple Town, Steambirds and Leap Day. In fact, it’s one of the reasons I was attracted to work there in the first place!

We do this by prototyping a lot of ideas, iterating quickly, and (ideally) killing readily. All as fully remote company. People in the company have talked about the details of this procedure publicly before, but I took a shot at it when asked to speak at Falmouth University the other week. The presentation (~30min) and Q&A (~20min) were filmed, and can be watched for free here:

http://onair.falmouth.ac.uk/Mediasite/Play/28dd23b44a3c43abbd60ecb6d49969bd1d?catalog=0a29d6d3-1b0b-405b-8976-0d080ce607e8

Here’s a direct link to the slides:
https://docs.google.com/presentation/d/1cCpuFpduP3NX2ZxJY_QaQ_eYxdwKug1tpkkiROI5Xyo/pub?start=false&loop=false&delayms=3000#slide=id.p

And here’s a similar (but longer) presentation by Daniel, David and Ryan at GDC 2013:
http://www.gdcvault.com/play/1018263/How-to-Make-an-Original ($)

Hope you find it interesting!

Practical Unit Tests, GDC 2014

I gave my talk yesterday on how bad unit tests can slow iteration, and how you can fix them. The turnout was fantastic, and we had a great roundtable afterwards in the wrap-up room. It was 8 years since the last test-focused GDC talk, and hopefully the organisers will now realise there’s an appetite for more on this topic.

I want to do a recording of the session next week (it’s only 25 minutes) and post it up here, but in the meantime the slides are below. Hopefully they make some sense without the audio.

Come see me speak at GDC 2014

Come see me speak at GDC 2014

That time of the year again, when I’m flip-flopping between wishing GDC was tomorrow, because I can’t wait to catch up with friends and colleagues, and just a few more weeks away, so I had just some more time to polish my talk.

This year I’m talking about unit testing, or more specifically how bad unit tests can ruin a project just as much as no unit tests. It’s aimed at devs who know what unit testing is – it’s not a tutorial session or a guide – but you don’t have to be an expert.

The talk is on Thursday, 11:30am in South Hall 304. See you there!

Your Favourite Free To Play Game Hasn’t Been Made Yet

There’s been a lot of criticism over the last few days of the new Dungeon Keeper free-to-play mobile game from EA. By all reports, it features some egregious and long countdown locks, that can be opened early with in-game currency, which can of course be topped up with real money in-app purchases. These types of IAPs are intended to both aid retention (as people come back to see their timers complete) and frustrate the players into spending.

Dungeon Keeper Ad

It’s a frequent pattern in high-grossing f2p games, and it’s not that weird to see it here, a high-profile title from a successful f2p publisher. The complaints haven’t been so much that this example is particularly bad, but coupled with a well-loved IP, it’s the straw that broke the camel’s back. I absolutely agree; as I mentioned above it’s a mechanic that has little to do with crafting an experience for the player. It’s a crutch for actually having a fun replayable game – if you make a good game, you don’t need to worry about retention mecahnics! I’m glad this kind of crap is being called out.

But let’s not go overboard. Here’s some quotes from the articles above:

“…if you are one of the very few developers who are still making real games (without the in-app purchase crap), I salute you. You are my heroes.”

“It’s greed, is all free-to-play games are.”

“Every interaction, mechanic, and screen must be expressly designed to convert scroungers into payers.”

We’re told that all f2p games are worthless, that any IAP is immoral, and that the designers of f2p games are borderline criminals.

Can everyone take a second and calm down?

There are design practices that we should shine a light on, and there are business practices we should frown upon, but we should not throw out a whole type of entertainment with the bath water.

Let’s ignore that there are high-profile f2p games on PC that always seem exempted from these arguments – maybe because they’re not aimed at the derided “casual” market? Let’s also ignore that there are successful but lower-profile f2p games on mobile that don’t use distasteful mechanics like countdown locks, despite me being lead developer for one. And finally let’s ignore the design reasons for creating a f2p game, like exploring the mechanics of large communities on a budget.

Instead, let’s explore an analogy.

CAUTION: ANALOGIES AHEAD

No one would argue that film and television are identical art forms. Many TV stations are free at the point of delivery, compared to the up-front cost of cinema, and are instead funded by ads. The individual episodes are shorter than films, yet have storylines that may last for years. Some shows, like soaps, essentially never stop. They both use sound and vision, but that’s about as much as TV and film have in common. The Sopranos is a very different masterpiece to Goodfellas.

You don’t think that ad funding affects how TV writers do their job? Do we stomp and scream at the writers of House because most ad breaks are prefixed by a mini cliffhanger? Aren’t they compromising their art for the sake of ad revenue? Why aren’t there blog posts decrying all television, ever, because Lost was designed to keep you watching week after week? Isn’t that as egregious as Dungeon Keeper’s countdown locks, which apparently condemn all other f2p games?

Actually I bet, if wordpress and medium had been around in the early 1940s, we would have seen posts complaining that television was nothing but a greedy cashgrab, compared to the “artistic purity” of cinema. But to shut down the medium then, when it was so young, would have robbed us of Cosmos, of West Wing, of the Simpsons, of Fraiser, of Monty Python, of Doctor Who, of Star Trek.

And is cinema really that artistically pure? Are the big releases not as commercially driven and “greedy” as any f2p game? Do they not compromise their vision for lower age ratings, or to include product placement? Let’s campaign to have them shut down as well!

In reality, upfront-cost video games and f2p games are different mediums, with different constraints. They’re both interactive entertainment, but don’t be fooled because they both have pixels and polygons. They serve different purposes, and have different potentials.

I must be clear I’m not arguing that Dungeon Keeper’s design is ok because TV has ads, nor that Dungeon Keeper should be given a free ride because f2p is a young medium. F2P is still finding its feet, and yes many of its baby steps have been in the wrong direction, but if we can bring it to a sensible maturity, we’ll have a richer world for all. Keep hating on bad f2p games, as well as on bad upfront-cost games, but don’t conflate them with the entire medium.

Edit: reworded conclusion slightly

Get Into Games Programming from GamesWest

I just spoke at GamesWest at UWS about starting a career as a professional games programmer. Here is my keynote slide deck, although it’s not very wordy:

Keynote 2013: https://dl.dropboxusercontent.com/u/13683832/get%20into%20game%20programming.zip

If there’s interest I can do a recording of this presentation. Shout at me on twitter!

How to build Unity3D scripts from the command line

I use emacs for day-to-day programming, including writing unity scripts. For a while I made changes then tabbed over to the unity editor to watch them compile and check for errors. This was, imaginably, tedious. Anyone not coding in MonoDevelop has the same issue, and even the MonoDevelop folks (god have mercy upon their souls) have issues.

It turns out there are multiple ways to compile your unity scripts from the command line, in such a way that most editors can be told to do it and parse the output for errors, greatly speeding up iteration time. I don’t think there’s anything new here, I’m just collating information that seems spread out across the net like horcruxes.

The first way to build your script files is the easiest to get working, but is cumbersome and slow. We’ll just pass a range of cryptic command line arguments to our Unity executable, it’ll launch without an editor front end, compile the scripts, dump any errors to the console, and quit. Cue up some regex and your editor can pull in the errors. Here is the command:

    path/to/unity -batchmode -logFile -projectPath path/to/project/dir -quit

On OSX you can find the unity executable at /Applications/Unity/Unity.app/Contents/MacOS/Unity.

You can read more about unity command line arguments in the unity docs, but let’s go through the options here:

  • -batchmode: stops any front end from loading
  • -logFile: tells unity what to do with any output. Without this parameter it’s all thrown away, which is of no use to our text editors. The docs above will tell you this argument needs a file parameter, but it’s undocumented that if you omit the parameter, all output, including errors, gets spat out to the terminal. That’s just what we want!
  • -projectPath: the path to the root of your project, where you find the sln file and the Assets directory
  • -quit: tells unity to exit straight after compiling.

This works on every platform, and has the nice feature of pulling in new files before compiling, so we don’t need a separate step for that. However it is slow as a milk cart going uphill, because it has to load and boot a lot of the unity runtime before it can do the simple thing of finding mono and asking it to compile a solution file for us.

It is also annoying because it doesn’t work if the unity editor is currently open. Even when in a heavy coding session there’s normally a need to run the game or tweak inspector values at regular intervals, and opening and closing the editor is not going to help you stay in a nice flow. Having said that, this drawback might not matter on a continuous integration server, so it might be the way to go.

So since this is just asking mono to compile for us in a round-about way, how do we cut out the middle man? The mono command-line compilation tool is called mdtool, and it ships with Unity. On OSX you can find it at /Applications/Unity/MonoDevelop.app/Contents/MacOS/mdtool. Here’s the command to give it:

    path/to/mdtool build path/to/project/project.sln

Here we just need to give mdtool the command build, and the path to the sln file. This is super-quick compared to the Unity approach, and works with the editor still open, but unfortunately won’t pick up newly added files. You’ll still have to tab to the editor for those to be picked up. However that’s relatively uncommon, so isn’t too much of a bother.

However when I tried this with the mdtool that ships with unity, I got very strange errors. They used to be relatively compact, but nowadays there are segfaults and all kinds of stuff going on. I did some casual googling over a few weeks, and couldn’t find a solution. But there was a workaround: install the latest Xamarin Studio (a free mono dev environment based on MonoDevelop), and use its mdtool. On OSX that’s at /Applications/Xamarin\ Studio.app/Contents/MacOS/mdtool.

So there you go: with one of these two approaches, you should be able to compile your unity scripts on the command line, find the errors and connect them to your editor of choice. These should all work on windows as well, but if someone can confirm in the comments that would be great.

If you’re interested, my emacs unity helper functions, including some flycheck compilers using both mdtool and unity, can be found on github.

Anatomy of a unit test

In my spare time I maintain a unit testing library built for Unity3D. It’s called UnTest, it’s open-sourced under the MIT license, and you can download it here.

In the Unity3D community forums announcing UnTest, _Shockwave asked for an example of some real-life unit tests written with this framework. UnTest is very xUnit-flavoured, so they follow a standard pattern, but I thought it would be a good excuse to talk about good unit testing practice.

Much of my unit testing approach is from Roy Osherove’s Art of Unit Testing, which is a very readable and practical book on unit testing. It’s aimed at .Net, so highly applicable for Unity3D development. The Art of Unit Testing website also has some recorded conference lectures from Osherove that are also worth watching. If you want to get better at writing unit tests, these are great resources.

The unit test I’m going to dissect is below. It’s a real-life test from a production behaviour tree system. It’s not really important here to understand what a behaviour tree or a selection node is, as much as the patterns and conventions I followed. Good unit tests are readable, maintainable and trustworthy. As we walk through the test, I’ll explain how these qualities apply, and how to maximise them.

    [Test]
    void UpdatePath_OneActionChild_AddsSelAndChildToPath() {

        var actionNode = FakeAction.CreateStub();
        m_selNode.AddChild(actionNode);

        m_selNode.UpdatePath(m_path, NodeResult.Fails, null);

        var selectionThenAction = new ITreeNode[] {
            m_selNode, actionNode };
        Assert.IsEqualSequence(m_path, selectionThenAction);
    }

To increase readability, the first thing to note is the context of the file you can’t see. It’s in a file called SelectionNodeTests.cs, so I instantly know this test applies to the SelectionNode class. There’s only one class in this file, with the same name, so there’s no chance of confusion.

The name of the function follows a consistent convention throughout the codebase: FunctionUndertest_Context_ExpectedResult. There are many naming conventions you could follow, this is the one I do. Context is how we set up the world before running the function. In this case, we’re adding a single action node to the selection node. ExpectedResult is how we want the function to behave; here we want the selection node and the action node to be added to the path.

It’s not important how long the name of this function is, since it’s never called from anywhere. The more readable and informative you can make the function name, the easier it will be to figure out what went wrong when it fails.

The unit test is split into three sections following the AAA pattern: Arrange, Act, Assert.

Arrange, where I set up the selection for testing:

    var actionNode = FakeAction.CreateStub();
    m_selNode.AddChild(actionNode);

All I need to do is create an action node and add it to the selection node.

Act, where I execute the function we want to test:

    m_selNode.UpdatePath(m_path, NodeResult.Fails, null);

Assert, where I check to see that end condition we expected has been fulfilled:

    var selectionThenAction = new ITreeNode[] {
        m_selNode, actionNode };
    Assert.IsEqualSequence(m_path, selectionThenAction);

Here I construct what I expect the path to be, and assert that it matches the actual path using an Assert library provided by UnTest.

By following this layout, all tests can be easily scanned. When time comes to fix or update a test, developers can dive in quickly.

You’ll notice I could compact the Assert section into one line:

    Assert.IsEqualSequence(m_path, new ITreeNode[] { m_selNode, actionNode });

The reason I keep it separate is to avoid “magic numbers”. It’s too easy to write code like, Assert.IsEqual(result, 5). The writer may know what this 5 means, but it would be much better for future readers to put it in a named variable and write Assert.IsEqual(result, hypotenuseLength).

Now this test is as readable as possible, how did I make it maintainable too? You’ll notice that by improving readability I’ve gone some way to also helping maintainability, as something that’s easier to read is also easier to understand, and therefore is easier to maintain. But there are other things I do as well.

Check out the first line:

    var actionNode = FakeAction.CreateStub();

I need an action to put into the selection node. I could use an existing IAction concrete class, but then any bugs in that concrete class might cause this test to fail. I’ll cover more why that’s bad later, but just pretend it sucks.

I could derive a new class from IAction, which I could keep simple enough to avoid bugs, but then I’d have to maintain that whenever the Action class interface changed. It’s much easier to use a “mocking framework” to do most of the hard work for me.

A mocking framework is a library that can be used to construct a new type at runtime that derives from Action and just does the right thing (among many other things). Then any changes are picked up for me automatically, and I have less code to maintain. If that sounds like magic, that’s because it is.

There’s a mocking framework behind that FakeAction.CreateStub() call, but since it’s such a common use case in this test suite I’ve wrapped it up in a helper function.

Any mocking framework that works with mono will work with Unity3D. I use Moq. The latest version is here. I’ve mirrored this in a unitypackage here for easy importing to Unity.

To further isolate myself from changes, I’m constructing the member variables m_selNode and m_path in a setup function (not shown). This function is run automatically before every test, and makes new SelectionNode and Path objects. This is not only handy, because they’re used in every test in the class, but also isolates the unit tests from changes to the constructor signatures. Other commonly-used functions can also be hidden behind helper functions, but it’s best not to hide your function-under-test for readability reasons.

The final thing I need to do is make the test “trustworthy”.

By going through the maintainable and readable steps, I’ve made sure this test depends on the minimum amount of game code. When this test fails, hopefully it will only be because the function under test, UpdatePath(), had an error.

The more game code you depend on, the closer your test slips along the spectrum from unit to integration test. Integration tests check how systems connect together, rather than single assumptions. They have their place in a testing plan, but here I’m trying to write a unit test. A great rule of thumb is that a single line of buggy code should cause failures in the minimum of unit tests, and ideally only one. If lots fail, that’s because the code isn’t isolated properly and you’ve ended up with some integration tests.

Some of my early unit tests, from F1 2011, created a whole world for the AI to move in and recorded the results, rather than mocking surrounding code like we have here. The end result was that a single bug in the world code could cause many many tests to fail. That makes it hard to track down the root cause of the bug, and meant I had probably written integration tests instead of unit tests.

When this test does fail, it will be deterministic. There’s no dependency here on databases, network services, or random number generators. There’s nothing worse than unit tests that fail once in a blue moon, because they erode developer trust in the test suite. That’s how you end up with swathes of tests commented out, and wasted engineering time.

Now you understand why I’ve written this real-life unit test in this way, and why it’s important your unit tests are readable, maintainable and trustworthy. Like any type of programming, writing good unit tests takes practice and perseverance. They’re truly the foundation of your project, giving you the freedom to restructure at will and the confidence that your game code is high quality. But like any foundation, if they’re not well engineered the whole edifice comes rapidly crumbling down. Take the time to follow up with the resources I linked above, and you will hopefully avoid that situation.