In a global market as challenging and competitive as mobile gaming, the one surefire way for a company to stand out is quality. A company that can ensure that gamers will get more value out of their games than out of the average competitor’s is a company with confidence in its product.
However, providing quality is easier said than done. Here at Outfit7, we have hundreds of millions of daily users who run our games on 15,000 different types of mobile devices worldwide. If we did all of our testing manually, that would involve testing every possible combination of game and mobile device, costing us thousands of hours of labor.
So how do we avoid turning quality assurance into a nightmare? We use automated game testing solutions. In short, we’ve developed a system to make our games play themselves.
First, we start with a testing tool. Our automated framework of choice is Appium. This open-source tool is a solution that allows us to scale to thousands of potential use cases and also test advanced techniques in the latest input-driven mobile games.
Since we use the Unity game engine (and lately our own bespoke engine, Starlite), our games are not compatible to either of the major platforms (native Android or iOS) that Appium is familiar with right out of the box. So we took Appium a step further, because without any customization, all Appium sees of our project hierarchy is the game engine’s native container for the whole game, denying us the possibility to individually test elements inside of our games.
To overcome this setback, we developed a Unity and Starlite code library to work with Appium. We include this library in the code for each of our games. When Appium runs the game with the library installed, it triggers a light JSON-based communication protocol to get the status of the game on the Appium server. The library doesn’t interfere with the game code, but it is able to control it in the background while monitoring play and informing Appium of any changes.
Samples of O7TestData.json and O7TestResult.json communication files.
Once a communication channel is established, Appium can instruct the game’s library to start a quality control test inside the game while it is still running. Through this simple protocol, Appium can send instructions including test mode, data, language, and parameters for especially detailed testing—in fact, instructions that are detailed enough to make the game play itself! Meanwhile, the library returns the exact results of each test in real time. To achieve such synchronisation, we use a simplified version of the two-way handshake protocol with ACK (acknowledge) and SYN (sync) JSON parameters.
System communication scheme with Appium.
The result: a stable and automated quality assurance solution that is still flexible enough to work in either Unity or our in-house engine, Starlite.
The benefits of automation: higher quality for much less work
We now have a platform capable of running tests on multiple devices in parallel, producing up to 1,500 localization tests per day. That means we save an entire workday of testing for each one of our game updates—which adds up quickly considering how many games we produce. The time we now save with this allows us to put our focus towards improving all other aspects, which brings the quality of our games even higher and higher. Hint: this time scales with every new app and every usage of the automation system feature.
The possibilities for expanding this system are enormous. In the future, we will add features like exploratory testing or even image recognition. Our system may seem highly advanced today, but we expect it to become the industry standard within a few years. Mobile games are becoming increasingly complex and numerous and the number of supported devices is growing. Soon, quality assurance setups like ours will become mandatory across the industry, and with that the QA skillset will expand as well. After all, the mobile gaming world is simply too competitive for gamers to settle for anything less than the highest quality.