As part of my work at Ninja Blocks, we’re creating a physical Linux device that lives in the user’s home - the Ninja Sphere. As we’re close to a public release, we’ve been trying to make the setup process as reliable as possible. This is a difficult challenge, because there are a huge number of moving parts that make up the system. Reliability is especially critical during the initial setup, since we can’t send down an update because the device doesn’t yet have an internet connection.

When a user first opens the Ninja Sphere app, they start by pairing their device to a WiFi network, which is done by securing a link using a passcode over Bluetooth Low Energy (where available on the phone). Once the device is on WiFi, switching over to an HTTP REST API for downloading any updates for the device and pairing with the cloud.

I took up the task of writing automated testing for this whole process, which looks something like this (excuse the early prototype plastic warpage, the long pauses, and the blurry picture):

Sphere Automation

Under normal circumstances, the Sphere itself has just one physical connection: power. Everything is done over a wireless medium of some sort. During the testing process, once you identify which Sphere you want to pair with, the Sphere challenges the phone to initiate a secure connection by entering a passcode displayed on the top of the device. If the codes match, the handshake succeeds and WiFi credentials can be submitted.

To automate this process, the phone-side automation needs to know this passcode. Short of having some machine vision work out which code was being displayed, the easiest method is via a side channel. Thankfully a serial console is exposed via a USB port, so we can log in and retrieve the pairing code using our physical access to the device, then inject this into the automation so everything moves along smoothly and without user input.

I chose to wrap this process up into a little class that uses node-serialport and stream-expect to log in, sudo, and then run a set of arbitrary commands on the device to retrieve the passcode.

Phone Automation

The phone side of the test process is implemented using Appium, which automates WebDriver-based testing. Our mobile app is split into a platform native set of screens, and a set of screens implemented in a WebView and shared with the Android port. In Appium terms, this is a “hybrid” app, and is supported conveniently by switching “contexts” between the native tree of UI elements and the WebView tree. When you’re in the WebView context, you can use normal CSS selectors to find elements and perform tests (just like when using Selenium or any other WebDriver implementation).

Switching to the WebView is a little bizarre, but not hard. I used a slightly more readable function than used in their documentation (supposedly this works any time where there is just a single WebView since the native context always comes first):

function switchToWebView(contexts) {
  return browser.context(contexts[1]); // choose the webview context
}

// ...

browser.contexts().then(switchToWebView)

Once in the WebView contexts, tests look something like this:

// wait until we're on the welcome page and continue
.waitForElementByCssSelector('.page-welcome sphere-animation', wd.asserters.isDisplayed, 10000).isDisplayed()
.sleep(2000)

.then(function() {
  console.log('Waiting for sphere to come up ...');
  return this.pairingCodePromise;
}.bind(this))

// click on the big continue button
.elementByCssSelector('.page-welcome sphere-animation').click()
.sleep(2000)

This part of the testing should be the easiest part of the excercise, given the number of people doing mobile application testing these days. Unfortunately it seems like nobody tests the testers, since the test process actually fails occasionally just because Appium, ios_webkit_debug_proxy, Instruments, or some other arbitrary part of the process breaks. Thankfully though it’s fairly easy to tell when the test procedure is at fault because you get floods of Instruments errors or connection refused events from Appium or its helpers.

Becoming the Adversary

Now pairing completes successfully unless Appium itself fails, everything’s done, right? Well, unfortunately it’s all too easy to forget that in the real world a user is unlikely to keep their phone awake for over 3 minutes while everything happens. The app is going to get closed, put to the background, or even crash. Can we recover? The next step is to create more tests, based off the successful one, that inject these disruptions throughout the process.

Rather than copy-pasting the entire thing, I chose to add hooks throughout the script where I could hook in to the promise chain and mix things up. Here’s what a chunk of the test looks like:

// enter wifi credentials
.waitForElementByCssSelector('.page-spheramidwifi .wifi-password', wd.asserters.isDisplayed, 20000)
.then(injectHook('wifi-intent'))
.elementByCssSelector('.page-spheramidwifi .wifi-password').sendKeys(wifiPassword)
.elementByCssSelector('.page-spheramidwifi .wifi-continue').click()
.then(injectHook('wifi-sent'))
.sleep(5000)

And here’s an example test case that hooks on wifi-sent (right after the WiFi credentials have been sent off) and injects the disruption of leaving the app for a bit and coming back:

var test = new basicTest.PairingTest();

test.hook('wifi-sent', function(_, driver) {
  return driver.sleep(10000)
    .backgroundApp(10) // 10 seconds in the background, then come back
    .contexts().then(switchToWebView);
});

test.performTest();

This means we can now test that leaving the app at any point is recoverable!