Intro

We’ve been building mobile web apps using JQuery Mobile with the main target being webkit-based mobile browsers - Android and iPhone, basically. We’re big fans of functional testing, so we spent some time figuring out how to get a good testing infrastructure set up for mobile app development. That’s what I’ll be sharing here.

The approach

Our team members have a lot of experience using Cucumber to write automated acceptance tests, and we’re big fans. In previous projects we were building native iOS apps and testing them using Cucumber and Frank. We wanted to bring over some of the testing infrastructure we already had set up, so we decided on a similar approach. We still use Cucumber to drive tests in the iOS simulator, but instead of talking to a native app using Frank we are talking to our mobile web app via Selenium 2’s remote WebDriver protocol. In order to do this we run a special iOS app provided as part of Selenium 2 called iWebDriver which hosts an embedded UIWebView and listens for WebDriver commands, in much the same way as a Frankified native app listens for Frankly commands. We launch the iWebDriver app using the same SimLauncher gem which we’ve previously used to launch our native app for pre-commit and CI testing. To make our lives easier we use Capybara, which provides a nice ruby DSL for driving web apps and some useful cucumber step definitions out of the box. We configure Capybara to use the selenium-webdriver gem to drive the iWebDriver-hosted web app via the WebDriver remote protocol.

Selenium 2 == WebDriver

Note that when I talk about Selenium here, I am not referring to the Selenium 1.0 system (Selenium RC, Selenium IDE and friends). We are using Selenium 2.0, which is essentially the WebDriver system plus some Selenium backwards compatibility stuff which isn’t used in this setup. We don’t run any kind of Selenium server, we simply run iWebDriver in the iOS simulator. iWebDriver exposes the WebDriver remote API, and our cucumber tests drive our mobile app directly within iWebDriver using that API.

WebDriver on mobile is still bleeding edge

On the whole we have been happy using WebDriver and the iOS simulator to drive our functional tests. However, the WebDriver infrastructure for mobile platforms does feel a little rough around the edges. We have rare cases where our CI test run will fail, apparently due to iWebDriver crashing. When we were first getting our app under test we saw what appeared to be an iWebDriver bug where it was overwriting the global _ variable that underscore.js uses. We worked around that by using underscore.js in no-conflict mode. We also had to add a small but hacky monkey-patch to the selenium-webdriver gem to work around a bug in the way CocoaHTTPServer (the embedded HTTP server that iWebDriver uses) handled a POST with an empty body. That bug has been fixed in more recent releases of CocoaHTTPServer, but frustratingly we got no response when we reported the issue and suggested an upgrade of the dependency to resolve the issue. UPDATE: Jari Bakken notes in the comments that a similar workaround to our monkey-patch has now been added to the selenium-webdriver gem. We also found that iWebDriver locks up when showing a javascript alert. We worked around this by injecting a stub version of window.alert() in our web page at the start of each test. This worked, but is obviously less than ideal.

We briefly experimented with using the Android WebDriver client, but it lacked support for CSS-based selectors at the time. That may have changed since. If you are just using Capybara this is not an issue since Capybara uses XPath under the hood. However we quickly found that our use of JQuery Mobile meant we needed to write a fair amount of custom selectors, and our automation engineers had a preference for CSS. Ideally we would have run our test suite against both the iOS simulator and the Android emulator, but this lack of CSS support led to us deciding to just test on iOS. Given that both platforms are webkit-based this was an acceptable tradeoff.

The last drawback worth mentioning around using the iOS simulator is that it can only be run under OS X, and it doesn’t seem possible to run multiple instances of the simulator at once. This means that parallelizing your tests will involve running multiple OS X host machines, either physical boxes (we have used mac minis successfully) or virtual machines.

Recommendations

All in all I would still recommend a testing approach similar to what I’ve outlined. The extra safety-net that our suite of functional tests provided was just as valuable as in a non-mobile project. While they are not totally polished, iWebDriver and Cucumber are the best way I know of currently to build that safety net.

One thing we didn’t do which I would recommend is to also have your test suite execute against a desktop webkit-based browser, using ChromeDriver for example. Tests will run a lot quicker on that platform, giving you faster feedback. You will likely need to make minor modifications to the way your tests are written, but if you’re running tests in ChromeDriver from day one then you should be able to tackle any small issues as and when they arise. That said, I would absolutely make sure you are also running your full test suite against at least one mobile platform as frequently as you can - there will be issues in mobile webkit browsers that aren’t apparent on a desktop browser.