...Which leads me to the topic at hand, one where I believe we can help. When it comes to mobile test automation, it all starts with one simple decision. Native framework or third-party tools?
What does that mean? As leading mobile platforms, iOS and Android — and their creators, Apple and Google — offer a tool for mobile UI test automation (as well as for unit tests). For Android, the tool is called Espresso; for iOS, it’s XCUITest. On the other hand, we have representatives of open source: third-party solutions like Appium or Calabash, both of which have become QA industry standards for many reasons.
Back to the decision of native vs. third-party. If you’re stuck in a position of uncertainty, the information and examples below should help.
Why Third-Party (Appium)
We’re using Appium as the third-party example because this solution has been most frequently used at STRV.
In short, the benefits are:
- It is the current QA industry standard.
- Allows for cross-platform tests.
- It comes with a big community and tons of learning materials.
According to many resources, Appium is the QA community’s favorite tool when it comes to mobile test automation. To see for yourself, try searching for Appium and then for XCUITest on Udemy. There are a few reasons for this popularity.
Firstly, Appium is the indirect follower of Selenium — Appium’s older, more mature brother. Both of these tools rely on the WebDriver API. Today, Selenium is still the no.1 used tool for web test automation, so it’s no surprise that QA analysts who had been using this tool adopted Appium when the mobile test automation trend started.
The second reason is that Appium provides the ability to write one test scenario that is executable on both Android and iOS. This is supposed to save QA analysts time when writing test cases. But the reality is often not as sweet as it may look. Only a small percentage of apps are truly identical on iOS and Android. More often than not, QA analysts must write up a lot of exceptions for the tests in order to run on both platforms. So whether or not the time saving benefit actually applies is arguable.
What I personally see as a big advantage of Appium is its community and how many learning materials are out there on the Internet. This is obviously great for the teams in which QA is not closely integrated with the developers; the community support allows QA analysts to solve a lot of the struggles and issues they come across before and during test automation, in both the earlier and advanced phases.
Why Native
After reading the section above, you might be wondering why anyone would decide to take an approach other than Appium? After all, it’s been established as the industry standard, it has a solid QA fanbase and there is a solid chance that your new QA hire will know the basic principles because he/she already knows Selenium. These are my thoughts:
- Native is much faster and more reliable.
- It allows QA analysts to use the same tech stack and language as developers.
- It provides better overall collaboration with app developers.
- The test code is a part of the source code (in the same repository).
- It is easier to integrate with CI/CD.
Running the test in native tools like XCUITest and Espresso is much faster than running them in Appium. On top of that, tests in native tools are much less flaky. The reason is that Appium is actually running on XCUITest and Espresso frameworks under the hood. And the extra layer of code — which is meant to allow you to run one test on both platforms — is actually slowing down the test execution and sometimes making tests fail for no obvious reason.
It is true that you can use many languages to work with Appium, such as Ruby, Python, JavaScript, PHP or C#. But is this what your team needs? Wouldn’t it be better if QA analysts used the same language as developers? Especially in agile teams, this is very important and crucial when it comes to the cooperation between QA and devs — which is why I see the fact that XCUITest only uses Swift, and that Espresso only uses Kotlin, as a big advantage. Can you imagine an iOS developer helping a QA analyst fix his tests written in Java or Python? A QA analyst having a different tech stack than developers is immediately anti-agile.
Because XCUITest can be run with command-line tools such as xcodebuild, you can easily plug these tests into your CI pipeline. There is no need to build a separate app or separate app structure for the UI tests, since they run with your existing app in the same way as unit tests.
Example: XCUITest on a Real Project
This is how the test group for login functionality may look. Tests are written with the BDD (behavior driven development) approach, making them easily readable by anyone non-technical.
class LoginUITest: UITestBase {
func testLogIn () {
givenAppIsLaunched()
whenUserSignsIn()
thenUserIsSignedInAndLogsOut()
}
func testLogInWithNonExistingAccount() {
givenAppIsLaunched()
whenUserSignsInWithNonExistingAccount()
thenUserNavigatesToOnboarding()
}
func testRegister() {
givenAppIsLaunched()
whenUserCreatesNewAccount()
whenNewUserIsCreated()
thenUserLogsOut()
}Tests consist of reusable steps. The main idea of that is to be able to reuse some parts of the test in a different test. All of the tests are isolated, meaning that they don’t depend on each other. Inside of the test, there are specific instructions which are executed.
func whenUserSignsIn() {
XCTContext.runActivity(named: "When user signs in") { _ in
Onboarding.getStarted.element.tap()
Onboarding.loginButton.element.tap()
Onboarding.topTextField.element.tap()
Onboarding.topTextField.element.typeText("lukas.smetana@strv.com")
Onboarding.bottomTextField.element.tap()
Onboarding.bottomTextField.element.typeText("TestPW1")
Onboarding.continueButton.element.tap()
cancelNotificationScreen()
}
}XCUITest also allows using XCTContext, which improves reporting and allows anyone to see exactly where the test failed in the report.
So, Should You Use Native Frameworks for Test Automation?
Test automation is inevitable on large and long-term projects. Manual testing can be highly effective on smaller projects and at a smaller scale but, as the project grows, manual testing gets very time-consuming and therefore costly.
The best thing about automated tests in XCUITest is that everyone can run them — developer, product owner and anyone on the client-side. This allows every stakeholder to have a real-time idea of the current application state.
There is no right or wrong answer when it comes to which automation framework you should choose. These are just ideas and thoughts based on the QA experience we’ve had at STRV. While there are teams that choose to select one framework, there are many teams that will combine two. So, whether you should use Espresso, XCUITest and/or Appium depends on your needs. Make sure to really understand what your project requires, discuss with experts if needed and go from there.
