Mutation tests on Android. Are they worth it?
Result of a study conducted at Natura&CO on mutation tests on Android with Kotlin in 2020
Translation in Portuguese here 🇧🇷
As an Android consultant at ThoughtWorks, I had the privilege of following Natura&CO’s digital transformation closely. A continuous and necessary transformation for Natura to differentiate themselves in the market and to scale the development of their digital products. The interesting thing about this type of organizational transformation is that whoever applies it tends to gain better guidelines for good software development, better delivery processes, and consequently, greater quality in the product released.
The reflexes of this digital transformation arrived in the team I am working on, Natura’s financial tribe. Here, we had the chance to conduct an investigation into a type of automated test that I believe to be very interesting and at the same time unknown to many people in the Android world. These are the mutation tests. Our goal was to understand whether this kind of test could be added to our delivery process and increase the quality of our Android app.
In the specific case of this article, we will talk about mutation tests on Android with Kotlin.
Fundamentally, this type of test aims to answer a single, direct, and complex question:
The automated tests of your project are really well implemented?
Mutation tests aim to guarantee the quality of your test suite. The goal is to check if your automated tests are actually able to detect faults or not.
Your production code will be tested directly by traditional Android tools such as JUnit, Espresso, Robolectric, etc … and indirectly tested by mutation testing tools.
In this article, we will share the tools we analyzed and the results of this investigation!
How mutation tests work?
To ensure that your automated tests are well implemented, firstly a mutation test will apply a random change to a specific point in your production code. This automatically generated change is called a mutation.
After applying the change, the mutation test tool will trigger your automated test suite on top of the original code and also on top of the changed code.
Considering that all tests performed on top of the original code passed, when the tool performs the same tests on the mutated code, the following scenarios can happen:
- 1# Tests performed on top of the original code passed but the same tests performed on top of the mutated code failed. This indicates that the tests failed for unwanted changes, which leads to the conclusion that the mutation applied has been captured and the test is well implemented 👍 ✅
- 2# Tests performed on top of the original code passed and tests performed on top of mutated also passed. This indicates that your test is poorly implemented because even with a random change in your production code your test still passed and did not identify anything 👎 🔴
- 3# No tests covered the mutated area, indicating that your test suite is not covering parts of your code that can be susceptible to bugs 👎 🔴
Realize that when we run a test on mutated code, failure is a positive sign. This means that possible bugs in that area can be identified and testing is guaranteeing the quality and behavior of the code.
An interesting point worth mentioning is that no additional code needs to be written to apply this analysis. We just need to set up the tool correctly and it will make use of everything that already exists in the project for you. As a result, the mutation testing tool can also generate reports describing how many mutations have been captured and how many have not been:
To end the explanation, I would like to share a sentence that I like very much and generates an interesting thought about the process of writing tests:
“Never trust a test you haven’t personally seen fail”
— Unknown Author
What tools can we use to apply mutation tests on Android?
To apply mutation tests in the Java and JVM (Java Virtual Machine) environment, we have some options available. Many of them are unknown in the software engineering area and do not have examples of real utilization in large-scale systems.
Among the tools researched in this case study, only one stood out as a possible candidate to enter in a large-scale Android project. That tool was Pitest. In addition to being the tool with the best documentation and easier configuration, it was also the best ranked in the surveyed benchmarks. For this reason, we chose Pitest as the main tool to be analyzed in this proof of concept.
For illustration, below you can find an example comparing various mutation testing tools for Java, including Pit(est):
Challenges we found with Pitest on Android
Pitest is a state of the art tool for mutation testing with Java and the JVM (as stated on their main page). However, a big challenge we have in Android is: modern applications are no longer using Java, and Android does not run on the JVM.
As we know, Kotlin is the official language recommended by Google for new Android projects. In addition, Android has its own virtual machine, which is a lighter version of the JVM running DEX bytecodes.
When trying to apply Pitest on Android, we soon realized some limitations (that other people have also experienced):
Since Pitest is an open-source and constantly updated project, the creators and the community tried to solve these problems by creating specific plugins for each situation. One plugin to integrate Pitest with Android and the other to integrate Pitest with Kotlin. Unfortunately, no specific solution was developed to meet the needs of Android projects with Kotlin 😢
Despite the limitations presented, the investigation continued to see what happened if we applied the Pitest Android plugin to an Android project that is completely written in Kotlin.
Results of mutation testing on an Android Kotlin project using Pitest Android Plugin
The scope of this investigation consisted of analyzing a project entirely written in Kotlin, using all the functionalities the language has to offer. We always try to write an idiomatic code with strong use of Coroutines.
How did Pitest (in its current state) behave in this scope?
Spoiler: unfortunately the Pitest's response (using Android plugin) was not satisfactory for the context of an Android project with Kotlin. For this reason, I discourage the practice of mutation testing in a large-scale Android project using the tools we have available today (Out / 2020).
Pitest did not work very well when applying mutations considering some Kotlin keywords. One of them was the when keyword which ended up generating the following error in some scenarios:
The class com.example.demo.TestClass$WhenMappings does not contain a source debug information.All classes must be compiled with source and line number debug information
We also had problems with some other classes that used annotations like @Parcelize in the code, generating a similar error.
The class com.example.demo.Model$Creator does not contain a source debug information.All classes must be compiled with source and line number debug information
For the case of Coroutines, it is clear that the specific Kotlin plugin does not yet support this functionality.
There are articles sharing similar problems when trying to apply mutation tests to Android projects with Kotlin. An alternative to work around these problems would be to ignore some specific classes and only consider simpler classes when applying mutations. When making this kind of adjustment, the tool worked correctly in some cases.
Given the result we had, we faced the following dilemma: should we adopt a new tool in our delivery pipeline to apply mutation tests just for simple classes that do not use many Kotlin resources? Or would be better to just not use mutation tests given the effort we would have to maintain the tool?
Due to the difficulties found, we concluded that the effort to maintain a new tool would be higher than the value brought by it. Therefore, we discourage including a mutation testing tool in a large-scale Android project with Kotlin at the moment.
Since we don’t have mutation tests in our pipeline, how do we ensure that our tests are well implemented?
Ensuring that tests are well implemented is a difficult task that requires discipline. However, some actions can help this path to be less painful. For this you can:
- Promote good practices when using unit tests in Java and Kotlin.
- Implement a good testing pyramid strategy that can be followed faithfully by your team.
- Favor testing behavior instead of testing implementation. A good resource on this topic is the book Unit Testing Principles, Practices, and Patterns written by Vladimir Khorikov.
- Be a person next to the quality metrics of your team.
- Strengthen the practice of TDD. If you are unable to apply TDD to your team, at least make the test fail once before considering it done.
Final words
Despite not recommending the use of mutation testing tools in large-scale Android projects with Kotlin today, we recognize the great work the collaborators of Pitest are doing for our community. Maybe with a little more energy and more people contributing to these awesome projects, we could have a more robust tool to use in our Android projects in the near future.
References
- http://thatsabug.com/blog/gentle_intro_mutation_testing/
- https://blog.activelylazy.co.uk/2017/01/24/never-trust-a-passing-test/
- https://medium.com/@s4n1ty/experimenting-with-mutation-testing-and-kotlin-b515d77e85b5
- https://blog.frankel.ch/experimental-kotlin-mutation-testing/
- https://pitest.org/
- https://arxiv.org/pdf/1707.09038.pdf
- https://blog.trifork.com/2016/09/07/adequacy-of-android-unit-tests/
We hope that this article has contributed to something in your technical journey. Please, share any questions or feedback in the comments section below! Thank you!
Also thanks to Poliana Florencio, Lucas Valadão, Lucas de Souza da Conceição and Eduardodribas for reviewing this article!
