Mutation tests on Android. Are they worth it?

Result of a study conducted at Natura&CO on mutation tests on Android with Kotlin in 2020

The automated tests of your project are really well implemented?

How mutation tests work?

Examples of mutations applied to production code that verifies whether a number is negative or not
Diagram showing how mutation tests work
Mutation test report example. From 5 mutations applied in the production code, only 3 were captured.

What tools can we use to apply mutation tests on Android?

Results of a benchmark conducted by Adnan Shaout

Challenges we found with Pitest on Android

Pitest GitHub issue about Android execution
Pitest GitHub issue about Kotlin execution

Results of mutation testing on an Android Kotlin project using Pitest Android Plugin

The class com.example.demo.TestClass$WhenMappings does not contain a source debug information.All classes must be compiled with source and line number debug information
The class com.example.demo.Model$Creator does not contain a source debug information.All classes must be compiled with source and line number debug information

Since we don’t have mutation tests in our pipeline, how do we ensure that our tests are well implemented?

Final words

References

Android engineer and test automation enthusiast. Working @Wise and formerly @ThoughtWorks. Twitter profile: @phellipeafsilva