My friend Tuan Hoang has recently published a very nice blog post about selective testing in iOS.

Tokopedia team has also published a very informative blog post about this topic, where they claimed that selective testing would make their unit test job 8 times faster 🤯.

It’s a great technique to quickly smoke-test relevant tests that are important for the changes you are making. I think the killer use case for this technique is a pre-commit hook, where all relevant tests will be executed quickly to ensure that our code changes will not break any tests in the relevant modules. This would save a lot of time and CI resources for the team because developers often forget to rerun the tests or check relevant tests when making changes, and tend to commit their changes directly to the remote repository.

XcodeSelectiveTesting

In this blog post, I’m not using the tool created by Tuan Hoang but another tool called XcodeSelectiveTesting.

In principle, both tools are working in the same way:

• Getting the change sets
• Find the affected targets by analysing the dependency graph
• Adjust the test plan to enable only affected test targets
• Run the test plan

The following illustration is taken from XcodeSelectiveTesting to illustrate that changes in Login module will only trigger tests affected test targets: Login, LoginUI and MainApp

This is a very clever process to execute only related tests from related packages, which are affected by the current code changes.

You can follow the instruction from XcodeSelectiveTesting to install the tool and test if it is working correcly.

Prepare Xcode project

We will need to create a specific SelectiveUnitTests.xctestplan for XcodeSelectiveTesting. The purpose of this special testplan is to separate the normal UnitTests.xctestplan, where all tests are enabled, from SelectiveUnitTests.xctestplan, where only relevant tests are enabled by XcodeSelectiveTesting.

We’ll then add it into our the Scheme, which is used to execute unit tests.

Git Pre-commit Hook

The orignal XcodeSelectTesting repo currently has an issue that makes it impossible to be used in Git pre-commit hook. I’m using a forked version of the tool with minor changes so that the tool can query staged change sets from the git repository.

We can then create the following smoke.sh script to run XcodeSelectiveTesting and relevant tests:

cp -fR TestPlans/SelectiveUnitTests.xctestplan /tmp/SelectiveUnitTests.xctestplan

xcode-selective-test --test-plan TestPlans/SelectiveUnitTests.xctestplan

xcodebuild \
    -workspace MyApp.xcworkspace \
    -scheme MyApp \
    -destination 'platform=iOS Simulator,name=iPhone 15 Pro' \
    -parallel-testing-worker-count 2 \
    -enableCodeCoverage NO \
    -testPlan SelectiveUnitTests \
    test | xcbeautify

cp -fR /tmp/SelectiveUnitTests.xctestplan TestPlans/SelectiveUnitTests.xctestplan

The reason we need to backup the SelectiveUnitTests.xctestplan is that, we don’t want anyone in the team to commit any change to the default settings of the SelectiveUnitTests.xctestplan, otherwise it will create a lot of git merge conflicts because everyone is working on different packages at the same time.

Then in the pre-commit hook script, we can easily invoke this script whenever we try to commit the code locally.

Benchmarking

When using XcodeSelectiveTesting on the (moderately big) codebase at my job, it could greatly improve my confidence to commit code changes to the remote repository. The time needs to run the relevant tests are depending on where the changes are, but in the best case, it cuts off half of the required time, where the change happens in a top layer package.

• All tests: 5m22s (4870 test cases)
• Selective Tests: 2m14s (186 test cases)

If we consider that our Unit Test job might take up to 20 minutes on our CI runner, running selective test locally to avoid surprised failing tests and wasting CI resources is a very much welcome solution.

Conclusion

Selective testing is a great tool to accelerate testing process. When combining this technique with a pre-commit hook, it increases the confidence to commit code changes to remote repositories automatically. Also it helps to avoid wasting CI resources developers forget to adapt the tests after changing the code, which happens quite often at my team. The main drawback for this technique is that it would take longer for the code to land on remote repositories, because the project needs to be built and tests need to be exected on every commit.