Quick Tip: Running tests repeatedly in Xcode
Making flaky tests fail
2 min read
Have you ever had a flaky test in your codebase? I did. Let me tell you, it isn't enjoyable. The main problem is (who would have thought it?) the flakyness. It fails randomly on the CI, but if you run it locally, everything seems fine. A simple "Works on my machine" sounds tempting, doesn't it? But that does not solve the problem. So how do you make a test fail, that only fails on every 500th run?
How about running it more than 500 times?
Run your test a thousand times (or more) with a few clicks
It's really simple. Just right-click your test in the "Test Navigator" in Xcode and select
Run "testMyAwesomeTest()" repeatedly..., choose how many repetitions you want and hit "Run". That's it.
Sure, you can also play with the other options, but that's basically it.
Doing the same with the command line
Of course, you can do the same using the command line. That might be handy if you want to run the test repeatedly on your build agent via
For my test project the command would be
xcodebuild test -scheme "running-tests" -destination 'platform=iOS Simulator,name=iPhone 14,OS=16.1' -only-testing "running-testsTests/FizzBuzzServiceTests/testFizzBuzzMagic" -test-iterations 1000
The interesting bits here are:
xcodebuild test-> let's run some tests
-only-testing "..."-> I just want to run this single (flaky) test. Not the entire test suite
-test-iterations 1000-> let's run the test 1000 times
And again: That's it. 🚀
One more thing
Of course, flaky tests can be much trickier to reproduce than just "it fails randomly. Just run it really often". If you want to dive deeper into "flaky tests", Antoine van der Lee has a great blog post on that matter.
Happy coding 🧑💻