How to Re-run RSpec Tests Again After Failures
In the course of working with Selenium and Appium via RSpec tests (while testing iOS apps), I occasionally was running into issues where Appium would lose its connection with the iOS simulator and the RSpec test would fail.
The test failures had nothing to do with the iOS app itself, or even with the RSpec tests. The problems existed somewhere between Selenium, Appium, and the iOS simulator. Something would just randomly break on one of those layers, and my code’s connection to the iOS simulator would essentially be lost. The only way to fix this problem is to manually close the iOS simulator and re-launch it via Appium. After that, all the RSpec tests would proceed merrily along their way without problem… or at least until Appium glitched again.
As you can imagine, with a large test suite, this can be rather annoying. If the driver glitches halfway through, you end up having to reset everything and re-run the entire suite from scratch.
This made me wonder… Would it be possible to catch the random driver errors and random Appium glitches, then programmatically reset the iOS simulator and re-run the failing test?
I did a bit of research and came up with a solution, which I will share here. Hopefully someone out there will get some benefit out of it:
spec_helper.rb (partial):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
RSpec.configure do |config| # Rerun the rspec example if we hit a transient driver error config.around(:each) do |example| retry_error_types = [ Selenium::WebDriver::Error::WebDriverError, Selenium::WebDriver::Error::NoSuchDriverError, Selenium::WebDriver::Error::UnknownError, Selenium::WebDriver::Error::JavascriptError ] retry_count = 3 retry_count.times do |i| # example is just a Proc. @example is the current RSpec::Core::Example example.run if @example.exception.nil? || !retry_error_types.include?(@example.exception.class) break # Stop the retry loop and proceed to the next spec end # If we got to this point, then a retry-able exception has been thrown by the spec e_line = @example.instance_variable_get '@example_block' puts "Error (#{@example.exception.class} - #{@example.exception}) occurred while running rspec example (#{e_line}" binding.pry if config.pry_on_errors if i < retry_count @example.instance_variable_set('@exception', nil) puts "Re-running rspec example (#{e_line}. Retry count #{i+1} of #{retry_count}" end end end end |
Essentially, what we are doing is running each RSpec test inside of an “around” block, and if the example fails for some driver-related reason, we reset and retry (up to three times) before moving on to the next test in the suite. Nifty, eh?
————————————-
01/30/2015 – Update to account for recent rspec changes
There have been some recent changes to the way how rspec handles examples and exceptions, so I have updated my code accordingly:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# Re-run the rspec example if we hit a transient error (such as a database threadlock) config.around(:each) do |example| retry_error_types = [ Selenium::WebDriver::Error::WebDriverError, Selenium::WebDriver::Error::NoSuchDriverError, Selenium::WebDriver::Error::UnknownError, Selenium::WebDriver::Error::JavascriptError ] retry_count = 3 retry_count.times do |i| example.run real_example = example.example if real_example.exception.nil? || !retry_error_types.include?(real_example.exception.class) break # Stop the retry loop and proceed to the next spec end # If we got to this point, then a retry-able exception has been thrown by the spec e_line = real_example.instance_variable_get '@example_block' puts "Error (#{real_example.exception.class} - #{real_example.exception}) occurred while running rspec example (#{e_line}" if i < retry_count example.instance_variable_set('@exception', nil) puts "Re-running rspec example (#{e_line}. Retry count #{i+1} of #{retry_count}" end end end |
Are you re-setting the example result anywhere? I’m curious when an example is re-ran and it passes whether the suite result is pass or fail.
Yes, it resets the exception on the rspec example to nil before it retries, as seen in this line of code:
@example.instance_variable_set(‘@exception’, nil)
This means that if the retry succeeds, there is no exception on the example, and the suite passes.
Thanks for the great code.
I am just starting to use rspec and webdriver.
Thanks again for sharing
Thanks for this. It helped me out greatly without having to incorporate a gem or something else.
Hi
Thanks for the awesome code.
I have just started programming and this has been really helpful.
It has stopped my allure reporting working.
In allure I get – undefined method ‘example’ for #
Sorry to ask but would you be able to explain why that is happening?
thanks
Mick
I was having some troubles recently where retries would succeed but still be counted as failures, showing at the error report at the end. The trouble seemed to be coming from resetting @excecption to nil – the code “example.instance_variable_set(‘@exception’, nil)” didn’t seem to be working. When I changed it to “real_example.instance_variable_set(‘@exception’, nil)”, it worked fine.
Was anyone else having this issue?