Skip to content

Commit

Permalink
Fix flaky testing for Minitst/MultipleAssertions
Browse files Browse the repository at this point in the history
The shared `@cop` instance variable of `AssertOffense` module
causes flaky tests due to state changes.

The `AssertOffense` testing module needs to be updated to allow for tests
that are independent in configuration between tests.
It is necessary to prevent race conditions caused by shared configurations.

This is a workaround to skip tests until the issue is resolved in `AssertOffense` module.
  • Loading branch information
koic committed Mar 4, 2024
1 parent 0faaa25 commit 0f481c8
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 11 deletions.
2 changes: 0 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,5 +84,3 @@ jobs:
bundler-cache: true
- name: test
run: bundle exec rake test
env:
ERROR_ON_TEST_FAILURE: "false"
4 changes: 1 addition & 3 deletions Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,7 @@ if RUBY_ENGINE == 'jruby' || RuboCop::Platform.windows?
else
desc 'Run tests'
task :test do
error_on_failure = ENV.fetch('ERROR_ON_TEST_FAILURE', 'true') != 'false'

system("bundle exec minitest-queue #{Dir.glob('test/**/*_test.rb').shelljoin}", exception: error_on_failure)
sh("bundle exec minitest-queue #{Dir.glob('test/**/*_test.rb').shelljoin}")
end
end

Expand Down
20 changes: 14 additions & 6 deletions test/rubocop/cop/minitest/multiple_assertions_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -186,9 +186,9 @@ class FooTest < ActiveSupport::TestCase
err = assert_raises React::ServerRendering::PrerenderError do
@renderer.render("NonExistentComponent", {}, nil)
end
assert_match(/NonExistentComponent/, err.to_s, "it names the component")
assert_match(/\n/, err.to_s, "it includes the multi-line backtrace")
end
end
Expand All @@ -205,9 +205,9 @@ class FooTest < ActiveSupport::TestCase
assert_equal 1, 1
assert_equal 1, 1
end
assert_match(/NonExistentComponent/, err.to_s, "it names the component")
assert_match(/\n/, err.to_s, "it includes the multi-line backtrace")
end
end
Expand All @@ -231,9 +231,9 @@ class FooTest < ActiveSupport::TestCase
assert_equal 1, 1
end
end
assert_match(/NonExistentComponent/, err.to_s, "it names the component")
assert_match(/\n/, err.to_s, "it includes the multi-line backtrace")
end
end
Expand Down Expand Up @@ -292,6 +292,8 @@ def test_asserts_once
end

def test_generates_a_todo_based_on_the_worst_violation
skip 'FIXME: The shared `@cop` instance variable causes flaky tests due to state changes.'

inspect_source(<<-RUBY, @cop, 'test/foo_test.rb')
class FooTest < Minitest::Test
def test_asserts_once
Expand Down Expand Up @@ -533,6 +535,8 @@ class FooTest < ActiveSupport::TestCase
end

def test_registers_offense_when_complex_structure_with_multiple_assertions
skip 'FIXME: The shared `@cop` instance variable causes flaky tests due to state changes.'

configure_max_assertions(2)

assert_offense(<<~RUBY)
Expand Down Expand Up @@ -572,6 +576,8 @@ class FooTest < ActiveSupport::TestCase
end

def test_registers_offense_when_complex_structure_with_assignments_and_multiple_assertions
skip 'FIXME: The shared `@cop` instance variable causes flaky tests due to state changes.'

configure_max_assertions(2)

assert_offense(<<~RUBY)
Expand Down Expand Up @@ -611,6 +617,8 @@ class FooTest < ActiveSupport::TestCase
end

def test_registers_offense_when_complex_multiple_assignment_structure_and_multiple_assertions
skip 'FIXME: The shared `@cop` instance variable causes flaky tests due to state changes.'

configure_max_assertions(2)

assert_offense(<<~RUBY)
Expand Down

0 comments on commit 0f481c8

Please sign in to comment.