Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new ParallelTests::RSpec::VerboseFormatter for enhanced debugging #926

Merged
merged 1 commit into from Dec 24, 2023

Conversation

jdufresne
Copy link
Contributor

The new formatter allows users of parallel_rspec to more easily obtain detailed information about the examples running as they occur in real time.

Background:

In my project, it was observed that an RSpec example would hang intermittently on CI. At first, no team member was able to make this reliably fail enough on CI to facilitate debugging and no one was able to reproduce locally at all. The hang only occurred with parallel tests running and never with just the "rspec" command.

In an attempt to gather more information, we began using the RSpec "documentation" formatter which outputs the "description", "context", and "it" strings. Unfortunately, when running in parallel, the output from different processes will interleave with one another making it extremely difficult to understand which example is responsible for the observed hang.

Solution:

Build a "documentation-like" formatter that plays nicely with the parallel_tests framework. The new formatter always ouputs all information on a single line to avoid important details from different processes interleaving with one another. This means the "describe" and "context" strings will be repeated across multiple "it" blocks.

To further assist, the formatter include the PID and parallel process number to help identify which subprocess hangs and which one continues. The output will look something like:

[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar

In the output above, it is clear the "Foo foo" example is still running.

Using this formatter, our project's team was able to successfully identify the example causing the intermittent hang and then address it. As such, I'm hoping that this may be useful for a larger audience.


Thank you for your contribution!

Checklist

  • Feature branch is up-to-date with master (if not - rebase it).
  • Added tests.
  • Added an entry to the Changelog if the new
    code introduces user-observable changes.
  • Update Readme.md when cli options are changed

The new formatter allows users of parallel_rspec to more easily obtain
detailed information about the examples running as they occur in real
time.

Background:

In my project, it was observed that an RSpec example would hang
intermittently on CI. At first, no team member was able to make this
reliably fail enough on CI to facilitate debugging and no one was able
to reproduce locally at all. The hang only occurred with parallel tests
running and never with just the "rspec" command.

In an attempt to gather more information, we began using the RSpec
"documentation" formatter which outputs the "description", "context",
and "it" strings. Unfortunately, when running in parallel, the output
from different processes will interleave with one another making it
extremely difficult to understand which example is responsible for the
observed hang.

Solution:

Build a "documentation-like" formatter that plays nicely with the
parallel_tests framework. The new formatter always ouputs all
information on a single line to avoid important details from different
processes interleaving with one another. This means the "describe" and
"context" strings will be repeated across multiple "it" blocks.

To further assist, the formatter include the PID and parallel process
number to help identify which subprocess hangs and which one continues.
The output will look something like:

```
[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar
```

In the output above, it is clear the "Foo foo" example is still running.

Using this formatter, our project's team was able to successfully
identify the example causing the intermittent hang and then address it.
As such, I'm hoping that this may be useful for a larger audience.
@grosser
Copy link
Owner

grosser commented Dec 24, 2023

awesome, looks nice!

@grosser grosser merged commit 7291dfd into grosser:master Dec 24, 2023
13 checks passed
@jdufresne jdufresne deleted the verbose-formatter branch December 25, 2023 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants