tests: split DDP & common + running xdist

[Borda]

What does this PR do?

Identify DDP tests and split testing into two steps:

• run DDP in a single thread as it was until now
• the rest of the tests try to run in parallel

In addition move unitest steps back to the main workflow

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

---

📚 Documentation preview 📚: https://torchmetrics--2127.org.readthedocs.build/en/2127/

[Borda]

Seems like this does not even start properly, could be because of?

torchmetrics/tests/unittests/conftest.py

Lines 55 to 73 in 886c09b

	def pytest_sessionstart():
	"""Global initialization of multiprocessing pool.
	Runs before any test.
	"""
	pool = Pool(processes=NUM_PROCESSES)
	pool.starmap(setup_ddp, [(rank, NUM_PROCESSES) for rank in range(NUM_PROCESSES)])
	pytest.pool = pool
	def pytest_sessionfinish():
	"""Correctly closes the global multiprocessing pool.
	Runs after all tests.
	"""
	pytest.pool.close()
	pytest.pool.join()

cc: @SkafteNicki

[SkafteNicki]

Seems like this does not even start properly, could be because of?

Yeah that would definitely be the problem, if I remember correctly you cannot start a child process inside a child process. So I am not sure if we will be able to use xdist for testing.

[SkafteNicki]

I can get it to run locally, but it is complaining with error

AttributeError: module pytest has no attribute pool

meaning that we need to refactor the code where we initialize the pool used for multiprocessing testing if we want this PR do land.

[Borda]

Pivoting and splitting into two steps can eventually be different jobs.
so all DDPs are marked and will run as they are now; the rest will be run xdist

BTW, mowing unittest action to the main workflow as we do not use conda env anymore so, having it separately does not make sense anymore, and it will be easier to read/follow steps

[codecov]

Codecov Report

Merging #2127 (b3e39f5) into master (41e942c) will decrease coverage by 54%.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #2127     +/-   ##
========================================
- Coverage      87%     33%    -54%     
========================================
  Files         303     303             
  Lines       17059   17059             
========================================
- Hits        14821    5613   -9208     
- Misses       2238   11446   +9208     

[Borda]

this is stated from master:

so seems we get 15-40min boost on GPU

[SkafteNicki]

Fantastic work @Borda

[lantiga]

Great job @Borda ⚡️

[stancld]

@Borda Wuaah, this is amazing! 🥇