Issue warning/error if 'testpaths' does not match any files/folders

[nicoddemus]

We should issue a warning (or even an error?) if testpaths does not match any files or folders.

I think an error is reasonable, even if it might break some incorrectly-configured suite out there.

---

Originally posted by @nicoddemus in #11006 (comment)

This is not really a bug, but an intended (albeit questionable) behavior:

The values of testpaths are actually globs, so globbing for tests in the root yields nothing. Given it finds nothing, pytest will behave as if called from the command-line without any parameters, which makes it search recursively from the current directory looking for python_files to collect.

pytest/src/_pytest/config/__init__.py

Lines 1382 to 1384 in 739408b

	args = []
	for path in testpaths:
	args.extend(sorted(glob.iglob(path, recursive=True)))

If you create the tests directory, then pytest will correctly search in that directory only.

I agree those 2 facts are surprising:

1. The fact that testpaths is a glob. This is documented but easy to overlook, probably we should add a glob to the example there.
2. pytest silently not finding anything, and then proceeding as usual.

I don't think we can do anything more for 1, but for 2 seems like we should at least emit a warning if testpaths is defined but does not match anything.

[RonnyPfannschmidt]

i propose start with warning, elevate to error in 8.x

[nicoddemus]

Agree, sounds good. 👍

[kenny-y-dev]

Hello!

Adding a warning for this, will submit a PR shortly