Exclude .arpa tld names from domains() strategy

[jenstroeger]

Fixes #3567, closes #3582.

@Zac-HD this is a Draft PR for now because we need to discuss a few things:

• turns out that the list of default top-level domain names did not contain special-use domain names, except arpa:

  hypothesis/hypothesis-python/src/hypothesis/provisional.py

  Line 44 in 600fc8a

  TOP_LEVEL_DOMAINS = ["COM"] + sorted(_tlds[1:], key=len)

  Should I add them?
• Alternatively we’d remove arpa from that list and inverse the logic when creating the list to draw from in the PR.
• Hardwire the special-use domain names, or use a vendor file?
• I wasn’t sure about your preference re import naming.

Things to do for me:

• Add tests, and
• Add to AUTHORS.rst.

[Zac-HD]

Wow, this is even more of a rabbithole than I expected. In writing this comment I consulted the following sources:

• https://en.wikipedia.org/wiki/.arpa, https://www.iana.org/domains/arpa, and https://www.rfc-editor.org/rfc/rfc3172.html to understand the use and management of the arpa top-level domain
• https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.txt as an authoritative (?) list of special-use domains which doesn't include the arpa top-level domain (just several subdomains)
• https://data.iana.org/TLD/tlds-alpha-by-domain.txt, which explicitly includes arpa as a top-level domain but not any special-use domains such as invalid, test, example, localhost, etc.

Based on https://www.iana.org/domains/arpa I'm willing to say that it's "special-use enough for me" and suggest that we discard ARPA when loading tlds from our vendored file. I think then we can avoid adding the allow_special_use argument discussed in the issue, though adding a domains= argument to st.emails() would still be nice.

[jenstroeger]

Fairnuff, I adjusted the PR accordingly — what says you now, @Zac-HD?

Should we split this PR into two because it’s now technically two separate changes?

Also, would you like me to squash all commits into a single one, or will you squash on merge?

[Zac-HD]

I think they're sufficiently related to keep as one PR, and whatever is easiest for you - I gave up on having a particularly clean git history years back, since it's just not viable when mentoring newbies 🙂

[jenstroeger]

@Zac-HD I could use a little hand-holding with testing: when I run make check-coverage as described here I get this errror:

coverage: commands[0]> python -bb -X dev -m coverage run --rcfile=.coveragerc --source=hypothesis -m pytest -n0 --ff tests/cover tests/conjecture tests/datetime tests/numpy tests/pandas tests/lark tests/redis tests/dpcontracts tests/codemods tests/typing_extensions
ERROR: while parsing the following warning configuration:

  ignore::hypothesis.errors.NonInteractiveExampleWarning

This error occurred:

Traceback (most recent call last):
  File "/.../hypothesis/hypothesis-python/.tox/coverage/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1690, in parse_warning_filter
    category: Type[Warning] = _resolve_warning_category(category_)
  File "/.../hypothesis/hypothesis-python/.tox/coverage/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1728, in _resolve_warning_category
    m = __import__(module, None, None, [klass])
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/.../hypothesis/hypothesis-python/.tox/coverage/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
...
  File "/.../hypothesis/hypothesis-python/.tox/coverage/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/.../hypothesis/hypothesis-python/.tox/coverage/lib/python3.10/site-packages/hypothesis/provisional.py", line 147, in <module>
    st.lists(
AttributeError: partially initialized module 'hypothesis.strategies' has no attribute 'lists' (most likely due to a circular import)

I just rebased on the latest Hypothesis master branch. Is there anything I need to do/know to run tests for this PR?

[Zac-HD]

OK, the problem is that hypothesis.provisional has a top-level import of hypothesis.strategies, and vice-versa... and the solution is annoyingly subtle - we can't practically modify .provisional, so we need to import it inside a function... even though we're using it as a default argument. Via the public API I'd use @st.composite, but with internals we can use LazyStrategy directly and save a tiny bit of overhead for subsequent draws:

def domains():
    import hypothesis.provisional
    return hypothesis.provisional.domains()

def emails(
    *,
    domains: st.SearchStrategy[str] = LazyStrategy(domains, (), {}),
) -> SearchStrategy[str]:

and while it's not the easiest reading, I think that's the best solution we're going to get.

[Zac-HD]

Thanks Jens!

[jenstroeger]

Thank you for wrapping up the PR, @Zac-HD! I promise, one day I’ll actually finish one…