Integrate manpage builds into PEP 517 build backend

[webknjaz]

SUMMARY

This patch creates a thin wrapper around the setuptools' PEP 517 build backend in-tree. It features an ability to request generating the manpage files in the process of building a source distribution. This toggle is implemented using the config_settings mechanism of PEP 517.
One must explicitly pass it a CLI option to the build front-end to trigger said behavior. Roughly, the packagers are expected to use the following call:

python -m build --config-setting=--build-manpages

This option has no effect on building wheels.

Related to: #79469 (comment)

ISSUE TYPE

• Maintenance Pull Request
• Packaging Pull Request

COMPONENT NAME

packaging/pep517_backend
pyproject.toml
setup.cfg

ADDITIONAL INFORMATION

$ python -m build --config-setting=--build-manpages --sdist
[...]
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools >= 39.2.0, wheel)
* Getting dependencies for sdist...
running egg_info
writing lib/ansible_core.egg-info/PKG-INFO
writing dependency_links to lib/ansible_core.egg-info/dependency_links.txt
writing entry points to lib/ansible_core.egg-info/entry_points.txt
writing requirements to lib/ansible_core.egg-info/requires.txt
writing top-level names to lib/ansible_core.egg-info/top_level.txt
reading manifest file 'lib/ansible_core.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'COPYING'
writing manifest file 'lib/ansible_core.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (docutils, jinja2, pyyaml, straight.plugin)
* Building sdist...
running sdist
running egg_info
[...]
Writing ansible-core-2.15.0.dev0/setup.cfg
Creating tar archive
removing 'ansible-core-2.15.0.dev0' (and everything under it)
Successfully built ansible-core-2.15.0.dev0.tar.gz

[Spredzy]

I am not knowledgeable enough to +1 the code implementation, but the idea behind it. Anything that can be baked in the code directly, rather than needing documentation is a big +1

[nitzmahone]

I'm not opposed to this in principle, but it does feel like a slippery slope in some ways- back to the bad old days of custom DistUtils commands that we're still cleaning up in other projects 😆 . But there's definitely some benefit in being able to have declarative (and dynamic) build deps that we can test, so I'm like +0.75 for the overall thing if we can make it a little more robust.

[nitzmahone]

Why not just do this inline in our interpreter with docutils.core.publish_file (or one of the other ones that takes string input, since we've already read the source in to hack on it anyway)? Locating the wrapper script that basically does the same thing feels brittle...

[webknjaz]

Because I was converting a series of calls across makefiles and wasn't implementing this from scratch. So I never checked what APIs are available. FWIW, I was trying to scope this effort to reproduce what makefiles do w/o changes, so it's easier to verify that it's doing the same thing. Optimization/refactoring could be a separate PR IMO.

[nitzmahone]

This appears to do the same thing sans subprocess, script and intermediate file:

    from docutils.core import publish_file
    from docutils.writers import manpage
    from io import StringIO

    templated_rst_doc = rst_in.read_text().replace('%VERSION%', version_number)
    rst_in.unlink()

    publish_file(source=StringIO(templated_rst_doc), destination_path=rst_in.with_suffix('').with_suffix(''), writer=manpage.Writer())

[webknjaz]

@nitzmahone still, I wouldn't want to couple refactoring of the manpage builds with the packaging aspects. How do you feel about having this in a separate PR?

P.S. Pro tip: StringIO/BytesIO should be used as context managers, otherwise, they need to be closed explicitly to clean up the resources correctly. This is not always a problem, but is a nice principle to follow for consistency with other I/O interactions.

[nitzmahone]

I may be mostly projecting my own behavior here 😉, but "I'll fix it later" will probably be written on my tombstone. I can live with it "as-is", but would definitely like to see it simplified with direct calls and fewer/no writes into the source tree at some point.

[webknjaz]

For me, it may end up being a never-ending PR refactoring that needs more and more testing, though. It'll end up blocked for longer this way.

[nitzmahone]

This probably wouldn't be necessary if the docutils publish was called directly instead of as a subprocess (see below)

[webknjaz]

Can we agree to move this to a separate PR?

[mattclay]

I'm fine with making this change in another PR.

[webknjaz]

I rewrote the smoke testing in Python and tightened the pins. With this, It think that they main blocking concerns are addressed now. I'd like to keep things that are out of the scope separate.

[webknjaz]

@mattclay I think this is ready for the next round of reviews.