Sphinx: Add inventory decoder for Sphinx

[amotl]

About

Operational tasks about decoding Sphinx inventories.

References

• Add project registry and tooling crate/crate-docs#104
• Crate.io Sphinx Inventory Labels crate/crate-docs#105

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 31 lines in your changes are missing coverage. Please review.

Project coverage is 60.48%. Comparing base (486252e) to head (878a31f).

Files	Patch %	Lines
pueblo/sphinx/inventory.py	0.00%	31 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #73       +/-   ##
===========================================
- Coverage   89.81%   60.48%   -29.33%     
===========================================
  Files          22       23        +1     
  Lines         746      777       +31     
===========================================
- Hits          670      470      -200     
- Misses         76      307      +231     

Flag	Coverage Δ
main	?
ngr	60.48% <0.00%> (-2.52%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[amotl]

It looks like those discussions, tools, and patches, revolve around the same topics. Maybe Markdown output has not been considered yet?

Problem statements

• Intersphinx undefined label while objects.inv lists it sphinx-doc/sphinx#6659
• Implement intersphinx entries web viewer (devtools?) sphinx-doc/sphinx#8211
• Make intersphinx (a.k.a. external references) more user friendly

Solutions

• https://github.com/bskinn/sphobjinv
• https://github.com/webknjaz/intersphinx-untangled
• https://webknjaz.github.io/intersphinx-untangled/
• Add a new references builder sphinx-doc/sphinx#12190
• Sphinx: Add inventory decoder for Sphinx #73

Proposals

• [FR] Have a special page listing linkable Spihnx objects per site readthedocs/readthedocs.org#7481
• Textual TUI for browsing inventories bskinn/sphobjinv#274

NB: This post has been edited multiple times to progressively include relevant references. Feel free to suggest further edits.

/cc @bskinn, @webknjaz, @chrisjsewell

[webknjaz]

@amotl I also consider sphinx-doc/sphinx#8211 related

[webknjaz]

Maybe Markdown output has not been considered yet?

output? Why?

[amotl]

Dear @webknjaz,

thanks a stack for your reply, and for suggesting the reference to your post sphinx-doc/sphinx#8211, I've added it to the summary note above.

Markdown FTW

Maybe Markdown output has not been considered yet?

output? Why?

We usually reach out to Markdown generation these days, because the output can be used both as a text-only variant, and for upgrading to HTML by sending it through a corresponding Markdown renderer. In this way, text output from utility programs becomes super versatile, because it can be easily shared on all systems understanding Markdown, like GitHub and Discourse ¹, which we use most of the time.

By using Markdown, it was easy to convey different sections of output into a single output file. In this case, the inventory decoder needed to take into account that we wanted to include intersphinx references for multiple projects, effectively all inventory files listed in sphinx-inventories.txt.

This has now converged into crate/crate-docs#105, not quite reaching the goal yet, because the generated Markdown became too large to be shared on behalf of a GitHub comment 🙈.

General Thoughts

We discovered the resources shared above per #73 (comment), including intersphinx-untangled, only after writing this little program just out of the need that we needed it, for the same reason you outlined at sphinx-doc/sphinx#8211, mostly frustration about needing to go the extra mile each time again and again to discover corresponding link targets from a huge documentation tree spanning multiple projects.

So, we are effectively doing corresponding orentiation flights only in retrospective, now, disovering your work on the same topic. Thank you so much!

Now, taking both sphobjinv and intersphinx-untangled into consideration, and also citing @chrisjsewell from https://github.com/orgs/sphinx-doc/discussions/12152:

I feel it would be nicer if there was a CLI that could just be pointed at the conf.py and output all mappings (possibly with some "filtering" options).

Is the actual problem in (shippable) tooling here, that both sphobjinv and intersphinx-untangled work on a per-project level, but both @chrisjsewell and we are looking for a multi-project intersphinx summary output of some sorts?

With kind regards,
Andreas.

Footnotes

1. But also Grafana. ↩

[bskinn]

sphobjinv author here, thanks for pinging!

We usually reach out to Markdown generation these days, because the output can be used both as a text-only variant, and for upgrading to HTML by sending it through a corresponding Markdown renderer.

Do you mean that you would like something like the following, which would generate a Markdown-formatted report of the objects in the inventory:

$ sphobjinv convert markdown objects.inv objects.md

If so, that should be feasible, I'd be happy to take it as a feature request.

Is the actual problem in (shippable) tooling here, that both sphobjinv and intersphinx-untangled work on a per-project level

For sphobjinv, yes, I think that's right --- though, a more precise way to say it is that it works on a "per-objects.inv level." You point it at the generated inventory file from one specific documentation set.

It would be straightforward to write a wrapper that parses intersphinx_mapping out of conf.py and uses sphobjinv to create a consolidated references list (as proposed in https://github.com/orgs/sphinx-doc/discussions/12152). And, I can see where Markdown output from sphobjinv would be helpful to implement that wrapper.

I don't consider it within sphobjinv's purview to actually be that consolidating wrapper, though.

[amotl]

Dear @bskinn,

thanks a stack for your swift reply.

Markdown Support

We appreciate your understanding about Markdown output, and look forward for corresponding support in sphobjinv. If the code in this PR, and the blueprint at cratedb-sphinx-inventory-nodocs.md is helpful for that, please take it into consideration but also recognize it is just a minimal variant what could be done.

For example, at the cost of increased file size, output could optionally use Markdown tables instead of just wrapping the main output verbatim into text blocks. When looking at MyST Markdown, other output variants might also become viable in future iterations.

Multi-Project Support

It would be straightforward to write a wrapper [to handle multi-project] processing. [...] [However,] I don't consider it within sphobjinv's purview to actually be that consolidating wrapper, though.

We will be happy to explore this on behalf of pueblo.sphinx (this PR), by pulling in sphobjinv. Excellent! Because we have the need for standalone usage (i.e. without access to conf.py), we consider this wrapper to receive the following features aka. variants to seed a list of projects:

• Parse intersphinx_mapping out of conf.py, like suggested by @chrisjsewell.
• Use a list of paths or URLs to multiple objects.inv files, like sphinx-inventories.txt, either by providing a file, or on stdin/cmdline. This reflects the use case we have, based on the code in tasks.py.
• Autodiscovery-like features on top of both variants: For example, a conf.py can be local or on a remote URL, possibly within a doc or docs folder within a Git project pointed at. Similarly, an objects.inv file can be located within a {doc,docs}/_build/ folder within a typical Git project, or within a /en/latest URL path when hosted on RTD, for example. That would be just a bunch of convenience heuristics on top for people who want them.
• Consider the use case of @webknjaz, to run a multi-tenant static intersphinx generator operation, on behalf of a list of projects like sphinx-websites.yml, based on relevant code to generate a static website, if he doesn't have any objections about it?
• Consider how the new references builder in Sphinx by @chrisjsewell could be used in this context. -- Add a new references builder sphinx-doc/sphinx#12190

In this spirit, we are offering pueblo.sphinx as an incubation area for corresponding freestyle evaluations which would probably never make it into Sphinx core, and, as you've outlined, would also go beyond the capacities of sphobjinv. So, until there is no better place where to curate such code, pueblo will be happy to host it and provide it to the community, by pulling in a few code fragments from our and other pens, enumerated above, and using sphobjinv as the main workhorse.

wdyt?

With kind regards,
Andreas.

[bskinn]

@amotl:

We appreciate your understanding about Markdown output, and look forward for corresponding support in sphobjinv.

Actually, now that I think about it further, I'm less certain whether direct Markdown output support in sphobjinv makes sense. My main purpose for the project is utilitarian output, for others to use as fits their needs, and not trying to customize the output formatting of sphobjinv itself to various users' use-cases.

So, to make sure you're aware: sphobjinv has an API as well as a CLI. I need to improve my documentation quite a bit to make that clearer... but, for example, you can obtain object representations of inventories via the following:

>>> import sphobjinv as soi
>>> inv_local = soi.Inventory('/path/to/local/objects.inv')
>>> inv_remote = soi.Inventory(url='https://example.com/doc/latest/objects.inv')

As a specific example:

>>> import sphobjinv as soi
>>> ( inv := soi.Inventory(url="https://sphobjinv.readthedocs.io/en/latest/objects.inv") )
Inventory(project='sphobjinv', version='2.3', source_type=<SourceTypes.URL: 'url'>)
>>> ( obj := inv.objects[0] )
DataObjStr(name='sphobjinv.cli.convert', domain='py', role='module', priority='0', uri='cli/implementation/convert.html#module-$', dispname='-')
>>> print(f"\n## {obj.name}\n\n{obj.domain}:{obj.role} ({obj.priority})\nURI Within Docset: {obj.uri_expanded}\nDefault
display name: {obj.dispname_expanded}")

## sphobjinv.cli.convert

py:module (0)
URI Within Docset: cli/implementation/convert.html#module-sphobjinv.cli.convert
Default display name: sphobjinv.cli.convert

So, I think sphobjinv already gives you everything you need to programmatically craft the Markdown you want---and, without having to worry about the sphobjinv release schedule.

[bskinn]

we consider this wrapper to receive the following features aka. variants to seed a list of projects:

All of those make sense! And, it seems to me there's already interest out there in multiple of those possible modes of multi-docset cross-reference compilation. Seems like a great thing to put energy into, to me.

[amotl]

Thanks a stack for your guidance.

So, I think sphobjinv already gives you everything you need to programmatically craft the Markdown you want.

After actually learning about sphobjinv, we already figured that would be the right approach. We will go down that path on behalf of pueblo.sphinx utilizing sphobjinv.Inventory, exploring both areas of Markdown/HTML output, and the multi-project processing details outlined above. We will keep you posted and appreciate any feedback on relevant outcomes. Thank you so much!

[bskinn]

👍Good luck! Please ping me any time with questions.

[amotl]

Filtering / DWIM

Regarding DWIM, as you can see in tasks.py, we already need filtering capacities (omit_documents, labels_only). In this spirit, let me also reference that item of yours, mostly because it also mentions the "webapp" idea, which resonates with us.

• Textual TUI for browsing inventories bskinn/sphobjinv#274

Other Intersphinx Community Operations

In order to further increase community support and leverage, fragments of your nice operations could also be absorbed into code instead of hosting it on behalf of a text file in a gist, if you like that idea. My thinking is that if you already have a tool in hand that takes care about those topics in a generalized way, it would feel like an artifical boundary why it would only work on local projects.

• https://github.com/bskinn/intersphinx-gist
• https://gist.github.com/bskinn/0e164963428d4b51017cebdb6cda5209

In the same spirit like @webknjaz curated a list of popular Python projects on behalf of intersphinx-untangled in YAML format, and you did on behalf of the intersphinx_mappings.txt Gist, those lists could also be written down or wrapped through Python, in order to provide easy command-line access, right?

So, instead of a possible reference research scenario like that:

wget https://matplotlib.org/stable/objects.inv
sphobjinv suggest objects.inv draw

What about such a command, effectively wrapping sphobjinv and the intersphinx_mappings gist into another variant of DWIM?

anansi suggest matplotlib draw

[bskinn]

So, instead of a possible reference research scenario like that:

wget https://matplotlib.org/stable/objects.inv
sphobjinv suggest objects.inv draw

If I'm understanding you correctly---the sphobjinv CLI is already set up to pull from remote locations using the -u flag, and the CLI also auto-searches the provided URL for likely locations of the objects.inv, so in most cases a user can paste any link from the remote documentation set. E.g.,:

>sphobjinv suggest -u https://matplotlib.org/stable/users/explain/quick_start.html#parts-of-a-figure draw

Attempting https://matplotlib.org/stable/users/explain/quick_start.html#parts-of-a-figure ...
  ... no recognized inventory.
Attempting "https://matplotlib.org/stable/users/explain/quick_start.html/objects.inv" ...
  ... HTTP error: 404 Not Found.
Attempting "https://matplotlib.org/stable/users/explain/objects.inv" ...
  ... HTTP error: 404 Not Found.
Attempting "https://matplotlib.org/stable/users/objects.inv" ...
  ... HTTP error: 404 Not Found.
Attempting "https://matplotlib.org/stable/objects.inv" ...
  ... inventory found.

------------------------------------------------------------------------

The intersphinx_mapping for this docset is LIKELY:

  (https://matplotlib.org/stable/, None)

------------------------------------------------------------------------

Project: Matplotlib
Version: 3.8.3

10431 objects in inventory.

------------------------------------------------------------------------

1 result found at/above current threshold of 75.

:std:label:`draw_idle`

So, if anansi is built to do the lookup from intersphinx_mapping, and thus turn anansi suggest matplotlib draw into an internally delegated call to sphobjinv suggest -u https://matplotlib.org/stable/objects.inv draw... it should work pretty cleanly, as far as I can tell, without any need for an intermediate local download of the objects.inv.

[amotl]

thus turn anansi suggest into an internally delegated call to sphobjinv suggest ...

Yeah exactly like this. Thanks for telling me about the -u option. It looks like all the building blocks for this very detail are in place already, sweet!

the CLI also auto-searches the provided URL for likely locations of the objects.inv, so in most cases a user can paste any link from the remote documentation set.

Ah I see, it already has the auto-discovery on board. I should actually start using sphobjinv more instead of just extrapolating ;]. Thanks!

[bskinn]

Ah I see, it already has the auto-discovery on board.

Only if you provide it a link from within the docset where it lives on the web.

sphobjinv doesn't know how to introspect conf.py:intersphinx_mapping to go from project name to docset URL. (And, indeed---sphobjinv is meant to be able to run whether or not the user is in a Sphinx documentation directory tree; and, whether or not Sphinx is in the current Python environment. sphobjinv does not depend on Sphinx at runtime.)

[amotl]

Hi again,

thank you very much for this excellent conversation.

• We converged the wish list items into the Linksmith RFC. Please let us know if you think we missed an important item you always dreamed of. It would be sad to leave anything behind.
• linksmith 0.0.1 has been released, and Linksmith Usage has minimal usage docs.
• We are closing this PR, because corresponding concerns will now be handled within linksmith.
• Please create issues for any other wishes in this regard, both on ingress flexibility, and output formatting. Thanks! 🍀

Cheers,
Andreas.