Change (or drop) removal of YAML front matter from Markdown before rendering

[fperez]

Problem

Today, the default (as shown in this JupyterLab-MyST issue behavior is for YAML frontmatter to be entirely stripped out of the document before reaching the renderer. Since we now have renderers, such as JupyterLab-MyST, that can properly render frontmatter, perhaps we should reconsider this decision.

As it stands, unless the user changes the default, they won't see frontmatter in MyST-rendered documents.

Proposed Solution

I'm not 100% sure, but it seems to me that we should send the entire document to the renderers, and leave it to them to strip out parts of a file they can't handle. For example, right now if the default renderer were to improve its frontmatter handling, it still wouldn't benefit users as the frontmatter isn't making it to the renderer at all.

I'm inclined to suggest removing that setting altogether, but at least switching the default to False seems better to me.

In any case, mostly opening this for discussion/input, as there's competing constraints at play.

Additional context

Relevant to this: #14200, #14203.

CC @rowanc1, @agoose77.

[rowanc1]

Thanks @fperez! This is something that is tripping up a lot of the users of the MyST extension as we make extensive use of YAML frontmatter for title, authors, affiliations, etc. It was unintuitive to us as extension authors that this is happening in the model, and there is no way to access this from the renderer once it is stripped off of the markdown value. Looking forward to any comments!

[fperez]

I also see that in #14200 @bollwyvl was dealing with front matter-heavy md docs; Nick, any thoughts on whether we should make any changes to this behavior as above?

[bollwyvl]

Welp... in my case, the JEP front matter was just breaking ToC, due to hard-coded whitespace "grammar" inconsistencies around a leading hr... but then ToC is entirely regex/heuristic-driven, not even using the marked parser.

At the implementation level: doing weird stuff to text/* documents, based on the MIME: seems un-neighborly. marked could certainly handle stripping it out, as its two-phase walk and render gives the implementer plenty of rope to hang themselves.

As to stripping it out: I guess I understand the motivation, sort of. It seems front matter is very frequently destructively consumed by downstreams, and not passed directly to end users.

Looking at specifications for "correct" frontmatter behavior: it's not part of the OG markdown spec, or GFM. So pretty much that means the Jekyll handling of it is probably the "canonical" usage, but that is purely a processing engine, and doesn't provide any kind of intermediate, partially-authored UI. So not a lot of help there.

To my original frustration with even having to do front matter on that issue, Jupyter's own JEPs: why expend effort having metadata if it doesn't appear anywhere in the output without a bunch of clicks to other sites? The underlying data in git, the PR history, and issue relationships is much stronger metadata. I guess if fully committed to embedding non-inline-printed metadata, I'd much rather see some use of a full standard, such as RDFa, or JSON-LD... but imagine most of these would be stripped out by most renderers. Spoil-sports.