Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Normalize parse and render options #145

Merged
merged 4 commits into from
Sep 3, 2021
Merged

Normalize parse and render options #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f02ea75
Normalized the parse and render options that are available in the par…
phillmv Aug 31, 2021
fd690b5
Merge branch 'main' into normalize-parse-render-options
phillmv Aug 31, 2021
2d0af64
Added missing commas.
phillmv Aug 31, 2021
62e70d9
Added very basic tests for exercising full range of parser, render op…
phillmv Sep 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
19 changes: 12 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,26 +130,31 @@ CommonMarker accepts the same options that CMark does, as symbols. Note that the
| Name | Description
| ----------------------------- | -----------
| `:DEFAULT` | The default parsing system.
| `:SOURCEPOS` | Include source position in nodes
| `:UNSAFE` | Allow raw/custom HTML and unsafe links.
| `:FOOTNOTES` | Parse footnotes.
| `:LIBERAL_HTML_TAG` | Support liberal parsing of inline HTML tags.
| `:VALIDATE_UTF8` | Replace illegal sequences with the replacement character `U+FFFD`.
| `:SMART` | Use smart punctuation (curly quotes, etc.).
| `:LIBERAL_HTML_TAG` | Support liberal parsing of inline HTML tags.
| `:FOOTNOTES` | Parse footnotes.
| `:STRIKETHROUGH_DOUBLE_TILDE` | Parse strikethroughs by double tildes (compatibility with [redcarpet](https://github.com/vmg/redcarpet))
| `:VALIDATE_UTF8` | Replace illegal sequences with the replacement character `U+FFFD`.

### Render options

| Name | Description |
| ------------------ | ----------- |
| `:DEFAULT` | The default rendering system. |
| `:UNSAFE` | Allow raw/custom HTML and unsafe links. |
| `:GITHUB_PRE_LANG` | Use GitHub-style `<pre lang>` for fenced code blocks. |
| `:SOURCEPOS` | Include source position in rendered HTML. |
| `:HARDBREAKS` | Treat `\n` as hardbreaks (by adding `<br/>`). |
| `:UNSAFE` | Allow raw/custom HTML and unsafe links. |
| `:NOBREAKS` | Translate `\n` in the source to a single whitespace. |
| `:SOURCEPOS` | Include source position in rendered HTML. |
| `:VALIDATE_UTF8` | Replace illegal sequences with the replacement character `U+FFFD`. |
| `:SMART` | Use smart punctuation (curly quotes, etc.). |
| `:GITHUB_PRE_LANG` | Use GitHub-style `<pre lang>` for fenced code blocks. |
| `:LIBERAL_HTML_TAG` | Support liberal parsing of inline HTML tags. |
| `:FOOTNOTES` | Render footnotes. |
| `:STRIKETHROUGH_DOUBLE_TILDE` | Parse strikethroughs by double tildes (compatibility with [redcarpet](https://github.com/vmg/redcarpet)) |
| `:TABLE_PREFER_STYLE_ATTRIBUTES` | Use `style` insted of `align` for table cells. |
| `:FULL_INFO_STRING` | Include full info strings of code blocks in separate attribute. |
| `:FOOTNOTES` | Render footnotes. |

### Passing options

Expand Down
11 changes: 8 additions & 3 deletions lib/commonmarker/config.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,28 @@ module Config
OPTS = {
parse: {
DEFAULT: 0,
SOURCEPOS: (1 << 1),
Copy link
Collaborator Author

@phillmv phillmv Aug 31, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NEW parse option:

As far as I can tell, we check for SOURCEPOS in inlines.c, and if the flag is set we annotate the node with sourcepos info, thereby allowing someone implementing their own renderer to also add sourcepos info to their output.

UNSAFE: (1 << 17),
VALIDATE_UTF8: (1 << 9),
SMART: (1 << 10),
LIBERAL_HTML_TAG: (1 << 12),
FOOTNOTES: (1 << 13),
STRIKETHROUGH_DOUBLE_TILDE: (1 << 14),
UNSAFE: (1 << 17)
}.freeze,
render: {
DEFAULT: 0,
SOURCEPOS: (1 << 1),
HARDBREAKS: (1 << 2),
UNSAFE: (1 << 17),
NOBREAKS: (1 << 4),
VALIDATE_UTF8: (1 << 9),
Copy link
Collaborator Author

@phillmv phillmv Aug 31, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied over all of the parse options and resorted the render hash. As described in the PR description, since the render method also initializes the parser with the same options integer, all of the parse options should be available as render options as well.

SMART: (1 << 10),
GITHUB_PRE_LANG: (1 << 11),
LIBERAL_HTML_TAG: (1 << 12),
FOOTNOTES: (1 << 13),
STRIKETHROUGH_DOUBLE_TILDE: (1 << 14),
TABLE_PREFER_STYLE_ATTRIBUTES: (1 << 15),
FULL_INFO_STRING: (1 << 16),
UNSAFE: (1 << 17),
FOOTNOTES: (1 << 13)
}.freeze,
format: %i[html xml commonmark plaintext].freeze
}.freeze
Expand Down
17 changes: 17 additions & 0 deletions test/test_basics.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,21 @@ def test_markdown_to_html
html = CommonMarker.render_html('Hi *there*')
assert_equal "<p>Hi <em>there</em></p>\n", html
end

# basic test that just checks if every option is accepted & no errors are thrown
def test_accept_every_option
text = "Hello **world** -- how are _you_ today? I'm ~~fine~~, ~yourself~?"
parse_opt = [:SOURCEPOS, :UNSAFE, :VALIDATE_UTF8, :SMART, :LIBERAL_HTML_TAG, :FOOTNOTES, :STRIKETHROUGH_DOUBLE_TILDE]
render_opt = parse_opt + [:HARDBREAKS, :NOBREAKS, :GITHUB_PRE_LANG, :TABLE_PREFER_STYLE_ATTRIBUTES, :FULL_INFO_STRING]

extensions = %i[table tasklist strikethrough autolink tagfilter]

assert_equal "<p>Hello <strong>world</strong> – how are <em>you</em> today? I’m <del>fine</del>, ~yourself~?</p>\n", CommonMarker.render_doc(text, parse_opt, extensions).to_html

# note how tho the doc returned has sourcepos info, by default the renderer
# won't emit it. for that we need to pass in the render opt
assert_equal "<p data-sourcepos=\"1:1-1:65\">Hello <strong>world</strong> – how are <em>you</em> today? I’m <del>fine</del>, ~yourself~?</p>\n", CommonMarker.render_doc(text, parse_opt, extensions).to_html(render_opt, extensions)

assert_equal "<p data-sourcepos=\"1:1-1:65\">Hello <strong>world</strong> – how are <em>you</em> today? I’m <del>fine</del>, ~yourself~?</p>\n", CommonMarker.render_html(text, parse_opt, extensions)
end
end
3 changes: 3 additions & 0 deletions test/test_extensions.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ def test_uses_specified_extensions
doc = CommonMarker.render_doc('~a~ ~~b~~ ~~~c~~~', :STRIKETHROUGH_DOUBLE_TILDE, [:strikethrough])
assert_equal("<p>~a~ <del>b</del> ~~~c~~~</p>\n", doc.to_html)

html = CommonMarker.render_html('~a~ ~~b~~ ~~~c~~~', :STRIKETHROUGH_DOUBLE_TILDE, [:strikethrough])
assert_equal("<p>~a~ <del>b</del> ~~~c~~~</p>\n", html)

CommonMarker.render_html(@markdown, :DEFAULT, %i[table strikethrough]).tap do |out|
refute_includes out, '| a'
refute_includes out, '| <strong>x</strong>'
Expand Down
5 changes: 0 additions & 5 deletions test/test_maliciousness.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,6 @@ def test_rendering_with_bad_type
CommonMarker.render_html(nil)
end

err = assert_raises TypeError do
CommonMarker.render_html("foo \n baz", [:SMART])
end
assert_equal('option \':SMART\' does not exist for CommonMarker::Config::OPTS[:render]', err.message)

assert_raises TypeError do
CommonMarker.render_doc("foo \n baz", 123)
end
Expand Down
7 changes: 5 additions & 2 deletions test/test_smartpunct.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@ class SmartPunctTest < Minitest::Test

smart_punct.each do |testcase|
doc = CommonMarker.render_doc(testcase[:markdown], :SMART)
html = CommonMarker.render_html(testcase[:markdown], :SMART)

define_method("test_smart_punct_example_#{testcase[:example]}") do
actual = doc.to_html.strip
doc_rendered = doc.to_html.strip
html_rendered = html.strip

assert_equal testcase[:html], actual, testcase[:markdown]
assert_equal testcase[:html], doc_rendered, testcase[:markdown]
assert_equal testcase[:html], html_rendered, testcase[:markdown]
end
end

Expand Down