Fix tags in comment

[peterzhu2118]

AFL caught a case where Liquid tags in comments were not treated as plain text. The problem is that tags like {%%} do not match on FullToken so we get parsing errors like:

Liquid::SyntaxError: Liquid syntax error (line 1): Tag '{%%}' was not properly terminated with regexp: /\%\}/

[dylanahsmith]

This doesn't match the behaviour of liquid-c (https://github.com/Shopify/liquid-c/blob/8a9a0498efbe7566b576d3e1eed8310c84587429/ext/liquid_c/block.c#L157-L166). It looks like the difference is that liquid-c yields the markup as an unknown tag, which the comment tag can then ignore.

[peterzhu2118]

I updated it to yield instead to behave like liquid-c.

[dylanahsmith]

liquid-c still calls this if the invalid tag token isn't terminated with %}.

[peterzhu2118]

That is true. Should we just change liquid-c to be consistent with liquid (and call BlockBody.raise_missing_tag_terminator instead of yield)? Should we also change how comments are handled because I think everything inside a comment block should be escaped?

[dylanahsmith]

This doesn't have to be one or the other. We can just check for if token.end_with?('%}') to only call BlockBody.raise_missing_tag_terminator only if the tag terminator is actually missing, which avoids breaking backwards compatibility with liquid templates and provides the more appropriate error message.

Should we also change how comments are handled because I think everything inside a comment block should be escaped?

We have explicitly allowed tags with an invalid tag name (#256) so it seems very likely that this is relied upon.

[peterzhu2118]

But the problem is that for a tag that looks like {% comment %}{% " %}{% endcomment %} is not matched on FullToken (and token.end_with?('%}') is true for {% " %}) in liquid so BlockBody.raise_missing_tag_terminator is called. But in liquid-c, it's yielded back to the caller (from the part you linked in the comment above).

[dylanahsmith]

Exactly, so you want to fix that problem, but not introduce another inconsistency when the tag terminator is actually missing. That's why I am recommending adding an if token.end_with?('%}') conditional here.

For example, Liquid::Template.parse('{% foo') should result in the syntax error Liquid syntax error: Tag '{%' was not properly terminated with regexp: /\%\}/ and not Liquid syntax error: Unknown tag '{%' as the current change would change it to without liquid-c. We could also use a test for the error messages for in this case, rather than just asserting that it results in a syntax error.

[peterzhu2118]

An I see, that makes sense. Thanks for clarifying!