Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hy doesn't lex unicode literals #1126

Closed
Anteru opened this issue Aug 31, 2019 · 0 comments · Fixed by #2670
Closed

Hy doesn't lex unicode literals #1126

Anteru opened this issue Aug 31, 2019 · 0 comments · Fixed by #2670
Labels
good first issue Good for newcomers T-bug type: a bug X-imported imported from Bitbucket

Comments

@Anteru
Copy link
Collaborator

Anteru commented Aug 31, 2019

(Original issue 1422 created by vodik on 2018-03-04T23:26:20.628206+00:00)

I've updated the Hy documentation to use hylang in its code blocks, and we've found a few places where it fails to lex valid Hy code. Most of the problems revolve around REPL output, which is understandable (trying to figure out what's the best practice here), but there are a few snippets that fail.

Consider the following snippet:

=> (deftag ↻ [code]
...  (setv op (last code) params (list (butlast code)))
...  `(~op ~@params))
=> #↻(1 2 3 +)
6

When pushing this through pymentize, I hit this:

*****************************************************************
An unhandled exception occurred while highlighting.
Please report the whole traceback to the issue tracker at
<https://bitbucket.org/birkenfeld/pygments-main/issues>.
*****************************************************************

=> (deftag Traceback (most recent call last):
  File "/home/simon/.virtualenvs/hy-eeIqtnUk/bin/pygmentize", line 11, in <module>
    load_entry_point('Pygments', 'console_scripts', 'pygmentize')()
  File "/home/simon/src/pygments-main/pygments/cmdline.py", line 548, in main
    return main_inner(popts, args, usage)
  File "/home/simon/src/pygments-main/pygments/cmdline.py", line 512, in main_inner
    highlight(code, lexer, fmter, outfile)
  File "/home/simon/src/pygments-main/pygments/__init__.py", line 85, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/home/simon/src/pygments-main/pygments/__init__.py", line 67, in format
    formatter.format(tokens, outfile)
  File "/home/simon/src/pygments-main/pygments/formatters/terminal.py", line 101, in format
    return Formatter.format(self, tokensource, outfile)
  File "/home/simon/src/pygments-main/pygments/formatter.py", line 95, in format
    return self.format_unencoded(tokensource, outfile)
  File "/home/simon/src/pygments-main/pygments/formatters/terminal.py", line 121, in format_unencoded
    for ttype, value in tokensource:
  File "/home/simon/src/pygments-main/pygments/filter.py", line 20, in _apply
    for token in filter_.filter(lexer, stream):
  File "/home/simon/src/pygments-main/pygments/filters/__init__.py", line 196, in filter
    raise self.exception(value)
pygments.filters.ErrorToken: ↻

Link to Hy issue: hylang/hy#1522

@Anteru Anteru added T-bug type: a bug X-imported imported from Bitbucket trivial labels Aug 31, 2019
@Anteru Anteru added good first issue Good for newcomers and removed trivial labels Nov 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers T-bug type: a bug X-imported imported from Bitbucket
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant