Reading encoded emojis in HTML attributes

[mcantrell]

I'm using jsoup to parse images that link to emoji images. These images have the actual emoji characters encoded in the alt attributes:

<img alt="&#55357;&#56495;" src="https://acme.com/images/emojis/100_percent.png">

When I parse the attribute using jsoup, the resulting string is garbage:

    @Test
    void parseImageEmojiAttribute() {
        String html = "<div><img alt=\"&#55357;&#56495;\" src=\"https://acme.com/images/emojis/100_percent.png\"></div>";
        Document document = Jsoup.parse(html);
        Element img = document.select("img").get(0);
        assertEquals("\uD83D\uDCAF", img.attr("alt"));
    }

Is there a configuration setting that I'm missing which could help? Failing that, is there a way that I can get the raw value of the attribute so I can parse it myself? For instance, Apache commons text has a utility which can decode the value:

    @Test
    void parseHtmlEncodedEmoji() {
        String emoji = StringEscapeUtils.unescapeHtml4("��");
        assertEquals("\uD83D\uDCAF", emoji);
    }

[mcantrell]

The provided Parser.unescapeEntities util has similar issues

    @Test
    void parseHtmlEncodedEmoji() {
        String emoji = Parser.unescapeEntities("��", false);
        assertNotEquals("\uFFFD\uFFFD", emoji, "UTF-8 replacement characters");
        assertEquals("\uD83D\uDCAF", emoji, "Invalid emoji");
    }

[jhy]

Thanks for the clear report, fixed!

This was failing to decode and returning the replacement character due to an explicit range check it was hitting: (>= 0xD800 && <= 0xDFFF). That check has been there since 8749726 (12 years ago!) when I reimplemented to follow the HTML5 spec. But I can't see why that range was explicitly excluded - perhaps it was previously reserved in Unicode and then opened later.

The single-point version (💯) worked OK originally.

Have fixed by removing that range as it is now valid and not excluded in the current spec.

I note that Chrome seemingly treats this as we previously did; �� in Chrome renders as two replacement characters. The single point renders correctly.

[jhy]

Fixing commit is: 954c46a