why doesn't \p{Emoji}+ match all emoji?

[phungleson]

What version of regex are you using?

1.7.1

Describe the bug at a high level.

I wrote a regex to capture single emoji like this ^\p{Emoji}+$

It captures most of the emojis (in apple eco) but some aren't, some fails such as:

Activities: 🕹️...
Flags: 🏴󠁧󠁢󠁥󠁮󠁧󠁿🏴‍☠️ ...
Travel and places: 🏔️ ...
Symbols: ⬆️☯️ ...
and many more...

I can list them all if it is helpful.

What are the steps to reproduce the behavior?

Explained.

What is the actual behavior?

Explained.

What is the expected behavior?

Ideally it should try to capture as many emojis as possible.

Or maybe it is impossible?

[BurntSushi]

In the future, when filing issues, it would be helpful to provide code that I can run. That way, I can be sure I know what it is you're talking about.

With that said, my guess here is that you're conflating \p{Emoji} with the concept of "emoji." \p{Emoji} is a Unicode property, and it is one component of the Unicode emoji technical standard. Actually detecting and extracting emoji requires implementing that UTS, which will use \p{Emoji} for sure, but it is nowhere close to sufficient to implement. Appendix A of UTS#51 outlines the various properties related to emoji, and Emoji is merely one of them. Presumably one would need to use all of them to implement proper emoji extraction. (I note that I have never myself implemented UTS#51, and have not read the standard in depth.)

It is beyond the scope of this crate specifically to provide an implementation of UTS#51.

Now if you have a program that executes a regex using a Unicode property such as \p{Emoji} and it is failing to match a codepoint that you believe is in Emoji, then I'd ask that you open a new issue with at least the following:

• A real program that I can run, including a Cargo.toml, that reproduces your problem.
• Some sort of citation or reference to the relevant Unicode data demonstrating that there is some kind of mismatch between the property you're using and its implementation in the regex crate.

For example, just checking your first "joystick" example, we can see that it is behaving as expected. We need to do two things for that. First is write a program to look at actual behavior, and second we need to check whether its output is consistent with Unicode's definition of the relevant properties. So here's the program:

use regex::Regex;

fn main() {
    let s = "🕹️";
    for cp in s.chars() {
        println!("{:X}", u32::from(cp));
    }
    
    let re = Regex::new(r"^\p{Emoji}+$").unwrap();
    println!("all in emoji? {:?}", re.is_match(s));
    
    let re = Regex::new(r"^[\p{Emoji}\p{Emoji_Component}]+$").unwrap();
    println!("all in emoji or emoji_component? {:?}", re.is_match(s));
}

Output:

1F579
FE0F
all in emoji? false
all in emoji or emoji_component? true

I had a suspicion that your emoji were actually composed of multiple codepoints, and that it was likely that one of them wasn't in Emoji but perhaps in another property. Indeed, that is the case here. But how do we know for certain and that the regex crate doesn't just have a bug somewhere? For that---the second part of our process---you have to download the Unicode Character Database (UCD), find the property definition and confirm its membership:

$ mkdir ucd15.0.0
$ cd ucd15.0.0
$ curl -LO https://www.unicode.org/Public/zipped/15.0.0/UCD.zip
$ unzip UCD.zip
$ rg 1F579 emoji/emoji-data.txt
249:1F573..1F579  ; Emoji                # E0.7   [7] (🕳️..🕹️)    hole..joystick
1095:1F573..1F579  ; Extended_Pictographic# E0.7   [7] (🕳️..🕹️)    hole..joystick
$ rg FE0F emoji/emoji-data.txt
794:FE0F          ; Emoji_Component      # E0.0   [1] ()        VARIATION SELECTOR-16

(I got somewhat lucky that the codepoints used here appear explicitly in the file. They may not, as the file is a sequence of ranges. So lookup might take a little more work than a simple grep query.)

As we can see, U+FE0F is in Emoji_Component but not Emoji. Thus, the behavior of the program above is correct, and at least for your first example, it should not match ^\p{Emoji}+$.

[phungleson]

Hey @BurntSushi thanks a lot for the explanation, I guess when I open the issue I was thinking of something much simpler.

I guess I need to do quite a bit of read up, and play around with your suggestion and let you know if I found something.

Cheers,