[BUG] Wrong way to get list of supported encodings

[nickspring]

Describe the bug
It looks like you get list of encodings from encoding.aliases module.
aliases(as one would/should expect) contains several cases where different keys are mapped to the same value e.g. 1252 and windows_1252 are both mapped to cp1252. You could save time if instead of aliases.keys() you use set(aliases.values()).

BUT THERE'S A WORSE PROBLEM: aliases don't contain codecs that don't have aliases (like cp856, cp874, cp875, cp737, and koi8_u).

To Reproduce
List of encodings, supported by this library:
https://charset-normalizer.readthedocs.io/en/latest/user/support.html#supported-encodings
It's declared that the library supports all encodings which are supported by Python.
But there is no, for example, KOI8-U.

Expected behavior
But KOI8-U is supported (but just it doesn't have an alias):
https://docs.python.org/3.11/library/codecs.html#standard-encodings

[Ousret]

Hello,

It looks like you get list of encodings from encoding.aliases module.

Yes.

You could save time if instead of aliases.keys() you use set(aliases.values()).

Already is the case.

BUT THERE'S A WORSE PROBLEM: aliases don't contain codecs that don't have aliases (like cp856, cp874, cp875, cp737, and koi8_u).

IT IS TRUE WE MISSED 9 ENCODINGS! 💯 (Please avoid caps lock in the future)

Otherwise, it is a good catch. They are still very kind rare encoding (depending on various factors), but still, valuable. Will add a patch to include them.

Regards,

[nickspring]

Sorry, it was copied from Stackoverflow :) I didn't realize that there is an information about set(aliases.values()) and Caps lock is from the original post. Actually I'm working on porting this library to other language and go through all sources.

[Ousret]

Actually I'm working on porting this library to other language and go through all sources.

Good! I am interested in any progress or proof of concept. Is it rust? golang? c?

[nickspring]

It's Rust. At the time I have ported cd/md/utils/models. So api and cli are waiting for implementation (ETA 1-3 weeks).

[nickspring]

It's Rust. At the time I have ported cd/md/utils/models. So api and cli are waiting for implementation (ETA 1-3 weeks).

Sorry for offtopic, I can't find any other possibility to contact you. https://github.com/nickspring/charset-normalizer-rs there is a Rust version of library. I tried to port with maximum compatibility. I believe I did everything correctly with copyrights and licences :) I would be grateful if you add link to th Rust version.

[Ousret]

Good work. I briefly tested it and it seems fine.
The immediate remark is that it did not build on alpine due to some incompatible dependencies.

I can mention it, of course.

[nickspring]

Good work. I briefly tested it and it seems fine. The immediate remark is that it did not build on alpine due to some incompatible dependencies.

I can mention it, of course.

apk add musl-dev should help.

Whole command:

docker run --rm -it -v .:/app -w /app --entrypoint sh rust:1.71.0-alpine -c 'apk add musl-dev && cargo run --bin performance --features performance --release'