Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Which encoding is this using? #5

Closed
ricardomatias opened this issue May 2, 2023 · 3 comments · Fixed by #8
Closed

Which encoding is this using? #5

ricardomatias opened this issue May 2, 2023 · 3 comments · Fixed by #8
Labels
released on @beta released

Comments

@ricardomatias
Copy link

Thanks for your effort with this fork, it's so much faster. It's not clear to me which encoding is being used, is it cl100k_base or r50k_base?

According to this link, gpt-3-encoder is using r50k_base, but I don't see any mention of it in the library.
@niieani
Copy link
Owner

niieani commented May 21, 2023

This is using r50k_base, but I'm working making it customizable in v2.0.

@niieani niieani mentioned this issue May 21, 2023
Closed
@niieani niieani mentioned this issue May 23, 2023
Merged
niieani added a commit that referenced this issue May 23, 2023
@niieani
feat: complete rewrite to support different models
eedd944 
BREAKING CHANGE: default encoder is now GPT3.5 / GPT4

fixes #5
fixes #6
@github-actions
Copy link

🎉 This issue has been resolved in version 2.0.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions
Copy link

🎉 This issue has been resolved in version 2.0.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
released on @beta released
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: complete rewrite to support different models niieani/gpt-tokenizer
2 participants
@niieani @ricardomatias