Skip to content

master-82c4ac7
Compare
Choose a tag to compare
@github-actions github-actions released this 08 Jun 11:40
master-82c4ac7
82c4ac7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode. 
Add support for the world tokenizer (#86)

* Add support for the world tokenizer

* Move tokenizer logic to rwkv_tokenizer.py

* Added test for the tokenizer
Assets 7