Hugging Face
Models
Datasets
Spaces
Community
Docs
Enterprise
Pricing
Log In
Sign Up
ggunio
/
intelligent-tokenizer-v6
like
0
PyTorch
flores200
11 languages
intelligent_tokenizer
tokenization
byte-level
neural-tokenizer
pattern-learning
vocabulary-free
Eval Results
License:
mit
Model card
Files
Files and versions
xet
Community
main
intelligent-tokenizer-v6
425 MB
1 contributor
History:
19 commits
ggunio
Update README.md
a1634b9
verified
4 months ago
core
Upload core/unified_model.py with huggingface_hub
4 months ago
src
Upload src/core/byte_tokenizer_v6.py with huggingface_hub
4 months ago
.gitattributes
1.52 kB
initial commit
4 months ago
README.md
2.87 kB
Update README.md
4 months ago
app.py
13.9 kB
Upload app.py with huggingface_hub
4 months ago
config.json
394 Bytes
Upload config.json with huggingface_hub
4 months ago
demo_poc.py
9.88 kB
Upload demo_poc.py with huggingface_hub
4 months ago
inference.py
9.67 kB
Upload inference.py with huggingface_hub
4 months ago
pytorch_model.bin
425 MB
xet
Upload pytorch_model.bin with huggingface_hub
4 months ago
requirements.txt
34 Bytes
Upload requirements.txt with huggingface_hub
4 months ago
