Hugging Face
Models
Datasets
Spaces
Buckets
new
Docs
Enterprise
Pricing
Log In
Sign Up
toksuite
/
tokenmonster-englishcode-32000-consistent-v1
like
0
Follow
TokSuite
7
Text Generation
Transformers
Safetensors
5 languages
llama
toksuite
tokenization
tokenmonster
subword
research
robustness
text-generation-inference
arxiv:
2512.20757
License:
mit
Model card
Files
Files and versions
xet
Community
Deploy
Use this model
main
tokenmonster-englishcode-32000-consistent-v1
2.83 GB
Ctrl+K
Ctrl+K
2 contributors
History:
16 commits
gsaltintas
Update README.md
e504c56
verified
4 months ago
.gitattributes
Safe
1.64 kB
Upload model-performance-comparison.png
4 months ago
README.md
Safe
8.19 kB
Update README.md
4 months ago
config.json
Safe
609 Bytes
Upload config
8 months ago
generation_config.json
Safe
150 Bytes
Upload model files
8 months ago
model-performance-comparison.png
Safe
279 kB
xet
Upload model-performance-comparison.png
4 months ago
model.safetensors
Safe
2.83 GB
xet
Upload model files
8 months ago
tokenizer.json
Safe
684 kB
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1_vocab.json - Upload model files
8 months ago
tokenizer_config.json
Safe
92 Bytes
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1_info.json - Upload model files
8 months ago
tokenmonster--englishcode-32000-consistent-v1.yaml
Safe
84 Bytes
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1.yaml - Upload model files
8 months ago
tokenmonster--englishcode-32000-consistent-v1_super_mapping.json
Safe
528 kB
Upload tokenizer file tokenmonster--englishcode-32000-consistent-v1_super_mapping.json - Upload model files
8 months ago
toksuite-logo.png
Safe
1 MB
xet
Upload toksuite-logo.png
4 months ago
