Hugging Face's logo

raptorkwok
/
pyctokenizer

Translation
Safetensors
Chinese
bart
Model card Files
xet
 Community

Configuration Parsing Warning:Config file tokenizer_config.json cannot be fetched (too big)

Cantonese Tokenizer

This is a Cantonese sentence tokenizer based on BART Chinese. It can be used along with our CCPC Parallel Corpus dataset.

  • Token Count: 161,278 (including 110,007 multi-character Chinese vocabularies)

Usage 

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("raptorkwok/pyctokenizer")
print(tokenizer.tokenize("ζˆ‘ε“‹εŽ»ε’—ε°–ζ²™ε’€η‡ι†«η”Ÿε‘€οΌ"))
# Output: ['ζˆ‘ε“‹', '去', 'ε’—', 'ε°–ζ²™ε’€', 'η‡ι†«η”Ÿ', 'ε‘€', '！']
Downloads last month
9
Safetensors
Model size
0.2B params
Tensor type
F32
Β·
Inference Providers NEW
Translation
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for raptorkwok/pyctokenizer

Base model

OpenMOSS-Team/bart-base-chinese
Finetuned
(9)
this model

Dataset used to train raptorkwok/pyctokenizer