[THUDM/ChatGLM-6B][BUG/Help] ice_text.model词表长度与config里设置不一致

ice_text.model词表长度130344

>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
>>> len(tokenizer.get_vocab())
130344

config:

"vocab_size": 130528

模型参数:

transformer.word_embeddings.embedding_table torch.Size([130528, 4096]) torch.float16
lm_head.weight torch.Size([130528, 4096]) torch.float16

词表长度不一致导致有时会生成词表外的词，然后索引越界退出

词表大小与config、模型参数一致

>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True)
>>> len(tokenizer.get_vocab())
130344

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :