You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I was working on implementing the XLNet language model in Julia. It uses the sentence-piece as the tokenizer.
I need the vocabulary for the tokenizer, which is stored inside spiece.model file.
I did refer to this Issue #121, but it only tells about modification of the model.
Could you tell me, how I should obtain the vocabulary in a .vocab format?
The text was updated successfully, but these errors were encountered:
SambhawDrag
changed the title
How to get the XLNet vocabulary from spiec.model file and store it to a .vocab file?
How to get the XLNet vocabulary from spiece.model file and store it to a .vocab file?
Jul 2, 2021
Hi, I was working on implementing the XLNet language model in Julia. It uses the sentence-piece as the tokenizer.
I need the vocabulary for the tokenizer, which is stored inside spiece.model file.
I did refer to this Issue #121, but it only tells about modification of the model.
Could you tell me, how I should obtain the vocabulary in a .vocab format?
The text was updated successfully, but these errors were encountered: