tokenize on sentence, not all document

[ngawangtrinley]

Description

Tokenizing is done on all the text each time. This makes the editor very slow. The waiting time grows exponentially. The maximum typing latency should be around ##, now it's much bigger.

The tokenizer is triggered by the syllable marker '་', the phrase marker '།'. and return char. At the moment the tokenizer processes all the document in one go. It should be tokenizing one phrase at a time. A phrase is a string of char separated by a '།'.

How to reproduce

1. Paste the text from here and try to edit
2. paste the text sentence by sentence and check the waiting time

Proposed solution

• find a way to tokenize and highlight in the background (is it even possible with python?)
• find phrase span, resegment the phrase at each trigger char
• get n syllable context around the cursor, segment the context at each trigger