zlacker

You could, but you wouldn't when those keywords can all change in equivalent contexts.

>>muyuu+(OP)
What do you mean?

`public` might have a token by itself, even though you can have `pub` occurring in other contexts, too.

>>muyuu+(OP)
The BPE or wordpiece tokenization algorithm will greedily take the longest valid token prefix. So if your text starts with “public static void main” it will try to find the longest token which matches that prefix. Even if “public” is a token, it will prefer to tokenize “public static” together.

>>eru+W
I meant that it wouldn't be efficient to agglomerate tokens in that way and that's why the system won't do it

>>janals+1x2
yes, but then you have both alternatives as tokens, which nullifies GP's argument