zlacker

[parent] [thread] 0 comments
1. agnish+(OP)[view] [source] 2026-01-01 02:20:11
> LLMs tokenize English words efficiently. Symbols like {, }, === fragment into multiple tokens. Words like "plus", "minus", "if" are single tokens.

The insight seems flawed. I think LLMs are just as capable of understanding these symbols as tokens as they are English words. I am not convinced that this is a better idea than writing code with a ton of comments

[go to top]