this post was submitted on 05 Oct 2023
11 points (78.9% liked)
ChatGPT
8929 readers
1 users here now
Unofficial ChatGPT community to discuss anything ChatGPT
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
This is really helpful, thanks. I followed up until the part about token IDs.
How and why is token text converted to an ID? Is the ID for a specific word always the same?
What are tokens and how to count them?