NLP BasicsLesson 2
Lesson 27 min
Tokenisation
Breaking text into the pieces a model actually processes.
What you will learn
- ✓Why text must be tokenised
- ✓Sub-word tokenisation (BPE)
- ✓How tokens affect cost and limits
Explanation
Before any NLP model can work, text is split into tokens. Splitting on whole words struggles with rare words and typos, so modern models use sub-word tokenisation like BPE (Byte-Pair Encoding).
BPE breaks rare words into common pieces — 'tokenisation' might become 'token' + 'isation' — so the model handles anything, even new words.
Tokens are also the unit of cost and context limits in LLM APIs.
Real-world use
Emoji and rare names still 'work' in chatbots because sub-word tokenisers break them into known pieces.
Common mistakes
- • Assuming one token equals one word — sub-word tokens are usually smaller.
Practice
Guess how 'unbelievably' might be split into sub-word tokens.
Knowledge check
0/1 answered1. Why use sub-word tokenisation (BPE)?
Answer all questions to check.