Generative AILesson 2
Lesson 28 min
Tokens & Embeddings
How models turn words into numbers they can compute on.
What you will learn
- ✓What a token is
- ✓What embeddings represent
- ✓Why this matters for cost and search
Explanation
Models do not see words — they see numbers. Text is first split into tokens (roughly word-pieces; 'studying' might be 'study' + 'ing').
Each token is mapped to an embedding: a list of numbers (a vector) that captures meaning, so similar words sit close together in that number-space.
Tokens matter practically: APIs charge per token, and context limits are measured in tokens. Embeddings power semantic search and RAG.
Real-world use
Search that understands meaning ('cheap laptop' matching 'budget notebook') uses embeddings, not keyword matching.
Common mistakes
- • Confusing tokens with words — a token is usually smaller than a whole word.
Practice
Estimate how many tokens a 100-word paragraph is (hint: ~1.3 tokens per word).
Knowledge check
0/2 answered1. An embedding is...
2. Why do tokens matter practically?
Answer all questions to check.