Data Chunking

A whole document is usually too big and too broad to embed and search effectively, this lesson splits each document into small, focused pieces, called chunks.

Why chunk at all?

Embedding an entire multi-page document into a single vector loses precision, the vector ends up representing an average of many different topics. Splitting into smaller chunks means each vector represents one focused idea, which makes retrieval far more accurate: a query about "password reset" should match a small chunk about password reset, not an entire FAQ document that also happens to mention pricing and onboarding.

A simple word-based chunker

Production systems typically aim for chunks of around 200-500 tokens. This course uses much smaller chunks (a handful of words) so you can see the effect clearly on short example text:

def chunk_text(text, chunk_size=12, overlap=3):
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunk = ' '.join(words[start:end])
        chunks.append(chunk)
        start += chunk_size - overlap
    return chunks

text = (
    'Kodstigen is a learning platform for programming courses. '
    'New students should start with the path that matches their goals, '
    'for example Python for general programming or DevOps for infrastructure.'
)

for i, chunk in enumerate(chunk_text(text)):
    print(f'Chunk {i}: {chunk!r}')

Why the overlap?

Without overlap, a sentence that happens to straddle two chunk boundaries gets cut in half, one chunk ends mid-thought and the next picks up without its earlier context. overlap=3 repeats the last 3 words of one chunk at the start of the next, so a query matching that boundary sentence still finds enough surrounding context in at least one of the two chunks.

def test_boundary():
    boundary_text = 'one two three four five six seven eight nine ten eleven twelve thirteen fourteen'
    chunks = chunk_text(boundary_text, chunk_size=6, overlap=2)
    for chunk in chunks:
        print(chunk)

test_boundary()

Notice each chunk's last two words reappear as the next chunk's first two words, that repetition is deliberate.