Data Collection

Retrieval-Augmented Generation (RAG) is how you get an LLM to answer questions using your knowledge, not just whatever it happened to memorize during training. Instead of retraining the model, you retrieve relevant pieces of your own data at question-time and hand them to the model as extra context. This course builds a complete, working (if simplified) RAG pipeline in 8 steps, this lesson covers the first: gathering the raw knowledge.

Why RAG at all?

An LLM's knowledge is frozen at training time and limited to its context window, it can't answer questions about your company's private docs, yesterday's support tickets, or a codebase it's never seen. RAG fixes this by retrieving the relevant pieces of your own data and inserting them into the prompt, so the model answers grounded in real, current, specific information instead of guessing from what it half-remembers.

Gathering the raw knowledge

Real sources are typically PDFs, docs, APIs, websites, and databases, a mix of structured and unstructured data. For this course, each "document" is just a Python string with a bit of metadata, so every lesson's code runs directly in your browser with no external files needed:

raw_documents = [
    {
        'source': 'onboarding.md',
        'text': 'Kodstigen is a learning platform for programming courses. New students should start with the path that matches their goals, for example Python for general programming or DevOps for infrastructure.',
    },
    {
        'source': 'faq.md',
        'text': 'To reset your password, go to Settings and click Forgot Password. Password reset emails are sent from no-reply@codeforge.dev and expire after one hour.',
    },
    {
        'source': 'billing.md',
        'text': 'Kodstigen is currently free for all students. Instructors can submit courses and challenges for review, once approved they appear in the public catalog.',
    },
]

for doc in raw_documents:
    print(doc['source'], '-', len(doc['text']), 'characters')

Structured vs. unstructured

• Unstructured data (the kind this course focuses on) is free-form text: markdown docs, articles, chat transcripts. There's no fixed schema, just prose.
• Structured data (rows in a database, JSON API responses) has a predictable shape, you'd typically convert it into descriptive text first (e.g. "Order #4821, shipped 2026-01-03, total $42.50") before it can flow through the same pipeline as unstructured text.