Every chunk now has a vector, but a list of vectors floating around isn't useful on its own, it needs somewhere to live alongside the original text it came from, ready to be searched. This lesson builds a minimal vector store.
Production tools
At real scale (millions of chunks), you'd reach for a dedicated vector database like Pinecone, FAISS, or Weaviate, tools built specifically to store embeddings and run similarity search fast, even across huge datasets, with persistence to disk and metadata filtering built in.
A minimal in-memory version
For this course's scale (a handful of chunks), a plain Python list holding every chunk's text, vector, and metadata together is enough to demonstrate the same core idea:
Pythonshares state with other Python blocks
import re
from collections import Counter
deftoy_embed(text):
words = re.findall(r"[a-z']+", text.lower())return Counter(words)classVectorStore:def__init__(self):
self.entries =[]# each entry: {'text': ..., 'vector': ..., 'metadata': ...}defadd(self, text, vector, metadata=None):
self.entries.append({'text': text,'vector': vector,'metadata': metadata or{},})def__len__(self):returnlen(self.entries)
store = VectorStore()
store.add('Password reset emails expire after one hour.',
toy_embed('Password reset emails expire after one hour.'),
metadata={'source':'faq.md'},)print(f'Store now has {len(store)} entr{"y"iflen(store)==1else"ies"}.')
Storing every chunk from every document
Putting the last three lessons together, every document gets chunked, every chunk gets embedded, and every (chunk, vector) pair gets added to the store along with which document it came from:
Pythonshares state with other Python blocks
defchunk_text(text, chunk_size=12, overlap=3):
words = text.split()
chunks =[]
start =0while start <len(words):
end = start + chunk_size
chunks.append(' '.join(words[start:end]))
start += chunk_size - overlap
return chunks
raw_documents =[{'source':'onboarding.md','text':'Kodstigen is a learning platform for programming courses. New students should start with the path that matches their goals.'},{'source':'faq.md','text':'To reset your password, go to Settings and click Forgot Password. Password reset emails expire after one hour.'},{'source':'billing.md','text':'Kodstigen is currently free for all students. Instructors can submit courses and challenges for review.'},]
store = VectorStore()for doc in raw_documents:for chunk in chunk_text(doc['text']):
store.add(chunk, toy_embed(chunk), metadata={'source': doc['source']})print(f'Indexed {len(store)} chunks from {len(raw_documents)} documents.')
Keeping metadata (like which file a chunk came from) alongside the vector is what lets a real RAG system cite its sources later, "according to faq.md, ...", rather than just returning an answer with no way to trace where it came from.
TIP
The core idea a vector database adds beyond "a list with vectors in it" is fast similarity search at scale, techniques like approximate nearest-neighbor indexing that make searching millions of vectors take milliseconds instead of scanning every single one. This course's simple linear scan (next lesson) is correct but wouldn't stay fast forever, which is exactly why those tools exist.
📝 Vector Storage Quiz
Passing score: 70%
1.Why does the VectorStore keep metadata (like the source filename) alongside each vector?
2.Pinecone, FAISS, and Weaviate are all examples of dedicated vector database tools.
3.A key advantage a real vector database adds over a plain list is fast similarity search at ____.
