Gmail RAG Pipeline
The problem
My inbox had become a useful but unsearchable archive. Investment newsletters, job threads, guitar deals, company research I’d saved for later. All there, but keyword search only returns what you already know to search for. And sending email content to an external API wasn’t an option. Too much context in there that I didn’t want leaving the machine.
The fix was local semantic search: BGE embeddings, ChromaDB, no external API calls.
What it does
You type a query (“job offer from last month”, “guitar deals”, “Sequoia portfolio companies”) and get back the most relevant email chunks with source metadata: sender, subject, date. Everything runs on the local machine.
Architecture
Ingestion pipeline
Five stages, each implemented as a LangChain Runnable:
Gmail fetch. Emails are pulled via the Gmail API (OAuth 2.0, read-only scope). The MIME tree is walked to extract plain text, falling back to BeautifulSoup HTML parsing if no plain-text part exists.
Cleaning. EmailReplyParser strips quoted reply chains before anything hits the vector store. Unsubscribe footers and signature markers are cut. Only the original message content gets indexed.
Chunking. Cleaned text is split into 1,500-character chunks with 200-character overlap using RecursiveCharacterTextSplitter. Each chunk is prefixed with the email’s headers (From, Subject, Date) so the retriever has source context when a chunk comes back in isolation.
Embedding. BGE-base-en-v1.5 (768-dimensional, L2-normalized). The same model carries over into the APD system that came out of this work.
Storage. Chunks go to ChromaDB with stable IDs:
{source_id}-{chunk_index}. Re-running on the same inbox doesn’t add duplicates.
Key design decisions
Reply stripping before embedding. Reply chains quote the original message under a different sender. Without stripping them, the same content shows up in multiple vectors and pulls similarity rankings toward repeated text instead of unique signal.
Header prepended to each chunk. A retrieved chunk rarely comes with the full email around it. Prepending sender, subject, and date to every chunk means results are readable without a separate metadata lookup.
Stable chunk IDs. IDs built from source message ID plus chunk index make the pipeline idempotent. Run it again on an updated inbox and only new emails get added.
What this became
This project established the core pattern that the Agentic Process Discovery system later extended. APD kept the same ingestion approach, BGE embeddings, and ChromaDB storage, then added WhatsApp as a second source, multi-stage query expansion, and an LLM synthesis step that turns retrieved messages into structured process narratives rather than just returning chunks.
Technologies:
Python LangChain ChromaDB BGE-base-en-v1.5 Gmail API BeautifulSoup Docker
Concepts / Algorithms:
RAG Vector embeddings Semantic search Text chunking Email parsing Embedding-based retrieval