← SELECTED WORK
Python / ChromaDB / sentence-transformersSummer 2026

The Unofficial Guide

Find the failure before fixing the answer.

What I built.
How I evaluated it.

  • Built a retrieval-augmented generation pipeline over a custom document corpus, selecting chunk size and overlap from observed document structure rather than defaults and documenting the reasoning behind each value.
  • Compared fixed-size, semantic, and recursive chunking strategies on retrieval quality for multi-section queries, tracing every retrieved chunk back to its source file and producing function.
  • Ran a before-and-after evaluation assigning verdicts to generated answers, diagnosing failures as retrieval versus generation problems, and validating one targeted fix against the same query set.
  • Documented remaining failure modes and their causes instead of reporting only the improved runs.

The bullets above are reported in the full source resume ↗. Repository findings and any differences are identified separately below.

How it fits together.

Ingestion preserves each Reddit comment as a source-labeled chunk and splits long comments into overlapping sentence-aware passages. ChromaDB attaches the same MiniLM embedding function to both indexing and querying, using cosine search over a local persistent collection. Generation discards distant chunks, refuses without an LLM call when none remain, and builds the source list from metadata rather than model-written citations.

sentence-transformersChromaDBGroqpython-dotenvGradioBM25

Choices with a reason.

  1. 01

    Comment-boundary chunks with a long-comment split

    Alternative considered: Blind fixed-size windows or merging unrelated comments

    A comment is the document’s natural unit of meaning; preserving it avoids splitting one opinion or diluting several opinions into a single embedding.

    Source ↗
  2. 02

    Small local MiniLM embeddings

    Alternative considered: Larger hosted or multilingual embedding models

    Short English comments suit inexpensive local embeddings; the documented trade-off is weaker recall on domain slang, long passages and other languages.

    Source ↗
  3. 03

    Keep the cosine-distance threshold at 0.6

    Alternative considered: Tighten the threshold to roughly 0.5

    The documented choice avoids masking the known retrieval failure, while the empty-context gate still prevents an unnecessary generation call.

    Source ↗

The result and the method.

Documented ingestion change: split comments longer than 250 words into roughly 200-word passages with 40-word overlap.

Largest chunk 864247 words. The resulting corpus contains 325 chunks across 15 files. This measures chunk size, not answer accuracy.

Source ↗

Five-question evaluation harness; sources and cosine distances are printed beside the generated answer for human review.

The Stat 134 question refused despite relevant material being in the corpus: top-four retrieval favored unrelated courses and a contentless comment. Human retrieval-quality and answer-accuracy columns are still unfilled, so no scored before/after gain is claimed.

Source ↗

What still needs work.

Excerpts from the repository’s own notes. These limits belong beside the results.

  • the answer is in the corpus but retrieval surfaced a contentless one liner
    Source ↗
  • English-only with a small (~256-token) input window
    Source ↗

Read the code.

PythonTest count not established

No tracked Python test_ functions found; evaluation scripts and notebooks are described separately. Source ↗

  1. ingest.py

    Inspect comment delimiters, long-comment splitting and source metadata.

  2. generator.py

    Follow the distance gate and programmatic source attribution.

  3. evaluate.py

    Read the five fixed questions and the human-review boundary.

Open repository ↗
  1. 01

    Establish a baseline.

  2. 02

    Make a targeted change.

  3. 03

    Re-measure on the same set.

  4. 04

    Document what still fails.

DockWise AI