Skip to project detailsSkip to navigation
← SELECTED WORK
Python / PyTorch / scikit-learn / JupyterSummer 2026

TakeMeter

Better labels. More meaningful evaluation.

View repository ↗AI Engineer resume ↗

What I built.
How I evaluated it.

  • Designed a label taxonomy for community post classification with written definitions, worked examples, and an explicit decision rule for the hardest boundary between adjacent labels.
  • Built and hand-labeled the training dataset, then trained a classifier locally and benchmarked it against a zero-shot baseline rather than reporting accuracy in isolation.
  • Evaluated with a per-class confusion matrix and an inter-annotator agreement report, using disagreement between labelers to expose where the taxonomy itself was ambiguous.
  • Diagnosed misclassifications to specific taxonomy and data issues, applied one targeted improvement, and re-measured on the same held-out set to confirm the gain was real.

The bullets above are reported in the full source resume ↗. Repository findings and any differences are identified separately below.

01 / SYSTEM ARCHITECTURE

How it fits together.

Local scripts clean the Reddit dataset, validate its four-label taxonomy and run the Groq baseline. The notebook fine-tunes DistilBERT on a stratified 305 / 66 / 66 train-validation-test split using a Colab T4. Evaluation compares the small classifier with the zero-shot baseline and reports per-class errors, so minority-class collapse remains visible.

README.md ↗README.md ↗
FROM THE DEPENDENCY FILES
pandasscikit-learnGroqpython-dotenv
requirements-local.txt ↗
02 / ENGINEERING DECISIONS

Choices with a reason.

  1. 01

    Label the author’s primary purpose

    Alternative considered: Assign a class from ticker mentions or surface vocabulary

    The boundary rule distinguishes evidence-backed trade reasoning, emotional group action and humor, even when all mention the same stock.

    Source ↗
  2. 02

    Inverse-frequency class-weighted loss

    Alternative considered: Unweighted cross-entropy

    The initial model ignored the two smaller classes; weights derived from training data increase their contribution without using test labels.

    Source ↗
  3. 03

    Select the best checkpoint by macro-F1

    Alternative considered: Select by overall accuracy

    Each class must matter in an imbalanced four-way task, rather than letting majority predictions dominate model selection.

    Source ↗
03 / EVALUATION RECORD

The result and the method.

Documented first run versus the revised DistilBERT run on the locked 66-example test set; loss weighting, epoch count and checkpoint selection changed together.

Accuracy 0.561 → 0.742; macro-F1 0.35 → 0.70. Meme F1 rose from 0.00 to 0.53 and community-meta F1 from 0.00 to 0.67. This is a combined revision, not an isolated weighting ablation.

Source ↗

Zero-shot Groq comparison with an incomplete baseline run.

Groq accuracy 0.847 and macro-F1 0.82 cover 59 / 66 examples because seven requests failed at the daily token limit. DistilBERT accuracy 0.742 and macro-F1 0.70 cover all 66; the denominators are not matched.

Source ↗
04 / DOCUMENTED LIMITS

What still needs work.

Excerpts from the repository’s own notes. These limits belong beside the results.

05 / EXPLORE THE IMPLEMENTATION

Read the code.

PythonTest count not established

No tracked Python test_ functions found; evaluation scripts and notebooks are described separately. Source ↗

  1. planning.md ↗

    Read the four-label taxonomy and difficult boundary examples first.

  2. AI201 Project 3 Takemeter Starter.ipynb ↗

    Inspect the Colab training run, class weights and stored evaluation output.

  3. scripts/validate_dataset.py ↗

    Check schema and label validation before reproducing the split.

Open repository ↗
THE EVALUATION PRACTICE
  1. 01

    Establish a baseline.

  2. 02

    Make a targeted change.

  3. 03

    Re-measure on the same set.

  4. 04

    Document what still fails.

NEXT PROJECT

The Unofficial Guide ↗