The row count was mostly repetition
The distillation dataset began with 104,272 captured records. After normalization, exact deduplication, near-duplicate clustering, and behavior labeling, 868 meaningfully distinct target behaviors remained. The original number mostly measured how often the collection pipeline saw the same work.
| Stage | Count |
|---|---|
| Captured records | 104,272 |
| Distinct target behaviors | 868 |
The reduction is not automatically a problem. Repetition can stabilize a frequent behavior. The mistake is believing that 104,272 examples provide 104,272 worth of coverage. Repeated examples of the same success are not a substitute for new failure modes, domains, or reasoning patterns.
Measure diversity by the decisions it covers
Useful diversity includes different intents, domains, output schemas, ambiguity patterns, negative examples, and escalation cases. Embedding distance can flag similarity. Labels tied to actual routing or planning decisions make the metric something a review can argue about.
One class retained a near-duplication score of 0.698 while most others approached zero. Keeping that outlier visible was more informative than a single average, which would have hidden the weak slice. Averages are how duplicated data looks healthy.
Bad targets can collapse the model while loss falls
Short, repetitive targets taught the fine-tuned model to emit terse fragments and malformed structure. Training loss kept falling because the model was learning the supplied distribution. The objective was working. The target design was wrong.
Before training, inspect target-length distributions, schema validity, rare classes, and a sample of near-duplicate clusters. A dataset can be technically clean and still impoverished in behavior. If loss is the only number you watch, you can ship that poverty with confidence.
Small lifts require small claims
Early evaluations used roughly sixty examples per slice and produced modest, noisy improvements. Those results support continued experimentation. They do not support a claim that the distilled model is ready to replace the teacher. Confidence intervals and repeated runs should appear beside the lift, or the lift will grow in the retelling.
The next dataset improvement should target observed errors rather than add undifferentiated rows. Record the failure, assign it to a missing or weak behavior cluster, add counterexamples, and re-run an untouched holdout. More rows of what you already have will not fix what you never collected.