GUIDE · AIunipedia Community
Data leakage occurs when model development uses information that would not be available at prediction time. The resulting score can look excellent while the deployed system fails on genuinely new cases.
A classic example is learning a preprocessing transformation from the complete dataset before evaluating a split. Another is allowing near-duplicate records from the same customer or document to appear on both sides of the evaluation boundary.
Leakage can make an impressive precision, recall or F1 score unreliable. The same separation principle matters when building an LLM evaluation set.
Audit the information path
For each feature and preprocessing step, ask when the value becomes available and which partition was used to learn it. Inspect identifiers, timestamps, duplicates and any label-derived fields.
Consider whether your split should respect time, customer, patient, document or organisation boundaries. A random row split is not automatically appropriate for every problem.
A small exercise
Sketch a model predicting whether a support enquiry will need escalation. Remove any feature produced only after the escalation decision. Then describe how you would evaluate the model on future enquiries rather than a convenient reshuffle of the past.
Keep the final test set separate from feature selection and repeated tuning. If leakage is discovered, revisit the experiment design instead of merely adjusting the reported number.
Discussion: What seemingly harmless field or preprocessing step has caused leakage in a project you can discuss publicly?
Reference: scikit-learn common pitfalls and data leakage.
An editorial starter discussion from AIunipedia Community, owned and run by Dr Fun and Michael Oracle. Share your own experience and help improve the explanation.
Continue exploring
Precision, Recall and F1: Choose the Right Metric · Build an LLM Evaluation Set You Can Trust
Browse: Machine Learning & Models · All learning paths
Tip: open a related discussion in a new tab to keep your place here. On a phone, touch and hold the link for your browser’s options.