Fred Hohman
/ Ph.D. Candidate at GT

Understanding and Visualizing Data Iteration in Machine Learning

Fred Hohman, Kanit Wongsuphasawat, Mary Beth Kery, Kayur Patel

The Chameleon interface integrates multiple coordinated views to help practitioners explore their evolving datasets. The Data Version Timeline (A) lists data versions across the top of the interface and allows users to select a primary and a secondary version to visualize below. The Sidebar (B) shows version summaries and multiples views that visualize changing instance predictions. Practitioners can use the sibebar views to filter data in the Feature View (C), which visualizes each feature of a dataset as a histogram with both selected data versions, faceted by performance and the train/testing split. Dataset and feature names are redacted for anonymity.

Abstract

Successful machine learning (ML) applications require iterations on both modeling and the underlying data. While prior visualization tools for ML primarily focus on modeling, our interviews with 23 ML practitioners reveal that they improve model performance frequently by iterating on their data (e.g., collecting new data, adding labels) rather than their models. We also identify common types of data iterations and associated analysis tasks and challenges. To help attribute data iterations to model performance, we design a collection of interactive visualizations and integrate them into a prototype, Chameleon, that lets users compare data features, training/testing splits, and performance across data versions. We present two case studies where developers apply Chameleon to their own evolving datasets on production ML projects. Our interface helps them verify data collection efforts, find failure cases stretching across data versions, capture data processing changes that impacted performance, and identify opportunities for future data iterations.

Citation

Understanding and Visualizing Data Iteration in Machine Learning
Fred Hohman, Kanit Wongsuphasawat, Mary Beth Kery, Kayur Patel
ACM Conference on Human Factors in Computing Systems (CHI). Honolulu, HI, USA, 2020.
Project PDF Video Preview

BibTeX


@inproceedings{hohman2020understanding,
  title={Understanding and Visualizing Data Iteration in Machine Learning},
  author={Hohman, Fred and Wongsuphasawat, Kanit and Kery, Mary Beth and Patel, Kayur},
  booktitle={Proceedings of the SIGCHI Conference on Human Factors in Computing Systems},
  year={2020},
  organization={ACM}
}