cleanlab helps youcleandata andlabels by automatically detecting issues in a ML dataset. To facilitatemachine learning with messy, real-world data,this data-centric AI package uses yourexistingmodels to estimate dataset problems that can be fixed to train evenbettermodels.
Examples of various issues in Cat/Dog datasetautomatically detectedby cleanlab via this code:
lab=cleanlab.Datalab(data=dataset,label="column_name_for_labels")
# Fit any ML model, get its feature_embeddings & pred_probs for your data
lab.find_issues(features=feature_embeddings,pred_probs=pred_probs)
lab.report()- Use cleanlab to automatically check every:image,text,audio,ortabulardataset.
- Use cleanlab to automatically:detect data issues (outliers, duplicates, label errors, etc),train robust models,infer consensus + annotator-quality for multi-annotator data,suggest data to (re)label next (active learning).
While this open-source packagefindsdata issues, its utility depends on you having: a good existing ML model + an interface to efficientlyfixthese issues in your dataset. Providing all these pieces,Cleanlab Studiois a Data Curation platform tofind and fixproblems in any {image, text, tabular} dataset. Cleanlab Studioautomaticallyruns optimized algorithms from this package on top ofAutoML & Foundationmodels fit to your data, and presents detected issues (+ AI-suggested fixes) in an intelligentdata correction interface.
Try it for free!Adopting Cleanlab Studio enables users of this package to:
- Work 100x faster (1 min to analyze your raw data withzerocode or ML work; optionally usePython API)
- Produce better-quality data (10x moretypes of issuesauto detected &correctedvia built-in AI)
- Accomplish more (auto-label data, deploy ML instantly, audit LLM inputs/outputs, moderate content,...)
- Monitor incoming data and detect issues in real-time (integrate your data pipeline on anEnterprise plan)
This cleanlab package runs on Python 3.8+ and supports Linux, macOS, as well as Windows.
- Get startedhere!Install via
piporconda. - Developers who install the bleeding-edge from source should refer tothis master branch documentation.
Practicing data-centric AI can look like this:
- Train initial ML model on original dataset.
- Utilize this model to diagnose data issues (via cleanlab methods) and improve the dataset.
- Train the same model on the improved dataset.
- Try various modeling techniques to further improve performance.
Most folks jump from Step 1 → 4, but you may achieve big gains withoutanychange to your modeling code by using cleanlab! Continuously boost performance by iterating Steps 2 → 4 (and try to evaluate withcleaneddata).
All features of cleanlab work withany datasetandany model.Yes, any model: PyTorch, Tensorflow, Keras, JAX, HuggingFace, OpenAI, XGBoost, scikit-learn, etc.
cleanlab is useful across a wide variety of Machine Learning tasks. Specific tasks this data-centric AI package offers dedicated functionality for include:
- Binary and multi-class classification
- Multi-label classification(e.g. image/document tagging)
- Token classification(e.g. entity recognition in text)
- Regression(predicting numerical column in a dataset)
- Image segmentation(images with per-pixel annotations)
- Object detection(images with bounding box annotations)
- Classification with data labeled by multiple annotators
- Active learning with multiple annotators(suggest which data to label or re-label to improve model most)
- Outlier detection(identify atypical data that appears out of distribution)
For other ML tasks, cleanlab can still help you improve your dataset if appropriately applied. See ourExample NotebooksandBlog.
Beyond automatically catchingall sorts of issueslurking in your data, this data-centric AI package helps you deal withnoisy labelsand train morerobust ML models. Here's an example:
# cleanlab works with **any classifier**. Yup, you can use PyTorch/TensorFlow/OpenAI/XGBoost/etc.
cl=cleanlab.classification.CleanLearning(sklearn.YourFavoriteClassifier())
# cleanlab finds data and label issues in **any dataset**... in ONE line of code!
label_issues=cl.find_label_issues(data,labels)
# cleanlab trains a robust version of your model that works more reliably with noisy data.
cl.fit(data,labels)
# cleanlab estimates the predictions you would have gotten if you had trained with *no* label issues.
cl.predict(test_data)
# A universal data-centric AI tool, cleanlab quantifies class-level issues and overall data quality, for any dataset.
cleanlab.dataset.health_summary(labels,confident_joint=cl.confident_joint)cleanlabcleans your data'slabels via state-of-the-artconfident learningalgorithms, published in thispaperandblog.See some of the datasets cleaned with cleanlab atlabelerrors.
cleanlab is:
- backed by theory-- withprovable guaranteesof exact label noise estimation, even with imperfect models.
- fast-- code is parallelized and scalable.
- easy to use-- one line of code to find mislabeled data, bad annotators, outliers, or train noise-robust models.
- general-- works withany dataset(text, image, tabular, audio,...) +any model(PyTorch, OpenAI, XGBoost,...)
Examples of incorrect given labels in various image datasetsfound and correctedusing cleanlab. While these examples are from image datasets, this also works for text, audio, tabular data.
cleanlab is based on peer-reviewed research. Here are relevant papers to cite if you use this package:
Confident Learning (JAIR '21)(click to show bibtex)
@article{northcutt2021confidentlearning,
title={Confident Learning: Estimating Uncertainty in Dataset Labels},
author={Curtis G. Northcutt and Lu Jiang and Isaac L. Chuang},
journal={Journal of Artificial Intelligence Research (JAIR)},
volume={70},
pages={1373--1411},
year={2021}
}
Rank Pruning (UAI '17)(click to show bibtex)
@inproceedings{northcutt2017rankpruning,
author={Northcutt, Curtis G. and Wu, Tailin and Chuang, Isaac L.},
title={Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels},
booktitle = {Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence},
series = {UAI'17},
year = {2017},
location = {Sydney, Australia},
numpages = {10},
url = {http://auai.org/uai2017/proceedings/papers/35.pdf},
publisher = {AUAI Press},
}
Label Quality Scoring (ICML '22)(click to show bibtex)
@inproceedings{kuan2022labelquality,
title={Model-agnostic label quality scoring to detect real-world label errors},
author={Kuan, Johnson and Mueller, Jonas},
booktitle={ICML DataPerf Workshop},
year={2022}
}
Out-of-Distribution Detection (ICML '22)(click to show bibtex)
@inproceedings{kuan2022ood,
title={Back to the Basics: Revisiting Out-of-Distribution Detection Baselines},
author={Kuan, Johnson and Mueller, Jonas},
booktitle={ICML Workshop on Principles of Distribution Shift},
year={2022}
}
Token Classification Label Errors (NeurIPS '22)(click to show bibtex)
@inproceedings{wang2022tokenerrors,
title={Detecting label errors in token classification data},
author={Wang, Wei-Chen and Mueller, Jonas},
booktitle={NeurIPS Workshop on Interactive Learning for Natural Language Processing (InterNLP)},
year={2022}
}
CROWDLAB for Data with Multiple Annotators (NeurIPS '22)(click to show bibtex)
@inproceedings{goh2022crowdlab,
title={CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators},
author={Goh, Hui Wen and Tkachenko, Ulyana and Mueller, Jonas},
booktitle={NeurIPS Human in the Loop Learning Workshop},
year={2022}
}
ActiveLab: Active learning with data re-labeling (ICLR '23)(click to show bibtex)
@inproceedings{goh2023activelab,
title={ActiveLab: Active Learning with Re-Labeling by Multiple Annotators},
author={Goh, Hui Wen and Mueller, Jonas},
booktitle={ICLR Workshop on Trustworthy ML},
year={2023}
}
Incorrect Annotations in Multi-Label Classification (ICLR '23)(click to show bibtex)
@inproceedings{thyagarajan2023multilabel,
title={Identifying Incorrect Annotations in Multi-Label Classification Data},
author={Thyagarajan, Aditya and Snorrason, Elías and Northcutt, Curtis and Mueller, Jonas},
booktitle={ICLR Workshop on Trustworthy ML},
year={2023}
}
Detecting Dataset Drift and Non-IID Sampling (ICML '23)(click to show bibtex)
@inproceedings{cummings2023drift,
title={Detecting Dataset Drift and Non-IID Sampling via k-Nearest Neighbors},
author={Cummings, Jesse and Snorrason, Elías and Mueller, Jonas},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
Detecting Errors in Numerical Data (ICML '23)(click to show bibtex)
@inproceedings{zhou2023errors,
title={Detecting Errors in Numerical Data via any Regression Model},
author={Zhou, Hang and Mueller, Jonas and Kumar, Mayank and Wang, Jane-Ling and Lei, Jing},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
ObjectLab: Mislabeled Images in Object Detection Data (ICML '23)(click to show bibtex)
@inproceedings{tkachenko2023objectlab,
title={ObjectLab: Automated Diagnosis of Mislabeled Images in Object Detection Data},
author={Tkachenko, Ulyana and Thyagarajan, Aditya and Mueller, Jonas},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
Label Errors in Segmentation Data (ICML '23)(click to show bibtex)
@inproceedings{lad2023segmentation,
title={Estimating label quality and errors in semantic segmentation data via any model},
author={Lad, Vedang and Mueller, Jonas},
booktitle={ICML Workshop on Data-centric Machine Learning Research},
year={2023}
}
To understand/cite other cleanlab functionality not described above, check out ouradditional publications.
-
Example Notebooks demonstrating practical applications of this package
-
NeurIPS 2021 paper: Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
-
Learn, discuss, and shape the future of cleanlab in our 1000+ memberSlack community.
-
Interested in contributing? See thecontributing guide,development guide,andideas on useful contributions.We welcome your help building a standard open-source platform for data-centric AI!
-
Have questions? Check outour FAQ,Github Issues,orSlack.
-
Need professional help with your Data/AI project? Email us:[email protected]
For instance, we can help youmonitor incoming data and detect issues in real-time.
Copyright (c) 2017 Cleanlab Inc.
cleanlab is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
cleanlab is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
SeeGNU Affero General Public LICENSEfor details. You can email us to discuss licensing:[email protected]
Commercial licensing is available for teams and enterprises that want to use cleanlab in production workflows, but are unable to open-source their codeas is required by the current license.Please email us:[email protected]
