Data set
Adata set(ordataset) is a collection ofdata.In the case of tabular data, a data set corresponds to one or moredatabase tables,where everycolumnof a table represents a particularvariable,and eachrowcorresponds to a givenrecordof the data set in question. The data set lists values for each of the variables, such as for example height and weight of an object, for each member of the data set. Data sets can also consist of a collection of documents or files.[2]
In theopen datadiscipline, data set is the unit to measure the information released in a public open data repository. The Europeandata.europa.euportal aggregates more than a million data sets.[3]
Properties
[edit]Several characteristics define a data set's structure and properties. These include the number and types of the attributes or variables, and variousstatistical measuresapplicable to them, such asstandard deviationandkurtosis.[4]
The values may be numbers, such asreal numbersorintegers,for example representing a person's height in centimeters, but may also benominal data(i.e., not consisting ofnumericalvalues), for example representing a person's ethnicity. More generally, values may be of any of the kinds described as alevel of measurement.For each variable, the values are normally all of the same kind.Missing valuesmay exist, which must be indicated somehow.
Instatistics,data sets usually come from actual observations obtained bysamplingastatistical population,and each row corresponds to the observations on one element of that population. Data sets may further be generated byalgorithmsfor the purpose of testing certain kinds ofsoftware.Some modern statistical analysis software such asSPSSstill present their data in the classical data set fashion. If data is missing or suspicious animputationmethod may be used to complete a data set.[5]
Classics
[edit]Several classic data sets have been used extensively in thestatisticalliterature:
- Iris flower data set– Multivariate data set introduced byRonald Fisher(1936).[1]Provided online by University of California-Irvine Machine Learning Repository.[6]
- MNIST database– Images of handwritten digits commonly used to test classification, clustering, andimage processingalgorithms
- Categorical data analysis– Data sets used in the book,An Introduction to Categorical Data Analysis,provided onlineby UCLA Advanced Research Computing.[7]
- Robust statistics– Data sets used inRobust Regression and Outlier Detection(Rousseeuwand Leroy, 1968).Provided onlineat the University of Cologne.[8]
- Time series– Data used in Chatfield's book,The Analysis of Time Series,areprovided on-lineby StatLib.[9]
- Extreme values– Data used in the book,An Introduction to the Statistical Modeling of Extreme Valuesarea snapshot of the data as it was provided on-line by Stuart Coles,the book's author.
- Bayesian Data Analysis– Data used in the book areprovided on-line(archive link) byAndrew Gelman,one of the book's authors.
- TheBupa liver data– Used in several papers in themachine learning(data mining) literature.
- Anscombe's quartet– Small data set illustrating the importance of graphing the data to avoid statistical fallacies
Example
[edit]Loading datasets using Python:
pipinstalldatasets
fromdatasetsimportload_dataset
dataset=load_dataset(NAMEOFDATASET)
See also
[edit]- List of datasets for machine-learning research
- List of datasets in computer vision and image processing
- Data blending
- Data (computer science)
- Sampling
- Data store
- Interoperability
- Data collection system
References
[edit]- ^abFisher, R.A. (1963)."The Use of Multiple Measurements in Taxonomic Problems"(PDF).Annals of Eugenics.7(2): 179–188.doi:10.1111/j.1469-1809.1936.tb02137.x.hdl:2440/15227.Archived fromthe original(PDF)on 2011-09-28.Retrieved2007-05-22.
- ^Snijders, C.; Matzat, U.; Reips, U.-D. (2012)."'Big Data': Big gaps of knowledge in the field of Internet ".International Journal of Internet Science.7:1–5. Archived fromthe originalon 2019-11-23.Retrieved2017-02-10.
- ^"European open data portal".European open data portal.European Commission.Retrieved2016-09-23.
- ^Jan M. Żytkow, Jan Rauch (2000).Principles of data mining and knowledge discovery.Springer.ISBN978-3-540-66490-1.
- ^United Nations Statistical Commission; United Nations Economic Commission for Europe (2007).Statistical Data Editing: Impact on Data Quality: Volume 3 of Statistical Data Editing, Conference of European Statisticians Statistical standards and studies(PDF).United Nations Publications. p. 20.ISBN978-9211169522.
- ^"UCI Machine Learning Repository: Iris Data Set".Archivedfrom the original on 2023-04-26.Retrieved2023-05-02.
- ^"Textbook Examples An Introduction to Categorical Data Analysis by Alan Agresti".Archivedfrom the original on 2023-01-31.Retrieved2023-05-02.
- ^"The ROUSSEEUW datasets".Archived fromthe originalon 2005-02-07.
- ^"StatLib:: Data, Software and News from the Statistics Community".Archived fromthe originalon 2011-01-02.
External links
[edit]
- Data.gov– the U.S. Government's open data
- GCMD– the Global Change Master Directory containing over 34,000 descriptions of Earth science and environmental science data sets and services
- Humanitarian Data Exchange(HDX)– The Humanitarian Data Exchange (HDX) is an open humanitariandata sharingplatform managed by theUnited Nations Office for the Coordination of Humanitarian Affairs.
- NYC Open Data– free public data published by New York City agencies and other partners.
- Relational data set repositoryArchived2018-03-07 at theWayback Machine
- Research Pipeline– a wiki/website with links to data sets on many different topics
- StatLib–JASA Data Archive
- UCI– a machine learning repository
- UK Government Public Data
- World Bank Open Data– Free and open access to global development data byWorld Bank