Adata lakeis a system orrepository of datastored in its natural/raw format,[1]usually objectblobsor files. A data lake is usually a single store of data including raw copies of source system data, sensor data, social data etc.,[2]and transformed data used for tasks such asreporting,visualization,advanced analytics,andmachine learning.A data lake can includestructured datafromrelational databases(rows and columns), semi-structured data (CSV,logs,XML,JSON),unstructured data(emails,documents,PDFs), andbinary data(images,audio,video).[3]A data lake can be establishedon premises(within an organization's data centers) orin the cloud(usingcloud services).
Background
editJames Dixon, then chief technology officer atPentaho,coined the term by 2011[4]to contrast it withdata mart,which is a smaller repository of interesting attributes derived from raw data.[5]In promoting data lakes, he argued that data marts have several inherent problems, such asinformation siloing.PricewaterhouseCoopers(PwC) said that data lakes could "put an end to data silos".[6]In their study on data lakes they noted that enterprises were "starting to extract and place data for analytics into a single, Hadoop-based repository."
Examples
editMany companies usecloud storage servicessuch asGoogle Cloud StorageandAmazon S3or a distributed file system such asApache Hadoopdistributed file system (HDFS).[7]There is a gradual academic interest in the concept of data lakes. For example, Personal DataLake atCardiff Universityis a new type of data lake which aims at managingbig dataof individual users by providing a single point of collecting, organizing, and sharing personal data.[8]
Early data lakes, such as Hadoop 1.0, had limited capabilities because it only supported batch-oriented processing (Map Reduce). Interacting with it required expertise in Java, map reduce and higher-level tools likeApache Pig,Apache SparkandApache Hive(which were also originally batch-oriented).
Criticism
editPoorly-managed data lakes have been facetiously called data swamps.[9]
In June 2015, David Needle characterized "so-called data lakes" as "one of the more controversial ways to managebig data".[10]PwCwas also careful to note in their research that not all data lake initiatives are successful. They quote Sean Martin, CTO ofCambridge Semantics:
We see customers creating big data graveyards, dumping everything intoHadoop distributed file system(HDFS) and hoping to do something with it down the road. But then they just lose track of what’s there. The main challenge is not creating a data lake, but taking advantage of the opportunities it presents.[6]
They describe companies that build successful data lakes as gradually maturing their lake as they figure out which data andmetadataare important to the organization.
Another criticism is that the termdata lakeis not useful because it is used in so many different ways.[11]It may be used to refer to, for example: any tools or data management practices that are notdata warehouses;a particular technology for implementation; a raw data reservoir; a hub forETLoffload; or a central hub for self-service analytics.
While critiques of data lakes are warranted, in many cases they apply to other data projects as well.[12]For example, the definition ofdata warehouseis also changeable, and not all data warehouse efforts have been successful. In response to various critiques, McKinsey noted[13]that the data lake should be viewed as a service model for delivering business value within the enterprise, not a technology outcome.
Data lakehouses
editData lakehousesare a hybrid approach that can ingest a variety of raw data formats like a data lake, yet provideACIDtransactions and enforce data quality like adata warehouse.[14][15]A data lakehouse architecture attempts to address several criticisms of data lakes by adding data warehouse capabilities such as transaction support, schema enforcement, governance, and support for diverse workloads. According to Oracle, data lakehouses combine the "flexible storage of unstructured data from a data lake and the management features and tools from data warehouses".[16]
See also
editReferences
edit- ^"The growing importance of big data quality".The Data Roundtable.21 November 2016.Retrieved1 June2020.
- ^"What is a data lake?".aws.amazon.com.Retrieved12 October2020.
- ^Campbell, Chris."Top Five Differences between DataWarehouses and Data Lakes".Blue-Granite.com.Archived fromthe originalon 14 March 2016.
- ^Woods, Dan (21 July 2011)."Big data requires a big architecture".Forbes.
- ^Dixon, James (14 October 2010)."Pentaho, Hadoop, and Data Lakes".James Dixon’s Blog.James Dixon.Retrieved7 November2015.
If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.
- ^abStein, Brian; Morrison, Alan (2014).Data lakes and the promise of unsiloed data(PDF)(Report). Technology Forecast: Rethinking integration. PricewaterhouseCoopers.
- ^Tuulos, Ville (22 September 2015)."Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances".NextRoll.
- ^Walker, Coral; Alrehamy, Hassan (2015). "Personal Data Lake with Data Gravity Pull".2015 IEEE Fifth International Conference on Big Data and Cloud Computing.pp. 160–167.doi:10.1109/BDCloud.2015.62.ISBN978-1-4673-7183-4.S2CID18024161.
- ^Olavsrud, Thor (8 June 2017)."3 keys to keep your data lake from becoming a data swamp".CIO.Retrieved4 January2021.
- ^Needle, David (10 June 2015)."Hadoop Summit: Wrangling Big Data Requires Novel Tools, Techniques".Enterprise Apps.eWeek.Retrieved1 November2015.
Walter Maguire, chief field technologist at HP's Big Data Business Unit, discussed one of the more controversial ways to manage big data, so-called data lakes.
[permanent dead link] - ^"Are Data Lakes Fake News?".Sonra.8 August 2017.Retrieved10 August2017.
- ^Belov, Vladimir; Kosenkov, Alexander N.; Nikulchev, Evgeny (2021)."Experimental Characteristics Study of Data Storage Formats for Data Marts Development within Data Lakes".Applied Sciences.11(18): 8651.doi:10.3390/app11188651.
- ^"A smarter way to jump into data lakes".McKinsey.1 August 2017.
- ^What is a Data Lakehouse? | Databricks
- ^What is a Data Lakehouse? | Snowflake
- ^What is a Data Lakehouse? | Oracle