Grid computing

For the computer manufacturer, seeGrid Systems Corporation.

Not to be confused withCluster computing.

Grid computingis the use of widely distributedcomputerresourcesto reach a common goal. A computing grid can be thought of as adistributed systemwith non-interactive workloads that involve many files. Grid computing is distinguished from conventional high-performance computing systems such asclustercomputing in that grid computers have each node set to perform a different task/application. Grid computers also tend to be moreheterogeneousand geographically dispersed (thus not physically coupled) than cluster computers.Although a single grid can be dedicated to a particular application, commonly a grid is used for a variety of purposes. Grids are often constructed with general-purpose gridmiddlewaresoftware libraries. Grid sizes can be quite large.

Grids are a form ofdistributed computingcomposed of many networkedloosely coupledcomputers acting together to perform large tasks. For certain applications, distributed or grid computing can be seen as a special type ofparallel computingthat relies on complete computers (with onboard CPUs, storage, power supplies, network interfaces, etc.) connected to acomputer network(private or public) by a conventionalnetwork interface,such asEthernet.This is in contrast to the traditional notion of asupercomputer,which has many processors connected by a local high-speedcomputer bus.This technology has been applied to computationally intensive scientific, mathematical, and academic problems throughvolunteer computing,and it is used in commercial enterprises for such diverse applications asdrug discovery,economic forecasting,seismic analysis,andback officedata processing in support fore-commerceandWeb services.

Grid computing combines computers from multiple administrative domains to reach a common goal,to solve a single task, and may then disappear just as quickly. The size of a grid may vary from small—confined to a network of computer workstations within a corporation, for example—to large, public collaborations across many companies and networks. "The notion of a confined grid may also be known as an intra-nodes cooperation whereas the notion of a larger, wider grid may thus refer to an inter-nodes cooperation".

Coordinating applications on Grids can be a complex task, especially when coordinating the flow of information across distributed computing resources.Grid workflowsystems have been developed as a specialized form of aworkflow management systemdesigned specifically to compose and execute a series of computational or data manipulation steps, or a workflow, in the grid context.

Comparison of grids and conventional supercomputers

“Distributed” or “grid” computing in general is a special type ofparallel computingthat relies on complete computers (with onboard CPUs, storage, power supplies, network interfaces, etc.) connected to anetwork(private, public or theInternet) by a conventionalnetwork interfaceproducing commodity hardware, compared to the lower efficiency of designing and constructing a small number of custom supercomputers. The primary performance disadvantage is that the various processors and local storage areas do not have high-speed connections. This arrangement is thus well-suited to applications in which multiple parallel computations can take place independently, without the need to communicate intermediate results between processors.The high-endscalabilityof geographically dispersed grids is generally favorable, due to the low need for connectivity betweennodesrelative to the capacity of the public Internet.

There are also some differences between programming for a supercomputer and programming for a grid computing system. It can be costly and difficult to write programs that can run in the environment of a supercomputer, which may have a custom operating system, or require the program to addressconcurrencyissues. If a problem can be adequately parallelized, a “thin” layer of “grid” infrastructure can allow conventional, standalone programs, given a different part of the same problem, to run on multiple machines. This makes it possible to write and debug on a single conventional machine and eliminates complications due to multiple instances of the same program running in the same sharedmemoryand storage space at the same time.

Design considerations and variations

One feature of distributed grids is that they can be formed from computing resources belonging to one or multiple individuals or organizations (known as multipleadministrative domains). This can facilitate commercial transactions, as inutility computing,or make it easier to assemblevolunteer computingnetworks.

One disadvantage of this feature is that the computers which are actually performing the calculations might not be entirely trustworthy. The designers of the system must thus introduce measures to prevent malfunctions or malicious participants from producing false, misleading, or erroneous results, and from using the system as an attack vector. This often involves assigning work randomly to different nodes (presumably with different owners) and checking that at least two different nodes report the same answer for a given work unit. Discrepancies would identify malfunctioning and malicious nodes. However, due to the lack of central control over the hardware, there is no way to guarantee thatnodeswill not drop out of the network at random times. Some nodes (like laptops ordial-upInternet customers) may also be available for computation but not network communications for unpredictable periods. These variations can be accommodated by assigning large work units (thus reducing the need for continuous network connectivity) and reassigning work units when a given node fails to report its results in the expected time.

Another set of what could be termed social compatibility issues in the early days of grid computing related to the goals of grid developers to carry their innovation beyond the original field of high-performance computing and across disciplinary boundaries into new fields, like that of high-energy physics.

The impacts of trust and availability on performance and development difficulty can influence the choice of whether to deploy onto a dedicated cluster, to idle machines internal to the developing organization, or to an open external network of volunteers or contractors. In many cases, the participating nodes must trust the central system not to abuse the access that is being granted, by interfering with the operation of other programs, mangling stored information, transmitting private data, or creating new security holes. Other systems employ measures to reduce the amount of trust “client” nodes must place in the central system such as placing applications in virtual machines.

Public systems or those crossing administrative domains (including different departments in the same organization) often result in the need to run onheterogeneoussystems, using differentoperating systemsandhardware architectures.With many languages, there is a trade-off between investment in software development and the number of platforms that can be supported (and thus the size of the resulting network).Cross-platformlanguages can reduce the need to make this tradeoff, though potentially at the expense of high performance on any givennode(due to run-time interpretation or lack of optimization for the particular platform). Variousmiddlewareprojects have created generic infrastructure to allow diverse scientific and commercial projects to harness a particular associated grid or for the purpose of setting up new grids.BOINCis a common one for various academic projects seeking public volunteers; more are listed at theend of the article.

In fact, the middleware can be seen as a layer between the hardware and the software. On top of the middleware, a number of technical areas have to be considered, and these may or may not be middleware independent. Example areas includeSLAmanagement, Trust, and Security,Virtual organizationmanagement, License Management, Portals and Data Management. These technical areas may be taken care of in a commercial solution, though the cutting edge of each area is often found within specific research projects examining the field.

Market segmentation of the grid computing market

For the segmentation of the grid computing market, two perspectives need to be considered: the provider side and the user side:

The provider side

The overall grid market comprises several specific markets. These are the grid middleware market, the market for grid-enabled applications, theutility computingmarket, and the software-as-a-service (SaaS) market.

Gridmiddlewareis a specific software product, which enables the sharing of heterogeneous resources, and Virtual Organizations. It is installed and integrated into the existing infrastructure of the involved company or companies and provides a special layer placed among the heterogeneous infrastructure and the specific user applications. Major grid middlewares are Globus Toolkit,gLite,andUNICORE.

Utility computing is referred to as the provision of grid computing and applications as service either as an open grid utility or as a hosting solution for one organization or aVO.Major players in the utility computing market areSun Microsystems,IBM,andHP.

Grid-enabled applications are specific software applications that can utilize grid infrastructure. This is made possible by the use of grid middleware, as pointed out above.

Software as a service(SaaS) is “software that is owned, delivered and managed remotely by one or more providers.” (Gartner2007) Additionally, SaaS applications are based on a single set of common code and data definitions. They are consumed in a one-to-many model, and SaaS uses a Pay As You Go (PAYG) model or a subscription model that is based on usage. Providers of SaaS do not necessarily own the computing resources themselves, which are required to run their SaaS. Therefore, SaaS providers may draw upon the utility computing market. The utility computing market provides computing resources for SaaS providers.

The user side

For companies on the demand or user side of the grid computing market, the different segments have significant implications for their IT deployment strategy. The IT deployment strategy as well as the type of IT investments made are relevant aspects for potential grid users and play an important role for grid adoption.

CPU scavenging

CPU-scavenging,cycle-scavenging,orshared computingcreates a “grid” from the idle resources in a network of participants (whether worldwide or internal to an organization). Typically, this technique exploits the 'spare'instruction cyclesresulting from the intermittent inactivity that typically occurs at night, during lunch breaks, or even during the (comparatively minuscule, though numerous) moments of idle waiting that modern desktop CPU's experience throughout the day (when the computer is waiting on IO from the user, network, or storage). In practice, participating computers also donate some supporting amount of disk storage space, RAM, and network bandwidth, in addition to raw CPU power.

Manyvolunteer computingprojects, such asBOINC,use the CPU scavenging model. Sincenodesare likely to go "offline" from time to time, as their owners use their resources for their primary purpose, this model must be designed to handle such contingencies.

Creating anOpportunistic Environmentis another implementation of CPU-scavenging where special workload management system harvests the idle desktop computers for compute-intensive jobs, it also refers as Enterprise Desktop Grid (EDG). For instance,HTCondor(the open-source high-throughput computing software framework for coarse-grained distributed rationalization of computationally intensive tasks) can be configured to only use desktop machines where the keyboard and mouse are idle to effectively harness wasted CPU power from otherwise idle desktop workstations. Like other full-featured batch systems, HTCondor provides a job queueing mechanism, scheduling policy, priority scheme, resource monitoring, and resource management. It can be used to manage workload on a dedicated cluster of computers as well or it can seamlessly integrate both dedicated resources (rack-mounted clusters) and non-dedicated desktop machines (cycle scavenging) into one computing environment.

History

The termgrid computingoriginated in the early 1990s as ametaphorfor making computer power as easy to access as an electricpower grid.The power grid metaphor for accessible computing quickly became canonical whenIan FosterandCarl Kesselmanpublished their seminal work, "The Grid: Blueprint for a new computing infrastructure" (1999). This was preceded by decades by the metaphor ofutility computing(1961): computing as a public utility, analogous to the phone system.

CPU scavenging andvolunteer computingwere popularized beginning in 1997 bydistributed.netand later in 1999 bySETI@hometo harness the power of networked PCs worldwide, in order to solve CPU-intensive research problems.

The ideas of the grid (including those from distributed computing, object-oriented programming, and Web services) were brought together byIan FosterandSteve Tueckeof theUniversity of Chicago,andCarl Kesselmanof theUniversity of Southern California'sInformation Sciences Institute.The trio, who led the effort to create the Globus Toolkit, is widely regarded as the "fathers of the grid".The toolkit incorporates not just computation management but alsostorage management,security provisioning, data movement, monitoring, and a toolkit for developing additional services based on the same infrastructure, including agreement negotiation, notification mechanisms, trigger services, and information aggregation.While the Globus Toolkit remains the de facto standard for building grid solutions, a number of other tools have been built that answer some subset of services needed to create an enterprise or global grid.

In 2007 the termcloud computingcame into popularity, which is conceptually similar to the canonical Foster definition of grid computing (in terms of computing resources being consumed as electricity is from thepower grid) and earlier utility computing.

Progress

In November 2006, Seidel received theSidney Fernbach Awardat the Supercomputing Conference inTampa, Florida."For outstanding contributions to the development of software for HPC and Grid computing to enable the collaborative numerical investigation of complex problems in physics; in particular, modeling black hole collisions."This award, which is one of the highest honors in computing, was awarded for his achievements in numerical relativity.

Fastest virtual supercomputers

• As of March 2020,Folding@home– 1.1 exaFLOPS.
• As of April 7, 2020,BOINC– 29.8 PFLOPS.
• As of November 2019, IceCube via OSG – 350 fp32 PFLOPS.
• As of February 2018,Einstein@Home– 3.489 PFLOPS.
• As of April 7, 2020,SETI@Home– 1.11 PFLOPS.
• As of April 7, 2020,MilkyWay@Home– 1.465 PFLOPS.
• As of March 2019,GIMPS– 0.558 PFLOPS.

Also, as of March 2019, theBitcoin Networkhad a measured computing power equivalent to over 80,000exaFLOPS(Floating-point Operations Per Second).This measurement reflects the number of FLOPS required to equal the hash output of the Bitcoin network rather than its capacity for general floating-point arithmetic operations, since the elements of the Bitcoin network (Bitcoin miningASICs) perform only the specific cryptographic hash computation required by theBitcoinprotocol.

Projects and applications

Grid computing offers a way to solveGrand Challenge problemssuch asprotein folding,financialmodeling,earthquakesimulation, andclimate/weathermodeling, and was integral in enabling the Large Hadron Collider at CERN.Grids offer a way of using information technology resources optimally inside an organization. They also provide a means for offering information technology as autilityfor commercial and noncommercial clients, with those clients paying only for what they use, as with electricity or water.

As of October 2016, over 4 million machines running the open-sourceBerkeley Open Infrastructure for Network Computing(BOINC) platform are members of theWorld Community Grid.One of the projects using BOINC isSETI@home,which was using more than 400,000 computers to achieve 0.828TFLOPSas of October 2016. As of October 2016Folding@home,which is not part of BOINC, achieved more than 101 x86-equivalent petaflops on over 110,000 machines.

TheEuropean Unionfunded projects through theframework programmesof theEuropean Commission.BEinGRID(Business Experiments in Grid) was a research project funded by the European Commissionas anIntegrated Projectunder theSixth Framework Programme(FP6) sponsorship program. Started on June 1, 2006, the project ran 42 months, until November 2009. The project was coordinated byAtos Origin.According to the project fact sheet, their mission is “to establish effective routes to foster the adoption of grid computing across the EU and to stimulate research into innovative business models using Grid technologies”. To extract best practice and common themes from the experimental implementations, two groups of consultants are analyzing a series of pilots, one technical, one business. The project is significant not only for its long duration but also for its budget, which at 24.8 million Euros, is the largest of any FP6 integrated project. Of this, 15.7 million is provided by the European Commission and the remainder by its 98 contributing partner companies. Since the end of the project, the results of BEinGRID have been taken up and carried forward byIT-Tude.com.

The Enabling Grids for E-sciencE project, based in theEuropean Unionand included sites in Asia and the United States, was a follow-up project to the European DataGrid (EDG) and evolved into theEuropean Grid Infrastructure.This, along with theWorldwide LHC Computing Grid(WLCG), was developed to support experiments using theCERNLarge Hadron Collider.A list of active sites participating within WLCG can be found onlineas can real time monitoring of the EGEE infrastructure.The relevant software and documentation is also publicly accessible.There is speculation that dedicated fiber optic links, such as those installed by CERN to address the WLCG's data-intensive needs, may one day be available to home users thereby providing internet services at speeds up to 10,000 times faster than a traditional broadband connection.TheEuropean Grid Infrastructurehas been also used for other research activities and experiments such as the simulation of oncological clinical trials.

Thedistributed.netproject was started in 1997. TheNASA Advanced Supercomputing facility(NAS) rangenetic algorithmsusing theCondor cycle scavengerrunning on about 350Sun MicrosystemsandSGIworkstations.

In 2001,United Devicesoperated theUnited Devices Cancer Research Projectbased on itsGrid MPproduct, which cycle-scavenges on volunteer PCs connected to the Internet. The project ran on about 3.1 million machines before its close in 2007.

Definitions

Today there are many definitions ofgrid computing:

• In his article “What is the Grid? A Three Point Checklist”,Ian Fosterlists these primary attributes:
  • Computing resourcesare not administered centrally.
  • Open standardsare used.
  • Nontrivialquality of serviceis achieved.
• Plaszczak/Wellnerdefine grid technology as "the technology that enables resource virtualization, on-demand provisioning, and service (resource) sharing between organizations."
• IBM defines grid computing as “the ability, using a set of open standards and protocols, to gain access to applications and data, processing power, storage capacity and a vast array of other computing resources over the Internet. A grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of resources distributed across ‘multiple’ administrative domains based on their (resources) availability, capacity, performance, cost and users' quality-of-service requirements”.
• An earlier example of the notion of computing as the utility was in 1965 by MIT's Fernando Corbató. Corbató and the other designers of the Multics operating system envisioned a computer facility operating “like a power company or water company”.
• Buyya/Venugopaldefine grid as "a type of parallel and distributed system that enables the sharing, selection, and aggregation of geographically distributedautonomousresources dynamically at runtime depending on their availability, capability, performance, cost, and users' quality-of-service requirements ".
• CERN,one of the largest users of grid technology, talk ofThe Grid:“a service for sharing computer power and data storage capacity over theInternet.”

See also

List of grid computing projects

Related concepts

• High-throughput computing
• Cloud computing
• Code mobility
• Jungle computing
• Sensor grid
• Utility computing

Alliances and organizations

• Open Grid Forum(FormerlyGlobal Grid Forum)
• Object Management Group
• SHIWA project

Production grids

• European Grid Infrastructure
• Enabling Grids for E-sciencE
• INFN Production Grid
• NorduGrid
• OurGrid
• Sun Grid
• Techila
• Xgrid
• Univa Grid Engine

International projects

Name	Region	Start	End
European Grid Infrastructure(EGI)	Europe	May 2010	Dec 2014
Open Middleware Infrastructure Institute Europe(OMII-Europe)	Europe	May 2006	May 2008
Enabling Grids for E-sciencE(EGEE, EGEE II and EGEE III)	Europe	March 2004	April 2010
Grid enabled Remote Instrumentation with Distributed Control and Computation(GridCC)	Europe	September 2005	September 2008
European Middleware Initiative(EMI)	Europe	May 2010	active
KnowARC	Europe	June 2006	November 2009
Nordic Data Grid Facility	Scandinavia and Finland	June 2006	December 2012
World Community Grid	Global	November 2004	active
XtreemOS	Europe	June 2006	(May 2010) ext. to September 2010
OurGrid	Brazil	December 2004	active

National projects

• GridPP(UK)
• CNGrid(China)
• D-Grid(Germany)
• GARUDA(India)
• VECC(Calcutta,India)
• IsraGrid(Israel)
• INFN Grid(Italy)
• PL-Grid(Poland)
• National Grid Service(UK)
• Open Science Grid(USA)
• TeraGrid(USA)

Standards and APIs

• Distributed Resource Management Application API (DRMAA)
• A technology-agnostic information model for a uniform representation of Grid resources (GLUE)
• Grid Remote Procedure Call (GridRPC)
• Grid Security Infrastructure (GSI)
• Open Grid Services Architecture (OGSA)
• Common Object Request Broker Architecture (CORBA)
• Open Grid Services Infrastructure (OGSI)
• A Simple API for Grid Applications (SAGA)
• Web Services Resource Framework (WSRF)

Monitoring frameworks

• GStat

References

Bibliography

• Buyya, Rajkumar;Kris Bubendorfer(2009).Market Oriented Grid and Utility Computing.Wiley.ISBN978-0-470-28768-2.
• Benedict, Shajulin; Vasudevan (2008)."A Niched Pareto GA approach for scheduling scientific workflows in wireless Grids".Journal of Computing and Information Technology.16(2): 101.doi:10.2498/cit.1001122.
• Davies, Antony(June 2004)."Computational Intermediation and the Evolution of Computation as a Commodity"(PDF).Applied Economics.36(11): 1131.CiteSeerX10.1.1.506.6666.doi:10.1080/0003684042000247334.S2CID7309750.Archived fromthe original(PDF)on 2008-02-28.Retrieved2005-04-20.
• Foster, Ian;Carl Kesselman(1999).The Grid: Blueprint for a New Computing Infrastructure.Morgan Kaufmann Publishers.ISBN978-1-55860-475-9.
• Plaszczak, Pawel;Rich Wellner, Jr (2006).Grid Computing "The Savvy Manager's Guide".Morgan Kaufmann Publishers.ISBN978-0-12-742503-0.
• Berman, Fran;Anthony J. G. Hey;Geoffrey C. Fox(2003).Grid Computing: Making The Global Infrastructure a Reality.Wiley.ISBN978-0-470-85319-1.
• Li, Maozhen;Mark A. Baker (2005).The Grid: Core Technologies.Wiley.ISBN978-0-470-09417-4.Archived fromthe originalon 2007-10-28.Retrieved2005-04-26.
• Catlett, Charlie;Larry Smarr(June 1992)."Metacomputing".Communications of the ACM.35(6): 44–52.doi:10.1145/129888.129890.
• Smith, Roger (2005)."Grid Computing: A Brief Technology Analysis"(PDF).CTO Network Library. Archived fromthe original(PDF)on 2012-02-08.
• Buyya, Rajkumar (July 2005)."Grid Computing: Making the Global Cyberinfrastructure for eScience a Reality"(PDF).CSI Communications.29(1). Mumbai, India: Computer Society of India (CSI). Archived fromthe original(PDF)on 2006-02-28.Retrieved2006-02-19.
• Berstis, Viktors."Fundamentals of Grid Computing".IBM. Archived fromthe originalon 2012-02-04.
• Elkhatib, Yehia (2011).Monitoring, Analysing and Predicting Network Performance in Grids(PDF)(Ph.D.). Lancaster University. Archived fromthe original(PDF)on 2015-06-23.Retrieved2013-04-28.
• Ferreira, Luis; et al. (2016-09-30)."Grid Computing Products and Services".IBM.
• Ferreira, Luis; et al. (2016-09-30)."Introduction to Grid Computing with Globus".IBM.
• Jacob, Bart; et al. (2016-09-30)."Enabling Applications for Grid Computing".IBM.
• Ferreira, Luis; et al."Grid Services Programming and Application Enablement".IBM. Archived fromthe originalon 2012-02-04.
• Jacob, Bart; et al. (2016-09-30).Introduction to Grid Computing.IBM.
• Ferreira, Luis; et al. (2016-09-30)."Grid Computing in Research and Education".IBM.
• Ferreira, Luis; et al."Globus Toolkit 3.0 Quick Start".IBM. Archived fromthe originalon 2012-02-04.Retrieved2006-04-27.
• Surridge, Mike; et al."Experiences with GRIA – Industrial applications on a Web Services Grid"(PDF).IEEE. Archived fromthe original(PDF)on 2012-03-06.
• Stockinger, Heinz;et al. (October 2007)."Defining the Grid: A Snapshot on the Current View"(PDF).Supercomputing.42:3.doi:10.1007/s11227-006-0037-9.S2CID16019948.Archived fromthe original(PDF)on 2007-01-07.
• Global Grids and Software Toolkits: A Study of Four Grid Middleware Technologies
• The Grid Technology Cookbook
• Francesco Lelli, Eric Frizziero, Michele Gulmini, Gaetano Maron, Salvatore Orlando, Andrea Petrucci and Silvano Squizzato.The many faces of the integration of instruments and the grid.International Journal of Web and Grid Services 2007 – Vol. 3, No.3 pp. 239 – 266Electronic Edition
• Poess, Meikel;Nambiar, Raghunath(2005).Large Scale Data Warehouses on Grid(PDF).Archived fromthe original(PDF)on 2015-06-23.Retrieved2010-11-05.
• Pardi, Silvio;Francesco Palmieri (October 2010). "Towards a federated Metropolitan Area Grid environment: The SCoPE network-aware infrastructure".Future Generation Computer Systems.26(8): 1241–1256.doi:10.1016/j.future.2010.02.003.