Intel QuickPath Interconnect

TheIntel QuickPath Interconnect(QPI)is a point-to-pointprocessorinterconnectdeveloped byIntelwhich replaced thefront-side bus(FSB) inXeon,Itanium,and certain desktop platforms starting in 2008. It increased the scalability and available bandwidth. Prior to the name's announcement, Intel referred to it asCommon System Interface(CSI).Earlier incarnations were known as Yet Another Protocol (YAP) and YAP+.

QPI 1.1 is a significantly revamped version introduced withSandy Bridge-EP(Romleyplatform).

QPI was replaced byIntel Ultra Path Interconnect(UPI) inSkylake-SP Xeon processors based onLGA 3647socket.

Background

Although sometimes called a "bus", QPI is a point-to-point interconnect. It was designed to compete withHyperTransportthat had been used byAdvanced Micro Devices(AMD) since around 2003.Intel developed QPI at its Massachusetts Microprocessor Design Center (MMDC) by members of what had been theAlphaDevelopment Group, which Intel had acquired from Compaq and HP and in turn originally came fromDigital Equipment Corporation(DEC). Its development had been reported as early as 2004.

Intel first delivered it for desktop processors in November 2008 on theIntel Core i7-9xxandX58chipset. It was released in Xeon processors code-namedNehalemin March 2009 and Itanium processors in February 2010 (code named Tukwila).

It was supplanted by theIntel Ultra Path Interconnectstarting in 2017 on theXeonSkylake-SPplatforms.

Implementation

QPI is anuncorecomponent in Intel'sNehalemmicroarchitecture.

The QPI is an element of a system architecture that Intel calls theQuickPath architecturethat implements what Intel callsQuickPath technology.In its simplest form on a single-processor motherboard, a single QPI is used to connect the processor to the IO Hub (e.g., to connect anIntel Core i7to anX58). In more complex instances of the architecture, separate QPI link pairs connect one or more processors and one or more IO hubs or routing hubs in a network on the motherboard, allowing all of the components to access other components via the network. As with HyperTransport, the QuickPath Architecture assumes that the processors will have integratedmemory controllers,and enables anon-uniform memory access(NUMA) architecture.

Each QPI comprises two 20-lane point-to-point data links, one in each direction (full duplex), with a separate clock pair in each direction, for a total of 42 signals. Each signal is adifferential pair,so the total number of pins is 84. The 20 data lanes are divided onto four "quadrants" of 5 lanes each. The basic unit of transfer is the 80-bitflit,which has 8 bits for error detection, 8 bits for "link-layer header", and 64 bits for data. One 80-bit flit is transferred in two clock cycles (four 20-bit transfers, two per clock tick.) QPI bandwidths are advertised by computing the transfer of 64 bits (8 bytes) of data every two clock cycles in each direction.

Although the initial implementations use single four-quadrant links, the QPI specification permits other implementations. Each quadrant can be used independently. On high-reliability servers, a QPI link can operate in a degraded mode. If one or more of the 20+1 signals fails, the interface will operate using 10+1 or even 5+1 remaining signals, even reassigning the clock to a data signal if the clock fails.The initial Nehalem implementation used a full four-quadrant interface to achieve 25.6 GB/s (6.4GT/s × 1 byte × 4), which provides exactly double the theoretical bandwidth of Intel's 1600 MHz FSB used in the X48 chipset.

Although some high-end Core i7 processors expose QPI, other "mainstream" Nehalem desktop and mobile processors intended for single-socket boards (e.g.LGA 1156Core i3, Core i5, and other Core i7 processors from theLynnfield/Clarksfieldand successor families) do not expose QPI externally, because these processors are not intended to participate in multi-socket systems.

However, QPI is used internally on these chips to communicate with the "uncore",which is part of the chip containing memory controllers, CPU-sidePCI Expressand GPU, if present; the uncore may or may not be on the same die as the CPU core, for instance it is on a separate die in theWestmere-basedClarkdale/Arrandale.

In post-2009 single-socket chips starting with Lynnfield, Clarksfield, Clarkdale and Arrandale, the traditionalnorthbridgefunctions are integrated into these processors, which therefore communicate externally via the slowerDMIand PCI Express interfaces.

Thus, there is no need to incur the expense of exposing the (former) front-side bus interface via the processor socket.

Although the core–uncore QPI link is not present in desktop and mobileSandy Bridgeprocessors (as it was on Clarkdale, for example), the internal ring interconnect between on-die cores is also based on the principles behind QPI, at least as far ascache coherencyis concerned.

Frequency specifications

Being asynchronous circuitthe QPI operates at a clock rate of 2.4 GHz, 2.93 GHz, 3.2 GHz, 3.6 GHz, 4.0 GHz or 4.8 GHz (3.6 GHz and 4.0 GHz frequencies were introduced with the Sandy Bridge-E/EP platform and 4.8 GHz with the Haswell-E/EP platform). The clock rate for a particular link depends on the capabilities of the components at each end of the link and the signal characteristics of the signal path on the printed circuit board. The non-extreme Core i7 9xx processors are restricted to a 2.4 GHz frequency at stock reference clocks.

Bit transfers occur on both the rising and the falling edges of the clock, so the transfer rate is double the clock rate.

Intel describes the data throughput (in GB/s) by counting only the 64-bit data payload in each 80-bit flit. However, Intel then doubles the result because the unidirectional send and receive link pair can be simultaneously active. Thus, Intel describes a 20-lane QPI link pair (send and receive) with a 3.2 GHz clock as having a data rate of 25.6 GB/s. A clock rate of 2.4 GHz yields a data rate of 19.2 GB/s. More generally, by this definition a two-link 20-lane QPI transfers eight bytes per clock cycle, four in each direction.

The rate is computed as follows:

3.2 GHz

× 2 bits/Hz (double data rate)

× 16(20) (data bits/QPI link width)

× 2 (unidirectional send and receive operating simultaneously)

÷ 8 (bits/byte)

= 25.6 GB/s

Protocol layers

QPI is specified as afive-layer architecture,with separate physical, link, routing, transport, and protocol layers.In devices intended only for point-to-point QPI use with no forwarding, such as the Core i7-9xx and Xeon DP processors, the transport layer is not present and the routing layer is minimal.

Physical layer

The physical layer comprises the actual wiring and the differential transmitters and receivers, plus the lowest-level logic that transmits and receives the physical-layer unit. The physical-layer unit is the 20-bit "phit." The physical layer transmits a 20-bit "phit" using a single clock edge on 20 lanes when all 20 lanes are available, or on 10 or 5 lanes when the QPI is reconfigured due to a failure. Note that in addition to the data signals, a clock signal is forwarded from the transmitter to receiver (which simplifies clock recovery at the expense of additional pins).

Link layer

The link layer is responsible for sending and receiving 80-bit flits. Each flit is sent to the physical layer as four 20-bit phits. Each flit contains an 8-bit CRC generated by the link layer transmitter and a 72-bit payload. If the link layer receiver detects a CRC error, the receiver notifies the transmitter via a flit on the return link of the pair and the transmitter resends the flit. The link layer implementsflow controlusing a credit/debit scheme to prevent the receiver's buffer from overflowing. The link layer supports six different classes of message to permit the higher layers to distinguish data flits from non-data messages primarily for maintenance of cache coherence. In complex implementations of the QuickPath architecture, the link layer can be configured to maintain separate flows and flow control for the different classes. It is not clear if this is needed or implemented for single-processor and dual-processor implementations.

Routing layer

The routing layer sends a 72-bit unit consisting of an 8-bit header and a 64-bit payload. The header contains the destination and the message type. When the routing layer receives a unit, it examines its routing tables to determine if the unit has reached its destination. If so it is delivered to the next-higher layer. If not, it is sent on the correct outbound QPI. On a device with only one QPI, the routing layer is minimal. For more complex implementations, the routing layer's routing tables are more complex, and are modified dynamically to avoid failed QPI links.

Transport layer

The transport layer is not needed and is not present in devices that are intended for only point-to-point connections. This includes the Core i7. The transport layer sends and receives data across the QPI network from its peers on other devices that may not be directly connected (i.e., the data may have been routed through an intervening device.) the transport layer verifies that the data is complete, and if not, it requests retransmission from its peer.

Protocol layer

The protocol layer sends and receives packets on behalf of the device. A typical packet is a memory cache row. The protocol layer also participates in maintenance of cache coherence by sending and receiving relevant messages.

See also

• Elastic interface bus
• Front-side bus
• HyperTransport
• List of device bandwidths
• PCI Express
• RapidIO

External links

• An Introduction to the Intel QuickPath Interconnect
• Intel QuickPath Interconnect Overview(PDF)
• What you need to know about Intel’s Nehalem CPU,Ars Technica,April 9, 2008, by Jon Stokes
• First Look at Nehalem Microarchitecture: QPI Bus,November 2, 2008, by Ilya Gavrichenkov
• The Common System Interface: Intel’s Future Interconnect,August 28, 2007, by David Kanter