Learn About PCIe Expansion

In 2007 the PCI-SIG approved the PCI Express External Cabling Specification that defines how PCI Express can be implemented over a standard cable. This new capability allows the full bandwidth of the PCIe bus to be utilized within multiple chassis systems and small local networks. Over the past three years, hundreds of applications have been implemented using PCIe over cable for high-speed I/O, bus expansion and local networking. In these applications, using PCIe provides the benefits of high-performance, low power and reduced costs. Some examples are:

  • Medical – CAT scanners, PET scanners, dental scanners and other imaging
  • Office documentation centers – High-end copy centers
  • Entertainment – HD cameras, film editing and theater projectors
  • Instrumentation – Oscilloscopes and test equipment
  • Data Storage – Disk and flash arrays
  • HPC – GPU acceleration and inter-CPU communications
  • Military – signal acquisition and analysis
  • Other – PC expansion slots

Although PCIe is well known as the PC backplane interface standard, it is less known as a high-speed cabling interface. Previous parallel bus structures such as PCI-X couldn’t easily be routed over a cable due to signal integrity problems. The serial technology and embedded clocking used within PCIe allows it to be used at full speeds across a motherboard or over a cable.

 

PCIe cables
 

Figure 1-PCIe cables come in four sizes supporting x1, x4, x8, x16 link widths

The cable specification defines 4 cable connectors – for x1, x4, x8 and x16 links – providing a wide range of price and performance (Figure 1). At the low end, the x1 cable provides a 2.5Gb/s interface over a low-cost cable. At the high end, a x16 cable provides 128Gb/s over a more expensive cable.

Cable Adapters
Figure 2 – PCIe over cable adapter modules are available for desktop, laptop and industrial bus structures.

Adapter modules supporting the various cable sizes and are available in a number of form factors supporting desktops, laptops, CompactPCI, CompactPCIe, VMEbus, AMC and XMC. Both the PCIe cables and adapter cards are now available from multiple sources.

Gen 1, Gen 2 and Gen 3 Performance

One of the techniques used to create high performance with PCIe is the use of embedded clocking within each differential signal pair. By embedding the clock within the signal stream, there is no distortion or time delay between the clock signal and the data. The clock rate used within each PCIe signal is 2.5Ghz for Gen 1, 5Ghz for Gen 2 and 8Ghz for Gen 3. PCIe interface components automatically detect the speed capability of the other side of the link, and only utilize Gen 2 or Gen 3 timing if both sides are capable of that performance level. The performance of various PCIe lane widths is shown in Table 1. These performance levels are identical whether the bus is routed over a backplane or over the PCIe cable.

  Gen 1 Gen 2 Gen 3
         x1 2.5 Gb/s 5 Gb/s 8 Gb/s
         x4 10 Gb/s 20 Gb/s 32 Gb/s
         x8 20 Gb/s 40 Gb/s 64 Gb/s
        x16 40 Gb/s 80 Gb/s 128 Gb/s

 

Most PCs today have a mixture of Gen 1 and Gen 2 PCIe slots. Similarly, I/O boards are available in a mix of Gen 1 and Gen 2. Low cost boards, such as USB or Ethernet controllers are typically Gen 1, whereas higher performance boards such as RAID controllers and graphics cards are generally Gen 2. Gen 3 systems and add-in boards are expected to become available in 2011. The first systems supporting Gen 3 will be based on Intel’s Sandy Bridge CPU, which brings PCIe Gen 3 directly out of the CPU chip. The first Gen 3 add-in cards are expected to graphics boards from nvidia and AMD/ATI.

With its industry standardization, high performance and low costs, PCIe over cable is well suited for connection to high speed I/O, bus expansion and local networking. Many designers are choosing PCIe as a connection from a PC to their high-performance I/O device. This usage model is sometimes called “USB on steroids”, since the data rate of PCIe over cable is 20 to 80 times faster than USB. Designers are now developing a wide variety of I/O devices that directly connect to PCIe over cable. These applications include disk arrays, flash disk arrays, video editing subsystems, HD video cameras, hand-held scanning devices and many other data acquisition subsystems.

Usage Models

PCIe over cable is also being used as a simple, yet high performance means of bus expansion. In this model the PCIe backplane bus is conceptually stretched to operate over a cable to a second backplane physically located within a second chassis. These expansion chassis offer up to 20 additional add-in board slots for either laptops or servers. By using bridges and switches within the expansion chassis backplane, a variety of PCI, PCI-X and PCIe slots can be supported. Some of the most economical expansion chassis are based on passive backplanes that have direct connections for PCIe cables. These backplanes are designed in the ATX form factor, so they can be used within standard PC-compatible cases.

Board
 

Figure 3 – This ATX form factor expansion backplane contains direct PCIe cable connections.

A variation of the bus expansion usage model is the use of a standard PC connected to an industrial bus subsystem. This combination provides applications with the best of both worlds – a cost-effective, robust PC for the human interface and general computing and an industrial or instrumentation bus-based sub-system (such as PXIe, CompactPCIe, or VMEbus) for the specialized I/O. Since the connection rate of the PCIe cable is equal to the PCIe backplane rate, the performance of this repartitioned system is equal to the architecture where the CPU board is in the native form factor of the industrial bus.

PCIe can also be used for direct communications between PCs. In this model, PCIe over cable can be used for high-speed memory-to-memory transfers and/or used as a very fast TCP/IP connection, with application compatibility to standard Ethernet. Direct data transfers are used in designs that send data from one PC’s memory into another PC’s memory as rapidly as possible. This is particularly useful for multi-computing applications where one PC inputs data and then passes it to a second PC for data manipulation. This PC then passes the data to another PC for graphical display, and another PC for storage. In this example, the PCs could be physically separate communicating with PCIe over cable. The TCP/IP mode allows users to utilize TCP/IP applications to conveniently communicate between PCs, map and share disk drives, printers, etc.

Applications that are adopting PCIe over cable for local networking include medical imaging, document center internal data transfer, instrumentation data sharing and high-performance computing inter-networking.

Advantages versus other Cable Standards

Whether being used for high speed I/O, bus expansion or as a high-speed network, there are several advantages associated with PCIe over cable – low cost, low power, high bandwidth and software transparency. These advantages derive from the fact that today’s CPU components and chip sets directly drive PCIe as their expansion bus.

Thus, the PCIe cable adapter cards are quite simple. The adapters simply route the signals from the motherboard out to the PCIe cable connector and provide some signal conditioning to guarantee the signal integrity is met at the other end of the cable. Because these adapters are simple, they are inexpensive. Because they don’t convert the PCIe protocol into anything else, they are the high performance and don’t require any software drivers (for the I/O expansion models).

As an I/O expansion protocol, PCIe can be used in place of other protocols including: Starfabric, External SAS, Fiberchannel and USB. Compared to each of these, PCIe is the highest performance since there are no protocol conversions or timing delays. No cable connection can transfer data faster than the PCIe slot is capable of, and PCIe over cable is by definition at 100% performance. Any adapter that has to convert PCIe into a different protocol, send the data over the cable, and then convert it back to PCIe at the other end of the cable so that it can communicate with I/O devices, will necessarily be slower – both in throughput and latency – than a pure PCIe over cable solution.

For networking, the comparison is PCIe versus 1Gb and 10Gb Ethernet. 1Gb Ethernet is clearly the most well accepted and lowest cost solution. 10Gb Ethernet continues to demand high prices, particularly for switches, and is used sparingly.

PCIe over cable is much higher performance, but also more expensive than 1 Gb Ethernet. PCIe over cable compares favorably, however, to 10Gb Ethernet. In this comparison, PCIe spans a much broader performance range – 2.5Gb/s for a x1 Gen 1 up to 128Gb/s for a x16 Gen 3 – than 10Gb Ethernet. In general, a PCIe network will perform 2-4 times as fast as 10Gb Ethernet, while requiring half the power and costing half as much.

The PCI-SIG cable specification doesn’t specify fiber optic solutions for PCIe. Several vendors have however introduced active fiber optic cable products for PCIe. These cables plug into the standard PCIe over cable connector and then convert the signals into optical signals and send them over fiber optic cables. The primary advantages of these products is that the fiber cable can span considerably longer distances – typically up to 500 meters, and that the optical signals are completely immune to EMI interference.

Two types of applications are utilizing PCIe over cable with active fiber optic cables. First are outdoor data acquisition and security systems where the distance feature is key. The second is within heavy industry factory floor applications where the fiber optic natural immunity to EMI (from welding and other machines) is key.

PCIe as a Network

PCIe was originally defined to support CPU-to-I/O communications, with a single PC serving as the bus interface host. Multi-computing can be accomplished using PCIe through a combination of non-transparent bridging and CPU-to-CPU communications software. This technology expands the applicability of PCIe to a wide variety of high-end applications, including radar and sonar analysis, medical imaging, test and measurement and high-performance computing (HPC) inter-communications (Figure 4).

PCIe Connectivity
 

Figure 4 – PCIe can be used as a high-performance networking architecture, connecting multiple PCs with software transparency to Ethernet.

The use of PCIe as a networking architecture is beginning to gain significant traction. This applies both to inter-blade communications (within a blade server) as well as external PCIe over cable connections between servers. Large HPC clusters with hundreds of interconnected nodes are currently being designed based on PCIe networking.

The ability to run PCIe over cable at full performance with software transparency opens up a range of new application possibilities for CPU to I/O system re-partitioning. Low cost host bus adapters extend the PCIe bus structure to expansion chassis or dedicated PCIe I/O hardware. PCIe over cable provides a simple and low cost method for extending applications that need more I/O boards than will fit in a single chassis to a multi-chassis solution. PCIe over cable can also be used as a high performance peripheral connection – a super-fast USB of sorts. Designing compatible end-points is straight forward because the PCIe interface is available as a gate array library.

When CPU-to-CPU communications are added to PCIe, the cable interface can be used as a high performance cabled network. A x8 cabled network with Gen 2 timing will transfer data at 40Gb/s – or 40 times faster than today’s 1Gb/s Ethernet interfaces.