The Evolution of Data Center Interconnects: A Tutorial

Introduction

In the digital era, the rapid expansion of cloud computing and data architecture has led to surging demands on data center interconnects (DCIs). To understand the evolution of DCIs, we can broadly categorize them into three groups based on their reach and location relative to the data center:

1. Intra-data center interconnects (< 2km)

2. Campus data center interconnects (< 10km)

3. Metro data center interconnects (< 100km)

Figure 1: Example of different data center interconnect (DCI) links. Intra-data center interconnects usually have lengths up to 2kms, campus interconnects up to 10km, and metro DCIs up to 100km, although there can be longer DCIs.

As data centers become more complex and AI increases its demands, the intra-data center sector is increasing in complexity and variety.

The Decentralization of Data Centers

Smaller, locally placed data centers have the potential to minimize latency, overcome inconsistent connections, and store and compute data closer to the end-user. These benefits are causing the global market for edge data centers to explode, with PWC predicting that it will nearly triple from $4 billion in 2017 to $13.5 billion in 2024.

Meanwhile, the demands of new AI infrastructure are pushing data center power consumption to such a degree that the electrical power grid might be unable to sustain it. Longer data center interconnects enable a more decentralized system of data centers with branches in different geographical areas connected through high-speed optical fiber links to cope with the strain of data center clusters on power grids.

These trends push the data center industry to look for interoperable solutions for longer interconnects over distances of 80 to 120 km.

Coherent vs. Direct Detect Technology

For many years, there has been an expectation that the increasing capacity demands on data centers would reach a point where coherent technology would deliver a lower cost per bit and power per bit than direct detect technologies.

However, direct detect technology (both NRZ and PAM-4) continues to successfully overcome the challenges of coherent technology and will continue to dominate the intra-DCI space (also called data center fabric) in the coming years. In this space, links span less than two kilometers, and for particularly short links (< 300 meters), affordable multimode fiber (MMF) is frequently used.

The tendency towards data center decentralization will also impact the inside of data centers. Larger data centers can require longer interconnects from one building to another. So in a way, some links inside the data centers could become more like campus DCI links, which require single-mode fiber solutions, and in those spaces coherent might have a better chance of becoming competitive.

Coherent technology has established itself as the go-to solution for interconnecting data centers over longer distances, such as metro DCIs. The advances in electronic and photonic integration allowed coherent technology for metro DCIs to be miniaturized into QSFP-DD and OSFP form factors. This progress allowed the Optical Internetworking Forum (OIF) to create the 400ZR and ZR+ standards for 400G DWDM pluggable modules. After the success of 400ZR standardization, the data center industry and the OIF are starting to promote an 800ZR standard to enable the next generation of interconnects.

From basic networking to sophisticated, AI-enhanced architectures, DCIs have become the backbone of the digital infrastructure, enabling the seamless operation of global cloud services and data centers. The decision to use coherent or direct detection technology for DCIs boils down to the reach and capacity needs.

Reference

[1] EFFECT Photonics, "The Evolution of Data Center Interconnects," www.effectphotonics.com, May 8, 2024.