Introduction
The ever-increasing data demands of modern computing applications are pushing the limitations of traditional electrical interconnects. As Ethernet interfaces reach speeds of 800Gbps and beyond, the power and efficiency challenges of moving data over long electrical traces become untenable. This has led the photonics industry to develop a new interconnect architecture known as co-packaged optics (CPOs).
CPOs promise to deliver a five-fold increase in bandwidth compared to traditional pluggable optical connections. However, this new approach requires a significant rethinking of system architecture and implementation. In this tutorial, we'll explore the drivers behind the emergence of CPOs, the key technological advances, and the roadmap for this transformative interconnect technology.
The Limitations of Pluggable Optics
For the past three decades, the dominant approach for high-speed optical interconnects has been the use of pluggable optical modules. In this configuration, the switch or routing ASIC is located on the main board, while the optical transceiver modules are plugged into the front panel (Figure 1).
This approach worked well when data rates were lower. However, as Ethernet speeds have increased to 400GbE and beyond, the energy inefficiencies of this architecture have become glaringly apparent. The long electrical traces required to connect the ASIC to the pluggable optical modules result in significant power consumption due to the need for power-hungry SerDes (serializer/deserializer) circuits.
"The amount of energy required to move data from the switch chip — the ASIC — to the optical module on the front panel of the switch is becoming untenable, at about 8 picojoules per bit," said Scott Durrant, strategic marketing manager at Synopsys. "A switch that is carrying 400 gigabits Ethernet equates to about 100 watts of power within this chip to move data between this switch ASIC and the pluggable module at the front panel. That's manageable, but as switches increase in speed, the amount of energy required is going to grow to 200 or 400 watts. And that becomes something that's very difficult to cool."
The Move to On-Board and Co-Packaged Optics
To address the power and efficiency challenges of the pluggable optic approach, engineers began exploring alternative architectures. The first step was the concept of "on-board optics," where the optical modules were placed closer to the ASIC, rather than at the front panel.
While this approach showed promise, it didn't quite live up to expectations. "Unfortunately, it didn't quite work. Once all the desired capabilities were incorporated into the standard, the optical modules ironically came out larger than before and the power savings weren't achieved," explained Alexander Janta-Polczynski, senior advisory engineer for advanced semiconductor packaging at IBM.
The next evolution was the concept of "co-packaged optics," where the optical module is integrated directly onto the same substrate as the switch ASIC. This allows the data to travel over extremely short, low-loss electrical connections, significantly reducing power consumption.
"Co-packaging can drop the power level, because you're now transmitting on electrical traces that are right on the substrate and not very long," said Jeff Hutchins, who serves as a board member and vice chair of the PLL Working Group for Co-Packaging at the Optical Internetworking Forum (OIF).
The OIF recently published a standard for a 3.2Tb/s CPO module that targets Ethernet switching applications. This module utilizes 100Gbps electrical lanes and provides backward compatibility with 50Gbps lanes. It can enable an aggregate bandwidth of up to 51.2Tb/s in a switch or router.
Key Technical Advances in CPOs
The development of co-packaged optics has been enabled by several key technological advancements in the photonics industry. Let's explore some of the critical components and innovations that underpin this new interconnect architecture.
Integrated Photonics
At the heart of CPO modules is the use of integrated photonic circuits, often based on silicon photonics (SiPh) technology. These integrated photonic chips combine optical and electrical components on a single substrate, allowing for highly dense and efficient optical interfaces.
"All co-packaged optics are highly parallelized and integrated inside the switch router. They are typically implemented in SiPh with a pluggable-based laser, which is also plugged into the switch faceplate," explained Robert Maher, CTO for optical modules and coherent solutions at Infinera.
The integration of photonic components, such as modulators, detectors, and waveguides, alongside electronic circuits enables the extreme density and power efficiency required for CPO modules.
Pluggable Lasers
While the optical engine is integrated onto the same substrate as the ASIC, the laser source in a CPO module is often implemented as a separate pluggable component. This approach provides several advantages:
Reliability: Optical components, especially lasers, are highly sensitive to temperature variations, which can impact their performance and reliability. By separating the laser from the integrated photonic chip, the thermal management challenges are reduced.
Flexibility: Utilizing a pluggable laser allows for easier replacement and maintenance of the optical components, should any failures occur. This "socket" approach for the co-packaged assembly was a key requirement identified by the OIF.
Vendor Interoperability: The pluggable laser component enables a level of vendor interoperability, as the optical module can be sourced from one vendor while the laser is procured from another.
Electrical Interfaces and Interconnects
In addition to the photonic innovations, CPO modules also require high-performance electrical interfaces and interconnects. The OIF standard specifies a 32 x CEI-112G-XSR (extra short reach) host interface, which operates at extremely low power, less than 2 picojoules per bit, and can reach distances of up to 50mm.
These short-reach, low-power electrical connections between the ASIC and the optical engine are critical for achieving the energy efficiency and performance targets of CPO systems.
Thermal Management Challenges
One of the key challenges in CPO implementations is the increased power density in the co-packaged area. Unlike traditional pluggable optic configurations, where the optical modules are distributed across the front panel, the co-packaged approach concentrates a significant amount of power-hungry components in a small area.
"While CPO reduces the overall power consumption, the power density is very high in that CPO packaged area, because you just put a lot more stuff there than you had before. Because of the power density, the thermal management is a lot more complicated. It may have to be re-thought in some applications," warned Hutchins.
Innovative cooling solutions, such as advanced heatsinks, liquid cooling, or even integrated microcoolers, may be required to effectively manage the thermal challenges of CPO modules.
The Roadmap and Adoption of CPOs
Despite the significant technical progress in co-packaged optics, the adoption of this new interconnect architecture is not a foregone conclusion. There are still several factors that will shape the timeline and path to widespread deployment of CPOs.
Pluggable Optics Advancements
One of the key factors is the continued advancement of pluggable optical modules. According to Infinera's Rob Shore, "Last year at OFC, there was a ton of talk about co-packaged optics. You would have expected a significant amount of progress given all that talk, but it was almost the exact opposite. Most people were saying, 'Hey, we made so much progress on pluggable optics, bringing down the cost and enabling higher capacity, that it looks like it's probably pushed out the drive for co-packaged optics by a couple of years at least.'"
The pluggable optic vendors have been making significant strides in improving the power efficiency, density, and performance of their modules, which may delay the immediate need for a more radical architectural shift to CPOs.
Vendor Lock-in and Ecosystem Challenges
Another factor slowing the adoption of CPOs is the challenge of vendor lock-in and the established ecosystem around pluggable optics. As Infinera's Maher explained, "Challenges revolve around vendor lock-in, switch plus optics supplied by a single vendor, such as BRCM. Hyperscalers typically have a large amount of control over the vendors of their intra-DC pluggables. Meta and Google stated at OFC that they would stick with pluggables for the 50T class of switch routers."
The established relationships and supply chains around pluggable optics represent a significant barrier to the widespread adoption of CPOs, at least in the short term.
Reliability and Failure Modes
Another key challenge is the reliability and failure modes of CPO modules. As Maher noted, "Another challenge is the failure aspect of the co-packaged optics part. Each SiPh part carries 6.4T in a BRCM Tomohawk 5 box. If one lane goes down, you need to take the box out, open it up and replace the entire 6.4T receptacle. Current approach, if you have a bad plug, you bin it and replace with another one."
The ability to easily replace failed components is a significant advantage of pluggable optics, and CPO vendors will need to address these reliability and maintenance concerns to drive wider adoption.
The Role of Standards
The development of industry standards, such as the one published by the OIF, is crucial for the widespread adoption of co-packaged optics. As OIF's Janta-Polczynski stated, "A standard for CPO is essential for photonics's emergence as a disruptive technology."
The standardization of interfaces, form factors, and interoperability will help to create a robust ecosystem and reduce the barriers to entry for both vendors and customers.
Potential Alternatives to CPOs
While co-packaged optics are a significant step forward in optical interconnects, there are also alternative approaches being explored. One such example is NVIDIA's "linear drive optics" concept, which was presented at a recent OIF event.
"In both cases, one of the two DSPs is removed from the link, reducing power consumption. CPO moves the optics closer to the ASIC, but linear-drive pluggable optics remain at the front panel of the switch," as described by the OIF.
These alternative approaches may offer different trade-offs and advantages, and could potentially coexist or compete with CPO solutions in the future.
Conclusion
The emergence of co-packaged optics represents a transformative shift in high-speed optical interconnects. By integrating the optical engine directly with the switch ASIC, CPOs promise to deliver a significant improvement in power efficiency and bandwidth density.
The development of this new interconnect architecture has been enabled by advancements in integrated photonics, pluggable laser sources, and high-performance electrical interfaces. However, the adoption of CPOs is not without its challenges, including thermal management, reliability concerns, and existing ecosystem barriers.
As the industry continues to push the boundaries of data rate and power efficiency, co-packaged optics are poised to play a crucial role in the future of high-performance computing and networking. While the timeline for widespread adoption may be influenced by the continued progress of pluggable optics, the standardization efforts and technological advances in CPOs suggest that this interconnect revolution is well underway.
Kommentare