Co-Packaging Interoperability: Enabling Next-Generation Data Center Interconnects

Introduction

The rapid growth of cloud computing and data-intensive applications has driven the need for ever-increasing bandwidth and throughput in data center networks. To meet these demands, the industry is exploring new approaches to packaging and integrating photonic and electronic components, known as co-packaging. Co-packaging offers the potential for dramatic reductions in power consumption, size, and cost compared to traditional pluggable optical modules. However, realizing the full benefits of co-packaging requires a high degree of interoperability between the different com ponents and subsystems.

This tutorial will provide an overview of the work being done by the Optical Internetworking Forum (OIF) to enable robust co-packaging interoperability. We will cover the key aspects of the OIF Co-Packaging Framework, including the identified application spaces, potential interfaces for interoperability, and demonstrations of interoperable co-packaged solutions. Additionally, we will dive into the details of two critical enabling technologies - the 3.2Tb/s Optical Module and the External Laser Small Form Factor Pluggable (ELSFP) - and discuss the electrical interface standards being developed to support co-packaging.

The Co-Packaging Framework

The OIF's co-packaging framework project was initiated to explore the next generation industry needs for co-packaged solutions and forge an industry consensus on interoperable approaches. This framework document serves as a guiding light, identifying the key application spaces, interface points requiring interoperability, and follow-on implementation projects needed to nail down the specifics.

Application Spaces

The framework document identified several key application spaces that will benefit from co-packaging, as shown in Figure 1. These include:

• Switch Generation: Co-packaging can enable massive switch capacity (e.g. 51.2Tb/s) by integrating multiple high-density optical modules.

• AI/ML Accelerators: The tight integration of photonics and electronics enabled by co-packaging can accelerate the performance of AI and machine learning workloads.

• Specialized ASICs: Co-packaging can improve the power efficiency and performance of specialized application-specific integrated circuits (ASICs) used in data centers.

• Servers and Linecards: Co-packaged optical interfaces can bring the benefits of reduced power, size, and cost to server and linecard platforms.

Figure 1: Co-Packaging Application Spaces

Potential Interfaces for Interoperability

The framework document also identified the key interfaces that must be standardized to ensure interoperability between co-packaged components, as illustrated in Figure 2. These include:

• Electrical Interfaces: High-speed electrical interconnects between the electronic and photonic components, as well as power delivery and management.

• Optical Interfaces: The optical coupling between the photonic components and the external fiber optic links.

• Thermal: The thermal management of the co-packaged system, including heat dissipation and cooling.

• Mechanical and Environmental: The mechanical integration and environmental requirements for co-packaged modules.

• Reliability and Repairability: Ensuring the overall system reliability and the ability to repair or replace individual components.

Figure 2: Co-Packaging Architectures

Interoperability Demonstrations

To validate the co-packaging framework and drive progress towards interoperable solutions, the OIF has organized a series of interoperability demonstrations at industry events like the Optical Fiber Communication Conference (OFC). These demonstrations have showcased the integration of various co-packaged components and subsystems, including:

• 3.2Tb/s Optical Module: A high-density optical module designed for co-packaging applications.

• External Laser Small Form Factor Pluggable (ELSFP): A standardized approach for providing external laser power to co-packaged optical engines.

• Electrical Interfaces: Demonstrations of the CEI-112G-XSR-PAM4 and CEI-112G-XSR+-PAM4 standards for high-speed electrical interconnects in co-packaged and near-packaged systems.

Let's dive into the details of these key enabling technologies.

The 3.2Tb/s Optical Module

One of the critical components for realizing high-density co-packaged systems is the 3.2Tb/s Optical Module, which was initiated by the OIF in 2021. This module is designed to provide 32 channels of 112Gb/s PAM4 signals, supporting both 400G-DR4 and 400G-FR4 (including 200G mode) optical interfaces.

The 3.2Tb/s Optical Module measures 52.1mm x 35.1mm x 6.0mm and features a land grid array (LGA) socket interface, as shown in Figure 3. This socket-based approach enables the module to be easily integrated into a co-packaged assembly, with the option for a copper cable assembly connection.

Figure 3: 3.2Tb/s Optical Module Dimensions

The module's power budget is designed to support either an internal laser option (56W) or an external laser option (48W) using the ELSFP interface. The LGA pin map, shown in Figure 4, includes various supply rails (12V, 3.3V, 2.6V, 1.8V, 1.2V, 0.9V, 0.7V) as well as communication interfaces like SPI and the Common Management Interface Specification (CMIS).

Figure 4: 3.2Tb/s Optical Module LGA Pin Map

The 3.2Tb/s Optical Module integrates the optical components (lasers, modulators, photodetectors) and electronic components (drivers, trans-impedance amplifiers, control logic) in a 3D-stacked architecture to achieve the high channel density and power efficiency required for co-packaging applications.

External Laser Small Form Factor Pluggable (ELSFP)

A key challenge in co-packaged systems is the thermal management of the optical components, particularly the laser sources. The OIF's ELSFP project addresses this by defining a common external laser pluggable module that can be used to provide optical power to multiple co-packaged optical engines.

The ELSFP module, shown in Figure 5, features a pluggable form factor with a blind-mate optical connector that mates with the host system. This blind-mate connector, along with a system interlock, enables the use of high-power Class 3B and 4 lasers inside the ELSFP, improving the overall system safety.

Figure 5: ELSFP Module-Side Optical Connector

The ELSFP module can support a range of optical power classes, from "Super Low Power" (2dBm) to "Super High Power" (26dBm), to accommodate the needs of different co-packaged optical engines. The module also includes features like a 3.3V power supply, CMIS-based management, and the ability to feed multiple optical engines from a single ELSFP.

The ELSFP's pluggable form factor enables flexible system architectures, allowing for "hot-swap" replacement of individual laser modules if needed. This improves the overall system reliability and serviceability.

Electrical Interfaces for Co-Packaging

To enable robust and power-efficient electrical interconnects between the co-packaged components, the OIF is developing two key interface standards: CEI-112G-XSR-PAM4 and CEI-112G-XSR+-PAM4.

CEI-112G-XSR-PAM4

The CEI-112G-XSR-PAM4 standard defines the electrical interface for co-packaged optical modules, targeting baud rates from 36 to 58 Gbaud. This standard is optimized for low power consumption by leveraging the reduced channel loss and jitter budgets available in a co-packaged system, as compared to traditional pluggable optical modules.

The CEI-112G-XSR-PAM4 standard defines three channel categories, each with specific insertion loss and bit error rate (BER) requirements, as shown in Table 1. This allows system designers to choose the appropriate category based on the requirements of their application.

Table 1: CEI-112G-XSR-PAM4 Channel Categories

Category	IL at Nyquist (Max, dB)	BER (Max)
CATI	10	le-6
CAT2	10	1 e—8
CAT3	8	le-9

CEI-112G-XSR+-PAM4

While co-packaged optical modules offer significant benefits, the monolithic package approach can also introduce challenges in terms of availability, cost, and multi-vendor support. To address this, the industry is also exploring a "near-package optics" (NPO) architecture, which relies on advanced printed circuit board (PCB) technology for high-speed electrical routing without significant power penalty.

To support this NPO approach, the OIF is developing the CEI-112G-XSR+-PAM4 standard. This interface is optimized for Ethernet rates at 106.25Gbps, the key application for co-packaged and near-packaged systems. The CEI-112G-XSR+-PAM4 standard targets an insertion loss of less than 13dB at 26.5625GHz Nyquist, including up to one separable interconnect, to enable the lowest practical energy consumption (pJ/b) implementation.

Interoperability Demonstrations

The OIF has organized a series of interoperability demonstrations at industry events to validate the co-packaging framework and showcase the integration of various co-packaged components and subsystems.

ELSFP and CEI Linear Interface Demonstration

One such demonstration, shown in Figure 6, showcased the integration of the ELSFP module providing external laser power to a CEI-112G-Linear interface. In this setup, the ELSFP module delivered continuous-wave (CW) light to a Mach-Zehnder modulator, which was then connected to a CEI-112G-Linear electrical interface for data transmission.

Figure 6: ELSFP and CEI Linear Interface Demonstration

This demonstration highlighted the ability of the ELSFP module to provide laser power for single-mode, CEI-112G-Linear applications, a critical capability for co-packaged systems.

Active ELSFP and Blind Mate Connector Demonstration

Another demonstration, depicted in Figure 7, showcased the interoperability of active ELSFP modules with a blind-mate optical connector. In this setup, uncooled ELSFP modules from different vendors (Casela, O-net) were connected to a 2x8 PMF MT blind-mate connector, demonstrating the modularity and interoperability of the ELSFP approach.

Figure 7: Active ELSFP and Blind Mate Connector Demonstration

3.2Tb/s Co-Packaged Copper Cable Demonstration

The OIF also showcased a multi-vendor demonstration of the 3.2Tb/s Optical Module implemented using passive copper cable assemblies, as shown in Figure 8. This demonstration validated the interoperability of the 3.2Tb/s module with various copper cable approaches, including CCA-3.2T to QSFP-DD, CCA-3.2T to OSFP, and OSFP800 to QSFP-DD800 DAC.

Figure 8: 3.2Tb/s Co-Packaged Copper Cable Demonstration

These interoperability demonstrations have been crucial in validating the OIF's co-packaging framework and driving progress towards the realization of robust, interoperable co-packaged solutions.

Conclusion

The rapidly growing demand for bandwidth and throughput in data centers has driven the industry to explore co-packaging as a means to achieve significant improvements in power, size, and cost. However, realizing the full benefits of co-packaging requires a high degree of interoperability between the various components and subsystems.

The OIF's co-packaging framework has been instrumental in identifying the key application spaces, potential interfaces for interoperability, and the necessary follow-on implementation projects. The development of the 3.2Tb/s Optical Module and the ELSFP, along with the electrical interface standards like CEI-112G-XSR-PAM4 and CEI-112G-XSR+-PAM4, are critical enablers for realizing interoperable co-packaged solutions.

The series of interoperability demonstrations organized by the OIF have validated the framework and showcased the integration of these co-packaged components, paving the way for the widespread adoption of this transformative technology in data centers and beyond.

As the industry continues to push the boundaries of data center performance, the work done by the OIF on co-packaging interoperability will play a pivotal role in shaping the future of high-speed interconnects.

Reference

[1] S. Durrant, A. Janta-Polczynski, R. Maher, J. Hutchins, M. Lebby, "Co-Packaged Optics (CPOs): Revolutionizing Ethernet Connectivity for Enhanced Efficiency and Speed," Optical Internetworking Forum, [Publication Date]. Available: https://www.oiforum.com/wp-content/uploads/OIF-Co-Packaging-3.2T-Module-01.0.pdf. [Accessed: April 7, 2024].