The Cboe Futures Exchange (CFE) trading platform and the CGIF (Index) platform are housed in the Equinix data center in Secaucus, New Jersey. The secondary data centers are located at 350 Cermak Chicago, IL and CH3 in Elk Grove Village, IL. Trading Privilege Holders (TPH) and market data subscribers are strongly encouraged to establish connectivity to both locations to minimize service disruption in the event of an issue at either data center. Secaucus is the primary or hot site, with Chicago\Elk Grove Village being secondary or warm. A customer may receive market data from Chicago, and they may connect and heartbeat with order entry systems in Chicago. Orders submitted to Chicago will be rejected until Cboe declares the primary site in Secaucus down.

TPHs can request physical connectivity for Options on Futures (OOF) which will be collocated within the CFE production environment.

It is the customer's responsibility to select their telecommunications provider and arrange for connections to Cboe data centers.

Cboe supports the following network connectivity choices:

• IPSec VPN via the Internet (only for certification or test sessions);
• Co-location Cross-connect (i.e. for customers co-located in the same data center as the CSMI/CFE trading platform);
• Extranet connectivity (See Cboe Approved Extranet Providers for a list of approved Extranet Providers); and
• Private line Ethernet (circuit extension from a carrier to Cboe, see Carriers for a list of Carriers)

The following standard physical interface specifications are supported in ORD1 and CH1 located at 350 East Cermak. For other interface specifications contact noc@cboe.com.

The following standard physical interface specifications are supported in the NY4 and NY5 data center:

*Contact Cboe NOC for details on ER Optical Transceivers

• Customers may connect via an IPSec Virtual Private Network (VPN) over the Internet for access to order entry and unicast market data feeds for certification and test purposes only. LAN-to-LAN IPSec VPN supported.
• IP address of the host presented to Cboe must be registered.
• Customers must contact Cboe NOC for encryption details and to receive their pre-shared key.

Note: Cboe does not offer multicast market data feeds over VPN.

Customers may co-locate within the NY5 data center or a data center where a Cboe PoP is located and cross-connect to Cboe.

• Each physical port connection (1Gpbs and 10Gbps) within the Secaucus and Chicago\Elk Grove Village data centers will be subject to a monthly recurring charge. See the CFE Fee Schedule for more information regarding connectivity fees for current CFE environment.
• Cboe reserves the right to charge for one-time setup and monthly recurring fees incurred connecting TPHs or extranets. See the CFE Fee Schedule for more information regarding any setup or recurring fees that are currently applicable.
• TPHs may order the following physical connections for use with Options on Futures (OOF) in the certification and production environments:
  • 10G Layer 3 Ports - OOF and CFE unicast/multicast can be combined
  • 1G Layer 3 Ports - OOF and CFE unicast/multicast can be combined
  • 10G Layer 1 Ports - OOF multicast only
• 1G/10G L3 ports that combine access to both CFE and OOF will continue to be subject to physical port fees according to the CFE Fee Schedule.

With data center co-location, customers can terminate communications circuits, and establish a cross-connect to Cboe (or other destinations) in their space. This gives the maximum amount of control to the customer. This option is neutral for the customer and provides the greatest flexibility for the customer in determining when and to whom to connect. Customers interested in co-location services should contact the data center Point of Contact (see Connectivity Matrix for POC information).

Customers may provision connectivity to Cboe via an extranet.

• Extranets have provisioned redundant connections to Cboe for use by multiple customers.
• Contact information for a variety of extranet providers is found within Extranet Providers.

This method is an attractive alternative when:

• The customer would otherwise have to provision a long-haul private line;
• Outsourcing of network services and network management is an option; or
• The ease and speed of turn-up are important (when both the customer and Cboe have an existing connection to the extranet).

Customers may connect to Cboe via Private Line Ethernet.

• No co-location space is required. Cross-connect from Telco demarcation point to Cboe network via an Ethernet interface.
• Each physical port connections (1Gpbs and 10Gbps) within the Secaucus and Chicago data centers will be subject to a monthly recurring charge. Pricing for new physical ports is detailed in the CFE Fee Schedule.
• Cboe reserves the right to charge for one-time setup and monthly recurring fees incurred connecting TPHs or extranets. See the CFE Fee Schedule for more information regarding any setup or recurring fees that are currently applicable.
• TPHs may order the following physical connections for use with Options on Futures (OOF) in the certification and production environments:
  • 10G Layer 3 Ports - OOF and CFE unicast/multicast can be combined
  • 1G Layer 3 Ports - OOF and CFE unicast/multicast can be combined
  • 10G Layer 1 Ports - OOF multicast only
• 1G/10G L3 ports that combine access to both CFE and OOF will continue to be subject to physical port fees according to the CFE Fee Schedule.
• Contact your carrier of choice to arrange connectivity to Cboe. See Carriers for more information.

A Customer Web Portal account is required to request a new cross connect or decommission existing connectivity to Cboe. Please see your firm’s account administrator or contact the CFE Trade Desk (312-786-8700) for an account.

• Location (NY5, NY4, 350 Cermak, or Elk Grove Village)
• Number and speed of connections requested (1G or 10G)
• Registered BGP ASN (Cboe NOC can assign a private ASN)
• Networks advertised to Cboe (registered public IPs or Cboe assigned private addresses are accepted
• Network and billing contact information
• Transit IP address (Public or private range assigned by Cboe NOC)

Upon approval of cross connect request, Cboe NOC will provide a Letter of Authorization - Customer Facility Assignment with the Z-side cage, cabinet, panel, and port pair assignment. The customer requesting the cross connect is known as the A-Side.

The requesting customer submits a cross connect request with the appropriate data center provider:

• Equinix - NY5 and NY4
• Cboe - 350 Cermak and Elk Grove Village

The data center provider will need the LOA-CFA and the A-side details to complete the connection. The A-Side customer is responsible for any data center setup fees and monthly recurring costs associated with the cross connect. As the A-Side customer, the customer is also responsible for initiating troubleshooting requests with the data center provider in the event of a down cross connect.

Cross connects originating within either NY4 or NY5 data centers will be engineered to provide equivalent latency between customer demarcation points and the Cboe order entry gateways in NY5. Equal fiber pathway latency will be determined by OBR testing. WAN circuits originating outside Secaucus, NJ will also be subject to latency equalization.

• Cboe/Equinix Latency Equalization Infrastructure provides an equal optical length of fiber to customers connecting in the NY4 or NY5 Secaucus, NJ data centers.

• Fiber cross connects to customer cages are run from Equinix Latency Equalized Hubs using equal lengths of single-mode fiber to each customer cage. The additional slack fiber is stored on the Cboe side of the cross connect. Regardless of the customer's distance from the Equinix Latency Equalized Hub, the fiber spool length is identical.

• Latency Equalized Hubs within NY4/NY5 are connected via fiber trunks to the gateway servers in the NY5 data center. Since the NY5 Latency Equalized Hub is much closer to the gateway servers, these fiber trunks connect through additional lengths of fiber to replicate the distance between NY4 to NY5. It is imperative that customers order cross connects only in their local data centers. For example, the Letter of Authorization (LOA) for a customer cage located in NY4 will specify a NY4 demarcation. In this scenario, attempting to order a NY4-to-NY5 cross connect would greatly increase your fiber distance and overall latency to the matching engine in NY5, duplicating the latency equalized engineered distance.

• Customers with cages outside of NY4/NY5 Secaucus campus or WAN connections should order cross connects to the NY4 Latency Equalized Hub for the lowest latency.

• Every customer fiber path and trunk are tested end-to-end using a proprietary Optical Backscatter Reflectometer (OBR).

• Total one-way latency represented is measured from the customer fiber handoff to the Environment ToR switch egress point, prior to the Gateway server NIC, based on the average using 128byte packet size.

• Finally, the fiber patch between assigned demarcation point and the Cboe outside network device is adjusted to a custom length, ensuring the total fiber optical length is within acceptable tolerances.

Any party that maintains direct physical connectivity to CFE is required to maintain at least two (2) physical connections to the primary data centers due to the nature of 24 x 5 operations of CFE. Cboe NOC will work with CFE customers to provide maximum redundancy (fiber path, panel, and network devices). A CFE customer may have any combination of physical cross connects or PLE circuits in either primary data center. Extranet connectivity and L1 Market Data Distribution ports will not count towards a CFE customer's Connectivity Redundancy Requirement.

TPHs can submit a request to the Cboe NOC to relocate one or more switch ports to another switch or a different port. The following guidelines must be adhered to:

• Only CFE Layer 3, 10Gbps NY4 and NY5 cross connects are eligible for physical port relocation; 1Gbps Layer 3 and 10Gbps Layer 1 ports are excluded from relocation requests.
• Requests must be submitted in writing to noc@cboe.com.
• Requests must be submitted at least seven (7) calendar days prior to the requested move date. Moves can only be performed on Fridays after market close.
• Requests must include either the Equinix-assigned cross connect ID or the transit IP address to properly identify the connection being moved.
• The physical port being moved must have been in service for at least 90 days prior to the request. Physical ports are not eligible for relocation again for 90 days after a move.
• Relocation requests are limited to two (2) physical ports per request. TPHs must wait until the relocation request has been fulfilled before submitting subsequent requests.
• Port relocation requests will be randomly assigned from currently available ports. The Cboe NOC will not disclose the current switch port location or the new port number following relocation.
• Port relocation requests to another switch must stay on the same switch path type (i.e., primary to primary and secondary to secondary switches).
• All port relocation requests will require a transit IP address change.

Cboe performs scheduled maintenance to CFE and Indices connectivity infrastructure during a standing maintenance window every Friday from 8:00 p.m. - 10:00 p.m. ET. Customers should expect potential disruption to their connectivity during this window.

Cboe will not provide advance notification for maintenance activities performed within this standing window. Customers will be notified separately for any maintenance scheduled outside of this window in accordance with standard change notification procedures.

To minimize service disruption, Cboe will perform maintenance to the primary and secondary paths at separate times during the window, ensuring that not all access to CFE and Indices is simultaneously affected. Customers with connectivity to both the primary (Secaucus, NJ) and secondary (Chicago, IL and Elk Grove Village, IL) data centers are encouraged to ensure their systems are configured to utilize redundant paths during this window.

CFE monitors inbound bandwidth per 10 Gbps physical port at the TPH access switch level and will automatically disable any 10 Gbps physical network connection which receives inbound data exceeding a bandwidth limit of a 1 gigabit per second (1 Gbps) average measured over a two second time interval. This inbound bandwidth limit is based on observations of actual data traffic patterns and is equally applied across all CFE TPH 10 Gbps physical ports.

The bandwidth limit is an inbound data limit and thus does not apply to outbound data, such as market data, received through a physical port. However, if a physical network connection is disabled due to exceeding the inbound bandwidth limit, the physical network connection will not be able to submit inbound data or receive outbound data until it is re-enabled.

To re-enable an impacted physical network connection disabled due to exceeding the inbound bandwidth limit, a TPH must contact the Cboe NOC and provide a written postmortem outlining the issue that caused the bandwidth limit to be exceeded and the actions taken to remediate the issue. Any connection disabled a second time within the same trading day for exceeding the inbound bandwidth limit will be re-enabled only after an additional written postmortem is received and TPH testing has been performed with the Cboe NOC during standard CFE maintenance hours.

CFE utilizes Layer 1 replication devices between all customer latency equalized cross connects and L3 customer access switches to enable timestamping and packet capture at the network edge. Information collected by these devices will be used by Cboe for internal analysis and troubleshooting purposes. These L1 timestamping devices will add approximately 20ns of additional one-way latency. Clock synchronization for packet timestamping uses Cboe's internal White Rabbit time distribution network.

DC Location	Quantity	Bandwidth	Make	Model
NY4	4	1G	LDA	NeoTapX
NY5	4	1G	LDA	NeoTapX
NY4	16	10G	LDA	NeoTapX
NY5	8	10G	LDA	NeoTapX

CFE employs both 1G and 10G dedicated switches that serve as the network access point for TPH connections. The number of switches is largely based on customer demand.

*Switching mode is effectively store-and-forward one way due to speed change from 10G to 25G.

These switches mesh the customer access layer with the Environment core layer. The Data Core layer services two environments - CFE (Cboe Futures/Options on Futures) and Indices. Uplinks between layers are all port-channels using the vendor default hashing algorithm.

*Includes links between switches to environment cores as well as to 10G switches.

This layer uses dedicated switches for connecting the environment Top of Rack (ToR) switches to the DC Core layer. Uplinks between layers are all 4x25G port-channels using the vendor default hashing algorithm.

Switches at this layer connect the Cboe Futures and Options on Futures order gateways and market data feed servers. Hosts are connected to both a primary and secondary switch for redundancy using NIC teaming (Active/Passive). Uplinks between layers are all 4x25G port-channel using the vendor default hashing algorithm.

*Switching mode is effectively store-and-forward one way due to speed change 10G to 25G.

Cboe Futures and Options on Futures customers may order Layer 1 (L1) Market Data Only connections in the NY4/NY5 data centers. Similar to Layer 3 (L3) connections, they will be latency equalized to ensure the same total optical length between CFE customers.

• Only CFE Multicast PITCH feeds FC and FD are available via L1 Market Data connections. TCP PITCH , Spin, and Gap Repsonse Multicast are only available via the L3 path.
• Customer L1 devices are split evenly between the FC feed and FD feed path. A customer connected to an FC side enabled device will not receive FD side market data, and vice versa.
• L1 network devices do not support BGP or PIM. Once the connection is patched by Cboe NOC, the customer will receive market data only for the path for which their connection lands.
• Consistent with the provisions of the current CFE Fee Schedule, any party connecting to CFE through a physical port for a particular function is required to have a redundant physical port for that function, which may be the same type or a different type of physical port that allows for that function.
• L1 connections are offered in 10Gbps bandwidth only, 1Gpbs is not available.
• Cboe will impose a limit of eight cross connects, per TPH, per L1 network device. TPHs may distribute the eight connections across both CFE or OOF.
• L1 10G cross connects will be assigned random port numbers on the L1 device(s).
• Even though L1 data only flows over the receive side of the fiber optic pair, both the transmit and receive fibers are required to be connected for monitoring and administrative purposes.

• Only CFE OOF Multicast PITCH feeds OFC and OFD are available via L1 Market Data connections. TCP PITCH, Spin, and Gap Response Multicast are only available via the L3 path.
• Customer L1 devices are split evenly between the OFC feed and OFD feed path. A customer connected to an OFC side enabled device will not receive OFD side market data, and vice versa.
• L1 network devices do not support BGP or PIM. Once the connection is patched by Cboe NOC, the customer will receive market data only for the path for which their connection lands.
• L1 connections are offered in 10Gbps bandwidth only, 1Gpbs is not available.
• Cboe will impose a limit of eight cross connects, per TPH, per L1 network device. TPHs may distribute the eight connections across both CFE or OOF.
• L1 10G cross connects will be assigned random port numbers on the L1 device(s).
• Even though L1 data only flows over the receive side of the fiber optic pair, both the transmit and receive fibers are required to be connected for monitoring and administrative purposes.

A TPH may choose to convert one or more L3 cross connects to L1 connections. Cboe NOC personnel will move the fiber patch from the L3 switch to the applicable L1 network device.

• A conversion request must be made in writing by emailing noc@cboe.com. Please include:
  • The Equinix cross connect ID.
  • The peer IP address of the L3 connection(s) to be converted.
  • Requested L1 feed(s) - CFE FC/FD or OOF OFC/OFD
• Conversions from L3 to L1 will be performed only during a Friday maintenance window (8:00 - 11:00 p.m. ET).
• A conversion from L3 to L1 will restart the physical port start billing date. The 90-day minimum requirement will be based on the new start billing date.
• Conversions from L3 to L1 are not reversible as the L3 physical port configuration is deleted and the transit IP address(es) are reclaimed.
• Similar to a new L1 10G cross connect, converted L3 connections will be assigned a random port number on the L1 device(s).

Some telecommunications providers available within the Secaucus and Chicago data centers are listed below. This list is a summary and is not indicative of Cboe preference or recommendation. For telecommunications providers not included on the list, please contact the Cboe NOC to discuss.

Cboe has arrangements with several extranet providers to aggregate customer connectivity and provide low cost, value-added B2B services such as multicast market data feeds. Extranet providers are required to sign a Telecommunications Service Provider Agreement after meeting the requirements outlined in the Extranet Provider Manual.

Telecom carriers provide a dedicated circuit between customers in different data centers to a demarcation point in the Secaucus or Chicago data centers. The circuit is extended from the demarc to a Cboe network device.

It is recommended that customers use redundant connectivity via multiple telecommunications providers to each of the Cboe data centers.

Contact Cboe NOC for information about circuit ordering details (e.g. NPA-NXX, LOA/CFA requirements, demarcation information, etc.).

Please direct questions or comments regarding this manual to noc@cboe.com (913.815.7005). Cboe NOC is a one-call shop supporting customer and telecommunications providers during initial setup and continuing support of all connectivity issues.