Muhammad Marakkoottathil

Regional Practice Lead, DC and Cloud Infrastructure

GBM delivers transformational data center infrastructure solutions such as Software-Defined Networking, Hyperconverged Infrastructure, and many more to customers in the region.

As you may know, the data center infrastructure environment is evolving into all shapes and forms, and it is moving from centralized locations to colocation, remote branch offices, and the cloud. These are influenced by data center cost, flexibility, agility and application, and cloud dynamics.

Real-world Topology Analysis

The Cisco ACI multi-site solution is an architecture that interconnects multiple APIC cluster domains with their associated pods. It also interconnects separate regions (fabrics), and each is deployed as a single pod or multiple pods. The key software solution element is the multi-site orchestrator/management plane element known as the MSO. The MSO then helps centrally manage site-to-site policies across on-premises and cloud infrastructures.

From the above diagram, the control plane function between the DC1 and DC2 is done by forming the MP-BGP EVPN session between the SPINE nodes in each site. This idea allows the exchanging of MAC and IP address information for the endpoints that communicate across the sites. The control plane session can also provide encryption for the DCI link using the cloud-sec software function.

Also, the data plane functionality between the endpoints connected to DC1 and DC2 is implemented using site-to-site VxLAN tunnels.

Key Requirements & Considerations
•Latency Requirements:There are three different scenarios to consider when using ACI multi-site to interconnect various sites:

No Description Max Latency
1 Control and data plane traffic latency between the sites 50 ms RTT
2 The latency between the MSO cluster nodes – this is needed only when you place the MSO nodes across different locations 150 ms RTT
3 Finally, the latency between the MSO and the APIC clusters situated across the sites 1 sec

•Software Requirements: Please note, the MSO release 1.2(x) software hosting on VMware has the following resource requirements:

Topology Requirements

Software Resource requirements
Vmware ESXi 6.0 or later Minimum of eight virtual CPUs (vCPUs), 24 Gbps of memory, and 100 GB of disk space
Form Factors Description
Physical Bare-metal servers’ installation with an ISO from
Virtual For this there are two virtual appliance flavors are available 1) to be run on VMware ESXi hosts and the 2) on Linux KVM hypervisors.
Cloud Deployment A specific .ami file is available; this allows you to deploy a cluster of three CASE VMs directly in a specific AWS region.

Inter-Site Connections: Connections between sites can be made using the back-to-back connections between the spines switches, or in the case of multiple sites that should be routed through ISNs (Inter-Site Network Nodes). The ISN can be any L3-capable router or switch model that supports OSPF, sub-interfaces, and VLAN tagging.

MSO Orchestrator to the APIC Controllers Connectivity: The Cisco ACI Multi-Site Orchestrator cluster should sit outside the Cisco ACI fabric – for instance; it could be connected to the APIC using the OOB network

Cloud APIC Deployments: Cloud APIC is a software solution that needs to be part of your public cloud infrastructure if you need to extend your network and security policies. So ideally, the required policies are defined globally on the MSO and then distributed to the Cloud APIC. It accomplishes the task by transforming the received policy from local to cloud-native network policy. For example, it can automate the provisioning of the needed cloud-native resources, such as VPC/VNET, CSR 1000v deployments, security group rules, etc.


In conclusion, a multi-site solution is ideal if your data center is located on-prem or on the public cloud. Please note that you can also have a cloud-only deployment with MSO and cloud APIC. A multi-site solution allows optimized L2/L3 traffic to stretch across on-premises sites and L3 to multi-cloud. It also provides centralized management of policies using a software orchestrator called MSO. For more information, please reach out to the GBM team at for further discussion or support.

How GBM can help?

Enterprise driven cloud architecture requires, networking and security planning around scalability, resiliency, and also operational capabilities across the deployments. With GBM and its rich services portfolio to help customers in their digital transformation journey by delivering new technologies and integrating them into your existing infrastructure, and overcoming challenges related to adoption, operations, and maintenance.

Invested in a talented, skilled workforce to implement solutions that cater to customer’s specific, complex, and diverse business needs. With on-the ground, readily available expertise, GBM provides round-the-clock support to customers across the region. Through our professional services and support teams you can call on us for all the advice, design, deployment, and support capability you’ll ever need to successfully harness the powerful solutions.


• Cisco Cloud Application Centric Infrastructure Solution Overview
• Cisco Cloud ACI on Microsoft Azure White Paper
• Cisco ACI Multi-Site Architecture White Paper