Overlay vs underlay, BGP EVPN control plane, VXLAN encapsulation — how it actually works in operator data centres

1. What Is EVPN-VXLAN and Why Operators Use It

EVPN-VXLAN is the dominant data centre fabric technology for any operator deploying 5G core functions and MEC (Multi-access Edge Computing) nodes. Understanding it is no longer optional for a 5G transport architect — the line between WAN transport and DC fabric has disappeared in modern deployments.

VXLAN (Virtual Extensible LAN) solves a fundamental problem in operator data centres: how to extend Layer 2 segments across a routed underlay network without the spanning tree limitations of traditional VLANs. It encapsulates Ethernet frames in UDP/IP packets, creating a logical overlay that can span multiple physical locations. Each overlay segment is identified by a 24-bit VNI (VXLAN Network Identifier) — giving you over 16 million possible segments, compared to 4094 VLANs.

EVPN (Ethernet VPN, RFC 7432) is the control plane that makes VXLAN scalable. Instead of flooding to discover MAC addresses across VXLAN tunnels (which kills performance), EVPN uses BGP to distribute MAC and IP reachability information between VTEP (VXLAN Tunnel Endpoint) devices. This means a workload migration in a 5GC UPF cluster or a new VNF (Virtual Network Function) instantiation is automatically discovered by all relevant VTEPs without any manual configuration or flooding.

2. Real Network Architecture — Underlay and Overlay

The Two-Layer Model

Layer	Technology	Purpose	Protocols
Underlay (physical fabric)	IP-routed Clos/spine-leaf topology	Provides IP connectivity between all VTEPs — pure IP forwarding	IS-IS or OSPF for IGP, ECMP across all spine links, BFD for fast failure detection
Overlay (logical fabric)	VXLAN tunnels between VTEPs	Provides L2/L3 virtual networks for 5GC and MEC workloads	BGP EVPN for MAC/IP advertisement, VXLAN encapsulation for data plane

The underlay does not care about tenants, VNFs, or 5G interfaces. It just provides IP connectivity between leaf switches (VTEPs) via the spine. ECMP distributes traffic across all available spine links — a typical operator DC uses 2-4 spine switches for redundancy. The overlay handles all the tenant separation, L2 extension, and service chaining.

Spine-Leaf Topology for 5G Core DC

A typical 5GC DC uses a spine-leaf (Clos) fabric:

Leaf switches — connect to servers hosting VNFs (AMF, SMF, UPF, PCF etc.) and to the WAN border routers (which connect to the transport network). Each leaf is a VTEP — it originates and terminates VXLAN tunnels.
Spine switches — provide pure L3 transit between leaves. No VTEPs on spine in most modern designs. They run IS-IS/OSPF and ECMP — nothing else. Simplicity on the spine is intentional.
Border leaves — a dedicated pair of leaf switches that connect the DC fabric to the WAN/transport network. They run both EVPN-VXLAN (toward the fabric) and MPLS/SR (toward the WAN). They perform the hand-off between overlay and underlay transport domains.

Field Note: In a GCC operator deploying a 5GC DC in Muscat, the border leaf design was the most debated topic. The team initially proposed running VXLAN all the way to the WAN PE routers — which would have required VXLAN support on the PE. The better design, which they ultimately adopted, was to terminate VXLAN at the border leaf and hand off native IP (in the appropriate VRF) to the PE. This kept the WAN transport domain clean and separated DC fabric technology choices from WAN transport choices.

3. Step-by-Step Real Flow — MEC Deployment for Low-Latency Apps

Enterprise AR/VR Application at Muscat Ruwi MEC Node

An enterprise customer deploys an AR/VR application at a MEC node co-located with the 5G RAN aggregation hub at Ruwi. The application server runs as a VM on the MEC platform. The UPF is also deployed at Ruwi for local breakout. Here is how EVPN-VXLAN connects everything:

Step 1 — VNF Instantiation and MAC/IP Advertisement

The UPF VNF is instantiated on a server in the Ruwi MEC rack. The server connects to the leaf switch via a 25GE link. The leaf switch (acting as a VTEP) detects the new VM’s MAC address via ARP snooping or BGP EVPN Type-2 route from the hypervisor. It advertises this MAC/IP binding to all other VTEPs in the fabric via BGP EVPN.

Every other VTEP (border leaf, other MEC leaves, central DC leaves) now knows how to reach the UPF’s IP address — through the VXLAN tunnel to the Ruwi leaf VTEP. No flooding, no ARP storms. BGP handles MAC/IP distribution.

Step 2 — Enterprise Traffic Enters from gNB via N3

An enterprise UE sends data. The gNB CU-UP at Ruwi forwards GTP-U traffic to the local UPF via the N3 interface. The N3 path: gNB CU-UP → transport network (IP/MPLS backhaul) → border leaf at Ruwi MEC → VXLAN overlay → UPF VNF on MEC server.

The border leaf performs the domain hand-off: it receives the IP packet from the WAN (after MPLS label popping at the PE), looks up the destination UPF IP in its EVPN RIB, and encapsulates the packet in VXLAN with the appropriate VNI for the N3 service segment. It forwards the VXLAN packet to the leaf connected to the UPF server.

Step 3 — VXLAN Forwarding Within the Fabric

The spine-leaf fabric forwards the VXLAN-encapsulated packet via ECMP across spine links to the leaf where the UPF server is connected. The leaf decapsulates the VXLAN header, delivers the inner IP packet to the UPF server via the hypervisor vSwitch. The UPF processes the GTP-U packet, strips the tunnel, and forwards the application payload to the AR/VR server on the same MEC platform — potentially via another VXLAN segment within the same DC fabric.

Step 4 — Response Path and Symmetry

The AR/VR server responds. The packet goes: AR/VR server → hypervisor vSwitch → leaf VTEP → VXLAN encapsulation (using VNI for the N3 segment) → spine ECMP → border leaf → MPLS/IP toward gNB → GTP-U to CU-UP → UE. The symmetry of the path is important — EVPN ECMP ensures load balancing is consistent in both directions, preventing asymmetric forwarding that causes packet reordering.

Key insight: The UPF and the AR/VR application server never need to know about VXLAN. They see native IP interfaces. The overlay is completely transparent to the applications. This is the fundamental value of EVPN-VXLAN — it provides network virtualisation without requiring application awareness.

4. Practical Example — 5GC Multi-Tenant DC

A GCC operator hosts both its own 5GC functions and a B2B enterprise customer’s private 5G core in the same physical DC. EVPN-VXLAN provides complete isolation:

Tenant	VNI Range	VRF/L3VNI	Physical Isolation
Operator 5GC — N3 segment	VNI 10100-10199	L3VNI 10001 (operator_5GC_VRF)	None needed — logical isolation via BGP RT
Operator 5GC — N4 segment	VNI 10200-10299	L3VNI 10001 (same VRF, different L2VNI)	Intra-VRF routing — UPF to SMF traffic
Enterprise Private 5G — RAN	VNI 20100-20199	L3VNI 20001 (enterprise_VRF)	Separate RT — cannot reach operator VNFs
Enterprise Private 5G — Core	VNI 20200-20299	L3VNI 20001 (same enterprise VRF)	Completely isolated from operator core
MEC application platform	VNI 30100-30199	L3VNI 30001 (MEC_VRF)	Selective leaking to N6 only via border leaf policy

Each tenant has its own L3VNI (Layer 3 VNI — the VNI used for inter-subnet routing within a VRF), its own Route Target for BGP EVPN export/import, and complete logical isolation. A misconfigured VM in the enterprise tenant cannot reach the operator’s 5GC functions because the RT policies prevent route leakage between VRFs.

5. Key Parameters and Configurations

Parameter	Typical Value	What Goes Wrong If Misconfigured
VXLAN UDP port	4789 (IANA standard)	Using non-standard port breaks interoperability with hypervisors and firewalls
VNI assignment	L2VNI per segment, L3VNI per VRF	Duplicate VNI across tenants = traffic leakage between tenants
BGP EVPN Route Target	Operator-assigned per VRF (e.g. 65001:10001)	Wrong RT import/export = routes not distributed or leaked to wrong VRF
Underlay MTU	9150 bytes (9000 inner + 50 VXLAN overhead)	Fragmentation of VXLAN packets — throughput drop, asymmetric behaviour
ARP suppression	Enabled on all VTEPs	Disabled ARP suppression = ARP flooding across all VTEPs — DC-wide broadcast storm
VTEP IP source	Loopback address, advertised via IGP	Using physical interface IP = VTEP unreachable if interface goes down
BFD on underlay links	300ms/3x on all leaf-to-spine links	Slow failure detection = VXLAN traffic blackholed for seconds on link failure

6. Integration With the Transport Network — The Border Leaf Hand-off

How DC Fabric Connects to WAN MPLS

The border leaf is the critical integration point. It must bridge two worlds: EVPN-VXLAN overlay on the DC side and IP/MPLS VPN on the WAN side. The hand-off works as follows:

On the DC side: border leaf runs BGP EVPN, participates in the VXLAN overlay, and has L3VNIs for each tenant VRF. It knows the MAC/IP of every VNF in the DC via EVPN.
On the WAN side: border leaf connects to the PE router. It runs a separate BGP session with the PE for VPNv4/VPNv6 route exchange. Each DC VRF maps to a WAN VRF on the PE — same Route Target, so routes are automatically imported/exported.
Traffic hand-off: incoming WAN packet (MPLS-decapsulated at PE) arrives at border leaf as plain IP in the WAN VRF. Border leaf maps this to the DC VRF, performs EVPN lookup for the destination, encapsulates in VXLAN, and forwards into the fabric. Outgoing DC traffic: border leaf decapsulates VXLAN, performs WAN VRF routing, hands off to PE for MPLS encapsulation toward the RAN.

Design principle: Never run VXLAN all the way to the WAN PE router. Terminate VXLAN at the border leaf and hand off IP. This keeps the WAN transport domain simple and decouples DC technology choices from WAN equipment vendor choices. Mixing VXLAN and MPLS on the same device works technically but creates complex operational models.

7. Common Issues in the Field

MTU mismatch — the most frequent EVPN-VXLAN problem. VXLAN adds 50 bytes of overhead (outer IP 20 + UDP 8 + VXLAN 8 + inner Ethernet 14). If underlay links are not configured for jumbo frames (9150+ bytes), VXLAN packets are fragmented or silently dropped. Always configure MTU end-to-end before enabling VXLAN.
Duplicate VNI across tenants — in multi-tenant deployments, a provisioning error can assign the same VNI to two different VRFs. This causes cross-tenant traffic leakage that may not be immediately obvious. Implement automated VNI allocation with a centralised IPAM/VLAN management system.
ARP suppression not enabled — without ARP suppression, ARP requests are flooded to all VTEPs, consuming bandwidth and CPU across the entire DC fabric. In a large 5GC DC with hundreds of VNFs, this creates a measurable performance impact. Always enable ARP suppression.
BGP EVPN session flap — if the BGP session between a leaf and its route reflector (spine or dedicated RR) resets, all EVPN routes from that leaf are withdrawn. VNFs on that leaf become unreachable until BGP reconverges. Tune BGP BFD to detect session failures quickly and ensure route reflector redundancy.
Asymmetric routing on ECMP paths — if different spine links have different VTEP reachability information (e.g., one spine has stale BGP routes), VXLAN packets may take asymmetric paths. This causes VXLAN decapsulation failures on the receiving VTEP (which does not recognise the incoming VTEP as a known source). Ensure all spines have consistent EVPN routing tables.

8. Troubleshooting Approach

Check VTEP reachability first — from any leaf, ping the loopback address of the remote VTEP. If the underlay ping fails, VXLAN cannot function. Fix the IGP routing issue first.
Verify EVPN BGP routes — on any leaf, check that the MAC/IP routes for the target VNF are present in the EVPN RIB. Missing Type-2 routes mean the VNF’s leaf is not advertising correctly — check BGP session state and RT configuration.
Test VXLAN path with overlay ping — after verifying underlay, use VXLAN overlay ping (ping the VNF IP from within the same VRF on another leaf). This tests the complete VXLAN path including VNI lookup and decapsulation.
Check MTU across the path — use ‘ping size 8972’ (maximum payload before VXLAN overhead reaches 9000 bytes) from leaf to leaf. If this fails but smaller pings succeed, MTU is the problem.
Inspect VXLAN packet capture at leaf — a packet capture on the leaf’s uplink during a connectivity problem shows whether VXLAN packets are being generated, whether they carry the correct VNI, and whether the underlay destination IP matches the correct VTEP.

9. Design Recommendations — Consultant Level

Always use a dedicated loopback for VTEP source — never use a physical interface IP as the VTEP source. Physical interface failures would make the VTEP unreachable. A loopback advertised via IGP remains reachable as long as any path to the leaf exists.
Deploy route reflectors for BGP EVPN at scale — in a DC with 20+ leaf switches, full-mesh BGP between all VTEPs is unmanageable. Deploy dedicated route reflectors (typically on spine switches or dedicated RR VMs) to distribute EVPN routes centrally.
Implement VNI allocation governance from day one — use a centralised tool (NetBox, Nautobot, or custom IPAM) to allocate VNIs, L3VNIs, and Route Targets. Document every VNI assignment. Untracked VNI allocation in a multi-tenant environment is a security and operational risk.
Test ARP suppression and MAC mobility before go-live — ARP suppression eliminates flooding but requires all leaf switches to have correct MAC/IP bindings in their EVPN table. VNF migrations (MAC mobility) must be tested to confirm routes update correctly without traffic blackholing.
Design border leaf redundancy with MLAG or ESI — the border leaf is a single point of failure. Use MLAG (Multi-chassis Link Aggregation) or EVPN ESI (Ethernet Segment Identifier) for active-active border leaf redundancy. Test failover with live traffic before production.

10. Summary — Practical Takeaways

EVPN-VXLAN is the infrastructure that makes 5GC and MEC deployments operationally viable at scale. It provides multi-tenant isolation, scalable MAC/IP distribution without flooding, and seamless integration with the WAN MPLS domain via the border leaf hand-off. For any transport architect involved in 5G core or MEC design, EVPN-VXLAN is a mandatory competency.

The most common failure modes are MTU misconfiguration, duplicate VNI assignment, and missing ARP suppression. All three are preventable with proper design reviews and automated provisioning governance. The border leaf design — terminating VXLAN at the DC edge and handing off IP to the WAN PE — is the clean architecture that keeps both domains manageable.

Takeaway	Action
MTU must be 9150+ on underlay links	Configure jumbo frames on all spine-leaf links before enabling VXLAN
VNI allocation must be governed centrally	Use IPAM/CMDB for VNI assignment — no manual or ad-hoc allocation
Always enable ARP suppression on VTEPs	Prevent broadcast flooding — mandatory in any production EVPN-VXLAN DC
Terminate VXLAN at border leaf	Hand off IP to WAN PE — never run VXLAN on WAN PE routers
Test MAC mobility and border leaf failover	Verify VNF migration and border leaf redundancy under live traffic before production

EVPN-VXLAN in 5G Core and MEC Practical Deployment for Low-Latency Applications