A complete CCIE Service Provider mock lab has to cover two autonomous systems with different IGPs, MPLS transports, L2/L3 VPN services, SRv6, Flex-Algo, NG-mVPN, Inter-AS options, and CsC, and in general my idea is to try to fit as many features as possible into one topology, without restarting and rebuilding it. The main constraint, as usual, is physical resources: a limited amount of RAM and especially CPU.
This post covers a 25-node lab setup, a two-provider topology on a server with 128 GiB RAM.
The Topology
The baseline topology consists of two providers, SP-A (AS 64501) and SP-B (AS 64502), eight nodes in each. Three customers on top of them, nine nodes, which makes 25 in total.
Both providers have the same physical layout: four P routers in a ring with one diagonal across it, two PEs attached to the core, one route reflector, and one ASBR facing the other provider. The protocols are different – SP-A runs OSPF and LDP, SP-B runs IS-IS and segment routing.
Each customer has its own focus.
- CustA is L3VPN and multicast, with sites on both providers.
- CustB is L2VPN.
- CustC needs MPLS on its own CEs, because later it becomes a carrier inside SP-B for CsC.
A-ASBR and B-ASBR are connected by a single link, and all Inter-AS options are built across it.
topology.clab.yml · Addressing: ipam.md · Baselines: lab_configs/Technologies and platform specifics
The SPv5.1 blueprint requires hands-on exposure to features that are segregated by platform. Flex-Algo, SRv6 with uSID and IGW support, Tree-SID mcast, and NG-mVPN Profiles 27/28/29 are exclusive to IOS-XR. Everything else: IS-IS-SR with TI-LFA, classic LDP, EVPN in all its forms, Inter-AS Options A/B/C, CsC, mVPN Profiles 0-17 – runs equally well on IOS-XE.
Initially, I was thinking about a full-XR topology: 14 XRv9k nodes across both providers, but unfortunately the RAM requirements were too tight.
The RAM Probe
Before committing to any topology, I probed XRv9k:
6 GB: XRv9k boots the QEMU VM, but the rest of the things fall apart with repetitive OOM restarts.
10 GB: Boot survived to the Press RETURN prompt, but vrnetlab’s launch.py crashed with a UnicodeDecodeError while reading XR’s console output. XR was running inside QEMU, but containerlab never received the clab user or SSH access.
After attempting 12 GB and 14 GB decided to stick with 14, since it was also in the clab recommended parameters. However, it wasn’t allowing me to use XR-only topology.
The Platform Decision
The decision was to run XR only where SPv5.1 requires XR, and CSR1000v everywhere else. On paper, it is supposed to cover:
- SR-MPLS base with prefix-SIDs
- IS-IS-SR with TI-LFA (adj-SID + prefix-SID)
- EVPN: ELAN, VPWS, vlan-based, vlan-bundle, vlan-aware
- mVPN IPv4 address-family (the classic Rosen profile family)
- BGP-LU
send-labelfor CsC - Inter-AS Options A, B, C
The three features CSR plainly can’t do – SR-TE policy, Flex-Algo, SRv6 – all live in SP-B, so XR will be used on exactly three nodes there: B-P1 (XRv9k, central P-router), B-PE1 (XRv9k, SRv6 endpoint), and B-PE2 (XRv9k, SRv6 endpoint).
Final node allocation:
| Role | Image | Count | Per-node RAM | Total |
|---|---|---|---|---|
| XR transit + SRv6 path | XRv9k | 3 | 14 GiB | 42 GiB |
| SP infra + CustC CEs | CSR1000v | 14 | 4 GiB | 56 GiB |
| Route reflectors (A-RR, B-RR) | XRd-CP | 2 | 1 GiB | 2 GiB |
| Customer CEs (CustA, CustB) | IOL | 6 | 0.5 GiB | 3 GiB |
| Total | 25 | 103 GiB |
Host: 125 GiB usable of the 128 GiB installed → ~22 GiB headroom at steady state.
The Containerlab nuances
1. XRd rejects ethN link names
Every other kind (cisco_xrv9k, cisco_csr1000v, cisco_iol) uses <node>:ethN in the links: section of the topology YAML. cisco_xrd alone requires <node>:Gi0-0-0-N (dashes, 0-indexed). The error:
ERROR: Cisco XRd interface name "eth1" doesn't match the required pattern
The pattern is documented in containerlab’s XRd kind page, which I missed in the beginning.
2. Startup-config injection is platform-specific
Without any startup-config, all four platforms come up with a usable clab-mgmt VRF and SSH access. Provide a startup-config and behavior splits:
| Kind | With startup-config | Consequence |
|---|---|---|
cisco_xrv9k, cisco_csr1000v | vrnetlab merges default mgmt + SSH on top of your config | No action needed — baselines can be data-plane only |
cisco_xrd | mgmt interface preserved, but no default user, SSH + vty dropped | Baseline must include username clab secret 10 <sha512> + line default / transport input ssh + ssh server vrf clab-mgmt |
cisco_iol | full replace — your config overwrites the entire default template including hostname, user, SSH, crypto, VRF, routes | Use a filename containing .partial → clab appends to its iol.cfg.tmpl instead of replacing it |
3. IOL interface numbering offset
IOL maps eth0 → Ethernet0/0 reserved for management. Data interfaces start at Ethernet0/1. IOL containers reached healthy status but remained unreachable over the data plane.
Managing the Memory Spike
Booting 25 nodes simultaneously would exceed available RAM during the spike phase. CSR1000v consumes roughly 4 GiB at steady state but can spike to 6+ GiB during boot. Added startup delay to ease the process.
| Kind | startup-delay: | Reasoning |
|---|---|---|
cisco_iol | 0 s | Fast boot |
cisco_xrd | 30 s | 1-2 min boot, 1 GiB each – done before CSRs |
cisco_csr1000v | 300 s | 14 nodes spike ~73 GiB peak |
cisco_xrv9k | 600 s | 3 nodes × 14 GiB spike ~55 GiB |
Full lab should be ready in about 22-25 minutes.
Swap is disabled on the host to avoid its usage, since XR devices can easily fall off into a swap, and this can cause unpredictable behavior on the control-plane level.
The Baseline Configs
Every node’s baseline is IP-only: hostname, Loopback0 with both v4 (10.x.0.N/32) and v6 (2001:DB8:x::N/128), data interfaces with /30 + /64 pairs, and the clab-mgmt VRF.
This will be a foundation layer where I’m planning to add the other layers of the configuration.
An excerpt from A-P1. All 25 baselines and the topology file are in the ccie_sp_lab repo:
hostname A-P1!ipv6 unicast-routing!interface Loopback0 ip address 10.1.0.1 255.255.255.255 ipv6 address 2001:DB8:1::1/128!interface GigabitEthernet2 description to A-P2 ip address 10.1.1.1 255.255.255.252 ipv6 address 2001:DB8:1:10::1/64 no shutdown
At baseline, show ip interface brief on any node shows only directly connected prefixes and Loopback0. The RIB is empty. Everything else is built from there.
What Each Provider Runs
SP-A (AS 64501) – the legacy style. Seven CSR1000v nodes plus one XRd-CP route reflector. OSPFv2 + OSPFv3, multi-area. LDP transport. Classic L3VPN, VPWS, VPLS, mVPN Profiles 0/11/13/14.
SP-B (AS 64502) – the modern style. Three XRv9k nodes, four CSR1000v, one XRd-CP (RR + SR-PCE). IS-IS Level-2 only, dual-stack with wide metrics. The service plan: SR-MPLS, SR-TE, Flex-Algo, SRv6 with uSID, EVPN-VPWS multi-homed, etc.
The Inter-AS link between A-ASBR and B-ASBR is classic only – nothing overly complicated across the boundary. The labs exercise Options A, B, and C across that link, plus CsC and VPNv4/v6. Will try to mirror how real operators connect SP domains that evolved independently.
Next steps
The topology file, the IPAM, and all 25 baseline configs are on GitHub: maugli13/ccie_sp_lab.
With the topology deployed and all 25 nodes accessible over SSH from the lab host, the next step is IGP Foundation. SP-A gets dual-stack OSPF. SP-B gets dual-stack IS-IS. Then optimizations that turn working IGPs into production-grade ones.
Yet another Blog 
Leave a comment