T3 Scale Design: Take Two
4 min read

T3 Scale Design: Take Two

A quiet mind

After I hit publish on my blog post last night, I went to bed. While laying there, a simpler way to implement my table-policy design suddenly came to me; One that eliminates the need of a L1/L2 ABR.

What I came up with was a hierarchical Route-Reflector design. Where the T1 routers, that border the two rings, act as RR's for the T3 routers. They would also originate a default route for the routers to use, instead of the IS-IS attached-bit.

Configuration

First, we had to restore the Level-2 Topology to both rings.

RP/0/0/CPU0:XRv-9#show isis topology 


IS-IS zeal paths to IPv4 Unicast (Level-2) routers
System Id       Metric  Next-Hop        Interface       SNPA          
XRv-1           30      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-2           20      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-3           20      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-4           30      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-5           40      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-6           40      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-7           10      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-8           10      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-9           --    
RR-1            50      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
RR-1            50      XRv-8           Gi0/0/0/0       0ca7.981d.0002
RP/0/0/CPU0:XRv-9#

Next, we establish IPv4-unicast sessions between the Ring Border Routers & the Tier 3 PE(s). We also need to add the config of default-originate to ensure that they specifically advertise a default route to this PE.

RP/0/0/CPU0:XRv-2#show run formal router bgp | i .9

router bgp 6275 neighbor 10.255.255.9 
router bgp 6275 neighbor 10.255.255.9 remote-as 6275
router bgp 6275 neighbor 10.255.255.9 update-source Loopback0
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast 
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS in
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-reflector-client
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS out
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast default-originate
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast soft-reconfiguration inbound always
RP/0/0/CPU0:XRv-2#
RP/0/0/CPU0:XRv-3#show run formal router bgp | i .9

router bgp 6275 neighbor 10.255.255.9 
router bgp 6275 neighbor 10.255.255.9 remote-as 6275
router bgp 6275 neighbor 10.255.255.9 update-source Loopback0
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast 
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS in
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-reflector-client
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS out
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast default-originate
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast soft-reconfiguration inbound always
RP/0/0/CPU0:XRv-3#

Great, so XRv-9 is now learning a default route. But the problem is that – by default – iBGP chooses the best possible single path. To ensure we're using both links on XRv-9, we need to enable multipathing. We also need to add the default route to our table policy to ensure it's installed in the FIB.

RP/0/0/CPU0:XRv-9#show rpl route-policy TBL_PLCY detail 

prefix-set TBL
  0.0.0.0/0,
  203.0.113.0/24 le 32,
  25.54.60.0/24 le 32,
  8.8.8.0/24 le 32
end-set
!
route-policy TBL_PLCY
  if destination in TBL then
    pass
  else
    drop
  endif
end-policy
!
RP/0/0/CPU0:XRv-9#
RP/0/0/CPU0:XRv-9#show run router bgp 6275 address-family ipv4 unicast 

router bgp 6275
 address-family ipv4 unicast
  table-policy TBL_PLCY
  maximum-paths ibgp 2
 !
!

RP/0/0/CPU0:XRv-9#

Now the prefix should be installed & ECMP aware.

RP/0/0/CPU0:XRv-9#show route bgp   


B*   0.0.0.0/0 [200/0] via 10.255.255.3, 00:54:33
               [200/0] via 10.255.255.2, 00:54:33
B    8.8.8.0/24 [200/0] via 10.255.255.1, 00:56:37
B    25.54.60.0/24 [20/0] via 203.0.113.6, 00:56:37
RP/0/0/CPU0:XRv-9#    

Conclusion

As usual, the best way to find the truth is by saying the wrong answer aloud.

This design is much more scalable and requires far less impact to a greenfield addition to a brownfield network. It still meets the main requirement – reducing IPv4 / IPv6 unicast FIB entries due to low prefix / ECMP scale. The Tier-3 routers still peer with the main Route Reflector for address-families such as l2vpn evpn, for L3VPN & L2VPN services.

RP/0/0/CPU0:XRv-9#show bgp l2vpn evpn summary  | b Spk      

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
10.255.255.10     0  6275     101      98        1    0    0 01:03:58          0

RP/0/0/CPU0:XRv-9#

Be sure to do this once in a while; Remember that perfect is the enemy of good. Go for the best design you can come up with at the moment. Socialize it around, sleep on it, and give it time. The topology is out there.