The Before Times

For a long time, I was using my trusty ER-X as my main router on my LAN.

For it's era, it was nearly perfect:

• Very low price tag
• Debian underpinnings that are easily exposed
• Really great CLI (due to the Vyatta fork) & okay GUI
• Hardware Switch for line-rate (1Gbps) switching between ports
• 24v Passive PoE with passthrough to power a Unifi AP with a single adapter
• All the good EdgeOS features (BGP, OSPF, IPSec, OSPF)
• Passive cooling

So, for a long time, my LAN looked like this:

I used my Odroid SBC as a Wireguard server (among other things, like syncthing & git). This allowed me to connect in, via the web, and access my LAN, the internet from my LAN, and also remote networks via always-on IPSec tunnels.

This allowed me to use JuiceSSH & Wireguard to remotely diagnose a DBIUA routing issue, while riding on a bullet train in Germany.

New Beginnings

Well, all good things come to an end and I moved to an apartment in a brand new purpose-built rental building. The big up-side was getting fiber to the unit internet service.

The big downside was that the MIPs chip in my ER-X can only handle about ~300 Mbps NAT offload. So, instead of bridging the provider's router, I decided to stick with it and let my ER-X collect some dust.

Aside from some yuck DHCP DORA wait-times, which was easy enough to solve with static addressing, the provider's router & separate AP wasn't bad at all.

The provider's portal even offered a lovely port-forwarding option, which allowed me to re-connect my VPN server.

Someday Always Comes

After many days of enjoying things as is, I found myself wanting my site-to-site IPSec tunnels back. I knew I could use Nebula attached routes, but I just didn't trust the little device I had on the LAN. I also preferred to keep my Nebula a safe overlay mesh, from host-to-host, with zero-trust policies.

I also wanted to use the ER-X as a 5 port desk switch. I intend to get a Minisforum desk server in the near future, so it will need a port.

So, as a temporary measure, I plugged my ER-X as-is and the network looked like this:

From the above, you should gather:

• My workstation was behind a double NAT (yuck)
• I would have to modify routes policies and routes both ways to allow connectivity between my VPN server & my ER-LAN / remote tunneled networks
• My Bandwidth was now limited :(

It Must Be Done

After pulling some large arch updates, I was annoyed to see my speed capped on my beautiful fiber connection. So I finally made the conversion.

Now my LAN had the following characteristics:

• I could pass traffic at 1Gbps line-rate to my LAN and beyond
• I attached my VPN tunnels to the SVI address of my ERX switch
• I utilized static routes to the SVI to reach the remote networks over my IPsec tunnels

basic config:
------
me@ERX:~$ show configuration commands | match switch0 | match inter
set interfaces switch switch0 address 10.88.99.235/24
set interfaces switch switch0 mtu 1500
set interfaces switch switch0 switch-port interface eth0
set interfaces switch switch0 switch-port interface eth1
set interfaces switch switch0 switch-port interface eth2
set interfaces switch switch0 switch-port interface eth3
set interfaces switch switch0 switch-port interface eth4
set interfaces switch switch0 switch-port vlan-aware disable
set vpn ipsec ipsec-interfaces interface switch0
me@ERX:~$ 

workstation routes:
------

[me@workstation ~]$ ip route | grep -E "def|99.23"
default via 10.88.99.254 dev enp4s0 proto static metric 100 
10.0.0.0/11 via 10.88.99.235 dev enp4s0 proto static metric 100 
10.10.0.0/23 via 10.88.99.235 dev enp4s0 proto static metric 100 
172.16.0.0/19 via 10.88.99.235 dev enp4s0 proto static metric 100 
[me@workstation ~]$ 

In the world of cattle vs pets, I sure did miss my little buddy.

The next-stage in the upgrade is to get an x86 2x 2.5Gbe router (my internet connection is actually 2Gbps down, 1Gbps up) . I will then upgrade my gateway to VyOS and use my ER-X as simply a managed switch.

Until then, I'm happy with this silly setup.

https://www.hpe.com/us/en/newsroom/press-release/2024/01/hpe-to-acquire-juniper-networks-to-accelerate-ai-driven-innovation.html

Re-route

It's a rather eerie coincidence that I sat down to write this post a year to the date of my post about waffling about CCIE / JNCIE . The waffling continued; However, I had done some reflection.

That reflection concluded that I've met/heard of so many Paper CCIE's but I've never met - nor heard of - a Paper JNCIE. In fact, all of the people I've met in my career, that have a JNCIE, have been excellent, if not superb. Looking at you Yufeng, Randy, Dakota, etc.

That made me think about pursuing the JNCIE this year. Especially given that it's practically free:

I think Juniper will keep this program going for the duration it would take me to hit JNCIE; However, I'm concerned about the value of Juniper branding, considering what's happened with other products swallowed by HP.

Just happy I didn't list it in my goals.

The Platform

Along with many of my colleagues, I have a lot of love for Juniper & JunOS.

• It feels like a platform made for operators that live in the terminal and actually solve problems.
• It has dedicated commands for verbose, terse & detailed outputs. Pattern matching & configuration layout has a great hierarchy
• It supports commit-based configuration on all platforms
• The information output is well structured, as it litters onto your screen.
• The feature support is often top-notch (automation, slicing, protocols, etc).
• The documentation has been relatively great
• Unshy about the FreeBSD underpinnings (or Linux with Junos Evolved).

In the argument of Pets vs Cattle, it's a treasured pet.

Fond Memories

I first truly worked with Juniper back in September of 2015, when I started working for a DC / MSP company. One side of our core was Juniper (EX & MX platform) along with a new offering featuring SRX boxes.

That exposure only increased as I migrated to working for an ISP. I have such fond memories on creating documentation for VLAN-Aware VPWS's (E-LINE) for a migration to QFX switches.

I also have fond memories tracing an obscure VRF across the country across several MX's and coming up with a strategy to absorb it into our Multi-tenant Private cloud solution, with overly complex stitching & route-policies.

Then there was that time I had 3-5 SRX's, two EX4200's in my employee co-location & this M20 sucking all that free electricity in my old apartment:

In general, I simply had a great time with Juniper; My love of Juniper is what made me favour the shells of VyOS & IOS-XR.

Looking Forward

I may still try to rush out the JNCIE. I think it could push me to do a lot of the studying I've been holding back on in the past few years; However, I don't think I'd consider it a ticket to the next stage in my career - I think that title is reserved for becoming a better coder / NetDev.

I also acknowledge that HP swallowing Juniper will be a slow process. Their ISP side still fills a gap that HP currently can't.

In the mean time, there's lots of discussion out there. Here's one I quite enjoyed from my RSS feeds:

HPE Buys Juniper

Personally Notable Achievements

The first thing I'd like to highlight of the things I did well & consistently:

• Quit centralized platforms and joined the Fediverse (Mastodon, Lemmy). Take that, enshitternet.
• I grew my RSS reader from ~20 feeds (mostly spammy news) to 86 feeds I actually enjoy reading. The small web is back. I'm sure this will grow in 2024
• I joined several mailing lists (GTALUG, NANOG) and forums (Practical ZFS & several Lemmy communities). ** still looking to meet a lobster to send me an invite **
• I've begun the habit of not being a passive viewer. I started emailing folks and - to my surprise - folks email back; This is the way the web should be.
• I took of the guard rails at work and allowed myself to stand out, despite the pitfalls of stepping on toes and being othered.
• I broke away from a Network / Linux focus. I've dabbeled more into web-development / CompSci / general coding / Systems Engineering, History (mainly Medieval / modern), and Tech Policy or privacy advocation.
• Hacked back distraction by using ad-free / purpose built services. Redefined how I Timeblock. I also used services like Distraction Free Youtube to end the rabbit holes.

Some Noteable Highlights

One of the things I did this year was take more of a focus on mentoring / guidance. I've provided & received as much insight online as I could – while trying not to be a reply guy. I've also taken on this mindset at work; I've really tried to being an encouraging guide to my colleagues. I just have to add, unfortunately, that I believe this is often negated by my work environment itself.

Another great great highlight was combining NANOG88 with a visit to my dear colleagues in Seattle. NANOG 88 had many great sessions and discussions. Some of those discussions inspired some of my projects for 2023 and beyond. I also enjoyed my time in Seattle. It's such a laid back and easy city for me to enjoy.

As for DBIUA / Orcas Island; I always have a wonderful time visiting my dear friends' slice of paradise. The deer population was down, but we got to see Orcas this time – on my colleagues pursuit boat, to boot. If my aspirations were different, I really wish I could settle down on Vancouver Island and be a stone's throw away from such a wonderful place.

I'm so grateful for the many accomodations the DBIUA team / community made for me. I hope to reward this with great projects.

The final thing I'd like to note is that I see the elements in which I have made change at work. Sadly, there was a recent incident where a seniour colleague of mine expressed that he didn't see the change, by making light of my tenure; However, I know better. It's really hard to be a beacon of change, when you're in an environment that is so resistant to it.

Nevertheless, I press on with a project I started in 2023, which I feel fills a very large void in our MTTR. I am proud as it is challenging me on all fronts of my knowledge (automation, reverse engineering, systems deployment, project management and gathering influence). I hope this project is successful in 2024 as it very well be my magnum opus / final endeavour, within this organization.

In summary, I'm happy to look deeper into the things I can personally bring to my org; However, I feel more and more as a Persona non Grata.

On a more personal non-career note, I also finally made it to Germany; This further solidified my itch to uproot my skills and export them to Europe.

Looking on to 2024

There's much to say about the remaining 358 days of the year. I will keep it short and sweet with some of my goals (these are not resolutions).

• Attend 3 conferences (I'm hoping for 2x NANOG & 1x another conference, such as DENOG)
• Speak at at least one of those conferences (lightning talk counts, but I'd prefer a full session)
• Contribute to a project I adore, such as Netbox or VyOS
• Produce a series of free guides, in the interest of making free stuff (a la Max Bock). Could be blog, could be video, could be both.
• Continue connecting with tech folks far and wide. One shouldn't be limited to their local tribe.
• Really take a deep dive into coding. A big stretch goal in this is to take on WebDev as a new hobby (with such great tooling and RSS feeds, it'd be a shame not to draw on such resources).
• Read as much as I can. Write as much as I can (be it journaling or on this blog).
• Read/Review five+ textbooks. I'll try to post back on these with my notes.
• Continue home projects / self-hosting. I moved in 2023, which allows me to get a fiber connection (2Gbps down, 1GBps) up. There is much to explore in 2.5Gbe, mirroring, filling up my patch closet with low-power servers.
• As per the above, I'd like to deploy more services. The fediverse has interesting projects (Pixelfeed, Peertube and Owncast). There's also front-ends like Individuous that will allow me to stop fighting Youtube Ad-block arms race and simply pull the content, ad/tracker/distraction free.
• Fill up my Bookmark manager (newly deployed) with notes & as many interesting articles as I can. Gone are the days of saving something in Telegram, only to struggle finding it later on. I'm using Linkding for this (thanks Tooters for your recommendations!).
• Do more for DBIUA. I believe some of our limitations can be undone – it just requires a little keyboard lube.

Cheers to 2024!

Bonus Thought

Instead of a year-end link dump, I think it's fair to say that 2023 was the beginning of the end for platforms. Xitter (pronounced Shitter) effectively died and many others are dying with it (Substack, etc.). It's perhaps time that we, once again, consider the web as an end-to-end service, rather than a select few private "Town Squares".

It's also been the rise of AI (or plausible sentence generators, as Doctorow would call them – being as "they're neither artificial or intelligent"). This is causing the world of SEO spam/low-effort garbage "content" to explode at a level we never fully imagined.

So, if you read anything this year, read this:

The Expanding Dark Forest and Generative AI

Proving you’re a human on a web flooded with generative AI content

Maggie AppletonLuciano Strika

Maggie nails it, especially with her diagram.

I'd like to expand upon this in a formal post, but I think 2024 will see a further pilgrimage of tech folk into the fediverse, small web and cozy web. I also believe that normal folk will continue to gather in the cozy web, leaving "platforms" and meeting their friends in digital living rooms.

I think we're at the start of something beautiful or quite horrific. I suppose it depends on your motives and your personal perspective.

When converted Zeal to a blog, I intended to write a lot more. I wanted to blog the various things I'm working on. Sadly, I've worked on many things I haven't blogged. I spend too much time making my posts "complete" , leaving too many posts in draft status.

So I intend to fix that... I am going to treat this blog more like technical notes.

I'm also going to take notes from these two gents:

Doctorow's Memex Method:

The Memex Method – Pluralistic: Daily links from Cory Doctorow

Pluralistic: Daily links from Cory Doctorow No trackers, no ads. Black type, white background. Privacy policy: we don't collect or retain any data at all ever period.Cory Doctorow

Nielson's method:

Things Learned Blogging

Writing about the big beautiful mess that is making things for the world wide web.

Jim Nielsen’s Blog Verified ($10/year for the domain)Jim Nielsen

This is likely my last post on Network Tracing. I just can't help myself, as recently I had a rather interesting case of where traceroute can be deceiving.

A user reported an issue where they experienced consistent small amounts of packet loss on their L2VPN LAN Extension. The issue started a few days ago and was still in effect. They provided proof via a ping, such as this:

patientuser@host:~$ ping 10.20.20.20 -c 50 | egrep -i "trans|rtt"
50 packets transmitted, 45 received, 10% packet loss, time 49187ms
rtt min/avg/max/mdev = 6.774/8.001/13.354/1.058 ms
patientuser@host:~$ 

My colleague decided to test it out. At first, the ping looked clean.... but uping the amount of echoes reproduced the loss.

RP/0/0/CPU0:XRv-1#ping 10.255.255.10 so lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.255.255.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#ping 10.255.255.10 so lo0 count 100 

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.255.255.10, timeout is 2 seconds:
!!!!!!!.!!!!!!!!!!!!!!!!!...!!!!!!!!!!!!!!.!!!!!!!!!.!!!!!!!!!!!!!!!!!
!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!
Success rate is 92 percent (92/100), round-trip min/avg/max = 1/5/9 ms
RP/0/0/CPU0:XRv-1#

So the loss seemed to be somewhere between the ingress PE and the egress as it traverses to the egress PE. He also confirmed that it was showing in MPLS OAM testing. He then called me over for some shoulder surfing support.

Spot The Problem

A quick traceroute, to determine where the issue was... across a few tests, things to go awry around hop 3:

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0         


Type escape sequence to abort.
Tracing the route to 10.255.255.10

 1  10.1.2.2 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 2  10.2.7.7 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 3  10.6.7.6 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  *   <---- here
 4  10.6.9.9 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 5  10.9.10.10 0 msec  *  0 msec 
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0 


Type escape sequence to abort.
Tracing the route to 10.255.255.10

 1  10.1.2.2 [MPLS: Label 16010 Exp 0] 9 msec  0 msec  0 msec 
 2  10.2.7.7 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 3  10.6.7.6 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 4  10.6.9.9 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  *   <---- hop 4 
 5  10.9.10.10 0 msec  *  0 msec 
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0 


Type escape sequence to abort.
Tracing the route to 10.255.255.10

 1  10.1.2.2 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 2  10.2.7.7 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 3  10.6.7.6 [MPLS: Label 16010 Exp 0] 0 msec  *  0 msec    <----- hop 3
 4  10.6.9.9 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
 5  10.9.10.10 0 msec  *  0 msec 
RP/0/0/CPU0:XRv-1#

In my lab, it's quite obvious where this PtP interface is between (router 6 & 7), but in our production... it's more arbitrary so we queried the LSDB:

RP/0/0/CPU0:XRv-1#show ip route 10.6.7.6 | i entry

Routing entry for 10.6.7.0/24
RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#show isis database detail | i "Host|10.6.7.0/24" | utility e$

  Hostname:       XRv-6
  Metric: 500        IP-Extended 10.6.7.0/24
--
  Hostname:       XRv-7
  Metric: 10         IP-Extended 10.6.7.0/24
RP/0/0/CPU0:XRv-1#

We then hopped to that router to test the ingress/egress adjacency:

RP/0/0/CPU0:XRv-7#show isis adjacency | i v-6

XRv-6          Gi0/0/0/3        0c83.0a4f.0006 Up    21   00:21:31 Yes None None
RP/0/0/CPU0:XRv-7#show arp | i 0/0/0/3

10.6.7.6        00:21:36   0c83.0a4f.0006  Dynamic    ARPA  GigabitEthernet0/0/0/3
10.6.7.7        -          0cf0.832b.0004  Interface  ARPA  GigabitEthernet0/0/0/3
RP/0/0/CPU0:XRv-7#ping 10.6.7.6 count 500 size 1472 donotfrag 

Type escape sequence to abort.
Sending 500, 1472-byte ICMP Echos to 10.6.7.6, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 100 percent (500/500), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-7#

Clean as a whistle... hmmm... we continued on & checked hop 4, as we saw something there:

RP/0/0/CPU0:XRv-6#show isis adjacency | i v-9

XRv-9          Gi0/0/0/3        0c06.7e75.0002 Up    26   00:22:44 Yes None None
RP/0/0/CPU0:XRv-6#show arp | i 0/0/3

10.6.9.6        -          0c83.0a4f.0004  Interface  ARPA  GigabitEthernet0/0/0/3
10.6.9.9        00:22:48   0c06.7e75.0002  Dynamic    ARPA  GigabitEthernet0/0/0/3
RP/0/0/CPU0:XRv-6#ping 10.6.9.9 count 500 size 1472 donotfrag 

Type escape sequence to abort.
Sending 500, 1472-byte ICMP Echos to 10.6.9.9, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 100 percent (500/500), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-6#

Also clean. To ensure we were in the right direction, we pinged back towards the ingress PE and also towards the egress PE. Sure enough, we were getting closer:

RP/0/0/CPU0:XRv-6#ping 10.255.255.10 count 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.255.255.10, timeout is 2 seconds:
!.!!!!!.!.!.!!!!!!!!!!!!!!!.!!!!!!!!!.!.
Success rate is 82 percent (33/40), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-6#ping 10.255.255.1 count 100 

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.255.255.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/2/19 ms
RP/0/0/CPU0:XRv-6#

We verified hop 5 (there were 6 hops in production) and moved onto the egress PE itself. Sure enough, we spotted the loss on a particular ingress interface:

RP/0/0/CPU0:XRv-10#show arp | i Dyna

10.5.10.5       00:24:36   0cde.5dc8.0004  Dynamic    ARPA  GigabitEthernet0/0/0/0
10.9.10.9       00:24:48   0c06.7e75.0004  Dynamic    ARPA  GigabitEthernet0/0/0/1
RP/0/0/CPU0:XRv-10#ping 10.9.10.9 count 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.9.10.9, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-10#


RP/0/0/CPU0:XRv-10#ping 10.5.10.5 count 100          

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.5.10.5, timeout is 2 seconds:
!!!!!!!.!!!!!!!!!!!!!!!!.!!!!!!.!!!.!!!!!!!.!!!!!.!!!!!.!!!!!!!!!!!.!!
!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!
Success rate is 90 percent (90/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-10#

What the FEC?

So, if you look closer at the beginning traces.... the issue seems to be more apparent at hop 5:

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0         

<snip>
 5  10.9.10.10 0 msec  *  0 msec 
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0 

<snip>
 5  10.9.10.10 0 msec  *  0 msec 
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0 

<snip>
 5  10.9.10.10 0 msec  *  0 msec 
RP/0/0/CPU0:XRv-1#

Suddenly it dawned on me as to why. As indicatd in the following RFCs: [3032] && [4950] , when TTL expires in an MPLS label in transit... the ingress LSR will inspect the underlying IP header, if applicable. It will utilize the Source IP in the header & deliver an ICMP TTL Exceeded (ICMP type 11, code 0) message to the source of the packet, now as the destination, with it's ingress interface as the source.

A potential problem is that the LSR may not know how to reach the source of the packet at all. In fact, the LSR may have almost no knowledge of anything outside of the MPLS core. Since the PE's likely have an idea of the VRF/L2 instance, the LSR will utilize the outer/transit label & re-generate a fresh packet towards the Z-end PE that the original transit label was targetting. Once the Z-End PE receives the payload, it will then forward it back over the core towards the source – as it knows exactly how to reach the node.

That means that many of the ICMP TTL expiry messages were being tunneled via the problematic Line-Card with the packet loss.

Captures Or It Didn't Happen

I've created some packet captures in my lab to demonstrate this:

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0 minttl 2 maxttl 2


Type escape sequence to abort.
Tracing the route to 10.255.255.10

 2  10.2.7.7 [MPLS: Label 16010 Exp 0] 0 msec  0 msec  0 msec 
RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0 minttl 2 maxttl 2


Type escape sequence to abort.
Tracing the route to 10.255.255.10

 2  10.2.7.7 [MPLS: Label 16010 Exp 0] 9 msec  0 msec  * 
RP/0/0/CPU0:XRv-1#

Router 7 generated an ICMP expiry to Router 1 & sourced it from it's ingress interface facing router 2

PHP is performed as the packet comes ingress into Router 10 and Router 10 is able to determine the best way to reach the destination, Router 1, via the core:

RP/0/0/CPU0:XRv-10#
RP/0/0/CPU0:XRv-10#show isis adjacency | i v-5

XRv-5          Gi0/0/0/0        0cde.5dc8.0004 Up    7    00:41:55 Yes None None
RP/0/0/CPU0:XRv-10#show interfaces gigabitEthernet 0/0/0/0 | i bia

  Hardware is GigabitEthernet, address is 0cd9.4a8c.0001 (bia 0cd9.4a8c.0001)
RP/0/0/CPU0:XRv-10#

Finally, Router 10 sends on the ICMP payload with the transit label of Router 1 (in this case, it's Node-SID):

Traceroute will then see the ICMP encapsulated headers/Datagram data and understand that it's for the relevant probe @ Hop 2.

Yet another reason why Traceroute is weird and sometimes it's best to roll up the sleeves and just verify hop by hop.

You've probably been there at least once. You're in a maintenance window. You make a change and the router is suddenly unreachable. You wait and wait, but it does not come back. You finally accept defeat & start the process for a ride-in, a DC tech, a truck roll, etc., etc.

It's not a good feeling. But, surely, it can be prevented. There are typically two ways this occurs:

1. The firmware/software upgrade.

Let's say you staged this upgrade – you should've, if you could've, anyway – but it breaks it production. Just another one for the Network Voodoo... move-on and laugh it off.

You can also use it as a lame excuse to buy out your competitors.

2. A config change....

This is one you really should try your best to prevent. Lab your topology changes as much as possible. Have strong peer review processes... but for everything else, there's another way.

Podcast Feedback

In 2021 I submitted this feedback to one of my favourite podcasts, 2.5 Admin, in regards to a fail-safe mechanism for most platforms:

2.5 Admins 60: Butter Fingers – 2.5 Admins

2.5 AdminsJoe Ressington

Joe, Jim & Alan do great work with this podcast. If you're a Network Professional, you should expand your horizons and tune-in.

The Mechanism

This is often called a confirmed commit. You make a change on a device. The device waits a set period, expecting you to confirm that things are alright. If no confirmation occurs, the device will perform the necessary steps to roll-back the configuration.

Let's look at an example from the main platform this is famous from, JunOS:

root@BB6> configure 
Entering configuration mode

[edit]
root@BB6# delete interfaces ge-0/0/0 

[edit]
root@BB6# show | compare 
[edit interfaces]
-   ge-0/0/0 {
-       description "PRODUCTION BREAKING CHANGE!!";
-       unit 0 {
-           family inet {
-               address 10.4.6.1/31;
-           }
-           family iso;
-           family mpls;
-       }
-   }

[edit]
root@BB6# commit confirmed 1  
commit confirmed will be automatically rolled back in 1 minutes unless confirmed
commit complete

# commit confirmed will be rolled back in 1 minute
[edit]
root@BB6#

Let's say that, as the description PRODUCTION BREAKING CHANGE!! implied, this breaks production and locks you out of the box. The method will save you and automatically rollback the change:

[edit]
                                                                               
Broadcast Message from root@BB6                                                
        (no tty) at 0:56 UTC...                                                
                                                                               
Commit was not confirmed; automatic rollback complete.                                                                               


[edit]
root@BB6# exit 
Exiting configuration mode

root@BB6> show configuration | display set | match 0/0/0 
set interfaces ge-0/0/0 description "PRODUCTION BREAKING CHANGE!!"
set interfaces ge-0/0/0 unit 0 family inet address 10.4.6.1/31
set interfaces ge-0/0/0 unit 0 family iso
set interfaces ge-0/0/0 unit 0 family mpls
set protocols mpls interface ge-0/0/0.0
set protocols isis interface ge-0/0/0.0 point-to-point

root@BB6> 

This convention also exists on IOS-XR:

RP/0/0/CPU0:BB4#configure 

RP/0/0/CPU0:BB4(config)#no router isis BB
RP/0/0/CPU0:BB4(config)#show commit changes diff 

Building configuration...
!! IOS XR Configuration 6.1.3
-  router isis BB
-   apply-group GROUP_TILFA
-   is-type level-2-only
-   net 49.0000.0000.0000.0004.00
-   address-family ipv4 unicast
-    metric-style wide
-    segment-routing mpls sr-prefer
    !
-   interface Loopback0
-    passive
-    address-family ipv4 unicast
-     prefix-sid index 4
     !
    !
-   interface GigabitEthernet0/0/0/0
-    point-to-point
-    address-family ipv4 unicast
    !
-   interface GigabitEthernet0/0/0/2
-    point-to-point
-    address-family ipv4 unicast
    !
-   interface GigabitEthernet0/0/0/4
-    point-to-point
-    address-family ipv4 unicast
    !
   !
end

RP/0/0/CPU0:BB4(config)#commit confirmed ?
  <30-65535>  Seconds until rollback unless there is a confirming commit
  minutes     Specify the rollback timer in the minutes
  show-error  Displays commit failures immediately
  <cr>        Commit the configuration changes via pseudo-atomic operation
RP/0/0/CPU0:BB4(config)#commit confirmed 30

RP/0/0/CPU0:BB4(config)#

RP/0/0/CPU0:BB4#show log | i "commit changes"

RP/0/0/CPU0:Jan 31 00:59:07.425 : config[65741]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user 'kazaii'. Use 'show configuration commit changes 1000000034' to view the changes. 
RP/0/0/CPU0:Jan 31 00:59:44.622 : cfgmgr_trial_confirm[65743]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user 'kazaii'. Use 'show configuration commit changes 1000000035' to view the changes. 
RP/0/0/CPU0:BB4#

RP/0/0/CPU0:BB4#show isis adja


IS-IS BB Level-2 adjacencies:
System Id      Interface        SNPA           State Hold Changed  NSF IPv4 IPv6
                                                                       BFD  BFD 
BB3            Gi0/0/0/0        *PtoP*         Up    28   00:02:06 No  None None
BB6            Gi0/0/0/4        *PtoP*         Up    18   00:02:06 Yes None None
BB5            Gi0/0/0/2        *PtoP*         Up    29   00:02:06 No  None None

Total adjacency count: 3
RP/0/0/CPU0:BB4#

For these platforms, and several others, things are really that elegant; You can utilize this feature to quickly get you out of Uh-Oh scenarios with limited impact.

You can also set it for a much broader timeframe – say, for the duration of your maintenance window. You can continually test your environment... knowing that the system will rollback for you.

Caveats

Sadly, sometimes it's less of a snapshot and more of a slapshot...

For other platforms like classic IOS, IOS-XE, VyOS, EdgeOS, and several others... the way this feature is implemented is to reboot the box in a set interval.

For IOS, you would perform the command:

reload in 15 – and the box would reload in 15 minutes.

Why is this such a stark difference? Well, because if you simply lock yourself out of management, while inline customer traffic is still flowing, you actually created a more serious outage; A tech-roll might've been preferred...

vyos@BB3# delete interfaces ethernet eth0
[edit]
vyos@BB3# show | compare
[edit interfaces]
-ethernet eth0 {
-    address 10.2.3.3/31
-    hw-id 0c:be:97:ba:00:00
-}
[edit]
vyos@BB3# commit-confirm 1
commit-confirm will automatically reboot in 1 minutes unless changes are confirmed.
Proceed? [y]y
Reboot scheduled for commit-confirm. Confirm your changes to cancel the reboot.
[edit]
vyos@BB3# 
[edit]
vyos@BB3# [  OK  ] Stopped /usr/bin/sg vyatta…/archive/config.boot-rollback.
[  OK  ] Stopped /usr/bin/sg vyatta…/archive/config.boot-rollback.
         Stopping Session 1 of user vyos.
[  OK  ] Removed slice system-modprobe.slice.
[  OK  ] Stopped target Graphical Interface.
[  OK  ] Stopped target Timers.
[  OK  ] Stopped Periodic ext4 Onli…ata Check for All Filesystems.
[  OK  ] Stopped Discard unused blocks once a week.
[  OK  ] Stopped Daily rotation of log files.

My Thoughts

The best method is really to take your time & do things right:

• Give yourself time.
• Lab things up.
• Test scenarios.
• Draw things out.
• Have peer reviews
• Before you commit, check the diff

Certifications spend way too much time focusing on configuring things & configuring them as fast as possible. There really should be more focus on ".. if I hit <Enter> now, what do I expect to happen?"

Maybe you should spend more time buildings tests & valdiation into your automation, before your automation becomes a distributed outage bot.

It's too dangerous to commit alone! Use a confirmed commit.

Once Upon a Night Shift Dreary

I recently performed a couple training sessions for my team. The subject of the training was "Network Core Tracing", as I've found – in regards to tracing in the core – there's a fair amount of inertia to just get started. This prolongs the outage and creates unnecessary MTTR growth.

I feel like a particular mentor of mine, Ray, would've smacked me in the back of the head if I didn't start tracing – once I've absolved the demarcs of guilt. He was a huge advocate of "Knowing the Flow". He was so right. So much can be found if you just... look.

He didn't care how I did it, as long as I knew all the potential paths of a particular flow – or sets of flows, if the application has multiple streams. Leave no hop unchecked. Leave no logs unread. At least until you've found the problem.

This really mattered in our environment. We deployed a lot of stateful firewalls with ACLs applied ingress on every interface; If a rule was missing, you really needed to consider where a flow would enter an interface. Although the firewall should've allowed the packets to flow in the return path, by being considered "related", sometimes there was issues. You had to be thorough.

How We Can Be Many Places At Once

I do understand the inertia. In a very widespread WAN environment – or a DC environment – it can seem that doing something as simple as logging into each node in the path, checking it's next hop in the FIB, and logging into those boxes poses a problem:

• Each box could have several diverse paths to the ultimate destination.
• Each of those diverse paths nodes could also have several diverse paths to the destination, causing the tree to grow exponentially.
• In an evironment where you have access controls, such as an one-time-password fob, it could seem like a very daunting – also time consuming –  task to login to 30 or 40 routers.

In the BGP-does-everything era, many people in modern DCs/Cloud environments know that BGP tables can give verbose information about why routes are selected, who they learned it from and where it originated from. Thus, it's not too hard to trace things down.

But what about in ISP environments, where we often rely on IGPs like OSPF & IS-IS? Well, those protocols also keep databases. Just like Google Maps, or the Kwisatz Haderach of Dune, you can search through the state & details of many links to trace out your network from practically anywhere.

What Do The Routers Know

IS-IS, being a Link-State Protocol, is very powerful. It advertises Link-State information via TLVs (type, length, value). Some very useful TLVs are:

• Type 135 - Extended IP Reachability (Prefixes the Router can reach directly)
• Type 132 - IP Interface Address (Essentially IPv4 Router-ID)
• Type 137 - Dynamic Name (Hostname of Router)
• Type 22 - Extended IS reachability (Adjacencies)

That's a lot of information to query from. More than enough to make sense of a traceroute, dynamically map out a network, or even keep track of topology changes.

Get Your Spice Coffee Ready For This One

Let's see if we can utilize these elements to trace from two routers. Let's say a customer of ours reports an issue from one site to another. They tell us the name of the first site, as it's their head-office. As for the other site, they are vague on details. They simply provide source & destination prefixes for us to work out.

We know the site they mentioned is connected to R1. We determine that the prefix is being learned via BGP neighbour 203.0.113.210 .  We check our IS-IS topology to find out who that is:

RP/0/0/CPU0:R1#show isis topology detail | i 203.0.113.210

R10 (203.0.113.210) [ucast 50 mcast <infinity>]
RP/0/0/CPU0:R1#

Now we check our next-hop towards R10:

RP/0/0/CPU0:R1#show ip route 203.0.113.210 | i "entry|via"

Routing entry for 203.0.113.210/32
  Known via "isis zeal", distance 115, metric 60, type level-2
    203.0.113.1, from 203.0.113.210, via GigabitEthernet0/0/0/0
RP/0/0/CPU0:R1#
RP/0/0/CPU0:R1#show isis adjacency | i 0/0/0/0

R2             Gi0/0/0/0        *PtoP*         Up    29   03:15:45 No  None None
RP/0/0/CPU0:R1#

Thank you very much, TLVs 137 & 132.

We now know this much about the topology:

Okay, so we said we won't login to any other router, right? So what do we do? We can utilize traceroute. Let's give that a try.

But before we do, there's something missing from my lab.

Lab Setup

As I mentioned in a previous post, I would be utilizing VyOS from now on as my P routers in my lab cores. This is because they're quicker to boot, maintain, and they support ECMP.... but with a catch.

When I first did my tests I found that ECMP in VyOS, by default, was only hashing on the L3 tuples of Src & Dst IP .....

RP/0/0/CPU0:R1#traceroute 203.0.113.210                   


Type escape sequence to abort.
Tracing the route to 203.0.113.210

 1  203.0.113.1 0 msec  0 msec  0 msec 
 2  203.0.113.3 0 msec  0 msec  0 msec 
 3  203.0.113.7 0 msec  0 msec  0 msec 
 4  203.0.113.17 0 msec  0 msec  0 msec 
 5  203.0.113.210 0 msec  0 msec  0 msec 
RP/0/0/CPU0:R1#
RP/0/0/CPU0:R1#traceroute 203.0.113.210                   


Type escape sequence to abort.
Tracing the route to 203.0.113.210

 1  203.0.113.1 0 msec  0 msec  0 msec 
 2  203.0.113.3 0 msec  0 msec  0 msec 
 3  203.0.113.7 0 msec  0 msec  0 msec 
 4  203.0.113.17 0 msec  0 msec  0 msec 
 5  203.0.113.210 0 msec  0 msec  0 msec 
RP/0/0/CPU0:R1#
RP/0/0/CPU0:R1#traceroute 203.0.113.210                   


Type escape sequence to abort.
Tracing the route to 203.0.113.210

 1  203.0.113.1 0 msec  0 msec  0 msec 
 2  203.0.113.3 0 msec  0 msec  0 msec 
 3  203.0.113.7 0 msec  0 msec  0 msec 
 4  203.0.113.17 0 msec  0 msec  0 msec 
 5  203.0.113.210 0 msec  0 msec  0 msec 
RP/0/0/CPU0:R1#

I confirmed this by adding 15 extra Loopbacks & sourcing my traces from those interfaces.

RP/0/0/CPU0:R1#show ip int br | i Lo  

Loopback0                      203.0.113.201   Up              Up       default 
Loopback230                    203.0.113.230   Up              Up       default 
Loopback231                    203.0.113.231   Up              Up       default 
Loopback232                    203.0.113.232   Up              Up       default 
Loopback233                    203.0.113.233   Up              Up       default 
Loopback234                    203.0.113.234   Up              Up       default 
Loopback235                    203.0.113.235   Up              Up       default 
Loopback236                    203.0.113.236   Up              Up       default 
Loopback237                    203.0.113.237   Up              Up       default 
Loopback238                    203.0.113.238   Up              Up       default 
Loopback239                    203.0.113.239   Up              Up       default 
Loopback240                    203.0.113.240   Up              Up       default 
Loopback241                    203.0.113.241   Up              Up       default 
Loopback242                    203.0.113.242   Up              Up       default 
Loopback243                    203.0.113.243   Up              Up       default 
Loopback244                    203.0.113.244   Up              Up       default 
Loopback245                    203.0.113.245   Up              Up       default 
RP/0/0/CPU0:R1#

That gave me the randomnesss I needed to sniff out that it was indeed using only those two tuples. As every probe, no matter how many, always hit the same hop... but each source IP revealed a new hop.

This is due to the default behaviour of the Linux kernel, as you can see in this great write-up here.

To summarize the fowarding behaviour:

for forwarded IPv4 packets (L3 hash){ Source Address, Destination Address }

for locally generated IPv4 packets (L4 hash) [3]{ Source Address, Destination Address, Protocol, Source Port, Destination Port }

However, with recently released Linux v4.12 selection of fields has changed a bit for IPv4 [4]. An L3 hash is used by default for both forwarded and locally generated traffic, but the user can choose to use the L4 hash, in both forward and local output path, with a new sysctl - net.ipv4.fib_multipath_hash_policy.

Since we are generating the traffic from our IOS-XR box, the VyOS routers in the middle are just forwarding with the L3 Hash. We need to enable the L4-hash for forwarding. In VyOS, this can be enabled using the following commands:

kazaii@R2# set system ip multipath 
Possible completions:
   ignore-unreachable-nexthops
                        Ignore next hops that are not in the ARP table
   layer4-hashing       Use layer 4 information for ECMP hashing

      
[edit]
kazaii@R2#
kazaii@R2# set system ip multipath layer4-hashing 
[edit]
kazaii@R2# commit;save;exit
Saving configuration to '/config/config.boot'...
Done
exit
kazaii@R2:~$ 

After that, ECMP hashing was now enabled for at least 5 tuples.... allowing Traceroute to sniff out the multiple hops.

How Does Traceroute Work Again?

If you find yourself scratching your head as to why the above mattered to Traceroute, and why the probes hit the same hop every time, you might want to review my favourite slide deck on how Traceroute works.

But I'll summarize the most relevant parts:

• Traceroute works by sending probes to find the various hops in a path
• It does this by incrementing the TTL by 1. When router forwards a packet, it decrements the TTL by 1 and sends it on. When TTL reaches 0, an ICMP TTL-Exceeded message is generated and sent back to the source.
• The source measures the delta between the receiving of that message against the sending of the probe to measure the round-trip-time to the hop.
• It also records the source IP of the Router that sent the message to be the presumed hop.
• It often sends 3 probes by default, in many implementations
• To encourage routers to hash / load-share the probes across multiple paths, it often utilizes UDP & increments the UDP destination port for each probe.
• Each probe is it's own individual test. No probe traces a complete path by itself – causing rather confusing results, sometimes known as the Paris Traceroute problem.

When you consider the above... even if the many hops in the network consider each probe a separate flow.... they are only hashing based on source & destination IP only. They do not care that each probe has a different UDP port...

RP/0/0/CPU0:R1#traceroute 203.0.113.210 minttl 4 maxttl 4 probe 20

Type escape sequence to abort.
Tracing the route to 203.0.113.210

 4  <snipped out for spoiler>
RP/0/0/CPU0:R1#  

Now that it's fixed, we can setup a test method that would test each hop of the network and ensure we sniff out as many links as we could imagine.

Parsing Method

Okay. So we can sniff out every end-point, at every particular hop in the network. How do we query the Link-State Database to determine what routers that goes through or to?

Well, we can utilize our Loopbacks as an example of what IP Prefixes & Hostnames look like in the LSDB (without spoiling what's to come):

RP/0/0/CPU0:R1#show isis database detail | i "Host|IP-Ext" | utility head -n 10

  Hostname:       R1
  Metric: 10         IP-Extended 203.0.113.0/31
  Metric: 0          IP-Extended 203.0.113.201/32
  Metric: 0          IP-Extended 203.0.113.230/32
  Metric: 0          IP-Extended 203.0.113.231/32
  Metric: 0          IP-Extended 203.0.113.232/32
  Metric: 0          IP-Extended 203.0.113.233/32
  Metric: 0          IP-Extended 203.0.113.234/32
  Metric: 0          IP-Extended 203.0.113.235/32
  Metric: 0          IP-Extended 203.0.113.236/32
RP/0/0/CPU0:R1#

Utilizing data like that, we can go a bit further and use the following command to find the exact IS's are attached to a prefix:

show isis database detail | i "Host|<Prefix>" | utility egrep -B1 "Ext"

With the above command, we will do the following: We will Pipe Include on the following patterns: Host, Prefix. We will then parse the results bit further with the egrep -B1 "Ext" , as egrep will show us the pattern that is 1 line before, due to the -B1 argument.

Since each Prefix is under the LSP entry of each router, the next line will always be the Hostname of the router that advertised it. Let me show the above example, if I match directly on 203.0.113.235/32

RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.235/32" | utility$

  Hostname:       R1
  Metric: 0          IP-Extended 203.0.113.235/32
RP/0/0/CPU0:R1#s

Thank you TLV 135. Thank you extended grep.

On With It

So first let's sniff out Hop 4, since we know from the first trace that we reached R10 in 5 hops.

RP/0/0/CPU0:R1#traceroute 203.0.113.210 minttl 4 maxttl 4 probe 20


Type escape sequence to abort.
Tracing the route to 203.0.113.210

 4  203.0.113.23 9 msec 
    203.0.113.27 0 msec 
    203.0.113.21 0 msec 
    203.0.113.23 0 msec 
    203.0.113.27 0 msec  0 msec 
    203.0.113.23 0 msec  *  0 msec 
    203.0.113.27 0 msec 
    203.0.113.23 0 msec 
    203.0.113.21 0 msec  * 
    203.0.113.27 0 msec 
    203.0.113.17 0 msec 
    203.0.113.23 0 msec 
    203.0.113.17 0 msec 
    203.0.113.23 0 msec 
    203.0.113.21 0 msec  * 
RP/0/0/CPU0:R1#

It hit the following addresses:

• 203.0.113.23
• 203.0.113.27
• 203.0.113.21
• 203.0.113.17

Let's look those up:

RP/0/0/CPU0:R1#show ip route 203.0.113.23 | i entry

Routing entry for 203.0.113.22/31
RP/0/0/CPU0:R1#show ip route 203.0.113.27 | i entry

Routing entry for 203.0.113.26/31
RP/0/0/CPU0:R1#show ip route 203.0.113.21 | i entry

Routing entry for 203.0.113.20/31
RP/0/0/CPU0:R1#show ip route 203.0.113.17 | i entry

Routing entry for 203.0.113.16/31
RP/0/0/CPU0:R1#

RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.22/31" | utility $

  Hostname:       R6
  Metric: 10         IP-Extended 203.0.113.22/31
--
  Hostname:       R9
  Metric: 10         IP-Extended 203.0.113.22/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.26/31" | utility $

  Hostname:       R7
  Metric: 10         IP-Extended 203.0.113.26/31
--
  Hostname:       R9
  Metric: 10         IP-Extended 203.0.113.26/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.20/31" | utility $

  Hostname:       R6
  Metric: 10         IP-Extended 203.0.113.20/31
--
  Hostname:       R8
  Metric: 10         IP-Extended 203.0.113.20/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.16/31" | utility $

  Hostname:       R5
  Metric: 10         IP-Extended 203.0.113.16/31
--
  Hostname:       R8
  Metric: 10         IP-Extended 203.0.113.16/31
RP/0/0/CPU0:R1#

Wow, that's a lot of data to process. But, if we consider what we already know.... The Z-end, which is Hop 5, is R10. So what's connected to R10 ?

We can query the adjacencies advertised into our LSDB to determine that. We will tweak our command above to be IS-Ext , instead of IP-Ext ... we will also limit our results to the LSP ID of R10. This always ends in .00-00 in most networks... but if you forget this, you can wildcard it:

show isis database detail <hostname>.* | i "<hostname>|IS-Ext"

RP/0/0/CPU0:R1#show isis database detail R10.* | i "R10|IS-Ext" 

R10.00-00             0x00000019   0x6096        902             0/0/0
  Hostname:       R10
  Metric: 10         IS-Extended R8.00
  Metric: 10         IS-Extended R9.00
RP/0/0/CPU0:R1#

So it's R8 & R9 that's directly connected to R10. With that information, along with the prefixes we sniffed out above... we can start mapping things out.

Thank you TLV 22.

We're getting pretty close. Let's trace Hop 3:

RP/0/0/CPU0:R1#traceroute 203.0.113.210 minttl 3 maxttl 3 probe 20


Type escape sequence to abort.
Tracing the route to 203.0.113.210

 3  203.0.113.15 0 msec 
    203.0.113.11 0 msec 
    203.0.113.9 0 msec  0 msec 
    203.0.113.13 0 msec 
    203.0.113.15 0 msec  0 msec  0 msec  0 msec 
    203.0.113.7 0 msec  * 
    203.0.113.9 0 msec 
    203.0.113.11 0 msec 
    203.0.113.13 0 msec 
    203.0.113.7 0 msec  0 msec 
    203.0.113.11 0 msec 
    203.0.113.13 0 msec 
    203.0.113.7 0 msec 
    203.0.113.15 0 msec 
RP/0/0/CPU0:R1#

That one was even more wild. It hit the following addresses:

• 203.0.113.15
• 203.0.113.11
• 203.0.113.9
• 203.0.113.13
• 203.0.113.7

Let's look them up:

P/0/0/CPU0:R1#show ip route 203.0.113.15 | i entry

Routing entry for 203.0.113.14/31
RP/0/0/CPU0:R1#show ip route 203.0.113.11 | i entry

Routing entry for 203.0.113.10/31
RP/0/0/CPU0:R1#show ip route 203.0.113.9 | i entry 

Routing entry for 203.0.113.8/31
RP/0/0/CPU0:R1#show ip route 203.0.113.13 | i entry

Routing entry for 203.0.113.12/31
RP/0/0/CPU0:R1#show ip route 203.0.113.7 | i entry 

Routing entry for 203.0.113.6/31
RP/0/0/CPU0:R1#

RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.14/31" | utility $

  Hostname:       R4
  Metric: 10         IP-Extended 203.0.113.14/31
--
  Hostname:       R7
  Metric: 10         IP-Extended 203.0.113.14/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.10/31" | utility $

  Hostname:       R3
  Metric: 10         IP-Extended 203.0.113.10/31
--
  Hostname:       R7
  Metric: 10         IP-Extended 203.0.113.10/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.8/31" | utility e$

  Hostname:       R3
  Metric: 10         IP-Extended 203.0.113.8/31
--
  Hostname:       R6
  Metric: 10         IP-Extended 203.0.113.8/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.12/31" | utility $

  Hostname:       R4
  Metric: 10         IP-Extended 203.0.113.12/31
--
  Hostname:       R6
  Metric: 10         IP-Extended 203.0.113.12/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.6/31" | utility e$

  Hostname:       R3
  Metric: 10         IP-Extended 203.0.113.6/31
--
  Hostname:       R5
  Metric: 10         IP-Extended 203.0.113.6/31
RP/0/0/CPU0:R1#

Once again, we take the routers we already know... and add the routers in our trace to those. Here's what it looks like now:

One more hop to go. Let's look at Hop 2:

RP/0/0/CPU0:R1#traceroute 203.0.113.210 minttl 2 maxttl 2 probe 20 

Type escape sequence to abort.
Tracing the route to 203.0.113.210

 2  203.0.113.3 0 msec 
    203.0.113.5 0 msec  0 msec  0 msec  0 msec 
    203.0.113.3 0 msec 
    203.0.113.5 0 msec 
    203.0.113.3 0 msec  0 msec  0 msec  0 msec  *  0 msec  0 msec 
    203.0.113.5 0 msec 
    203.0.113.3 0 msec 
    203.0.113.5 0 msec  * 
    203.0.113.3 0 msec  0 msec 
RP/0/0/CPU0:R1#

It hit the following addresses:

• 203.0.113.3
• 203.0.113.5

Let's see who those belong to:

RP/0/0/CPU0:R1#show ip route 203.0.113.3 | i entry

Routing entry for 203.0.113.2/31
RP/0/0/CPU0:R1#show ip route 203.0.113.5 | i entry

Routing entry for 203.0.113.4/31
RP/0/0/CPU0:R1#

RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.2/31" | utility e$

  Hostname:       R2
  Metric: 10         IP-Extended 203.0.113.2/31
--
  Hostname:       R3
  Metric: 10         IP-Extended 203.0.113.2/31
RP/0/0/CPU0:R1#show isis database detail | i "Host|203.0.113.4/31" | utility e$

  Hostname:       R2
  Metric: 10         IP-Extended 203.0.113.4/31
--
  Hostname:       R4
  Metric: 10         IP-Extended 203.0.113.4/31
RP/0/0/CPU0:R1#

Well, those are connected to our known first-hop, R2.

Thus the trace produced this topology:

Revealing the topology I built in GNS3:

A very smart cookie would point out that our diagram is missing two links in the topology... the ones between R6 & R7/R5.

The reason being is that Traceroute is determining the paths that would be taken from Source to Destination.... and those particular paths are one hope too many to be sniffed out. 6 hops to be exact, if you use your fingers to trace from R1 -> R2 -> R3 -> R5 -> R6 -> R9 -> R10.

This would be true if there were 200+ routers in your topology. You are utilizing a method that will trace out your path, not the entire WAN diagram, as that's what you're looking for anyway.

If our task was different, and we were trying to map out R6's adjacencies – say, if we were about to set our overload bit, and remove it from the path –  we could query it's LSP for IS-Extended reachability, to reveal those links:

RP/0/0/CPU0:R1#show isis database detail R6.* | i "Host|IS-Ext"  

  Hostname:       R6
  Metric: 10         IS-Extended R7.00 <----
  Metric: 10         IS-Extended R8.00 
  Metric: 10         IS-Extended R5.00 <----
  Metric: 10         IS-Extended R3.00
  Metric: 10         IS-Extended R4.00
  Metric: 10         IS-Extended R9.00
RP/0/0/CPU0:R1#

Now What?

Now that you know the path the packets will walk, you can start your tests.

• Perhaps use a packet-generator to to determine at which point the point the flow fails.
• You can capture on tap points on these links, if possible, to determine where the problem occurs.
• After seeing the topology, you may already know there is a common-point for other queries. It could be R6, which is the most touched router in this flow. Or up-to half of flows go through R9, and a customer directly connected to that router is also reporting recent issues.
• Now that you demystified the path, it might seem less daunting to login to these routers and do the necessary checks. You might find a Line Card that has reached it's scale limit and has been screaming for days-on-end in the logs.

Some Closing Notes

If the customer didn't tell you the A-End, you could've checked the VPN route-reflector to get R1's Loopback IP as your recursive next-hop. Then you could've queried that in the LSDB.

Of course, this example won't always apply 1:1. Maybe traceroute dies before you reach the LB address. Maybe TTL propagation is disabled to hide the core from traces. It still doesn't stop you from utilizing the costs to calculate the paths yourself. You could also utilize my other example to trace it the good old fashioned way.

Do Not Ponder, Weak and Weary

Ray was right all along. It's great & all to know all the nerd knobs of adjacencies, MTU padding, authentication, timers, BFD, TI-LFA, etc. ad-naseum. But when it comes right down to the wire, and you're looking for tangible data, it's even better to know what ol' Dijkstra produced for your particular graph. What the deck of your BGP tables delivered to your hand.

The more your trace your network, the better you know it. The faster you get. The more sure you are of your results. Do it whenever you can.

Know the Flow. Just get started.

Thanks, Ray.

When I was a wee sprout

Early in my career, in my first legitimate "Network" role, I witnessed a dear colleague achieve his CCIE. I could see him work hard at his goal, and the immense satisfaction passing gave him. Not only that, it seemed that his studies opened him up to many things I had never heard of in college. Things such as GNS3 (when it was still a dynamips-only beta), Unetlab, MPLS, etc. etc.... things I didn't see in my day to day at the small company we worked for.

He then moved on to work in a 'real' Network team at a big financial firm. It set-off the bug in my head that perhaps certification was the real deal, and I should give it a go.

I'm so happy I did. After studying for many hours and many days, I got my CCNA – soon to be followed by my CCNP. That certification on my resume, along with the exposure it gave me, definitely helped me to land & pass the technical interview I got for my first NOC role at a large Datacentre / MSP org.

Epiphany

As I continued my studies and certification journey, I quickly realized that is was the journey, not the destination, that really made the impact. Knowing the outline of CCNP, and reading up on it's various applications, helped me to understand the world outside of my current perview. It didn't mean that EIGRP & DMVPN were best – because I was now familiar with them, and familiar feels best – they were simply a tool in the toolbet, to be drawn upon, should the requirements deem fitting, or simply dictate.

It also was the fact that I put in the hours at the terminal, demystifying that arbitary domain-specific-language that Cisco & others implemented. This was better put towards getting my hands dirty at work, rather than prepping for a test.

After landing a position at a major nation-wide Telco, by circumstance of acquisition, I decided I would up my skills by not getting a CCIE, but by using CCIE's outline as a guideline for my studying. Time proved this method effective for me.

I also soon discovered that CCIE outlines were also a bit dated for the more emerging technologies I was working on, in a competitive Telco. In fact, it seemed that CCIE was somewhere between 2-5 years behind what we're pushing ahead today; Furthermore, when CCIE does ultimately catch up, it's rather shallow in it's exploration of new topics – Segment Routing comes to mind, as an example. This likely comes from Cisco's uncertainty in what products & implementations will actually take hold of the industry, and which ones will flop – as so many implementations do!

Source of Contention

Now I find myself at a different crossroad; I'd like to strengthen my paper portfolio (CV), along with my knowledge. The reason being is that I'd like to potentially leave this continent & move to Europe, for personal reasons.

I'd wager that a littled amplified 'Alphabet soup' would help me stand out in the crowd, especially in foreign lands. Perhaps there are other unconsidered benefits from the actualization of the full path as well.

Sadly, there are some negative associations I have with CCIE. The first one being is that I, and countless others, have encountered so many "Paper CCIEs". People who purportedly have – or had – the knowledge to pass the outline of CCIE, at the time of their certification. Yet, when it comes time to do the work, they can't troubleshoot their way out of a wet paper bag.  Even on the very protocols they seemingly had to 'master' to pass the troubleshooting portion of their certification.

I have many theories as to how this happens, but the simple explanation – that isn't malicious or nefarious in nature, anyway – is that a career in Networking is a Marathon, not a sprint. This can be clearly seen in some of my colleagues I've had.

I've met those that are old CCIE's, new CCIE's, Poly *IE's, etc. etc. The defining factor in their knowledge doesn't seem to be specifically the certification journey, but an extensive or intense career path; Were they a back-bencher or a minister of pushing packets? That seemed to always be the defining factor.

Thus, I ask myself – yet again – what the hell do I want this Cert for? Who do I want this for? It always seems to be not for me. But if it increases my chances to immigrate by, say, even 5% ? How badly do I want the end result of said immigration? Even if I know I'm simply playing the HR Alphabet game.

CCIE Pros:

1. CCIE is universally recognized
2. Reviewing the outline today, I feel confident I can master the topics, especially in CCIE-SP track.
3. I'd never have to think about this again.

CCIE Cons:

1. There are too many CCIEs, and too many 'paper' ones. Do technical managers even care? I know I wouldn't (But maybe the job market I am targetting does).
2. Many hours spent for what could be spent doing something I'd consider exceptional, like contributing to projects I believe in – such as FRR, VyOS & DBIUA.
3. Money I could invest in my homelab, etc.

JNCIE Pros:

1. Not many of these guys/gals/etc. out there.
2. Could demonstrate & vastly expand my Juniper knowledge – I'd consider myself intermediate today
3. I like their plaque better than the CCIE one

JNCIE Cons:

1. One must complete the whole track – from JNCIA to JNCIP – taking more time & money.
2. There seems to be more resources from the vendor, but less resources out there on the interwebz. I'd have to search for the tribe.
3. I am very annoyed by this one, but Juniper silently removed the vMX trial license. Meaning my 17.2 R2 image is the best I'll get. That's too old of an image for the JNCIE-SP track. I could grovel through my companies vendor relationship, but I'd rather not. They should take this page down

Conclusion

I am simply writing what I'm thinking. This could all be summarized as "Maybe". If I do decide to pursue this track, I will report back & hold myself accountable to my very small – maybe non-existent – readership.

I will say that I have great respect for many *IE's:

Russ White (who was CCAr as well), Ivan Pepelnjak, Packet Pushers Crew, countless co-workers, and the most technically talented person I've personally worked with, Randy – who let his JNCIE expire, if that says anything.

I also witnessed a valued member of the r/networking community be awarded his JNCIE recently. I can definitely say that the plaque is very nice.

There are plenty others who did not pursue certification that I consider industry experts, Jeremy Stretch being the most notable.

One thing is for certain.... Certification is not the reason I grew to respect those individuals. So I wouldn't expect anyone to respect me any more, should I get mine.

Networking:

Routing Will Never Be A Solved Problem

The Important There will never be perfect information There will always be an area of futzing to find optimization points across multiple competing objectives We are just beginning to dive into some areas of networking, so there is much discovery yet to happen

Internet DynamicsMark Seery

Russ’ Rules of Network Design

We have the twelve truths of networking, and possibly Akin’s Laws, but is there a set of rules for network design? I couldn’t find one, so I decided to create one, containing 18 laws I&…

rule 11 readerauthor page

Mean Time to Innocence is not Enough

A long time ago, I supported a wind speed detection system consisting of an impeller, a small electric generator, a 12 gauge cable running a few miles, and a voltmeter. The entire thing was calibra…

rule 11 readerauthor page

ISP Column - November 2022

November 2022

Living with Small Forwarding Tables « ipSpace.net blog

A friend of mine working for a mid-sized networking vendor sent me an intriguing question: We have a product using an old ASIC that has 12K forwarding entries, and would like to extend its lifetime. I know you were mentioning some useful tricks, would you happen to remember what they were? This c…

ipSpace Logo

Personal Development:

When You Find Yourself on Mount Stupid « ipSpace.net blog

The early October 2021 Facebook outage generated a predictable phenomenon – couch epidemiologists became experts in little-known Bridging the Gap Protocol (BGP), including its Introvert and Extrovert variants. Unfortunately, I also witnessed several unexpected trips to Mount Stupid by people who sho…

ipSpace Logo

’Tis the Season for Reexamining Your Values

How will you spend your precious time in the coming year? As one year ends and another begins, many of us won’t help but reflect on who we are and who we’d like to become. Most people aiming to emulate their ideal selves will resort to New Year’s resolutions—but more often than not, those are doomed…

Nir and FarNir Eyal

4 Mental Traps That Kill Productivity

Productivity has many enemies: too many meetings, external triggers like interruptions from coworkers, whether virtual or in person, and multitasking the wrong way, to name a few. But more often than not, it’s mental traps that trip us up.

Nir and FarNir Eyal

General:

ISP Column - October 2022

October 2022

Few People Get Promoted For Asking Difficult Questions

Research indicates that even when everyone within a group recognizes who the subject matter expert is, they defer to that member just 62% of the time; when they don’t, they listen to the most extro…

Paul Taylorthe Rotters

Asking Meaningful Questions: What Problem Are We Trying To Solve? - Packet Pushers

At some point in your career, you’ll likely participate in a project that is a technical and implementation success but is still a failure. That’s because the wrong solution was implemented. For example, after weeks or months of hard work you might successfully deploy a client-based VPN solution, bu…

Packet PushersEyvonne Sharp

What “Work” Looks Like

Writing about the big beautiful mess that is making things for the world wide web.

Jim Nielsen’s Blog Verified ($10/year for the domain)Jim Nielsen

Make Free Stuff

On web3, Wordle and the radical concept of building things for free.

Max BöckMax Böck

I Was Wrong About Mastodon – EscapingTech

I said that Mastodon moderation wouldn’t scale, it does. The cultural differences will likely continue to maintain a friendlier atmosphere regardless of size.

EscapingTechMarcus Hutchins

How to have a good internet experience in 8 easy steps

#1 - Stop having a bad faith interpretation of every thing you read If you think something someone said might have been something you disagree with, instead of starting an argument, ask them to clar…

Tumblr

Stop Talking to Each Other and Start Buying Things: Three Decades of Survival in the Desert of Social Media

I bet you’re wondering how we got here...

Welcome to GarbagetownCatherynne M. Valente

Protocols, Not Platforms: A Technological Approach to Free Speech

Knight First Amendment InstituteMike Masnick

Pluralistic: Freedom of reach IS freedom of speech (10 Dec 2022) – Pluralistic: Daily links from Cory Doctorow

Pluralistic: Daily links from Cory Doctorow No trackers, no ads. Black type, white background. Privacy policy: we don't collect or retain any data at all ever period.Cory Doctorow

On anti-crypto toxicity

Crypto is known for its toxicity towards outsiders. Similar attitudes are emerging from some who oppose crypto.

Some Closing Smirks:

Verified Personal Website

Writing about the big beautiful mess that is making things for the world wide web.

Jim Nielsen’s Blog Verified ($10/year for the domain)Jim Nielsen

Chemicals

xkcdAbout

I'll be posting more interesting articles that I read in 2023 on my Mastodon. be sure to follow me there for article links, boosts, and more short-form opinions/progress.

I wish you all a happy & safe 2023.

I recently went on a quick trip to Chicago, followed by a brief 'staycation'. I am very proud that I was able to effectively check-out from tech life. However, this also meant that I had 400 + articles to read in my RSS feed.

While tackling my RSS feed – and getting rid of the redundant feeds – I came across an interesting VyOS project update from November:

Thanks to our long-time contributor Cheeze-It, IS-IS segment routing support has been refactored and brought much closer to the OSPF segment routing implementation (T4739).

Very fascinating work from a familiar name on r/networking . Not only did he add the much needed ISIS-SR configuration to Vyos, he also helped the FRR team add the missing opaque LSA knob for the OSPF implementation.

I was very excited by the prospect of SR-MPLS now working in Vyos, as XRv & XRv9k can be quite beefy and clumsy to utilize in my labs. Vyos, on the other hand, is lean & mean as a CE/PE implementation.

I figured I'd put it to the test. I went to the Vyos site and downloaded the latest rolling image. I then imported it into GNS3 & plopped it directly into my TI-LFA / SR-TE refresher lab. After some configuration tweaking, it worked beautifully. The next part was to test it's potential as an LSR in my topology; Could I create a TI-LFA recovery path, using the SR labels/SIDs on the router?

To simulate a TI-LFA hard coverage scenario, I suspended some links and add a SRLG (shared risk link group) to a crucial node in the path.

Messy – I know – but it did the trick! The end topology looks a little more like this:

Let's consider the normal path for router 3 to reach router 10's loopback. The shortest path would be via router 6, then router 9. If the link towards router 6 failed, the natural second best path would be via router 7 or router 4.

But what if I told you that router 6's path to router 4 & router 9 had a shared risk.... such as a common DWDM shelf, a common fiber conduit, shared ASIC, shared line card, etc. We can program that into the device to protect against the common faults introduced by router 6.

First, here's the SR-MPLS & TI-LFA configuration on router 3:

RP/0/0/CPU0:XRv-3#show run formal | i "isis|mpls"

Building configuration...
 router isis '.*' 
 router isis '.*'  interface 'GigabitEthernet.*' 
 router isis '.*'  interface 'GigabitEthernet.*'  address-family ipv4 unicast 
 router isis '.*'  interface 'GigabitEthernet.*'  address-family ipv4 unicast  fast-reroute per-prefix
 router isis '.*'  interface 'GigabitEthernet.*'  address-family ipv4 unicast  fast-reroute per-prefix ti-lfa
ipv4 unnumbered mpls traffic-eng Loopback0
router isis zeal apply-group GROUP_TILFA
router isis zeal 
router isis zeal is-type level-2-only
router isis zeal net 49.0000.0000.0003.00
router isis zeal address-family ipv4 unicast 
router isis zeal address-family ipv4 unicast metric-style wide
router isis zeal address-family ipv4 unicast fast-reroute per-prefix tiebreaker node-protecting index 100
router isis zeal address-family ipv4 unicast fast-reroute per-prefix tiebreaker srlg-disjoint index 200
router isis zeal address-family ipv4 unicast microloop avoidance
router isis zeal address-family ipv4 unicast microloop avoidance rib-update-delay 65535
router isis zeal address-family ipv4 unicast advertise passive-only
router isis zeal address-family ipv4 unicast segment-routing mpls sr-prefer
router isis zeal interface Loopback0 
router isis zeal interface Loopback0 passive
router isis zeal interface Loopback0 address-family ipv4 unicast 
router isis zeal interface Loopback0 address-family ipv4 unicast prefix-sid index 3
router isis zeal interface GigabitEthernet0/0/0/0 
router isis zeal interface GigabitEthernet0/0/0/0 address-family ipv4 unicast 
router isis zeal interface GigabitEthernet0/0/0/1 
router isis zeal interface GigabitEthernet0/0/0/1 address-family ipv4 unicast 
router isis zeal interface GigabitEthernet0/0/0/2 
router isis zeal interface GigabitEthernet0/0/0/2 address-family ipv4 unicast 
router isis zeal interface GigabitEthernet0/0/0/3 
router isis zeal interface GigabitEthernet0/0/0/3 bfd minimum-interval 100
router isis zeal interface GigabitEthernet0/0/0/3 bfd multiplier 3
router isis zeal interface GigabitEthernet0/0/0/3 bfd fast-detect ipv4
router isis zeal interface GigabitEthernet0/0/0/3 address-family ipv4 unicast 
mpls traffic-eng 
mpls traffic-eng auto-tunnel p2p tunnel-id min 10000 max 14094
mpls ip-ttl-propagate disable
RP/0/0/CPU0:XRv-3#

The above configuration, especially the fast-reroute portion, should protect us if router 6 completely fails, or that common risk SRLG path. The microloop avoidance will implement this tunnel for a set time, in the assumption that the forwarding plane takes time to propagate.

Now let's look at the SRLG configuration on router 6:

RP/0/0/CPU0:XRv-6#show run formal srlg  | i "name|value"

srlg interface GigabitEthernet0/0/0/0 name CONDUIT
srlg interface GigabitEthernet0/0/0/3 name CONDUIT
srlg name CONDUIT value 100
RP/0/0/CPU0:XRv-6#
RP/0/0/CPU0:XRv-6#show srlg name CONDUIT                            

SRLG : CONDUIT
Value : 100

Interface:
 GigabitEthernet0/0/0/0
 GigabitEthernet0/0/0/3


RP/0/0/CPU0:XRv-6#
RP/0/0/CPU0:XRv-6#show isis adjacency | i "0/0/0/0|0/0/3"

XRv-4          Gi0/0/0/0        0cd9.5942.0002 Up    6    00:32:36 Yes None None
XRv-9          Gi0/0/0/3        0c06.7e75.0002 Up    25   00:32:35 Yes None None
RP/0/0/CPU0:XRv-6#  

Great. So the playing field is set.... but how does one setup SR-MPLS (in IS-IS) on VyOS? Much simpler than you think, thanks to Mr. Cheese.

vyos@Vyos-8:~$  show configuration commands | match "isis|mpls"
set protocols isis interface eth0 network
set protocols isis interface eth1 network
set protocols isis interface eth2 metric '40'
set protocols isis interface eth2 network
set protocols isis interface lo passive
set protocols isis level 'level-2'
set protocols isis net '49.0000.0000.0008.00'
set protocols isis segment-routing global-block high-label-value '24999'
set protocols isis segment-routing global-block low-label-value '16000'
set protocols isis segment-routing local-block high-label-value '104000'
set protocols isis segment-routing local-block low-label-value '100000'
set protocols isis segment-routing prefix 10.255.255.8/32 index value '8'
set protocols mpls interface 'eth0'
set protocols mpls interface 'eth1'
set protocols mpls interface 'eth2'
vyos@Vyos-8:~$
vyos@Vyos-8:~$ show interfaces 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             10.6.8.8/31                       u/u  
eth1             10.7.8.8/31                       u/u  
eth2             10.8.9.8/31                       u/u  
lo               127.0.0.1/8                       u/u  
                 10.255.255.8/32                        
                 ::1/128                                
vyos@Vyos-8:~$

I set the local-block, to be used by the Adjacency Labels, to a high value so they stand out from the Prefix SIDs.

It works:

vyos@Vyos-8:~$ show ip route isis | match label
I   10.255.255.1/32 [115/40] via 10.6.8.6, eth0 inactive, label 16001, weight 1, 00:01:11
                             via 10.7.8.7, eth1 inactive, label 16001, weight 1, 00:01:11
I   10.255.255.2/32 [115/30] via 10.6.8.6, eth0 inactive, label 16002, weight 1, 00:01:11
                             via 10.7.8.7, eth1 inactive, label 16002, weight 1, 00:01:11
I   10.255.255.3/32 [115/20] via 10.6.8.6, eth0 inactive, label 16003, weight 1, 00:06:15
                             via 10.7.8.7, eth1 inactive, label 16003, weight 1, 00:06:15
I   10.255.255.4/32 [115/20] via 10.6.8.6, eth0 inactive, label 16004, weight 1, 00:06:15
I   10.255.255.5/32 [115/20] via 10.6.8.6, eth0 inactive, label 16005, weight 1, 00:06:15
I   10.255.255.6/32 [115/10] via 10.6.8.6, eth0 inactive, label implicit-null, weight 1, 00:06:15
I   10.255.255.7/32 [115/10] via 10.7.8.7, eth1 inactive, label implicit-null, weight 1, 00:06:15
I   10.255.255.9/32 [115/20] via 10.6.8.6, eth0 inactive, label 16009, weight 1, 00:06:15
I   10.255.255.10/32 [115/30] via 10.6.8.6, eth0 inactive, label 16010, weight 1, 00:06:15
vyos@Vyos-8:~$ 
vyos@Vyos-8:~$ show mpls table
 Inbound Label  Type        Nexthop   Outbound Label  
 -----------------------------------------------------
 16001          SR (IS-IS)  10.7.8.7  16001           
 16002          SR (IS-IS)  10.7.8.7  16002           
 16003          SR (IS-IS)  10.7.8.7  16003           
 16003          SR (IS-IS)  10.6.8.6  16003           
 16004          SR (IS-IS)  10.6.8.6  16004           
 16005          SR (IS-IS)  10.6.8.6  16005           
 16006          SR (IS-IS)  10.6.8.6  implicit-null   
 16007          SR (IS-IS)  10.7.8.7  implicit-null   
 16009          SR (IS-IS)  10.6.8.6  16009           
 16010          SR (IS-IS)  10.6.8.6  16010           
 100000         SR (IS-IS)  10.8.9.9  implicit-null   
 100001         SR (IS-IS)  10.7.8.7  implicit-null   
 100002         SR (IS-IS)  10.6.8.6  implicit-null   

vyos@Vyos-8:~$ 

To further demonstrate the Microloop avoidance / node protection feature... I also set the Metric between Router 8 & 9 to be 40, rather than the default value of 10, in the topology. To demonstrates that router 3 is trying to really avoid sending the backup path via router 6.

set protocols isis interface eth2 metric '40'

vyos@Vyos-8:~$ show isis neighbor XRv-9 | match Inter
    Interface: eth2, Level: 2, State: Up, Expires in 28s
vyos@Vyos-8:~$

Now let's see what router 3 sees:

RP/0/0/CPU0:XRv-3#show isis fast-reroute 10.255.255.10/32


L2 10.255.255.10/32 [30/115]
     via 10.3.6.6, GigabitEthernet0/0/0/3, XRv-6, SRGB Base: 16000, Weight: 0
         Backup path: TI-LFA (node+srlg), via 10.3.7.7, GigabitEthernet0/0/0/1 XRv-7, SRGB Base: 16000, Weight: 0
           P node: Vyos-8.00 [10.255.255.8], Label: 16008
           Q node: XRv-9.00 [10.255.255.9], Label: 100000
           Prefix label: 16010
RP/0/0/CPU0:XRv-3#
RP/0/0/CPU0:XRv-3#show cef 10.255.255.10/32  | i "backup|16008"

   via 10.3.7.7/32, GigabitEthernet0/0/0/1, 9 dependencies, weight 0, class 0, backup (remote) [flags 0x8300]
     local label 16010      labels imposed {16008 100000 16010}
RP/0/0/CPU0:XRv-3#

You can see that an auto-engineered fast-reroute path is generated. Utilizing the following label stack:

• Node SID of 16008 to path to Router 8
• Adjacency SID on Router 8 to force the traffic over the less preferred link, via Router 9
• Bottom-of-stack is the label for the node SID of Router 10.

If this were to be combined with a VPN service label – which it likely would – that would mean a label depth of 4 protecting the payload.

Here's what the backup path looks like:

This proves that Vyos – in the to-be-released 1.4 version, anyway – is a viable SR-MPLS LSR replacement for XRv, Junos, EOS, etc. – at least in your lab! It also boots way faster and utilizes modest resources. Sadly, RFC 7432 implementation and TI-LFA are not implemented in FRR, as of yet:

vyos@Vyos-8:~$ vtysh

Hello, this is FRRouting (version 8.4.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

Vyos-8# show isis fast-reroute 
% Command incomplete: show isis fast-reroute 
Vyos-8# show isis fast-reroute summary 
Area VyOS:
 IS-IS L2 IPv4 Fast ReRoute summary:

 Protection \ Priority     Critical  High      Medium    Low       Total   
 --------------------------------------------------------------------------
 Classic LFA               0         0         0         0         0       
 Remote LFA                0         0         0         0         0       
 Topology Independent LFA  0         0         0         0         0       
 ECMP                      0         0         3         0         3       
 Unprotected               0         0         6         0         6       
 Protection coverage       0.00%     0.00%     33.33%    0.00%     33.33%  

(I also tried EVPN rt-5 in my lab, without luck)

VPNv4 Unicast + SR works just fine :)

This makes me happy as I can build larger topologies & boot/restart resources faster. Thanks to the VyOS team, FRR team,  &  Cheeze_it

After I hit publish on my blog post last night, I went to bed. While laying there, a simpler way to implement my table-policy design suddenly came to me; One that eliminates the need of a L1/L2 ABR.

What I came up with was a hierarchical Route-Reflector design. Where the T1 routers, that border the two rings, act as RR's for the T3 routers. They would also originate a default route for the routers to use, instead of the IS-IS attached-bit.

Configuration

First, we had to restore the Level-2 Topology to both rings.

RP/0/0/CPU0:XRv-9#show isis topology 


IS-IS zeal paths to IPv4 Unicast (Level-2) routers
System Id       Metric  Next-Hop        Interface       SNPA          
XRv-1           30      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-2           20      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-3           20      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-4           30      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-5           40      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-6           40      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-7           10      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
XRv-8           10      XRv-8           Gi0/0/0/0       0ca7.981d.0002
XRv-9           --    
RR-1            50      XRv-7           Gi0/0/0/1       0cfe.1ea7.0002
RR-1            50      XRv-8           Gi0/0/0/0       0ca7.981d.0002
RP/0/0/CPU0:XRv-9#

Next, we establish IPv4-unicast sessions between the Ring Border Routers & the Tier 3 PE(s). We also need to add the config of default-originate to ensure that they specifically advertise a default route to this PE.

RP/0/0/CPU0:XRv-2#show run formal router bgp | i .9

router bgp 6275 neighbor 10.255.255.9 
router bgp 6275 neighbor 10.255.255.9 remote-as 6275
router bgp 6275 neighbor 10.255.255.9 update-source Loopback0
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast 
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS in
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-reflector-client
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS out
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast default-originate
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast soft-reconfiguration inbound always
RP/0/0/CPU0:XRv-2#
RP/0/0/CPU0:XRv-3#show run formal router bgp | i .9

router bgp 6275 neighbor 10.255.255.9 
router bgp 6275 neighbor 10.255.255.9 remote-as 6275
router bgp 6275 neighbor 10.255.255.9 update-source Loopback0
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast 
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS in
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-reflector-client
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast route-policy PASS out
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast default-originate
router bgp 6275 neighbor 10.255.255.9 address-family ipv4 unicast soft-reconfiguration inbound always
RP/0/0/CPU0:XRv-3#

Great, so XRv-9 is now learning a default route. But the problem is that – by default – iBGP chooses the best possible single path. To ensure we're using both links on XRv-9, we need to enable multipathing. We also need to add the default route to our table policy to ensure it's installed in the FIB.

RP/0/0/CPU0:XRv-9#show rpl route-policy TBL_PLCY detail 

prefix-set TBL
  0.0.0.0/0,
  203.0.113.0/24 le 32,
  25.54.60.0/24 le 32,
  8.8.8.0/24 le 32
end-set
!
route-policy TBL_PLCY
  if destination in TBL then
    pass
  else
    drop
  endif
end-policy
!
RP/0/0/CPU0:XRv-9#
RP/0/0/CPU0:XRv-9#show run router bgp 6275 address-family ipv4 unicast 

router bgp 6275
 address-family ipv4 unicast
  table-policy TBL_PLCY
  maximum-paths ibgp 2
 !
!

RP/0/0/CPU0:XRv-9#

Now the prefix should be installed & ECMP aware.

RP/0/0/CPU0:XRv-9#show route bgp   


B*   0.0.0.0/0 [200/0] via 10.255.255.3, 00:54:33
               [200/0] via 10.255.255.2, 00:54:33
B    8.8.8.0/24 [200/0] via 10.255.255.1, 00:56:37
B    25.54.60.0/24 [20/0] via 203.0.113.6, 00:56:37
RP/0/0/CPU0:XRv-9#    

Conclusion

As usual, the best way to find the truth is by saying the wrong answer aloud.

This design is much more scalable and requires far less impact to a greenfield addition to a brownfield network. It still meets the main requirement – reducing IPv4 / IPv6 unicast FIB entries due to low prefix / ECMP scale. The Tier-3 routers still peer with the main Route Reflector for address-families such as l2vpn evpn, for L3VPN & L2VPN services.

RP/0/0/CPU0:XRv-9#show bgp l2vpn evpn summary  | b Spk      

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd
10.255.255.10     0  6275     101      98        1    0    0 01:03:58          0

RP/0/0/CPU0:XRv-9#

Be sure to do this once in a while; Remember that perfect is the enemy of good. Go for the best design you can come up with at the moment. Socialize it around, sleep on it, and give it time. The topology is out there.

I was recently discussing a design with a colleague in the industry. He was tasked with deciding on a platform for his Tier 3 access rings; These rings would be low bandwidth and potentially oversubscribed. The platform he decided on was the NCS500 series.

The main problem he was having was delivering full tables to clients connected to the T3 ring, as the NCS500 series routers could not handle full tables in FIB. They were also low scale in general – as also seen in the NCS55XX series – with 4k ECMP FEC entries limitation.

The solution he landed on was to introduce PWHE's (Pseudo Wire Head-ends) into his network; The NCS500 routers would act as a point-to-point l2vpn x-connect to the S-PE's (Switching PE, that terminates the PWHE's).

I listened to the main concern he had in mind: the ability to advertise full tables to customers. I proposed he consider a BGP table-policy instead. He wasn't fully convinced by the idea, so I've decided to lab it up to explore more and discover it's efficacy for myself.

Lab Topology

I've created the following topology to represent a Tier-1 ring with a Tier-3 ring attached:

For the above topology, XRv-1 to 6 are our Tier 1 ring. They are acting as bigger routers, capable of holding full tables and multiple VPN services. Routers XRv-7 to 9 are the Tier 3 ring. These will act as our low scale NCS500 series routers. RR-1 is simply our IPv4 unicast route-reflector.

The two Peer routers are simply advertising EBGP prefixes to our border routers, for demonstrating the sub-optimal routing from XRv-9's perspective. I also setup a Customer CE router as an internet customer, to demonstrate it's view of the topology.

Configuring PWHE

Now it's time to build the proposed topology. The concept of a PWHE's was fairly new to me; Thus, I had to refer to documentation.

First we create our PWHE interface & matching L2VPN x-connect on XRv-3, acting as the S-PE. You can see that XRv-3 has an established BGP session with the CUST-1 router:

RP/0/0/CPU0:XRv-3#show run formal | i "PW|xconnect"

Building configuration...
interface PW-Ether100 
interface PW-Ether100 ipv4 address 203.0.113.5 255.255.255.252
interface PW-Ether100 attach generic-interface-list gil1
l2vpn xconnect group pwhe 
l2vpn xconnect group pwhe p2p xc_main_interface 
l2vpn xconnect group pwhe p2p xc_main_interface interface PW-Ether100
l2vpn xconnect group pwhe p2p xc_main_interface neighbor ipv4 10.255.255.9 pw-id 100 
RP/0/0/CPU0:XRv-3#
RP/0/0/CPU0:XRv-3#show bgp sessions

Neighbor        VRF                   Spk    AS   InQ  OutQ  NBRState     NSRState
10.255.255.10   default                 0  6275     0     0  Established  None
203.0.113.6     default                 0   789     0     0  Established  None
RP/0/0/CPU0:XRv-3#
RP/0/0/CPU0:XRv-3#show arp  | i "Add|PW-"

Address         Age        Hardware Addr   State      Type  Interface
203.0.113.5     -          0257.cc67.a402  Interface  ARPA  PW-Ether100
203.0.113.6     00:03:29   0c05.f335.0000  Dynamic    ARPA  PW-Ether100
RP/0/0/CPU0:XRv-3#

XRv-9 is delivering this connection purely via L2VPN x-connect:

RP/0/RP0/CPU0:XRv-9#show run l2vpn
l2vpn
 xconnect group pwhe
  p2p xc_main_intf
   interface GigabitEthernet0/0/0/2
   neighbor ipv4 10.255.255.3 pw-id 100
   !
  !
 !
!

RP/0/RP0/CPU0:XRv-9

RP/0/RP0/CPU0:XRv-9#show mpls ldp neighbor brief 

Peer               GR  NSR  Up Time     Discovery   Addresses     Labels    
                                        ipv4  ipv6  ipv4  ipv6  ipv4   ipv6 
-----------------  --  ---  ----------  ----------  ----------  ------------
10.255.255.3:0     N   N    00:02:35    1     0     5     0     0      0    

RP/0/RP0/CPU0:XRv-9#
RP/0/RP0/CPU0:XRv-9#show l2vpn xconnect pw-id 100        
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
        SB = Standby, SR = Standby Ready, (PP) = Partially Programmed

XConnect                   Segment 1                       Segment 2                
Group      Name       ST   Description            ST       Description            ST    
------------------------   -----------------------------   -----------------------------
pwhe       xc_main_intf
                      UP   Gi0/0/0/2              UP       10.255.255.3    100    UP    
----------------------------------------------------------------------------------------
RP/0/RP0/CPU0:XRv-9#

\\ The BGP Upstream peers are also learning the advertised prefix from the customer

kazaii@CUST-1:~$ show ip route connected 

C>* 25.54.60.0/24 is directly connected, lo, 00:14:17
C>* 203.0.113.4/30 is directly connected, eth0, 00:14:17
kazaii@CUST-1:~$

kazaii@Peer-1:~$ show bgp ipv4 25.54.60.0 bestpath 
BGP routing table entry for 25.54.60.0/24
Paths: (2 available, best #2, table default)
  Advertised to non peer-group peers:
  1.1.1.2 70.20.31.2
  6275 789
    1.1.1.2 from 1.1.1.2 (10.255.255.5)
      Origin incomplete, valid, external, best (AS Path)
      Last update: Tue Jul 26 00:17:58 2022
kazaii@Peer-1:~$ 

It's worth noting that the above configuration created the pseudowire via LDP signalling. You can, however, utilize EVPN-VPWS .. to keep this BGP signalled and less overhead.

The use of EVPN for VPWS eliminates the need for signaling single-segment and multi-segment PWs for point-to-point Ethernet services. You can also configure the PWHE interface and a bridge domain access pseudowire using EVPN-VPWS.

Customer View over PWHE

Finally, the Customer Router is able to reach the PWHE interface and is learning the full table

kazaii@CUST-1:~$ ping 203.0.113.5 
PING 203.0.113.5 (203.0.113.5) 56(84) bytes of data.
64 bytes from 203.0.113.5: icmp_seq=1 ttl=255 time=2.31 ms
64 bytes from 203.0.113.5: icmp_seq=2 ttl=255 time=2.31 ms
64 bytes from 203.0.113.5: icmp_seq=3 ttl=255 time=1.85 ms
64 bytes from 203.0.113.5: icmp_seq=4 ttl=255 time=1.69 ms
^C
--- 203.0.113.5 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 8ms
rtt min/avg/max/mdev = 1.689/2.040/2.313/0.277 ms
kazaii@CUST-1:~$ show bgp summary 

IPv4 Unicast Summary:
BGP router identifier 25.54.60.1, local AS number 789 vrf-id 0
BGP table version 16
RIB entries 31, using 5952 bytes of memory
Peers 1, using 21 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
203.0.113.5     4       6275         6         7        0    0    0 00:01:28           14       16

Total number of neighbors 1
kazaii@CUST-1:~$ 

kazaii@CUST-1:~$ show ip route bgp
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

B>* 1.1.1.0/30 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 1.1.1.100/32 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 2.2.2.0/30 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 2.2.2.100/32 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 8.8.8.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 23.74.88.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 50.50.50.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 60.60.60.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 66.77.230.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 70.20.31.0/30 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 70.70.70.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 99.99.99.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 173.45.20.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
B>* 209.20.35.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:01:38
kazaii@CUST-1:~$ 

PWHE Caveats

That wasn't so hard. What's the problem? Well, there are a few:

• The L2VPN x-connect terminates on XRv-3. That means all flows go through XRv-8, even if the shortest path is via XRv-7 to Peer-2, for example. That's sub-optimal routing (even if it's just one extra hop).
• Load-balancing – both ECMP and over Bundle interfaces – is not supported by default in this setup. You need to configure FAT labels, also known as entropy labels. This adds more complexity to the design... (I would have demonstrated this, but my XRv routers do not support it).
• The lack of load-balancing causes under-utilized links. For example, if this was our only X-connect, the link to XRv-7 would be completely unused, unless there was a ring break. This would also be true for links in the Bundle.
• To balance X-connects over the links, you could program 50% of the links on XRv-2 and XRv-3... but this introduces a lot of state and planning. Especially as you can't predict the flow patterns of Customer 1 vs Customer 2, 3 etc... it would be better if ECMP hashing could balance the flows.

Here is the demonstration of the sub-optimal routing:

kazaii@CUST-1:~$ traceroute 99.99.99.99
traceroute to 99.99.99.99 (99.99.99.99), 30 hops max, 60 byte packets
 1  203.0.113.5 (203.0.113.5)  8.829 ms  8.621 ms  8.553 ms
 2  10.2.3.2 (10.2.3.2)  23.717 ms  23.535 ms  23.409 ms
 3  10.1.2.1 (10.1.2.1)  25.956 ms  25.807 ms  25.758 ms
 4  99.99.99.99 (99.99.99.99)  25.464 ms  25.319 ms  25.184 ms
kazaii@CUST-1:~$


kazaii@CUST-1:~$ traceroute  209.20.35.1
traceroute to 209.20.35.1 (209.20.35.1), 30 hops max, 60 byte packets
 1  203.0.113.5 (203.0.113.5)  6.066 ms  5.842 ms  5.618 ms
 2  10.3.4.4 (10.3.4.4)  11.545 ms  11.504 ms  11.365 ms
 3  10.4.5.5 (10.4.5.5)  13.402 ms  17.144 ms  17.106 ms
 4  209.20.35.1 (209.20.35.1)  16.977 ms  16.842 ms  16.820 ms
kazaii@CUST-1:~$



kazaii@Peer-1:~$ show configuration commands | match "address|bgp"
set interfaces ethernet eth0 address '1.1.1.1/30'
set interfaces ethernet eth1 address '70.20.31.1/30'
set interfaces loopback lo address '1.1.1.100/32'
set interfaces loopback lo address '50.50.50.50/24'
set interfaces loopback lo address '60.60.60.60/24'
set interfaces loopback lo address '70.70.70.70/24'
set interfaces loopback lo address '209.20.35.1/24'
set protocols bgp 123 address-family ipv4-unicast redistribute connected
set protocols bgp 123 neighbor 1.1.1.2 address-family ipv4-unicast
set protocols bgp 123 neighbor 1.1.1.2 remote-as '6275'
set protocols bgp 123 neighbor 70.20.31.2 address-family ipv4-unicast
set protocols bgp 123 neighbor 70.20.31.2 remote-as '222'
kazaii@Peer-1:~$


kazaii@Peer-2:~$ show configuration commands | match "address|bgp"
set interfaces ethernet eth0 address '2.2.2.1/30'
set interfaces ethernet eth1 address '70.20.31.2/30'
set interfaces loopback lo address '2.2.2.100/32'
set interfaces loopback lo address '66.77.230.1/24'
set interfaces loopback lo address '99.99.99.99/24'
set interfaces loopback lo address '23.74.88.1/24'
set interfaces loopback lo address '173.45.20.1/24'
set interfaces loopback lo address '8.8.8.8/24'
set protocols bgp 222 address-family ipv4-unicast redistribute connected
set protocols bgp 222 neighbor 2.2.2.2 address-family ipv4-unicast
set protocols bgp 222 neighbor 2.2.2.2 remote-as '6275'
set protocols bgp 222 neighbor 70.20.31.1 address-family ipv4-unicast
set protocols bgp 222 neighbor 70.20.31.1 remote-as '123'
kazaii@Peer-2:~$ 

As seen above, regardless of the shortest path to Peer 1 or 2, the traffic is going via XRv-3.

Alternative Proposal: Table Policy

My alternative proposal would be to keep the peering on XRv-9, and drop the Tier-3 ring down to an IS-IS Level 1 domain. This would set the attached-bit (default route) and you could filter entries from the FIB with a table-policy. Let's take a look.

XRv-2 & XRv-3 are the Level 1-2 routers (the ABR's). They are put into another area, as this is required to have the attached-bit set.

RP/0/0/CPU0:XRv-2# show run formal router isis | i "49.|prop|circu"

router isis zeal net 49.0001.0000.0000.0002.00
router isis zeal address-family ipv4 unicast propagate level 2 into level 1 route-policy LOO
router isis zeal interface GigabitEthernet0/0/0/0 circuit-type level-2-only
router isis zeal interface GigabitEthernet0/0/0/1 circuit-type level-2-only
router isis zeal interface GigabitEthernet0/0/0/2 circuit-type level-1
RP/0/0/CPU0:XRv-2#
RP/0/0/CPU0:XRv-2#show isis adjacency | i "Lev|XR"

IS-IS zeal Level-1 adjacencies:
XRv-7          Gi0/0/0/2        0cfe.1ea7.0001 Up    7    00:08:30 Yes None None
IS-IS zeal Level-2 adjacencies:
XRv-1          Gi0/0/0/0        0c7b.5763.0001 Up    28   00:09:35 Yes None None
XRv-3          Gi0/0/0/1        0c50.dac2.0001 Up    25   00:09:03 Yes None None
RP/0/0/CPU0:XRv-2#

RP/0/0/CPU0:XRv-3#show run formal router isis | i "49.|prop|circu"

router isis zeal net 49.0001.0000.0000.0003.00
router isis zeal address-family ipv4 unicast propagate level 2 into level 1 route-policy LOO
router isis zeal interface GigabitEthernet0/0/0/0 circuit-type level-2-only
router isis zeal interface GigabitEthernet0/0/0/1 circuit-type level-2-only
router isis zeal interface GigabitEthernet0/0/0/2 circuit-type level-1
RP/0/0/CPU0:XRv-3#
RP/0/0/CPU0:XRv-3#show isis adjacency | i "Lev|XR"

IS-IS zeal Level-1 adjacencies:
XRv-8          Gi0/0/0/2        0ca7.981d.0001 Up    9    00:07:42 Yes None None
IS-IS zeal Level-2 adjacencies:
XRv-2          Gi0/0/0/0        0cc6.415c.0002 Up    8    00:08:40 Yes None None
XRv-4          Gi0/0/0/1        0c9c.d3f4.0001 Up    7    00:08:40 Yes None None
RP/0/0/CPU0:XRv-3#

You can see the ABR's are advertising the default route, as the attached-bit is set:

RP/0/0/CPU0:XRv-9#show isis database | i "LSPID|1/0"

LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime  ATT/P/OL
XRv-2.00-00           0x00000007   0x4e2e        594             1/0/0
XRv-3.00-00           0x00000008   0x80ea        602             1/0/0
RP/0/0/CPU0:XRv-9#
RP/0/0/CPU0:XRv-9#show route isis


i*L1 0.0.0.0/0 [115/20] via 10.7.9.7, 00:10:10, GigabitEthernet0/0/0/1
               [115/20] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0
i ia 10.255.255.1/32 [115/30] via 10.7.9.7, 00:10:10, GigabitEthernet0/0/0/1
                     [115/40] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0 (!)
i L1 10.255.255.2/32 [115/20] via 10.7.9.7, 00:10:34, GigabitEthernet0/0/0/1
i L1 10.255.255.3/32 [115/20] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0
i ia 10.255.255.4/32 [115/40] via 10.7.9.7, 00:10:10, GigabitEthernet0/0/0/1 (!)
                     [115/30] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0
i ia 10.255.255.5/32 [115/50] via 10.7.9.7, 00:10:10, GigabitEthernet0/0/0/1 (!)
                     [115/40] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0
i ia 10.255.255.6/32 [115/40] via 10.7.9.7, 00:10:10, GigabitEthernet0/0/0/1
                     [115/50] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0 (!)
i L1 10.255.255.7/32 [115/10] via 10.7.9.7, 00:10:34, GigabitEthernet0/0/0/1
i L1 10.255.255.8/32 [115/10] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0
i ia 10.255.255.10/32 [115/50] via 10.7.9.7, 00:10:10, GigabitEthernet0/0/0/1
                      [115/50] via 10.8.9.8, 00:10:10, GigabitEthernet0/0/0/0
RP/0/0/CPU0:XRv-9#

XR-9 also has inter-area routes to the loopbacks from the Tier-1 ring. This is because the ABR's are propagating this through, as shown in the config above and a closer look at the policy:

RP/0/0/CPU0:XRv-3#show rpl route-policy LOO detail 
Tue Jul 26 02:51:52.019 UTC
prefix-set LOO
  10.255.255.0/24 le 32
end-set
!
route-policy LOO
  if destination in LOO then
    pass
  else
    drop
  endif
end-policy
!
RP/0/0/CPU0:XRv-3#

This allows XRv-9 to peer with other PE's to maintain L2VPN/L3VPN services. In this case, it is simply utilizing this policy to peer with the v4 route-reflector, to learn the BGP routing table.

RP/0/0/CPU0:XRv-9#show bgp sessions


Neighbor        VRF                   Spk    AS   InQ  OutQ  NBRState     NSRState
10.255.255.10   default                 0  6275     0     0  Established  None
203.0.113.6     default                 0   789     0     0  Established  None
RP/0/0/CPU0:XRv-9#

Limiting the FIB

Finally, to not overwhelm the little router's small TCAM, we are utilizing a table-policy that filters only the routes we want to install in FIB. The most crucial being the route back to the Customer... as only having the default-route would cause a routing loop ping-pong between the ABR's and XRv-9 (the ABR's learn the customer prefix via XRv-9, but XRv-9 doesn't install it in FIB so it's recursive lookup is via the default route to the ABR's)

RP/0/0/CPU0:XRv-9#show rpl route-policy TBL_PLCY detail 

prefix-set TBL
  203.0.113.0/24 le 32,
  25.54.60.0/24 le 32
end-set
!
route-policy TBL_PLCY
  if destination in TBL then
    pass
  else
    drop
  endif
end-policy
!
RP/0/0/CPU0:XRv-9#


RP/0/0/CPU0:XRv-9#show route bgp


B    25.54.60.0/24 [20/0] via 203.0.113.6, 00:11:14
RP/0/0/CPU0:XRv-9#

Despite there only being one prefix in FIB, XRv-9 still holds the full table in RIB:

RP/0/0/CPU0:XRv-9#show bgp ipv4 unicast | b Network

   Network            Next Hop            Metric LocPrf Weight Path
*>i1.1.1.0/30         10.255.255.5             0    100      0 123 ?
*>i1.1.1.100/32       10.255.255.5             0    100      0 123 ?
*>i2.2.2.0/30         10.255.255.1             0    100      0 222 ?
*>i2.2.2.100/32       10.255.255.1             0    100      0 222 ?
*>i8.8.8.0/24         10.255.255.1             0    100      0 222 ?
*>i23.74.88.0/24      10.255.255.1             0    100      0 222 ?
*> 25.54.60.0/24      203.0.113.6              0             0 789 ?
*>i50.50.50.0/24      10.255.255.5             0    100      0 123 ?
*>i60.60.60.0/24      10.255.255.5             0    100      0 123 ?
*>i66.77.230.0/24     10.255.255.1             0    100      0 222 ?
*>i70.20.31.0/30      10.255.255.5             0    100      0 123 ?
*>i70.70.70.0/24      10.255.255.5             0    100      0 123 ?
*>i99.99.99.0/24      10.255.255.1             0    100      0 222 ?
*>i173.45.20.0/24     10.255.255.1             0    100      0 222 ?
*> 203.0.113.4/30     203.0.113.6              0             0 789 ?
*>i209.20.35.0/24     10.255.255.5             0    100      0 123 ?

Processed 16 prefixes, 16 paths
RP/0/0/CPU0:XRv-9#

Thus it has the capability to advertise this table to CUST-1

kazaii@CUST-1:~$ show ip route bgp

B>* 1.1.1.0/30 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 1.1.1.100/32 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 2.2.2.0/30 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 2.2.2.100/32 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 8.8.8.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 23.74.88.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 50.50.50.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 60.60.60.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 66.77.230.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 70.20.31.0/30 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 70.70.70.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 99.99.99.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 173.45.20.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
B>* 209.20.35.0/24 [20/0] via 203.0.113.5, eth0, weight 1, 00:11:53
kazaii@CUST-1:~$

There, we solved our problem. We now have 7-tuple ECMP hashing and the link to XRv-7 is utilized. This design is fairly un-orthodox for a flat MPLS transport ring; However, it still is far less state than many L2VPN x-connects.

Table-Policy / IS-IS Level-1 Caveats

There are – of course – some caveats to consider...

First, there is load-balancing, but it still does not consider the true shortest path to the border routers; It simply does flow hashing to determine the next-hop to the ABR's. I'll use this perfectly imperfect example:

kazaii@CUST-1:~$ traceroute 209.20.35.1 
traceroute to 209.20.35.1 (209.20.35.1), 30 hops max, 60 byte packets
 1  203.0.113.5 (203.0.113.5)  0.908 ms  0.806 ms  0.786 ms
 2  10.8.9.8 (10.8.9.8)  9.258 ms  9.710 ms  9.690 ms
 3  10.3.8.3 (10.3.8.3)  7.198 ms  7.341 ms  10.798 ms
 4  10.3.4.4 (10.3.4.4)  18.422 ms  18.538 ms  18.689 ms
 5  10.4.5.5 (10.4.5.5)  19.807 ms  20.423 ms  20.400 ms
 6  209.20.35.1 (209.20.35.1)  20.490 ms * *
kazaii@CUST-1:~$ traceroute  60.60.60.60
traceroute to 60.60.60.60 (60.60.60.60), 30 hops max, 60 byte packets
 1  203.0.113.5 (203.0.113.5)  1.090 ms  0.989 ms  0.972 ms
 2  10.7.9.7 (10.7.9.7)  23.659 ms  24.971 ms  24.946 ms
 3  10.2.7.2 (10.2.7.2)  24.850 ms  25.107 ms  25.090 ms
 4  10.1.2.1 (10.1.2.1)  38.416 ms  38.583 ms  38.561 ms
 5  10.1.6.6 (10.1.6.6)  47.487 ms  47.463 ms  47.649 ms
 6  10.5.6.5 (10.5.6.5)  33.546 ms  33.672 ms  33.591 ms
 7  60.60.60.60 (60.60.60.60)  45.792 ms  23.508 ms  21.985 ms
kazaii@CUST-1:~$ 

If you look closely, the first flow went via the optimal path. As XRv-8 is the shortest path to Peer1. Sadly, the second flow went the sub-optimal path, via XRv-7

kazaii@Peer-1:~$ show interfaces | match 60.
                 60.60.60.60/24                         
kazaii@Peer-1:~$ show interfaces | match 209.20.
                 209.20.35.1/24                         
kazaii@Peer-1:~$ 

You might want to tackle this by being selective of your prefixes you install in FIB. For example, you could perform some Netflow sampling to determine where most customers are sending and receiving their traffic from. A low hanging fruit would be google's DNS:

RP/0/0/CPU0:XRv-9#show run prefix-set TBL

prefix-set TBL
  203.0.113.0/24 le 32,
  25.54.60.0/24 le 32,
  8.8.8.0/24 le 32
end-set
!

RP/0/0/CPU0:XRv-9#show route bgp


B    8.8.8.0/24 [200/0] via 10.255.255.1, 00:00:31
B    25.54.60.0/24 [20/0] via 203.0.113.6, 00:00:31
RP/0/0/CPU0:XRv-9#                         
RP/0/0/CPU0:XRv-9#
RP/0/0/CPU0:XRv-9#show cef 8.8.8.0 | i "/24|adja|hop"

8.8.8.0/24, version 150, internal 0x1000001 0x0 (ptr 0xa143c374) [1], 0x0 (0x0), 0x0 (0x0)
 local adjacency 10.7.9.7
    next hop 10.255.255.1/32 via 10.255.255.1/32
RP/0/0/CPU0:XRv-9#
RP/0/0/CPU0:XRv-9#traceroute 8.8.8.8 so lo0


Type escape sequence to abort.
Tracing the route to 8.8.8.8

 1  10.7.9.7 [MPLS: Label 16001 Exp 0] 9 msec  0 msec  0 msec 
 2  10.2.7.2 [MPLS: Label 16001 Exp 0] 0 msec  0 msec  0 msec 
 3  10.1.2.1 0 msec  0 msec  0 msec 
 4  8.8.8.8 0 msec  0 msec  0 msec 
RP/0/0/CPU0:XRv-9#

Secondly, since Segment Routing Fast-Reroute is calculated based on the IS-IS topology, we broke FRR:

RP/0/0/CPU0:XRv-9#show isis fast-reroute  | utility egrep -B2 "No"

       E - EIGRP, A - access/subscriber, M - mobile, a - application
       i - IS-IS (redistributed from another instance)
       D - Downstream, LC - Line card disjoint, NP - Node protecting
--
L1 10.255.255.2/32 [20/115]
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       No FRR backup
--
L1 10.255.255.3/32 [20/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       No FRR backup
--
L1 10.255.255.7/32 [10/115]
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       No FRR backup
--
L1 10.255.255.8/32 [10/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       No FRR backup
RP/0/0/CPU0:XRv-9#

That's because the Level-1 and Level-2 topologies are separate. The link between the ABR's is Level-2, thus breaking the topology ring. The simple fix is the make the circuit between the ABR's a Level 1-2 circuit. This adds a bit of overhead... but the Tier 1 router can handle it:

P/0/0/CPU0:XRv-2#show isis interface gigabitEthernet 0/0/0/1 | i 1-2 

  Circuit Type:             level-1-2
RP/0/0/CPU0:XRv-2#
RP/0/0/CPU0:XRv-2#show isis adjacency | i "Level|XRv-3"

IS-IS zeal Level-1 adjacencies:
XRv-3          Gi0/0/0/1        0c50.dac2.0001 Up    21   00:00:17 Yes None None
IS-IS zeal Level-2 adjacencies:
XRv-3          Gi0/0/0/1        0c50.dac2.0001 Up    28   00:46:31 Yes None None
RP/0/0/CPU0:XRv-2


RP/0/0/CPU0:XRv-3#show isis interface gigabitEthernet 0/0/0/0 | i 1- 

  Circuit Type:             level-1-2
RP/0/0/CPU0:XRv-3#
RP/0/0/CPU0:XRv-3#show isis adjacency | i "Level|XRv-2"

IS-IS zeal Level-1 adjacencies:
XRv-2          Gi0/0/0/0        0cc6.415c.0002 Up    8    00:04:37 Yes None None
IS-IS zeal Level-2 adjacencies:
XRv-2          Gi0/0/0/0        0cc6.415c.0002 Up    7    00:50:53 Yes None None
RP/0/0/CPU0:XRv-3#


RP/0/0/CPU0:XRv-9#show isis fast-reroute | utility egrep -i "/32|/0|backup"

IS-IS zeal IPv4 Unicast FRR backups
       P - Primary path, SRLG - SRLG disjoint, TM - Total metric via backup
df 0.0.0.0/0 [20/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       FRR backup via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0, Metric: 20
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       FRR backup via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0, Metric: 20
ia 10.255.255.1/32 [30/115]
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       FRR backup via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0, Metric: 40
L1 10.255.255.2/32 [20/115]
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       FRR backup via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0, Metric: 30
L1 10.255.255.3/32 [20/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       FRR backup via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0, Metric: 30
ia 10.255.255.4/32 [30/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       FRR backup via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0, Metric: 40
ia 10.255.255.5/32 [40/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       FRR backup via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0, Metric: 50
ia 10.255.255.6/32 [40/115]
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       FRR backup via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0, Metric: 50
L1 10.255.255.7/32 [10/115]
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
         Backup path: TI-LFA (srlg), via 10.8.9.8, GigabitEthernet0/0/0/0 XRv-8, SRGB Base: 16000, Weight: 0
L1 10.255.255.8/32 [10/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
         Backup path: TI-LFA (srlg), via 10.7.9.7, GigabitEthernet0/0/0/1 XRv-7, SRGB Base: 16000, Weight: 0
ia 10.255.255.10/32 [50/115]
     via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0
       FRR backup via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0, Metric: 50
     via 10.7.9.7, GigabitEthernet0/0/0/1, XRv-7, SRGB Base: 16000, Weight: 0
       FRR backup via 10.8.9.8, GigabitEthernet0/0/0/0, XRv-8, SRGB Base: 16000, Weight: 0, Metric: 50
RP/0/0/CPU0:XRv-9

Thirdly, we noted at the beginning the NCS540 has a lot of scale issues. Not just the total amount of prefixes it can hold, but also the 4096 ECMP FEC limitation.

You could combat this with a similar strategy of advertising a default route from the ABR's to the Tier 3 PE's in the L3VPN VRF. Then you could selectively import the prefixes with a route-policy & prefix-set

RP/0/0/CPU0:XRv-9(config)#show configuration 

Building configuration...
!! IOS XR Configuration 6.1.3
vrf TEST
 address-family ipv4 unicast
  import route-policy DEFAULT_ONLY
  import route-target
   6275:999
  !
  export route-target
   6275:999
  !
 !
!
end

RP/0/0/CPU0:XRv-9(config)#

The final drawback is that IS-IS area in the net address.... to configure that in a greenfield network? No problem. Configuring this in a production ring will cause IS-IS to re-form adjacencies, effectively causing ring breaks in our example topology. Be sure to do this during an outage window.

The things we do for CapEx

For me, this design is a lot easier to trace & troubleshoot than entropy labels & pseudowires. I suppose the much easier solution would be to spend more money on bigger routers. If only...

Let's utilize another hypothetical scenario:

Susan, a talented member of the sysadmin team, advises Mike, from the NOC, of intermittent slowness between two points in a client's managed network. Because Susan is a pro, she provides the source & destination IP, the application Network name, and some captures & outputs.

She advises that some application flows –  like curl – seem to be nearly instant in transmission; However, her current SSH session & icmp echo requests/replies seem to be plagued with delay. She also advises that she sees no indication of packet loss.

susan@Host-1:~$ ping 10.20.20.20 -c 3
PING 10.20.20.20 (10.20.20.20) 56(84) bytes of data.
64 bytes from 10.20.20.20: icmp_seq=1 ttl=60 time=208 ms
64 bytes from 10.20.20.20: icmp_seq=2 ttl=60 time=207 ms
64 bytes from 10.20.20.20: icmp_seq=3 ttl=60 time=273 ms

--- 10.20.20.20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 207.702/229.974/273.545/30.816 ms
susan@Host-1:~$ 

For this scenario, I am utilizing my SR refresher lab again. I have built a EVPN-SR Route-Type 5 (L3VPN) connection between XRv-1 & XRv-10

Mike knows that traceroute would produce less than desirable results:

RP/0/0/CPU0:XRv-1#traceroute 10.255.255.10 so lo0
Sun Jul 10 17:00:02.890 UTC

Type escape sequence to abort.
Tracing the route to 10.255.255.10

 1   * 
    10.9.10.10 0 msec  * 
RP/0/0/CPU0:XRv-1

Why is this so? Because someone decided to obfuscate the main thing traceroute relies on, TTL.

RP/0/0/CPU0:XRv-1#show run | i propa
Sun Jul 10 17:01:02.001 UTC
Building configuration...
mpls ip-ttl-propagate disable
RP/0/0/CPU0:XRv-1#

Mike now considers how he will sniff out this latency. He could check the logs of various routers for issues; Yet, how would he know which routers the flow would traverse? He knows that the equal cost paths of the routers provides multiple ways to reach the Z-end, but the ultimate path is determined by the 7-tuple hash calculation. He also knows that traceroute might cause him to miss something – due to the paris traceroute problem – even if he found a way to perform one.

Mike desperately hopes that the issue is perhaps with the CE routers, or the last mile, as that would be a toss-over to field ops.

RP/0/0/CPU0:XRv-1# ping 10.10.10.10  vrf CUST-1
Sun Jul 10 17:07:02.890 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.10, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-10#ping 10.20.20.20  vrf CUST-1
Sun Jul 10 17:07:37.458 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.20.20, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
RP/0/0/CPU0:XRv-10#

Clean as a whistle; Darn. It's time to put the work in.

Mike has to utilize commands that expose the actual forwarding state – i.e. CEF / FIB – rather than what the RIB advises could take place. He reviews Cisco IOS-XR documentation, the packet captures variables, and produces the following command:

show cef vrf CUST-1 exact-route 10.10.10.10 10.20.20.20 protocol tcp source-port 41890 destination-port 22 ingress-interface gigabitEthernet 0/0/0/1

RP/0/0/CPU0:XRv-1#show cef vrf CUST-1 exact-route 10.10.10.10 10.20.20.20 prot$
Sun Jul 10 17:28:54.670 UTC
10.20.20.0/24, version 5, internal 0x1000001 0x0 (ptr 0xa143c3f4) [1], 0x0 (0x0), 0x208 (0xa15832d0)
 Updated Jul 13 23:29:08.783 
 local adjacency 10.1.2.2
 Prefix Len 24, traffic index 0, precedence n/a, priority 3
   via GigabitEthernet0/0/0/0
   via 10.255.255.10/32, 3 dependencies, recursive [flags 0x6000]
    path-idx 0 NHID 0x0 [0xa15ebdf4 0x0]
    recursion-via-/32
    next hop VRF - 'default', table - 0xe0000000
    next hop 10.255.255.10/32 via 16010/0/21
     next hop 10.1.2.2/32 Gi0/0/0/0    labels imposed {16010 81539}
RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#ping 10.1.2.2 count 100                 
Sun Jul 10 17:28:55.990 UTC
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.1.2.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-1#

Mike then moves onto XRv-2. He then considers the next command he needs to consider the full packet flow, as XRv-2 is a P router simply forwarding the encapsulated packets; It does not have knowledge of the CUST-1 VRF.

He knows that 16010 is the outer label and 81539 is the service label – or the bottom label. All other parameters have not changed. He considers the following two commands:

show mpls forwarding exact-route label 16010 ipv4 10.10.10.10 10.20.20.20 protocol tcp source-port 41890 destination-port 22 ingress-interface gigabitEthernet 0/0/0/0 detail

show mpls forwarding exact-route label 16010 bottom-label 81539 protocol tcp source-port 41890 destination-port 22 ingress-interface gigabitEthernet 0/0/0/0 detail

They're not quite right.... so he scours his documentation and finds the following:

show mpls forwarding exact-route label-stack {16010 81539} ipv4 10.10.10.10 10.20.20.20 protocol tcp source-port 41890 destination-port 22 ingress-interface gigabitEthernet 0/0/0/0 detail

Perfect. Mike utilizes his IS-IS adjacency, to determine the ingress interface each router will receive the packet on. He then runs the above command and determines the next-hop. He will ping each hop and continue on, until he finds the problematic router or link.

RP/0/0/CPU0:XRv-2#show isis topology | utility egrep -iw XRv-1
Sun Jul 10 17:44:53.815 UTC
XRv-1           10      XRv-1           Gi0/0/0/0       0c97.2455.0001
RP/0/0/CPU0:XRv-2#show mpls forwarding exact-route label-stack {16010 81539} i$
Sun Jul 10 17:45:14.903 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes       
Label  Label       or ID              Interface                    Switched    
------ ----------- ------------------ ------------ --------------- ------------
16010  16010       SR Pfx (idx 10)    Gi0/0/0/2    10.2.7.7        N/A         
     Updated: Jul 10 16:23:42.658
     Via: Gi0/0/0/2, Next Hop: 10.2.7.7
     Label Stack (Top -> Bottom): { 16010 }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 14/18, MTU: 1500

RP/0/0/CPU0:XRv-2#ping 10.2.7.7 count 100 
Sun Jul 10 17:45:25.512 UTC
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.2.7.7, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-2#

RP/0/0/CPU0:XRv-7# show isis adjacency | i XRv-2                               
Sun Jul 10 17:45:46.211 UTC
XRv-2          Gi0/0/0/0        0c3a.f889.0003 Up    26   01:22:25 Yes None None
RP/0/0/CPU0:XRv-7#show mpls forwarding exact-route label-stack {16010 81539} i$
Sun Jul 10 17:45:52.241 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes       
Label  Label       or ID              Interface                    Switched    
------ ----------- ------------------ ------------ --------------- ------------
16010  16010       SR Pfx (idx 10)    Gi0/0/0/3    10.6.7.6        N/A         
     Updated: Jul 10 16:23:44.659
     Via: Gi0/0/0/3, Next Hop: 10.6.7.6
     Label Stack (Top -> Bottom): { 16010 }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 14/18, MTU: 1500

RP/0/0/CPU0:XRv-7#ping 10.6.7.6 count 100
Sun Jul 10 17:46:07.299 UTC
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.6.7.6, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-7#


RP/0/0/CPU0:XRv-6#show isis adjacency | i XRv-7
Sun Jul 10 17:46:30.248 UTC
XRv-7          Gi0/0/0/5        0cf0.832b.0004 Up    9    01:23:08 Yes None None
RP/0/0/CPU0:XRv-6#show mpls forwarding exact-route label-stack {16010 81539} i$
Sun Jul 10 17:46:34.228 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes       
Label  Label       or ID              Interface                    Switched    
------ ----------- ------------------ ------------ --------------- ------------
16010  16010       SR Pfx (idx 10)    Gi0/0/0/3    10.6.9.9        N/A         
     Updated: Jul 10 16:23:44.049
     Via: Gi0/0/0/3, Next Hop: 10.6.9.9
     Label Stack (Top -> Bottom): { 16010 }
     NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
     MAC/Encaps: 14/18, MTU: 1500

RP/0/0/CPU0:XRv-6#ping 10.6.9.9 count 100
Sun Jul 10 17:46:40.547 UTC
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.6.9.9, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 199/204/299 ms
RP/0/0/CPU0:XRv-6#

Wow. There it is. 200ms of latency! He quickly moves to the other router to see if the latency seems to be specific to the link, or the router:

RP/0/0/CPU0:XRv-9#ping 10.9.10.10 count 100 
Sun Jul 10 17:48:16.181 UTC
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.9.10.10, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-9#show isis adjacency | i XRv-6
Sun Jul 10 17:48:46.749 UTC
XRv-6          Gi0/0/0/1        0c83.0a4f.0004 Up    7    01:25:24 Yes None None
RP/0/0/CPU0:XRv-9#

RP/0/0/CPU0:XRv-10#  ping 10.9.10.9 count 100 
Sun Jul 10 17:48:22.900 UTC
Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.9.10.9, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/9 ms
RP/0/0/CPU0:XRv-10#

Both tests are clean. The issue does not appear to be XRv-9 itself. The issue seems to be when packets flow on the link between XRv-6 & XRv-9. Mike passes the ticket over to the TNOC. The TNOC finds an issue with one of the DWDM shelves and takes action.

After this is completed, Susan is happy to find that SSH is now responsive and her packets are flowing without issue:

susan@Host-1:~$ ping 10.20.20.1 -c 3
PING 10.20.20.1 (10.20.20.1) 56(84) bytes of data.
64 bytes from 10.20.20.1: icmp_seq=1 ttl=61 time=7.94 ms
64 bytes from 10.20.20.1: icmp_seq=2 ttl=61 time=8.39 ms
64 bytes from 10.20.20.1: icmp_seq=3 ttl=61 time=8.30 ms

--- 10.20.20.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 7.948/8.215/8.399/0.220 ms
susan@Host-1:~$ 

Ask yourself how you would trace a flow in your network

Whatever it was you 'didn't change', change it back
- Russ White

Hedge 134: Ten Things

One of the many reasons engineers should work for a vendor, consulting company, or someone other than a single network operator at some point in their career is to develop a larger view of network …

rule 11 readerauthor page

In my personal experience, I've found this insight to be true. Most network outages seem to be caused by lack of peer review, lack of testing, careless implementation, lack of change control oversight, and various other lacks of control over operations.

Example Scenario

Let me make a theoretical example to demonstrate. Picture an early afternoon on a beautiful July day. The NOC was out of their chairs, chatting, laughing, and discussing their activities for the upcoming weekend. Suddenly, the board went red. Moments later, tickets started to roll through the floodgates. Apps were down, customers were down –  there was now a Priority One incident in progress.

What's the problem? Just another day in Ops Life, no? Unfortunately, nobody had any clue what had happened. One eager operator scrolled through their alarm board and found no indication of fault; No LOS condition, adjacency lost, core device offline, etc.

What could have happened here?

Topology

To demonstrate this outage, I am using my GNS3 lab above, intended for my Segment Routing testing/refresher. This is typically a flat IS-IS topology; However, I've changed the adjacency between XRv-1 & XRv-2 from IS-IS to BGP.

I've also created a set amount of "Networks" to be advertised to the core via XRv-2. These networks are then summarized via an aggregate route. This method is not uncommon to advertise a set of prefixes from a DC, region, domain, etc.

RP/0/0/CPU0:XRv-1#show ip int br | i Loo  
Sat Jul  2 17:51:54.871 UTC
Loopback0                      10.255.255.1    Up              Up       default 
Loopback48                     10.48.48.1      Up              Up       default 
Loopback50                     10.50.50.1      Up              Up       default 
Loopback51                     10.50.51.1      Up              Up       default 
Loopback52                     10.50.52.1      Up              Up       default 
Loopback53                     10.50.53.1      Up              Up       default 
Loopback54                     10.50.54.1      Up              Up       default 
Loopback55                     10.50.55.1      Up              Up       default 
Loopback56                     10.50.56.1      Up              Up       default 
Loopback57                     10.50.57.1      Up              Up       default 
Loopback58                     10.50.58.1      Up              Up       default 
Loopback60                     10.60.60.1      Up              Up       default 
Loopback61                     10.60.61.1      Up              Up       default 
Loopback62                     10.60.62.1      Up              Up       default 
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1# show run router bgp 1 address-family ipv4 unicast     
Sat Jul  2 18:22:26.476 UTC
router bgp 1
 address-family ipv4 unicast
  aggregate-address 10.48.0.0/12 summary-only
  redistribute connected
 !
!

RP/0/0/CPU0:XRv-1#show bgp sessions
Sat Jul  2 18:24:15.598 UTC

Neighbor        VRF                   Spk    AS   InQ  OutQ  NBRState     NSRState
10.1.2.2        default                 0     2     0     0  Established  None
RP/0/0/CPU0:XRv-1#


RP/0/0/CPU0:XRv-2#show bgp ipv4 unicast neighbors 10.1.2.1 received routes | b Network
Sat Jul  2 18:27:02.247 UTC
   Network            Next Hop            Metric LocPrf Weight Path
*> 10.1.2.0/24        10.1.2.1                 0             0 1 ?
*> 10.48.0.0/12       10.1.2.1                               0 1 i
*> 10.255.255.1/32    10.1.2.1                 0             0 1 ?
*> 100.64.0.0/24      10.1.2.1                 0             0 1 ?

Processed 4 prefixes, 4 paths
RP/0/0/CPU0:XRv-2#

RP/0/0/CPU0:XRv-2#show run formal | i redist     
Sat Jul  2 18:27:36.445 UTC
Building configuration...
router isis zeal address-family ipv4 unicast redistribute bgp 2 level-2
router bgp 2 address-family ipv4 unicast redistribute isis zeal
RP/0/0/CPU0:XRv-2#

The summary-only configuration suppresses any prefixes, within the range of 10.48.0.0/12 [ 10.48.0.0 - 10.63.255.255 ], and advertises only the aggregate route.

The rest of the core sees this aggregate, due to XRv-2's redistribution. They can also reach the individual subnets contained within this aggregate.

RP/0/0/CPU0:XRv-10#show route 10.50.50.1
Sat Jul  2 17:51:33.993 UTC

Routing entry for 10.48.0.0/12
  Known via "isis zeal", distance 115, metric 40, type level-2
  Installed Jul  2 17:41:16.075 for 00:10:17
  Routing Descriptor Blocks
    10.5.10.5, from 10.255.255.2, via GigabitEthernet0/0/0/0, Protected
      Route metric is 40
    10.9.10.9, from 10.255.255.2, via GigabitEthernet0/0/0/1, Protected
      Route metric is 40
  No advertising protos. 
RP/0/0/CPU0:XRv-10#


RP/0/0/CPU0:XRv-10#ping 10.50.50.1 so lo0
Sat Jul  2 17:52:22.050 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.50.50.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms
RP/0/0/CPU0:XRv-10#
RP/0/0/CPU0:XRv-10#
RP/0/0/CPU0:XRv-10#ping 10.50.55.1 so lo0
Sat Jul  2 17:52:26.869 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.50.55.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/9/29 ms
RP/0/0/CPU0:XRv-10#


RP/0/0/CPU0:XRv-10#traceroute 10.50.50.1 so lo0 probe 1         
Sat Jul  2 17:53:18.956 UTC

Type escape sequence to abort.
Tracing the route to 10.50.50.1

 1  10.5.10.5 9 msec 
 2  10.4.5.4 0 msec 
 3  10.3.4.3 0 msec 
 4  10.2.3.2 0 msec 
 5  10.1.2.1 0 msec 
RP/0/0/CPU0:XRv-10#

So what happened?

Now begins the story of our outage lottery winner, John Doe. John Doe's blunder is very much inspired by a similar real-world scenario I witnessed. The topology, impact, and actions are changed, for the sake of discretion.

John Doe is in the provisioning team. He received a request to decommission the network of a service that had been cancelled. John Doe believed that the change approval process was cumbersome. Thus, he decided that he would quickly delete this network and update the relevant documentation.

John logged into XRv-1 and manually deleted the network. He then went to his long overdue lunch, as it was nearly 2 pm in the afternoon.

John did not inform the NOC he made this change. He did not utilize monitoring systems to see the impact he caused. He also did not validate the configuration he was implementing.

RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#configure 
Sat Jul  2 17:53:43.904 UTC
RP/0/0/CPU0:XRv-1(config)#no interface loopback 5*
RP/0/0/CPU0:XRv-1(config)#commit
Sat Jul  2 17:54:00.703 UTC
RP/0/0/CPU0:XRv-1(config)#end
RP/0/0/CPU0:XRv-1#

John was in and out of that configuration in ~17 seconds.

What was the blunder? John's intended Network to delete was the one attached to Lo58. What did he actually type? Loopback 5* . This '*' acts like a wildcard character. That means John unintentionally deleted networks 50 - 59 , along with network 58.

John's shift key just caused some major impact.

Troubleshooting

The NOC figures there is a bug, a failed ASIC, a hung router, etc. They engage escalation resource, Jenny Bourne. Jenny decides to trace from a relevant PE router, XRv-10, to the defined networks in outage.

RP/0/0/CPU0:XRv-10#traceroute 10.50.50.1 so lo0 probe 1 
Sat Jul  2 17:55:17.378 UTC

Type escape sequence to abort.
Tracing the route to 10.50.50.1

 1  10.5.10.5 0 msec 
 2  10.4.5.4 0 msec 
 3  10.3.4.3 0 msec 
 4  10.2.3.2 0 msec 
 5   * 
 6   * 
 7   * 
 8   * 
 9  
RP/0/0/CPU0:XRv-10#
RP/0/0/CPU0:XRv-10#ping 10.50.55.1 so lo0                          
Sat Jul  2 17:54:19.641 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.50.55.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/0/CPU0:XRv-10#ping 10.50.50.1 so lo0
Sat Jul  2 17:54:35.460 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.50.50.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/0/CPU0:XRv-10#

This trace shows that, in hop 4, packets get to the link between XRv-3 and XRv-2. Then it seems like XRv-2 or XRv-1 was dropping the traffic.

She checked Router 2:

RP/0/0/CPU0:XRv-2#show cef 10.50.50.0 detail  | i "version|via|Inter|Giga"
Sat Jul  2 18:35:02.247 UTC
10.48.0.0/12, version 51, internal 0x1000001 0x0 (ptr 0xa143c1f4) [1], 0x0 (0x0), 0x0 (0x0)
   via 10.1.2.1/32, 2 dependencies, recursive, bgp-ext [flags 0x6020]
    next hop 10.1.2.1/32 via 10.1.2.1/32
    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/0/0/0    10.1.2.1       
RP/0/0/CPU0:XRv-2

This looks alright. XRv-2 had a valid route to XRv-1 . She then checked router 1.

RP/0/0/CPU0:XRv-1#show route 10.50.50.1
Sat Jul  2 18:03:16.485 UTC

Routing entry for 10.48.0.0/12
  Known via "bgp 1", distance 200, metric 0, type locally generated
  Installed Jul  2 17:41:15.775 for 00:22:00
  Routing Descriptor Blocks
    directly connected, via Null0
      Route metric is 0
  No advertising protos. 
RP/0/0/CPU0:XRv-1#

Next-hop via Null0 , that means this router is throwing packets in the garbage. She then checked to see what's changed on the router.

RP/0/0/CPU0:XRv-1#show configuration commit list | utility egrep -C1 "~"  
Sat Jul  2 17:57:38.308 UTC
SNo. Label/ID              User      Line                Client      Time Stamp
~~~~ ~~~~~~~~              ~~~~      ~~~~                ~~~~~~      ~~~~~~~~~~
1    1000000064            jdoe    con0_0_CPU0         CLI         Sat Jul  2 17:54:00 2022
RP/0/0/CPU0:XRv-1#

Resolution

It appears her colleague, John, made a change. She checked to see what it entailed.

RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#configure 
Sat Jul  2 17:59:24.871 UTC
RP/0/0/CPU0:XRv-1(config)#
RP/0/0/CPU0:XRv-1(config)#load rollback changes last 1
Building configuration...
Loading.
611 bytes parsed in 1 sec (593)bytes/sec
RP/0/0/CPU0:XRv-1(config)#show commit changes diff 
Sat Jul  2 17:59:46.179 UTC
Building configuration...
!! IOS XR Configuration 6.1.3
+  interface Loopback50
+   ipv4 address 10.50.50.1 255.255.255.255
   !
+  interface Loopback51
+   ipv4 address 10.50.51.1 255.255.255.255
   !
+  interface Loopback52
+   ipv4 address 10.50.52.1 255.255.255.255
   !
+  interface Loopback53
+   ipv4 address 10.50.53.1 255.255.255.255
   !
+  interface Loopback54
+   ipv4 address 10.50.54.1 255.255.255.255
   !
+  interface Loopback55
+   ipv4 address 10.50.55.1 255.255.255.255
   !
+  interface Loopback56
+   ipv4 address 10.50.56.1 255.255.255.255
   !
+  interface Loopback57
+   ipv4 address 10.50.57.1 255.255.255.255
   !
+  interface Loopback58
+   ipv4 address 10.50.58.1 255.255.255.255
   !
end

RP/0/0/CPU0:XRv-1(config)#commit
Sat Jul  2 17:59:52.409 UTC
RP/0/0/CPU0:XRv-1(config)#end
RP/0/0/CPU0:XRv-1#

She observed that John deleted the networks above. Since customers & the NOC were complaining, and she recognized the networks to be valid for production, she did not hesistate to restore the prefixes.

Immediately, after the commit, she checked connectivity.

RP/0/0/CPU0:XRv-10#traceroute 10.50.50.1 so lo0 probe 1 
Sat Jul  2 17:59:57.298 UTC

Type escape sequence to abort.
Tracing the route to 10.50.50.1

 1  10.5.10.5 0 msec 
 2  10.4.5.4 0 msec 
 3  10.3.4.3 0 msec 
 4  10.2.3.2 0 msec 
 5  10.1.2.1 0 msec 
RP/0/0/CPU0:XRv-10#ping 10.50.50.1 so lo0               
Sat Jul  2 18:00:02.288 UTC
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.50.50.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms
RP/0/0/CPU0:XRv-10#  

Jenny confirmed with the NOC that the board is going green and in-band connectivity appeared restored.

Post-Mortem

Mr. Doe came back from a delicious lunch to find he caused quite a headache for his colleagues. They schedule a meeting and discuss some things that could be done better:

John could've done the following

• Informed anyone in operations, if not the NOC, that he is about to make a change. The details of the change do not matter, as long as someone knows he is working on the network.
• He could've ran that same show commit changes diff command Jenny had utilized, prior to hitting commit. That would've exposed the gravity of his typo.
• He could've added a comment to his change. In the event that this change was more complex, and Jenny had doubts that his change was the clear impact, she could've reviewed the change request / documentation related to his config to see what the scope and impact should be:

RP/0/0/CPU0:XRv-1#
RP/0/0/CPU0:XRv-1#configure 
Sat Jul  2 18:35:14.153 UTC
RP/0/0/CPU0:XRv-1(config)#no interface loopback 5*
RP/0/0/CPU0:XRv-1(config)#commit comment "CRQ1234"
Sat Jul  2 18:35:35.252 UTC
RP/0/0/CPU0:XRv-1(config)#end
RP/0/0/CPU0:XRv-1#


RP/0/0/CPU0:XRv-1#show configuration commit list 1 detail 
Sat Jul  2 18:36:12.599 UTC

   1) CommitId: 1000000069                 Label: NONE
      UserId:   jdoe                       Line:  con0_0_CPU0
      Client:   CLI                        Time:  Sat Jul  2 18:35:35 2022
      Comment:   "CRQ1234" 
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1

!!! Or: !!!

RP/0/0/CPU0:XRv-1#configure 
Sat Jul  2 18:38:13.711 UTC
RP/0/0/CPU0:XRv-1(config)#no interface loopback 5*
RP/0/0/CPU0:XRv-1(config)#commit comment "Decom Subscriber Net 58"
Sat Jul  2 18:38:25.040 UTC
RP/0/0/CPU0:XRv-1(config)#
RP/0/0/CPU0:XRv-1(config)#
RP/0/0/CPU0:XRv-1(config)#end
RP/0/0/CPU0:XRv-1#

RP/0/0/CPU0:XRv-1#show configuration commit list 1 detail 
Sat Jul  2 18:38:41.769 UTC

   1) CommitId: 1000000073                 Label: NONE
      UserId:   jdoe                       Line:  con0_0_CPU0
      Client:   CLI                        Time:  Sat Jul  2 18:38:25 2022
      Comment:   "Decom Subscriber Net 58" 
RP/0/0/CPU0:XRv-1#

What could the organization do better?

• Well, it would be great if the NOC received notifications when someone makes a change on a core device. That would've provided a timestamp & change location for the NOC to trace down, without Jenny's help. This could be SNMP traps, or Slack hooks.
• Make it clear to operators that the Network is a controlled utility, not a cowboy ranch. John should've created an implementation plan and reviewed it's contents. He then should've submitted a change request to be approved.
• The org should think harder about how they control changes. Maybe it's time to consider a CI/CD pipeline to decom provisioned resources. The pipeline could even add a comment with the job ID, or the commit id.... Of course, that introduces it's own set of problems and technical debt.

RP/0/0/CPU0:XRv-1#show configuration commit list 1 detail 
Sat Jul  2 18:38:41.769 UTC

   1) CommitId: 1000000073                 Label: NONE
      UserId:   ansible                       Line:  con0_0_CPU0
      Client:   CLI                        Time:  Sat Jul  2 18:38:25 2022
      Comment:   "0ea3815881" 
RP/0/0/CPU0:XRv-1#

Shoulda'-Woulda'-Coulda'. The only thing for certain would be that John owed Jenny a stiff drink.

When I wrote my last post, a long while ago, I had promised to complete the story;  I am now completing that promise in time for Christmas.

Let's start with the configuration

###[OSPF]###

#// protocol config

set protocols ospf area 0 network 10.0.0.0/8
set protocols ospf area 0 network 172.16.0.0/12
set protocols ospf passive-interface default
set protocols ospf passive-interface-exclude {interface name}
set protocols ospf parameters abr-type cisco
set protocols ospf parameters router-id {loopback IP}


#// optional config to advertise a default route to internet 

set protocols ospf default-information originate metric 10
set protocols ospf default-information originate metric-type 1


#// Interface config 

set interfaces ethernet {interface name} ip ospf authentication md5 key-id 1 md5-key {our key}
set interfaces ethernet {interface name} ip ospf cost {10, 50} 

### 10 for primary/members, 50 for backup

The above can tell us the following:

1. We are enabling OSPF on all interfaces that have an IP in the subnets above.
2. All interfaces are passive unless otherwise explicitly enabled (this means they will not form OSPF adjacencies or send Hello packets)
3. We are using authentication. This prevents rogue routers on point-to-multipoint radios joining our OSPF network
4. We are using static costing (metrics) to control traffic directions (as mentioned in the previous post)
5. The routers that are connected to the internet advertise a default route into OSPF. We've selected metric type 1 to increment the cost as it passes through hops in the network. This would matter most if we had multiple viable internet connections, at opposite ends of the core ring, or at the end of a spoke chain.

Now let's look at the Topology

Through trial and tribulation, we have threaded the needle again and again. The end results are two radio network core "Rings" that connect to our main internet connection, which is a fiber connection on Blakely island.

We also have a backup radio transit at our Water Tank location.

This ultimately collapses to make our core ring (with a shortcut path)

Connections to the members

Each PoP could have multiple relay sites daisy-chained below it. These points have no alternate paths, they are just means to reach members who are further away from one of our PoP's.

In the above diagram, if the connection between Su-PoP and the Watertank fails, there is an alternate path via Ne-PoP; However, if Relay 1 to Relay 2 fails, then everything south of Relay 2 is, unfortunately, down.

You have to imagine that off of every PoP, there is is at least one relay. Each relay has a PtMP radio that serves another relay, and potentially directly connected members.

What makes OSPF so dynamic?

All the time, OSPF routers are chatting. The two things they are sending are updates on the topology, and heartbeats – or keepalives – in the form of "Hello" packets.

Let's take a look:

root@wt-er:~# tcpdump -i eth0 host 224.0.0.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:44:26.228788 IP 10.0.0.7 > ospf-all.mcast.net: OSPFv2, Hello, length 64
17:44:26.229488 IP 10.0.0.40 > ospf-all.mcast.net: OSPFv2, Hello, length 64
17:44:29.205888 IP 10.0.0.4 > ospf-all.mcast.net: OSPFv2, Hello, length 64
17:44:29.214227 IP 10.0.0.13 > ospf-all.mcast.net: OSPFv2, Hello, length 64
17:44:31.221995 IP 10.0.0.3 > ospf-all.mcast.net: OSPFv2, Hello, length 64
17:44:31.469316 IP 10.0.0.40 > ospf-all.mcast.net: OSPFv2, LS-Update, length 92
^C
6 packets captured
52 packets received by filter
4 packets dropped by kernel
root@wt-er:~#

From this packet capture, we can see that this router at our tank location is receiving multiple keepalives from it's peers (on a point-to-multipoint radio).

By default, for broadcast or Point-to-Point interfaces, the interval in which OSPF routers send these hellos is 10 seconds and it will declare the link dead if a hello is not received within 40 seconds

We can also see that it's sending these keepalives to the multicast address of 224.0.0.5 (All OSPF Routers). Yes, multicast transport is required for OSPF to work – unless you encapsulate it.

If you're not familiar with multicast, think of it this way. In the song "Hey Ya!", André 3000 yells "Okay now, ladies..." and the only the women respond, "Yeah?". Well, now imagine he yells out "Hello, OSPF routers [224.0.0.5]". The routers would know it is they that are being addressed.

Broadcasts, in the case of ARP, are more like yelling "HEY, EVERYBODY!" ... and continuing with, ".... I'm looking for 10.0.0.25". Everyone, except for .25, then realizes their attention wasn't actually needed and discards the message from their brains.

Continuing on, Let's take a closer look at a Hello packet:

root@wt-er:~# tcpdump -i eth0 host 224.0.0.5 -vv 
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:52:54.281058 IP (tos 0xc0, ttl 1, id 36, offset 0, flags [DF], proto OSPF (89), length 100)
    10.0.0.7 > ospf-all.mcast.net: OSPFv2, Hello, length 64
        Router-ID 172.x.x.15, Backbone Area, Authentication Type: MD5 (2)
        Key-ID: 1, Auth-Length: 16, Crypto Sequence Number: 0x0017be2c
        Options [External]
          Hello Timer 10s, Dead Timer 40s, Mask 255.255.255.0, Priority 1
          Designated Router 10.0.0.40, Backup Designated Router 10.0.0.13
          Neighbor List:
            172.x.x.1
            172.x.x.7
            172.x.x.3
            172.x.x.8
            172.x.x.20

The above Hello packet shows many things:

1. A list of our neighbours' neighbours
2. It's network mask
3. What Auth type (MD5) and the Key-ID it matches (1)
4. Which Area it's in (backbone / Area 0)
5. It's router-ID (as we set in the config)
6. Who it's Designated Router & Backup DR are (more on that later)
   

For OSPF routers to start exchanging prefixes, some things must match:

1. Area ID
2. Authentication (if enabled)
3. Hello and Dead Intervals
4. MTU Size
5. Subnet mask (on broadcast interfaces only)

Redundancy

Redundancy comes into play when the primary path, calculated by lowest cost, fails. The OSPF router will wait until the dead timer expires and then re-calculate the best path it knows from the Link-State database.

We can see this below in the scenario where Mr. Tomato wants to send TPS reports to Mr. Skull Racer.

In the scenario, the link between PoP 3 and 4 is struck by lightning (very real possibility, considering we strap majority of our equipment to trees). PoP 3 waits up to 40 seconds to receive a hello packet from PoP 4, then it will declare the path dead and re-calculate to go via the "long-haul" path.

Beyond that, it will also do the neighbourly thing and update it's OSPF neighbours that it's "link state" has changed, thus causing a chain reaction in the Topology to re-calculate a new link-state topology.

A magnifying glass view

To understand how the topology is built, we can look at what happens when a new router comes onto the network. To demonstrate this, I built the following mini lab topology:

When an OSPF router first comes online, it must go through the various OSPF states to reach the "Full" state with it's neighbour.

Once they have proved Bi-directional connectivity (both routers have seen the other routers hello packet), they will both offer a DB Description update.

You can see in frame 6, that Router 7 has 442 bytes worth of information to inform Router 8 about. Here's what that consists of:

It's a list of all the Link-State Advertisements (LSAs) it has received. These LSAs were advertised by the routers in the topology.

For a closer look, we can see this on the router itself.

vyos@VyOS-7:~$ show ip ospf database 

       OSPF Router with ID (172.16.1.7)

                Router Link States (Area 0.0.0.0)

Link ID         ADV Router      Age  Seq#       CkSum  Link count
172.16.1.1      172.16.1.1       813 0x8000000e 0x70d7 4
172.16.1.2      172.16.1.2       803 0x80000009 0x4b41 3
172.16.1.3      172.16.1.3       802 0x8000000d 0x677b 4
172.16.1.4      172.16.1.4       791 0x8000000c 0xae48 4
172.16.1.5      172.16.1.5       742 0x80000008 0xee7e 3
172.16.1.6      172.16.1.6       803 0x80000008 0x5edd 3
172.16.1.7      172.16.1.7       820 0x80000008 0x79ce 3
172.16.1.8      172.16.1.8       711 0x80000005 0xe4b3 2

                Net Link States (Area 0.0.0.0)

Link ID         ADV Router      Age  Seq#       CkSum
10.1.2.2        172.16.1.2       994 0x80000002 0xc249
10.1.5.5        172.16.1.5        22 0x80000002 0x9b64
10.1.6.6        172.16.1.6      1323 0x80000002 0x8e6d
10.2.3.3        172.16.1.3       572 0x80000002 0xb352
10.3.4.4        172.16.1.4       991 0x80000002 0xa45b
10.3.6.6        172.16.1.6       763 0x80000002 0x8a6d
10.4.5.5        172.16.1.5       952 0x80000002 0x9564
10.4.7.7        172.16.1.7       690 0x80000002 0x7b76
10.7.8.8        172.16.1.8        71 0x80000002 0x687f

                AS External Link States

Link ID         ADV Router      Age  Seq#       CkSum  Route
0.0.0.0         172.16.1.2       723 0x80000004 0x186e E1 0.0.0.0/0 [0x0]

vyos@VyOS-7:~$

You can see it contains details about the routers on the topology, and the numbers of links they are responsible for. It also contains the prefixes, both internal and external (the default route we injected via our configuration), along with their metrics (cost).

Let's look at a prefix more precisely:

vyos@VyOS-7:~$ show ip ospf database network 10.4.5.5   

       OSPF Router with ID (172.16.1.7)

                Net Link States (Area 0.0.0.0)

  LS age: 1087
  Options: 0x2  : *|-|-|-|-|-|E|*
  LS Flags: 0x6  
  LS Type: network-LSA
  Link State ID: 10.4.5.5 (address of Designated Router)
  Advertising Router: 172.16.1.5
  LS Seq Number: 80000002
  Checksum: 0x9564
  Length: 32
  Network Mask: /24
        Attached Router: 172.16.1.4
        Attached Router: 172.16.1.5

vyos@VyOS-7:~$ 

As the the prefix itself subtley hints, we can see this prefix is a /24 that lives between Router 4 & 5. Once router 8 received this in it's database, and determines it's the best path, it can then install it in it's Routing table and Forwarding table.

In the exchange, you can see there are also LS Requests and LS Updates. This is the routers comparing their databases and making sure that they have the most correct & true information. After an LS Update is sent by a neighbour, the other neighbour will send back an LS Acknowledge, like an Ack to a Syn in TCP.

Periodically, on your topology, you will see LS Updates flood. Don't worry, as this is just neighbourly routers advising their neighbours what the current status of their links are.

Topology Changes

Finally, let's look at a topology change. To simulate a link loss, like in the skull & tomato link failure, I will shutdown the link between router 7 & 8. This will isolate router 8 from the network

vyos@VyOS-7:~$ show interfaces | match /24
eth0             10.4.7.7/24                       u/u  
eth1             10.7.8.7/24                       u/u  
vyos@VyOS-7:~$ configure 
[edit]
vyos@VyOS-7# set interfaces ethernet eth1 disable
[edit]
vyos@VyOS-7# show | compare
[edit interfaces ethernet eth1]
+disable
[edit]
vyos@VyOS-7# commit;save;exit
Saving configuration to '/config/config.boot'...
Done
exit
vyos@VyOS-7:~$ 

Now let's see what a packet capture between router 3 & router 4 looks like:

Router 4, who receives the update first, floods it to it's neighbours, including Router 3. This LS update states It has two updates for a Router-LSA & it's Network LSA. It now is responsible for links 10.4.7.7 & 172.16.1.7 (note: 10.7.8.7 is absent).

Also, note the LS Age on the LSAs. For the Router-LSA, it's LS age is 2 seconds, which is good. For the Network LSA of 10.7.8.7, it's LS Age is 3600 seconds, which is the maximum. This effectively tells the routers to delete this from their routing table.

Since Router 7 was the only viable way to reach this prefix from the topology, this route will effectively disappear. As will 172.16.1.8's prefix, as it was only listed as an attached router on this Network-LSA.

Now, let's restore the link and see what happens

vyos@VyOS-7:~$ configure 
[edit]
vyos@VyOS-7# delete interfaces ethernet eth1 disable 
[edit]
vyos@VyOS-7# show | compare
[edit interfaces ethernet eth1]
-disable
[edit]
vyos@VyOS-7# commit;save;exit
Saving configuration to '/config/config.boot'...
Done
exit
vyos@VyOS-7:~$ 

Router 4 then receives & floods an LS-Update from Router 7. This LS-Update advises that Router 7 now has link-state that includes 10.7.8.0/24 being reachable.

We'll also receive the LSA's from router 8, informing us that this prefix is just fine and we can continue to route to it, restoring connections for our members downstream of router 7

If there is a way, OSPF will find it

via GIPHY

Point-To-Multipoint Considerations

The last thing I'd like to mention is considerations of bandwidth and flows for PtMP links. When you consider the chatty nature of OSPF (Hellos, Flooding updates, etc.), there's a lot going on all the time.

When you imagine a broadcast domain that is setup on a PtMP radio, there could be a lot of routers connected. If each router needs to maintain a full adjancency, there's a lot of synchronization to be done.

This could cause bandwidth & packet/frame processing to grow exponentially, when there are more important packets to be pushed like Zoom packets.

To solve this, OSPF has a process called the Designated Router (DR) Election. This process essentially elects two routers to be responsible for performing the Sychronization of the Link-State DB with neigbours, drastically cutting down on noise.

In our Tank example, as we saw in our first tcpdump, there are a fair amount of routers on this segment.

me@wt-er:~$ show ip ospf neighbor | match "Neig|eth0"
Neighbor ID     Pri   State            Dead Time   Address         Interface           Instance ID
172.x.x.7       1   2-Way/DROther    00:00:31    10.0.0.3        eth0                    0
172.x.x.3       1   2-Way/DROther    00:00:29    10.0.0.4        eth0                    0
172.x.x.15      1   2-Way/DROther    00:00:34    10.0.0.7        eth0                    0
172.x.x.8       1   Full/Backup      00:00:30    10.0.0.13       eth0                    0
172.x.x.20      1   Full/DR          00:00:33    10.0.0.40       eth0                    0
me@wt-er:~$

By default, OSPF will select the router with the highest priority (you can manually configure this) as the DR and the second highest as the Backup DR (BDR). If all priorities are equal, it will then look for the highest router-id.

In the above output, .20 is selected as the DR because it has the highest Loopback / Router-ID.  Router 15 would typically be the BDR, but it may have not been online / reachable, at the time of DR election. If the DR election were to be done again, it would likely become the BDR.

You'll also notice that the other routers do not form a Full adjacency. They stay in the "2-way" state. Do not be alarmed when you see this. This is because they have two-way connectivity, as proven by their hellos, but they do not Exchange DBDs or LSAs.

Now consider the placement of your Designated router; If your PtMP radio has low bandwidth capability, you might want to be weary of where your DR is placed.

In my mock example above, Su-PoP has two relays connected on a PtMP radio. Relay 2 was selected as the DR (this would not be abnormal, as our router-id increments as we onboard routers farther down the chain).

As it's role entails, the DR must keep it's neighbours informed on the state of the Link-State topology. However, if it is doing so with Relay 1, which is also a PtMP spoke, traffic doubles as it hairpins through the radio at the PoP. This would make the 442 bytes we saw in a previous capture double to to 884 bytes as it hairpins through the radio (442 bytes in, and 442 bytes out).

This could be problematic, if you have an oversubscribed or low bandwidth capability radio, such as a 900Mhz radio.

This is also something to consider, should a user behind Relay 1 need to communicate directly with a user behind Relay 2. In this scenario, you might want to consider deploying dedicated radios between the PoP & these relay sites, allowing the router to handle the traffic with dedicated bandwidth, per link.

Conclusion

I will say that I am fairly happy with the state of the routing. Things go down, things come back up; OSPF finds a way. It's also relatively easy to on-board new sites and members, especially after some of the design optimizations Chris has made for the topology.

We get a lot of emails and inquiries from various people around the world, trying to replicate our setup. I hope this post answers most of your questions. I hope the hyperlinks I included expand on your knowledge even further. If not, please do reach out and I will be happy to elaborate.

If I could offer one piece of advice, It would be to keep it super simple (KISS model); Allow your configuration to be be greater for the sum of it's parts.