Q-Q Tunneling For Fun & Profit

• By Joe Astorino
• December 13, 2011
• 13 Comments

Introduction

 

Welcome back everybody! Last week we covered a tunneling technology known as l2protocol-tunneling or L2pt here on the blog.  If you missed that article, you may want to go through and get a handle on that technology before jumping into Q-Q tunneling.  The two usually go hand in hand, but I cover them separately to show that they are two independent technologies that just so happen to go nicely together.

Today, we will look at Q-Q tunneling and a bit later we will show how to integrate L2PT with your Q-Q tunnels.  What is the point of Q-Q tunneling you might ask?  It is mainly a SP technology used in metro ethernet that essentially transparently connects different customer sites that connect into the metro ethernet cloud by “tunneling” their traffic through the service provider network.  The cool thing about it is that the tunneling aspect of the technology allows the customer to send as many VLANs as they need to while also keeping their traffic pretty much isolated from the SP core.

 

How It Works

 

I usually like to start with a diagram of what things look like.  This helps me visualize what is happening.  Take a look at this, and don’t worry if you don’t get why certain VLANs are where, we will deal with that.  Let’s just go with the general concepts for now

 

 

Here we have Cat1 at customer site A and Cat4 at customer site B.  The customer has elected to connect these two sites together by means of a metro ethernet service provided by our little service provider here consisting of Cat2 and Cat3.  OK, cool.  The goal of the SP is to make Cat1 and Cat4 appear to be directly connected from their perspectives.  The SP wants to be as transparent as possible, and they also require that they keep their traffic isolated and secured from the customer traffic.  Well, how do we do that, and how do we keep the VLANs separated if the traffic from two different organizations needs to pass through the same set of devices.  This is where Q-Q comes in.

Q-Q works by double encapsulating ethernet frames with two 802.1Q tags, hence the name Q-Q (Q in Q).  When you think about it, it works a lot like MPLS L3 VPN actually so if you are familiar with that topic, this may come easier for you.  What happens is that the customer sends some traffic tagged with whatever VLAN it is they need.  When the SP edge switch receives that traffic, it keeps that existing “inner” tag and adds a a second “outer” tag to the frame.  The “inside” tag is the customer VLAN, while the “outside” tag is used to transport or “tunnel” the frame through the metro ethernet cloud.  When the frame gets to the egress edge switch, the outer tag is popped off, leaving just the customer tag.  VERY familiar if you know how MPLS VPNs work, right? One thing that is really cool about this technology for the SP is that they only need to assign a SINGLE VLAN for each customer.  In this example, Cat1 can be trunking 100 VLANs to Cat2 and it doesn’t matter….every single frame may have a different inner VLAN tag, but the outside VLAN tag for transport through the cloud is ALWAYS the same.  One customer, one VLAN.  It’s a beautiful thing.  From the perspective of the customer switches, they are basically connected directly together with a really long crossover cable : )

Now, if you are like me you need specifics and concrete examples.  You need to know exactly what everything on this diagram means, or you are pulling your hair out already. So, let’s get a little deep before we pull out a real configuration here.  Let’s just start at the beginning, with customer switches.  Very easy, clean and simple.  The customer configures a plain 802.1Q trunk and sends across whatever VLANs they want.  No tunneling configuration, nothing out of the ordinary.  Remember, to them it appears seamless and they have no idea their traffic is being tunneled.  The SP edge switch port facing the customer is where it gets “interesting.”  We don’t configure the customer facing edge ports as a dot1q trunk.  Instead, we use a special mode called a dot1q tunnel port.  This tells the switch that traffic received on this port will be tunneled by adding an extra “outside” tag.  How does it know what VLAN tag to put on the outside?  We use the switchport access vlan command and tell it. In this case, that VLAN is 99. The frames arrive on Cat2 tagged in this case with either VLAN 10, 20, or 30 from the customer.  Cat2 sees they are arriving on a dot1q tunnel port assigned with VLAN 99 so it keeps the inner tag and adds a second outer tag of 99 and sends it across a standard dot1q trunk to cat3.  Cat3 gets the frame in VLAN 99 and before sending it out the egress interface to Cat4 strips off the VLAN 99 tag.  Cat4 receives the frame as if nothing ever happened in between.  Nice.

There are a few other minor things to be aware of that we will look at when we do the configuration, but that is basically the theory.  Let’s get started on configuring this.  We are starting with a blank slate here.

 

Configuration

 

The first thing we are going to do is just configure the customer switches.  To give us something to test, we will configure an SVI in each of our VLANs 10, 20 and 30 as well.

Cat1

vtp mode trans
vlan 10,20,30
!
interface Vlan10
 ip address 192.168.10.1 255.255.255.0
!
interface Vlan20
 ip address 192.168.20.1 255.255.255.0
!
interface Vlan30
 ip address 192.168.30.1 255.255.255.0
!
interface FastEthernet0/24
 switchport trunk encapsulation dot1q
 switchport mode trunk

Cat4

vtp mode trans
vlan 10,20,30
!
interface Vlan10
 ip address 192.168.10.4 255.255.255.0
!
interface Vlan20
 ip address 192.168.20.4 255.255.255.0
!
interface Vlan30
 ip address 192.168.30.4 255.255.255.0
!
interface FastEthernet0/24
 switchport trunk encapsulation dot1q
 switchport mode trunk

OK, some quick verification shows all the SVIs are up/up and at least on our end we are trunking.

Cat1#sh ip int brie | i Vlan
Vlan1                  unassigned      YES unset  administratively down down
Vlan10                 192.168.10.1    YES manual up                    up
Vlan20                 192.168.20.1    YES manual up                    up
Vlan30                 192.168.30.1    YES manual up                    up

Cat1#sh int trunk

Port        Mode             Encapsulation  Status        Native vlan
Fa0/24      on               802.1q         trunking      1

Port        Vlans allowed on trunk
Fa0/24      1-4094

Port        Vlans allowed and active in management domain
Fa0/24      1,10,20,30

Port        Vlans in spanning tree forwarding state and not pruned
Fa0/24      1,10,20,30

Cat4#sh ip int brie | i Vlan
Vlan1                  unassigned      YES unset  administratively down down
Vlan10                 192.168.10.4    YES manual up                    up
Vlan20                 192.168.20.4    YES manual up                    up
Vlan30                 192.168.30.4    YES manual up                    up

Cat4#sh int trunk

Port        Mode             Encapsulation  Status        Native vlan
Fa0/24      on               802.1q         trunking      1

Port        Vlans allowed on trunk
Fa0/24      1-4094

Port        Vlans allowed and active in management domain
Fa0/24      1,10,20,30

Port        Vlans in spanning tree forwarding state and not pruned
Fa0/24      1,10,20,30

Nice. Let’s move on to our SP switch configurations.  The first thing we are going to do is change the system MTU of Cat2 and Cat3 to 1508 bytes and reload them.  Why?  Once you understand the technology (and I know you guys do now) it makes perfect sense.  A standard ethernet frame has a max size of 1500 bytes.  An 802.1Q tag is 4 bytes which gives us 1504.  Now, the Q-Q portion of the config adds a second outer tag which is also 4 bytes, so we need our SP switches to support up to a 1508 byte MTU for this to work at all.

Cat2(config)#system mtu 1508
!
reload

Cat3(config)#system mtu 1508
!
reload

Once they come back up, we can double check this with the show system mtu command

Cat2#show system mtu
System MTU size is 1508 bytes

Cat3#show system mtu
System MTU size is 1508 bytes

OK, rock on. On to the fun stuff. We need to do a few things on the SP side. We need to define the VLAN which will be used for our “outer” tag. We need to setup the dot1q-tunnel port, assign the customer VLAN to it, and finally setup a simple standard dot1q trunk between our two SP switches.

 

Cat2 & Cat3

vtp mode transparent
vlan 99
!
interface FastEthernet0/24
 switchport access vlan 99
 switchport mode dot1q-tunnel
 no cdp enable
!
interface FastEthernet0/19
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 99
 switchport mode trunk

Notice I did not add the customer VLANs 10,20 and 30 on the SP switches. Why? I don’t need them because we don’t really care what VLANs the customers use. Regardless, it is getting double encapsulated as VLAN 99 in OUR network. This is another great win for the SP because guess what? They don’t need to waste resources on their equipment defining and trunking all these customer VLANs. Furthermore, different customers could use the same VLANs. Every customer they have could be using VLANs 10,20 and 30 and it doesn’t matter to the SP switches. You may have also noticed the command no cdp enable on the customer facing ports. That is actually automatically added by IOS when you configure the edge port as a dot1q-tunnel. Makes sense because we don’t want Cat1 and Cat4 to see the SP switches at all in CDP. Remember, transparency. We may want the customer switches to see each other though…hmmmmm we’ll deal with that at the end. One more thing — You will notice that on the cat2/cat3 trunk I pruned every single VLAN except for 99. Why? It is cleaner that way and avoids potential problems in a larger environment. The ONLY purpose of that trunk in this setup is to carry VLAN 99, the customer traffic. I don’t need anything else.

Some quick verification boys…

Cat2#show dot1q-tunnel

dot1q-tunnel mode LAN Port(s)
-----------------------------
Fa0/24

Cat3#show dot1q-tunnel

dot1q-tunnel mode LAN Port(s)
-----------------------------
Fa0/24

Simple enough. Does it work?

Cat1#ping 192.168.10.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

Cat1#ping 192.168.20.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.4, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms

Cat1#ping 192.168.30.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.4, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/3/9 ms

That is a beautiful thing!

 

The Native VLAN

 

Good job guys…we got it working so now of course we need to talk about other “what if” scenarios, break it and ultimately talk about a solution. What if we simply do this on Cat2 and Cat3?

interface FastEthernet0/19
 switchport trunk native vlan 99

Cat1(config-if)#do ping 192.168.30.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.4, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Uh-Oh. Everything is broken and this will probably end up with a call to me at 2 AM on a Sunday morning. Why did it break? Cat1 sent us some tagged frames. Cat2 gets them just like before and sees the dot1q-tunnel access VLAN is 99. However, because the native VLAN on the SP trunk is also 99, it never adds the outside tag. Since the inside tag the customer added is still there Cat2 tries to send the frame over to Cat3 with just the customer tag. In this case, it can’t because for one we don’t have VLAN 10 defined on the SP switches and for two it is not allowed on the trunk. Let’s say it was though. Ultimately, it would not end up in the right place because the outside tag of 99 is missing. In a worst case scenario, imagine you had another customer, customer B and the VLAN the SP decided to use to tunnel their traffic through the SP happened to be 10. Now you have customer A traffic leaking to customer B and maybe a nasty lawsuit : ) Fun times, but not what I want to be doing with my weekends sooooo…how do we fix it?

1) We use ISL trunks everywhere inside the SP because they tag everything and have no concept of a native VLAN

2) We tag ALL 802.1Q frames, even the native VLAN. How? The vlan dot1q tag native commmand.

3) Always choose a native VLAN for the SP trunks that is NOT in the range of VLANs used to tunnel customer traffic

I really don’t like ISL trunks, and solution 3 is kind of boring so let’s try option 2.

Cat2(config)#vlan dot1q tag native

Cat3(config)#vlan dot1q tag native

How about now?

Cat1(config)#do ping 192.168.10.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/9 ms

Cat1(config)#do ping 192.168.20.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/8 ms

Cat1(config)#do ping 192.168.30.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms

Nice!

 

…And a pinch of L2PT

 

Everything is working awesome.  Our customer is connected end to end, they can pass all their VLANs, the VLANs are isolated in the SP core and life is good.  Except now some network manager who hasn’t touched a switch in a decade calls to complain because when he runs show cdp neighbor on Cat1 and Cat4 he doesn’t see his switch on the other end of “the cloud”.  You cringe at the use of the term “the cloud” and tell him you will see what you can do.  Remember, L2PT?  I told you it would come in handy?  Again, if you missed that blog, go back and review it now.  As I have said, L2PT is often used in conjunction with Q-Q tunnels and this is exactly why.  We are already tunneling the customer data and all their VLANs, but now we also want to tunnel control protocol traffic for CDP and perhaps STP, VTP, UDLD and even PaGP or LACP.  L2PT is control traffic. Q-Q is the customer data.  OK, well if you already understand L2PT by itself this is a piece of cake.

Let’s configure our SP to tunnel the customers CDP, VTP and STP traffic between Cat1 and Cat4.  Just for fun, let’s make Cat1 a VTP server and Cat2 a VTP client so we have something to test first

Cat1(config)#vtp mode server
Setting device to VTP SERVER mode
Cat1(config)#vtp version 2
Cat1(config)#vtp domain ccie
Changing VTP domain name from NULL to ccie

Cat4(config)#vtp version 2
Cat4(config)#vtp domain ccie
Cat4(config)#vtp mode client
Setting device to VTP CLIENT mode.

Cat2 & Cat3

interface FastEthernet0/24
 switchport access vlan 99
 switchport mode dot1q-tunnel
 l2protocol-tunnel cdp
 l2protocol-tunnel stp
 l2protocol-tunnel vtp

Give CDP a minute or so..(60 second timer by default). Did it work?

Cat2#sh l2protocol-tunnel
COS for Encapsulated Packets: 5
Drop Threshold for Encapsulated Packets: 0

Port       Protocol Shutdown  Drop      Encapsulation Decapsulation Drop
                    Threshold Threshold Counter       Counter       Counter
---------- -------- --------- --------- ------------- ------------- -------------
Fa0/24              cdp           ----      ----         1         0         0
                    stp           ----      ----         0       132         0
                    vtp           ----      ----         0         0         0
                    ---           ----      ----      ----      ----      ----
                    ---           ----      ----      ----      ----      ----
                    ---           ----      ----      ----      ----      ----

Cat3#sh l2protocol-tunnel
COS for Encapsulated Packets: 5
Drop Threshold for Encapsulated Packets: 0

Port       Protocol Shutdown  Drop      Encapsulation Decapsulation Drop
                    Threshold Threshold Counter       Counter       Counter
---------- -------- --------- --------- ------------- ------------- -------------
Fa0/24              cdp           ----      ----         1         2         0
                    stp           ----      ----       205         0         0
                    vtp           ----      ----         0         0         0
                    ---           ----      ----      ----      ----      ----
                    ---           ----      ----      ----      ----      ----
                    ---           ----      ----      ----      ----      ----

Looking very sweet so far…let’s check with our customer.

Cat1#show cdp neighbor | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Cat4             Fas 0/24          166              S I   WS-C3550- Fas 0/24

Cat4#sh cdp neigh | b Device
Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Cat1             Fas 0/24          157           S I      WS-C3560- Fas 0/24

Awesome!!! That should shut him up for a few weeks…now, what about VTP? If the VTP is being tunneled properly, when we add a VLAN to Cat1 it should propogate to Cat4 even though the SP has no concept of our VTP domain, etc.

Cat1(config)#vlan 50
Cat1(config-vlan)#name TESTME

As soon as we do that, we can see the VTP counters increase for L2PT on the SP switches

Cat2#sh l2protocol-tunnel | i vtp
                    vtp           ----      ----         4         3         0

Cat3#sh l2protocol-tunnel | i vtp
                    vtp           ----      ----         3         4         0

Did it actually work? Let’s check the customer client switch…You know it will take the users like 3 days to run “show vlan brief” so you just go to lunch and wait for them to call you…

Cat4#show vlan brief | i 50
50   TESTME                           active

Love it!

 

Summary

 

Q-Q tunneling is largely a service provider technology used to tunnel customer data between two sites in a metro ethernet environment while at the same time keeping the customer infrastructure logically separated from both the SP and other customers.  The technology uses double tagged 802.1Q frames so that all the VLANs tunneled from end to end for a particular customer are tunneled across the SP domain as a single VLAN.  The technology is often used with L2PT so that from the customer perspective their L2 control traffic is also passed transparently as if they were directly connected.  Don’t forget about your system MTU and native VLAN fun in the SP cloud and you should be good to go boys!  Thanks for reading!