1. Background
OUCS can offer transparent LAN services which preserve and extend a Unit's virtual LAN (VLAN) groupings and associated access privileges across the university backbone network. This is achieved with 802.1Q Tunnelling (also called 802.1Q-in-Q).
OUCS can allocate each unit a second VLAN, with no routed interface – a 'switched' VLAN. This is presented at your main site and any annexe sites you nominate. Many Units have this in place already. Usually you send your main VLAN as un-tagged traffic down this switched VLAN, enabling you to have a common subnet across your main site and annexes. The traffic has an 802.1Q tag added while it's inside the university backbone network to keep it separate from other Units' traffic. However the limitation is that only one VLAN can ever be sent between your sites.
If you would like to discuss this feature or request that we enable it for you, please contact networks@oucs.ox.ac.uk
2. Technical Details
When Q-in-Q tunnelling is enabled, the software adds an extra 802.1Q tag to your traffic at the ingress port of the switch (usually FroDo) at the edge of our network – regardless of whether or not it is already tagged. When the traffic leaves the egress port and enters another point in your network, the extra tag is removed.
3. Benefits
802.1Q-in-Q tunnelling allows all of your configured VLANs to be aggregated and backhauled over this single Backbone VLAN, which is a scalable solution. Without Q-in-Q, OUCS would have to assign a unique VLAN ID number to each of your individual VLANs which are required at more than one site this would rapidly consume the 4094-ID VLAN space supported by IEEE 802.1Q technology. In this way, encapsulating multiple 802.1Q VLANs into a single Backbone 802.1Q VLAN (hence the name, “Q in Q”) makes it possible for you to have up to 4094 VLANs present across your sites.
4. Layer 2 control plane traffic
To transport not only your data traffic but also Layer 2 control traffic (such as STP, Cisco Discovery Protocol (CDP), and VLAN Trunking Protocol (VTP)), we also configure Layer 2 Protocol Tunnelling, which is a separate feature. This means that to you, the backbone appears like a 'bump in the wire' and it is as if the two Unit switches were directly connected. For example the main site switch will be able to see all the annexe switches as 'CDP/LLDP neighbors' on the annexe port.
This leads to an additional benefit. A standard 'switched' VLAN merges the Spanning Tree Protocol (STP) of your VLAN and the Backbone VLAN. This can often lead to a Spanning Tree with a diameter of greater than seven. The default timers are set with the assumption that the limit of seven hops from the Root Bridge to any leaf will not be broken so this can lead to anomalous behaviour which can be difficult to troubleshoot. Enabling this feature creates many distinct STP instances, one for the switched VLAN crossing the backbone and one for each of your VLANs[1]. OUCS has complete control over the STP in the university backbone network, and you have control over the STP for each of your VLANs.
5. Disadvantages
There are several issues which we will hope to address in the next iteration of the backbone by providing an alternative solution.
- Best practice would be to route all traffic crossing the core of the university backbone network. This is because Q-in-Q creates a large layer 2 broadcast domain across all your sites and the university backbone network. Large broadcast domains are prone to issues and difficult to troubleshoot.
- With large broadcast domains, an issue in one area of the network will not be isolated to that segment potentially leading to network wide problems.
- As the source and destination MAC addresses are maintained when traffic traverses the university backbone network, any Network Access Control (NAC) or similar solution used by a Unit which results in a single source MAC address appearing from more than one place will cause MAC address table instability in the university backbone networks core and lead to anomalous behaviour.
- Some Unit Annexe sites are connected directly to a 6500. We have found that Q-in-Q tunnels between FroDos (Cisco 3750) and a 6500 have had issues. Therefore where possible we will migrate Unit annexe connections to an existing local FroDo when provisioning Q-in-Q.
6. Prerequisites
In order to mitigate the risk of anomalous behaviour we require the following:
- You must agree to run Spanning Tree Protocol (STP) on you network.
- You agree to disable Dynamic Trunking Protocol (DTP) on all ports connecting to a Backbone Tunnel interface. Switch ports connecting to the university backbone must be statically configured in either access, or VLAN trunk mode[2]. DTP has been found to behave unpredictably when switches are connected via a tunnel. This is especially an issue where there is more than one Annexe site.
- You must agree that implementing this for your network will not to your knowledge cause any MAC addresses to enter the backbone from more than one location. OUCS reserves the right to withdraw Q-in-Q tunnelling and/or Layer 2 Protocol Tunnelling if this occurs and cannot be resolved. Common causes are Firewalls or NAC solutions which forge MAC addresses.
- The MTU of Frames is slightly larger due to the additional 4 bytes in the header needed for the second 802.1Q tag. To accommodate this OUCS will need to reboot some FroDos before enabling Q-in-Q. This will normally be done during the Tuesday morning JANET at-risk period and announced via ITSS-Announce.
[1] This assumes you are using Cisco switches. Other vendors provide one instance of STP for the native (untagged) VLAN and one for all tagged VLANs. Since the tree is usually identical for all VLANs in these examples, this is not an issue.