1. Background

OUCS can offer transparent LAN services which preserve and extend a Unit's virtual LAN (VLAN) groupings and associated access privileges across the university backbone network. This is achieved with 802.1Q Tunnelling (also called 802.1Q-in-Q).

OUCS can allocate each unit a second VLAN, with no routed interface – a 'switched' VLAN. This is presented at your main site and any annexe sites you nominate. Many Units have this in place already. Usually you send your main VLAN as un-tagged traffic down this switched VLAN, enabling you to have a common subnet across your main site and annexes. The traffic has an 802.1Q tag added while it's inside the university backbone network to keep it separate from other Units' traffic. However the limitation is that only one VLAN can ever be sent between your sites.

If you would like to discuss this feature or request that we enable it for you, please contact networks@oucs.ox.ac.uk

2. Technical Details

When Q-in-Q tunnelling is enabled, the software adds an extra 802.1Q tag to your traffic at the ingress port of the switch (usually FroDo) at the edge of our network – regardless of whether or not it is already tagged. When the traffic leaves the egress port and enters another point in your network, the extra tag is removed.

3. Benefits

802.1Q-in-Q tunnelling allows all of your configured VLANs to be aggregated and backhauled over this single Backbone VLAN, which is a scalable solution. Without Q-in-Q, OUCS would have to assign a unique VLAN ID number to each of your individual VLANs which are required at more than one site this would rapidly consume the 4094-ID VLAN space supported by IEEE 802.1Q technology. In this way, encapsulating multiple 802.1Q VLANs into a single Backbone 802.1Q VLAN (hence the name, “Q in Q”) makes it possible for you to have up to 4094 VLANs present across your sites.

4. Layer 2 control plane traffic

Please note, we are considering withdrawing Layer 2 Protocol Tunnelling as it has lead to many issues. Please do not rely on it remaining available. This feature enables protocols which use a special multicast address to send traffic to adjacent devices only (such as STP, CDP etc) to pass these frames to devices which are not physically adjacent. However, with this feature enabled 'MAC flaps' are far more common in the backbone. This leads to packet loss for you and creates troubleshooting issues for us. In cases where we disable L2Protocol Tunnelling but leave Q-in-Q enabled, will will also filter BPDUs at the tunnel endpoints to prevent a large spanning-tree from forming across both your main site and annexe VLANs in the university backbone.

5. Disadvantages

There are several issues which we will hope to address in the next iteration of the backbone by providing an alternative solution.

  1. Best practice would be to route all traffic crossing the core of the university backbone network. This is because Q-in-Q creates a large layer 2 broadcast domain across all your sites and the university backbone network. Large broadcast domains are prone to issues and difficult to troubleshoot.
  2. With large broadcast domains, an issue in one area of the network will not be isolated to that segment potentially leading to network wide problems.
  3. As the source and destination MAC addresses are maintained when traffic traverses the university backbone network, any Network Access Control (NAC) or similar solution used by a Unit which results in a single source MAC address appearing from more than one place will cause MAC address table instability in the university backbone networks core and lead to anomalous behaviour.
  4. Some Unit Annexe sites are connected directly to a 6500. We have found that Q-in-Q tunnels between FroDos (Cisco 3750) and a 6500 have had issues. Therefore where possible we will migrate Unit annexe connections to an existing local FroDo when provisioning Q-in-Q.

6. Prerequisites

In order to mitigate the risk of anomalous behaviour we require the following:

  1. You must agree to run Spanning Tree Protocol (STP) on you network.
  2. You must agree not to run any private fibre between sites which are connected by a Q-in-Q tunnel.
  3. You agree to disable Dynamic Trunking Protocol (DTP) on all ports connecting to a Backbone Tunnel interface. Switch ports connecting to the university backbone must be statically configured in either access, or VLAN trunk mode[1]. DTP has been found to behave unpredictably when switches are connected via a tunnel. This is especially an issue where there is more than one Annexe site.
  4. You must agree that implementing this for your network will not to your knowledge cause any MAC addresses to enter the backbone from more than one location. OUCS reserves the right to withdraw Q-in-Q tunnelling and/or Layer 2 Protocol Tunnelling if this occurs and cannot be resolved. Common causes are Firewalls or NAC solutions which forge MAC addresses.
  5. The MTU of Frames is slightly larger due to the additional 4 bytes in the header needed for the second 802.1Q tag. To accommodate this OUCS will need to reboot some FroDos before enabling Q-in-Q. This will normally be done during the Tuesday morning JANET at-risk period and announced via ITSS-Announce.

[1] If and only if you are connecting to a 801.1Q tunnel.