Flowcontrol IEEE 802.3x

Old known stuff, probably for everybody who is somehow related to networking, but anyway, putting it here too.

IEEE 802.3x – Wikipedia link

If QoS is enabled and you like to prioritize the traffic, flow control needs to be disabled,as it doesn’t care about any higher level prioritization, just when ingress traffic is coming in faster than receiver can accept it, flow control will kick in and send pause frames until the ingress-egress rate will be equalized or ingress rate is lower than egress of that interface.

A bit more info from Dell FTOS 9 documentation about flow control :

 

flowcontrol

I would use it only for storage – for example iscsi traffic, in separated network, then it won’t do any harm.. probably 🙂

But of course no way of using it on trunk links, other switch facing links and etc.

 

Dell Networking VLT concepts

So what is a VLT and what does it does :

Virtual link trunking (VLT) allows physical links between two Dell switches to appear as a single virtual link to the network core or other
switches such as Edge, Access, or top-of-rack (ToR). As a result, the two physical switches appear as a single switch to the connected
devices.

Basically we are creating one logical switch out of two physical switches.

From the left we see how it looks when interconnected physically, from right how end device sees it.

vlt_concept

Configuration steps :

1.Enable spanning tree – RSTP and PVST supported  – step is optional, but nevertheless recommended.

configure

protocol spanning-tree rstp

bridge-priority 4096 (primary VLT switch)

bridge-priority 8192 (Secondary VLT switch)

no disable

Recommended to have a root bridge on VLT master and to set STP priority to secondary VLT switch in case if the first fails no to have topology change when other third unknown device would become a root.

2. Configure ports for VLTi link :

configure

interface range fortyGigE 0/56 , fortyGigE 0/60

no shutdown

interface port-channel 100

channel-member fortyGigE 0/56,60

no shutdown

3. Create VLT domain on both switches, don’t forget to create a backup-link

configure

vlt domain 1

primary-priority 10 (primary VLT switch)

primary-priority 20 (Secondary VLT switch)

back-up destination 192.168.0.2 (Primary VLT switch, management interface)

back-up destination 192.168.0.3  (Secondary VLT switch management interface)

peer-link port-channel 100

Backup links are needed to have a heartbeat messages flowing between two switches.

heartbeat

VLT also would work without the heartbeat but then you can encounter possible split brain scenario in case of VLTi link failure.

After configuring the VLT we should get the following picture :

shvltbrief.png

Now let’s attach a device to our VLT switches.

On both VLT members pick up a port for redundant connection :

interface port-channel xx

no ip address

switchport

channel-member tex/x/x

no shut

vlt-peer-lag port-channel 110

And you are ready to go.

You can tweak the stuff like dampening – just to give some time for routing and other protocols to get online after rebooting the switch ,as ports will get up faster and devices without knowing that routing protocol is not ready yet may black hole the traffic.

You can also play with spanning-tree metrics – to have interruption after reboot as small as possible.

VLT behavior :

vlt_behaviour

You can check that MACs are being synced using the command :

show mac-address-table count

Some of the of interesting points to remember (more you can find by downloading the user guide)

  • When you enable Layer 3 routing protocols on VLT peers, make sure the delay-restore timer is set to a value that allows sufcient time
    for all routes to establish adjacency and exchange all the L3 routes between the VLT peers before you enable the VLT ports.

  • RSTP and PVST is supported only, no other spanning-tree would work properly in vlt config

  • Stacking is not allowed when configuring the VLT.

  • If the source is connected to an orphan (non-spanned, non-VLT) port in a VLT peer, the receiver is connected to a VLT (spanned) portchannel, and the VLT port-channel link between the VLT peer connected to the source and ToR is down, trafc is duplicated due to
    route inconsistency between peers. To avoid this scenario, Dell Networking recommends confguring both the source and the receiver
    on a spanned VLT VLAN.

  • In a scenario where one hundred hosts are connected to a Peer1 on a non-VLT domain and trafc flows through Peer1 to Peer2; when
    you move these hosts from a non-VLT domain to a VLT domain and send ARP requests to Peer1, only half of these ARP requests reach
    Peer1, while the remaining half reach Peer2 (because of LAG hashing). The reason for this behavior is that Peer1 ignores the ARP
    requests that it receives on VLTi (ICL) and updates only the ARP requests that it receives on the local VLT. As a result, the remaining
    ARP requests still points to the Non-VLT links and trafc does not reach half of the hosts. To mitigate this issue, ensure that you
    confgure the following settings on both the Peers (Peer1 and Peer2):
    arp learn-enable and mac-address-table stationmove refresh-arp

  • Don’t use any VLAN config on VLTi – switch will match the vlans automatically

  • Don’t use Dynamic lang on VLTI – static is recommended

  • In a VLT domain, the following software features are supported on VLTi: link layer discovery protocol (LLDP), flow control, port
    monitoring, jumbo frames, and data center bridging (DCB)

  • If the link between the VLT peer switches is established, changing the VLT system MAC address or the VLT unit-id causes the link
    between the VLT peer switches to become disabled. However, removing the VLT system MAC address or the VLT unit-id may
    disable the VLT ports if you happen to confgure the unit ID or system MAC address on only one VLT peer at any time.

  • If the link between VLT peer switches is established, any change to the VLT system MAC address or unit-id fails if the changes
    made create a mismatch by causing the VLT unit-ID to be the same on both peers and/or the VLT system MAC address does not
    match on both peers

  • If VLTi connectivity with a peer is lost but the VLT backup connectivity indicates that the peer is still alive, the VLT ports on the
    Secondary peer are orphaned and are shut down.

    Also the L3 VLANS would be shut down too

Some failure scenarios :

failurescenarios

Overall VLT is a great thing for load balancing, redundancy and availability (you can upgrade the switches one by one without having a downtime) – In stack this wouldn’t be possible.

All info and images were taken from Dell User guide for S4048-ON switch, you can download it by following this link : http://downloads.dell.com/manuals/all-products/esuprt_ser_stor_net/esuprt_networking/esuprt_net_fxd_prt_swtchs/force10-s4048-on_administrator%20guide15_en-us.pdf  

In user guide you can find a lot of detailed info about all the possible switch OS functions and how to use/implement/troubleshoot them.

Dell VLT Peer-Routing

Some important points about VLT Peer routing technology.

Peer routing enables one VLT node to act as a proxy gateway for the other peer in a VLT domain. When you enable routing on VLT peers,
you can also enable the peer routing feature.  

In a nutshell, when peer-routing is enabled on both VLT switches you can load-balance, the L3 packets through both switches – as this allows a switch in VLT domain to forward traffic on behalf of its peer switch.

Example how VLT forwards the traffic without peer-routing enabled :

without_peer_routing

 

When you enable peer-routing :

with_peer_routing

Images taken from Configuration guide

Peer-routing helps to avoid sub-optimal routing, reduces the latency by avoiding another hop in traffic path, no need to have VRRP.

Keep in mind in case if switch – Peer-1 will fail with peer routing enabled, your traffic will still be forwarded without any interruption – but as you don’t have any virtual IP address any control or management plane requests won’t be answered by Switch-1’s peer.

So basically by enabling peer routing we have only one goal – redundancy and traffic sharing for L3 protocols.

During the bootup of VLT peer switches, a forwarding loop may occur until the VLT confgurations are applied on each switch and the
primary/secondary roles are determined.


To prevent the interfaces in the VLT interconnect trunk and RSTP-enabled VLT ports from entering a Forwarding state and creating a
traffic loop in a VLT domain, take the following steps.


1 Configure RSTP in the core network and on each peer switch as described in
Rapid Spanning Tree Protocol (RSTP).
Disabling RSTP on one VLT peer may result in a VLT domain failure.


2 Enable RSTP on each peer switch.
PROTOCOL SPANNING TREE RSTP mode
no disable

forwarding loop3 Configure each peer switch with a unique bridge priority.
PROTOCOL SPANNING TREE RSTP mode
bridge-priority

More info about peer-routing advantages comparing to VRRP.

https://hasanmansur.com/2016/06/09/vlt-peer-routing-and-routed-vlt/

Routed VLT v1.2 – document covers peer-routing in great details.