Unknown cause of disappearing CAM table entry and unicast flooding

Question

Posting this in response to the closed question: Network flooding without ff-ff-ff-ff-ff-ff (layer 2 broadcast) MAC address in destination?

I have a network that is experiencing the following problem: frames addressed to a particular machine (ip 192.168.107.125, mac bbbb.bbbb.bbbb) are sent to certain devices in the same VLAN.

For example, a wireshark capture on another machine (ip 192.168.107.10, mac aaaa.aaaa.aaaa) lists packets addressed to (ip 192.168.107.125, mac bbbb.bbbb.bbbb). The traffic is FTP traffic (including logins and passwords), so I am pretty sure that it has no place on 192.168.107.10.

I have also spotted that the mac address-table entry for bbbb.bbbb.bbbb is missing when the flooding happens. After a restart of bbbb.bbbb.bbbb the MAC entry comes back, but only for a while.

All pictured servers (network clients) are in the same VLAN

Switch models are shown on the diagram. Configuration is shown on the diagram and listed below.

EDIT 1: The device using the MAC that gets lost definitely emits frames before the address disappears - we have wireshark captures of this. In fact, the device continues to emit frames even after the disappearance so I would expect the switch to create a new MAC address-table entry.

I have checked the switch logs, and there are no signs of link flapping on the port that loses MAC of the connected client. For example, on one occasion when the MAC address was lost I have restarted the client device. In logs I could see the interface going through the designated->blocking->learning->forwarding stages (within 30 seconds) and then no messages appear about that interface. After the restart the MAC entry was still missing! Only moving the device to another port has made it appear for a while.

EDIT 2:

Nexus 1 config:

version 8.2(5)
feature-set fex
switchname Core
feature telnet
feature vrrp
feature scheduler
feature ospf
feature pim
feature msdp
feature eigrp
feature port-security
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature ptp
feature lldp
feature sla sender
feature sla responder
logging level aaa 5
logging level cdp 6
logging level hsrp 5
logging level interface-vlan 5
logging level monitor 6
logging level otm 5
logging level radius 5
logging level spanning-tree 6
logging level dhcp_snoop 5
logging level vpc 5
ip domain-lookup
service unsupported-transceiver
errdisable recovery cause link-flap
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause loopback
errdisable recovery cause storm-control
errdisable recovery cause security-violation
errdisable recovery cause psecure-violation
errdisable recovery cause vpc-peerlink
errdisable recovery cause failed-port-state
ip access-list accessblock121
  statistics per-entry
  11 deny ip 192.168.107.0/24 192.168.121.200/32 
  30 permit ip any any
ip access-list cape
  statistics per-entry
  10 permit icmp 192.168.120.125/32 192.168.107.152/32 
  20 permit ip any any
ip access-list tac
  statistics per-entry
  10 permit icmp 192.168.120.159/32 192.168.107.152/32 
  20 permit ip any any
time-range 02:07:00
ip dhcp snooping
service dhcp
ip dhcp relay
ipv6 dhcp relay
ipv6 dhcp guard policy DHCP_CLIENT
!
class-map type qos match-any VLAN_QOS
policy-map type qos NFLINT
  class class-default
    police cir 200 mbps bc 200 ms conform transmit violate drop
fex 42
  pinning max-links 1
  debounce time 0
  description FEX_42
fex 45
  pinning max-links 1
  debounce time 0
  description FEX_45
ip pim rp-address 192.169.180.3 group-list 224.0.0.0/4
ip pim auto-rp mapping-agent Vlan107
ip pim ssm range 232.0.0.0/8
ip pim auto-rp forward
ip pim pre-build-spt
ip igmp any-query-destination
vlan 107
  name EEE
spanning-tree vlan 107 priority 4096
vrf context keepalive
vrf context management
  ip route 0.0.0.0/0 192.168.121.254
vpc domain 10
  peer-switch
  role priority 1500
  peer-keepalive destination 192.168.145.14 source 192.168.145.13 vrf keepalive
  peer-gateway
  config-sync
  ip arp synchronize
cfs eth distribute
interface Vlan107
  description EEE1
  no shutdown
  mtu 9216
  no ip redirects
  ip address 192.168.107.252/24
  no ipv6 redirects
  ip ospf passive-interface
  ip pim sparse-mode
interface port-channel1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  vpc peer-link
interface port-channel42
  description FEX_42
  switchport
  switchport mode fex-fabric
  fex associate 42
  mtu 9216
interface port-channel45
  description FEX_45
  switchport
  switchport mode fex-fabric
  fex associate 45
  mtu 9216
interface Ethernet4/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown
interface Ethernet4/3
  description VPC KeepAlive Link
  vrf member keepalive
  ip address 192.168.145.13/24
  no shutdown
interface Ethernet5/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown
interface Ethernet5/45
  description FLOODING_ADDRESSED_HERE
  switchport
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown
interface Ethernet7/46
  description NO_FLOODING_HERE_1

  switchport
switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown
interface Ethernet42/1/10
  description NO_FLOODING_HERE_2
  switchport
  switchport access vlan 107
no shutdown
interface Ethernet45/1/10
  description NO_FLOODING_HERE_3
  switchport
  switchport access vlan 107
  no shutdown
logging logfile messages 6
no terminal log-all
line console
  terminal width  80
line vty
router ospf core
  network 192.168.107.0/24 area 0.0.0.0
monitor session 2 
  source interface Ethernet5/45 both
  destination interface Ethernet5/11
ip dhcp snooping vlan 107
scheduler logfile size 1024

Nexus 2 configuration:

version 8.2(5)
feature-set fex
hostname HOSTNAME
feature privilege
feature telnet
feature vrrp
feature scheduler
feature ospf
feature pim
feature msdp
feature eigrp
feature port-security
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature ptp
feature lldp
feature sla sender
feature sla responder
logging level aaa 5
logging level cdp 6
logging level hsrp 5
logging level interface-vlan 5
logging level monitor 6
logging level otm 5
logging level radius 5
logging level spanning-tree 6
logging level dhcp_snoop 5
logging level vpc 5
ip domain-lookup
service unsupported-transceiver
errdisable recovery cause link-flap
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause loopback
errdisable recovery cause storm-control
errdisable recovery cause security-violation
errdisable recovery cause psecure-violation
errdisable recovery cause vpc-peerlink
errdisable recovery cause failed-port-state
ip access-list accessblock121
  statistics per-entry

  11 deny ip 192.168.107.0/24 192.168.121.200/32

  30 permit ip any any
ip access-list cape
  statistics per-entry
  10 permit icmp 192.168.120.125/32 192.168.107.152/32 
  20 permit ip any any
ip access-list tac
  statistics per-entry
  10 permit icmp 192.168.120.159/32 192.168.107.152/32 
  20 permit ip any any
ip dhcp snooping
service dhcp
ip dhcp relay
ipv6 dhcp relay
ipv6 dhcp guard policy DHCP_CLIENT
!
class-map type qos match-all trustme
fex 48
  pinning max-links 1
  debounce time 0
  description FEX_48
fex 54
  pinning max-links 1
  debounce time 0
  description FEX_54
ntp server 192.168.140.13
ntp server 192.168.140.14
ip pim rp-address 192.169.180.3 group-list 224.0.0.0/4
ip pim auto-rp mapping-agent Vlan107
ip pim ssm range 232.0.0.0/8
ip pim auto-rp forward
ip pim pre-build-spt
ip igmp any-query-destination
vlan 107
  name EEE

vrf context keepalive
vrf context management
  ip route 0.0.0.0/0 192.168.121.254
vpc domain 10
  peer-switch
  role priority 1000
  peer-keepalive destination 192.168.145.13 source 192.168.145.14 vrf keepalive
  peer-gateway
  config-sync
  ip arp synchronize
cfs eth distribute
interface Vlan107
  description EEE1
  no shutdown
  mtu 9216
  no ip redirects
  ip address 192.168.107.254/24
  no ipv6 redirects
  ip ospf passive-interface
  ip pim sparse-mode
interface port-channel1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  vpc peer-link
interface port-channel48
  description FEX_48
  switchport
  switchport mode fex-fabric
  fex associate 48
  mtu 9216
interface port-channel54
  description FEX_54
  switchport
  switchport mode fex-fabric
  fex associate 54
  mtu 9216
  vpc 54
interface Ethernet4/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown
interface Ethernet4/3
  description VPC KeepAlive Link
  vrf member keepalive
  ip address 192.168.145.14/30
  no shutdown
interface Ethernet5/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown
interface Ethernet6/41
  description FEX_48
  switchport
  switchport mode fex-fabric
  fex associate 48
  mtu 9216
  channel-group 48
  no shutdown
interface Ethernet6/42
  description FEX_48
  switchport
  switchport mode fex-fabric
  fex associate 48
  mtu 9216
  channel-group 48
  no shutdown
interface Ethernet7/28
  description Link FEX54
  switchport
  switchport mode fex-fabric
  fex associate 54
  mtu 9216
  channel-group 54
  no shutdown
interface Ethernet48/1/3
  description FLOODING_RECEIVED_HERE_1
  switchport
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown
interface Ethernet48/1/8

  description FLOODING_RECEIVED_HERE_2
  switchport
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown
interface Ethernet54/1/10
  description FLOODING_RECEIVED_HERE_3
  switchport
  switchport access vlan 107
  no shutdown
logging logfile messages 6
no terminal log-all
line console
  terminal width  80
line vty
router eigrp 10
  router-id 192.168.133.253
  default-information originate
router ospf 1
router ospf core
  network 192.168.107.0/24 area 0.0.0.0
monitor session 2 
ip dhcp snooping vlan 107
scheduler logfile size 1024

EDIT 3: Hypothesis: the MAC address is missing because it ages out. Thank you, Zac67, I would like to test this further. When MAC address-table entry for bbbb.bbbb.bbbb was missing, I have exported ARP and CAM tables from both switches. ARP entry exists:

192.168.107.125 00:15:53 bbbb.bbbb.bbbb Vlan107

But CAM tables on BOTH switches don't contain this MAC entry! I know that if we see flooding only on one side one would conclude that the entry is missing only on that side, but that is not the case: flooding is happening only on one side, and both CAM tables are missing this entry.

Also, when the MAC address-table entry for bbbb.bbbb.bbbb was missing from the switch, I have taken a SPAN capture on the interface where this client was connected to, and saw the following:

I interpret this as proof that the switch has received frames with bbbb.bbbb.bbbb Source H/W address encapsulated within them. Even if the MAC entry has timed out, the switch should have re-created it, correct?

EDIT 4: A support case has been opened with Cisco by my colleagues. This is now under 'deeper' investigation, as they could not explain the cause of this behaviour straight away.

Please edit the question to include the switch configuration. Speculation and guessing are off-topic here, and that is all we could do without more information. — Ron Maupin, May 03 '21 at 12:33
@RonMaupin, I did my best to include the details and provided more information in this new question. What exactly would you like me to add? — Tony Sepia, May 03 '21 at 12:37
Something like this question. You should obfuscate any public addresses and passwords. — Ron Maupin, May 03 '21 at 12:40
@RonMaupin I have added the switch details. Can you please re-open the question? — Tony Sepia, May 04 '21 at 13:48
Re EDIT3: What's in the ARP table doesn't matter - the CAM table/source address table does. Regarding the SPAN capture: make sure that the source address of those frames matches the missing MAC and that the VLAN matches. If they do and the address is still missing from SAT, there's a bug in the switches. (You're not using group MACs, are you??) — Zac67, May 04 '21 at 16:17
@Zac67 thanks for having a look! I have included ARP because I wanted to make sure that the IP->MAC mapping was there for the applications. And Yes, I have manually checked with a colleague character-by-character, and it does match. We will try to contact Cisco.. — Tony Sepia, May 04 '21 at 16:20
The ARP table within the switch is only relevant if the packets need to be routed across VLANs. For L2 switching alone (=within a VLAN) it isn't used. — Zac67, May 04 '21 at 16:36
Did any answer help you? if so, you should accept the answer so that the question does not keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer. — Ron Maupin, Dec 23 '21 at 18:56
@RonMaupin Hi Ron. I greatly appreciate all the answers and comments given, as your time is precious. The problem still hasn't been resolved. I suspect that it's a bug in the switch firmware, but it hasn't been confirmed. I am happy to mark the given answer as 'accepted' if you think it's what I should do — Tony Sepia, Dec 23 '21 at 20:09
I did not actually post an answer. I was just doing a year-end cleanup to see if people forgot to accept an answer, or they found an answer and did not post it. This helps keep the number of old answers from popping up, looking for an answer when one has already been found. — Ron Maupin, Dec 23 '21 at 20:14

Zac67 · Answer 1 · 2021-05-04T14:36:12.517

frames addressed to a particular machine (ip 192.168.0.20, mac bbbb.bbbb.bbbb) are sent to certain devices in that VLAN.

Unless that MAC address is a multicast address, it needs to be unique in its VLAN - "certain devices" should be "a certain device".

[edit] You seem to be referring to devices attached to the right-hand Nexus - sorry about that.

a wireshark capture on another machine (ip 192.168.0.10, mac aaaa.aaaa.aaaa) lists packets addressed to (ip 192.168.0.20, mac bbbb.bbbb.bbbb). The traffic is FTP traffic (including logins and passwords), so I am pretty sure that it has no place on 192.168.0.10.

Switches don't care about IP addresses, the only thing relevant is the MAC address - source for learning, destination for forwarding.

As in your previous question, a MAC address is removed from its port association when

its associated port loses its link
it isn't seen for a certain time and aged out by the switch (that time is configurable usually)
it is seen as source on another port

When a destination MAC isn't present in its source address table, a switch floods that frame to all ports, like a broadcast, mimicking a repeater hub. However, all NICs not addressed by that MAC will simply ignore that frame - so it'll just waste bandwidth but won't cause issues.

So, make sure that MAC aging in the switches is configured appropriately and that the device using the MAC emits frames before the addressed is aged out. Also check the logs for any link flapping that might cause the MAC to be dropped prematurely. As a workaround, you could try triggering an ARP query every few minutes or so.

Since it's the right-side Nexus that losing the MAC address from its table, you should make sure that there's at least some traffic originating from 192.168.0.20 hitting that switch before the aging times out the address (defaults to 1800 seconds). Normally, somewhat frequent broadcasts (e.g. ARP) ensure reliable updating of all switches but that node might not do/need that. If you can't extend the aging, pinging across that switch in either direction should solve the problem.

You should try setting a longer MAC aging timeout:

mac address-table aging-time seconds [ vlan vlan_id ]

If that doesn't help, you might put a static mapping on the desired port:

mac address-table static address mac_addr vlan vlan_id [ interface type slot/port ]

See https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus7000/sw/layer2/command/cisco_nexus7000_layer2_command_ref/cisco_nexus7000_layer2_command_ref_chapter_011.html for details.

Sorry Zac67, you must have misunderstood me - the frames addressed to machines are sent to certain devices (plural, as the diagram shows with the case of flooding unknown unicast). And thank you for your other suggestions - I have added the edits to clarify these points — Tony Sepia, May 03 '21 at 10:08
Oh, you mean "all devices" (attached to the right-hand Nexus)? — Zac67, May 03 '21 at 10:14
Thank you, @Zac67! I have added EDIT 3 in response to your recent edits — Tony Sepia, May 04 '21 at 15:51

Unknown cause of disappearing CAM table entry and unicast flooding

1 Answers1