EtherChannel Mishaps

So we implemented Cisco's Virtual Switching System at work this week, and almost flawlessly I might add.  We ran into a small mishap on our final 2 switches while configuring the multi-chassis etherchannels.  Now we configured the channels correctly, but there was one statement that bit us:

switchport trunk allowed vlan [list]

This, on the server distribution switches, was different on the physical interfaces.  One of the physical interfaces had one more vlan in the list than the other.  Now, the channel came up, and looked good on the core end, and on the server distribution switch end...except that the port with the most vlans in the allowed list was suspended from the port channel (its actual state was up/down).  We began having intermittent connectivity problems to some of our servers.  Weird.  Upon investigation I found that suspended port on the server switches.  First I noticed the up/down state in a show ip int b; then the suspended port in a show interface status.  A quick look at the log showed the problem (different vlan/mask).  I quickly added the vlan manually to the port-channel interface and the physical interface came up, and began operations in the channel-group once again.  After a quick bit of labbing I confirmed the problem.  The intermittent connectivity was indeed because of the load-balancing algorithm, and etherchanneling protocol in use.  We were using src-dst-ip load balancing on both ends.  Because we used channel-group mode on, as our statement, the core end of the channel saw the suspended port as still in the channel (had we used PaGP we would not have had the problem).  Because it was still in the channel, the XOR calculation was still using the port index as a transmit link.  So when the XOR hit...the link was used to transmit packets.  When packets chased down the link, they were silently discarded by the suspended port.  Funny how things work!  Luckily, we were able to quickly identify the problem, and resolve it.  TAC confirmed our suspicions. 

XOR with 2 links...

gig0/1 -- index 0
gig0/2 -- index 1

Use the last bit of the IP address for XOR calculations (2^1 give you 2)

192.168.100.1 --> 192.168.100.2
1 in binary 00000001
2 in binary 00000010

XOR 1 and 0 = 1 (Use link gig0/2)

XOR breakdown
0 and 0 = 0
0 and 1 = 1
1 and 0 = 1
1 and 1 = 0