Dropping packets and seeing errors



  • We recently started seeing slow network response and dropping packets on a NSRP pair of firewalls.  The counter statistics show some errors.  I’v eliminated port speeds, cables, etc.  I can fail over to my backup firewall which goes into a second 3750 Cisco switch and I see the same errors.  Any suggestions on what I might need to look at.  This connection is coming from my ISP and they have been little to no help and it’s going to be very hard for me to sniff that connection.

    Hardware counters for interface ethernet1/1:
    in bytes      2343207298 | out bytes    2819415257 | early frame            0
    in packets      94004412 | out packets    84511282 | late frame            0
    in no buffer          0 | out no buffer          0 | re-xmt limit          0
    in overrun            0 | out underrun  1828716544 | drop vlan              0
    in coll err            0 | out coll err          0 | out cs lost            0
    in misc err            0 | out misc err  1828716544 |                       
    in dma err            0 | out bs pak            0 |                       
    in crc err            0 | out discard            0 |                       
    in align err          0 | out defer              0 |                       
    in short frame1828716544 | out heartbeat          0 |

    Hardware 64-bit counters for interface ethernet1/1:
    in bytes                  23818044926 |  out bytes                19999285923
    in ucast                      453654 |  out ucast                      398687
    in mcast                            0 |  out mcast                          0
    in bcast                            0 |  out bcast                          0

    Total flow counters for interface ethernet1/1:
    in bytes      161199877 | out bytes      84036489 | tcp proxy              0
    tear drop              0 | in vlan                0 | out vlan              0
    in permit      158840810 | out permit      10871696 | src route              0
    no g-parent            0 | ping of death          0 | no gate sess          0
    address spoof          0 | in icmp          245072 | no nat vector          0
    land attack            0 | in self                0 | no map                0
    icmp flood            0 | in un-auth            0 | no conn                0
    udp flood              0 | in unk prot            0 | no dip                0
    winnuke                0 | in vpn              522 | no gate                0
    port scan              0 | in other              0 | no xmit vpnf          0
    ip sweep              0 | no mac                0 | no route              3
    tcp out of seq        0 | mac relearn            0 | no frag sess          0
    wrong intf            0 | slow mac              0 | no frag netpak        0
    wrong slot            0 | trmng queue            0 | no sa                133
    icmp broadcast        0 | trmng drop            0 | no sa policy          0
    illegal pak          484 | tiny frag              0 | sa inactive            0
    url block              0 | syn frag              0 | sa policy deny        0
    encrypt fail          0 | connections        5089 | policy deny            0
    mp fail                0 | misc prot              0 | auth deny              0
    auth fail              0 | loopback drop        242 | big bkstr              0
    proc sess              0 | mal url                0 | sessn thresh          0
    invalid zone          0 | null zone              0 | no nsp-tunnel          0
    IP cls failure        0 | first pak frag        0 | unknown pak        7493
    multiauth drop        0 | multi-DIP drop        0 | ip chksum              0
    ip pak short          0 | ip tlen over          0 | ip pak trunc          0
    ip ver err            0 | ip hdlen err          0 | ip bad src            50
    ip bad prot            0 | icmp hdlen err        0 | udp hdlen err          0
    tcp hdlen err          0 | tcp offset err        0 | tiny tcp              0
    ip opt err            0 | tcp opt err            0 |



  • The short frames might be the VoIP traffic running across this link.  We did get the problem resolved.  It turned out to be a routing problem on the ISP side on an SRP link between to Cisco GSR routers.  This would have taken much longer to resolve if our ISP was not another division of our company.



  • Out overruns and out misc errs indicate that likely the firewall is sending packets faster than the upstream switch can handle. You mention that this only happened recently. Did anything change recently? How long has this worked in the past? What firewall platform and ScreenOS is this?

    Also I am curious about the in short frame. This seems to equal the values of the overruns. In short frames means the firewall is seeing packets smaller than 64 bytes. You might need to run some sort of sniffer or, depending on the firewall platform and how busy it is, possibly snoop on the firewall itself to find out what those short frames are.


 

28
Online

38.4k
Users

12.7k
Topics

44.5k
Posts