NetSS

Linux Firewalls Using iptables

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

In This Chapter

Chapter 11

Linux Firewalls Using iptables

What Is iptables?

Download And Install The Iptables Package

How To Get iptables Started

Packet Processing In iptables

Iptables Packet Flow Diagram

Processing For Packets Routed By The Firewall

Packet Processing For Data Received By The Firewall

Packet Processing For Data Sent By The Firewall

Targets And Jumps

Descriptions Of The Most Commonly Used Targets

Important Iptables Command Switch Operations

General Iptables Match Criteria

Common TCP and UDP Match Criteria

Common ICMP (Ping) Match Criteria

Common Match Extensions Criteria

Using User Defined Chains

Sample iptables Scripts

Basic Initialization

Allowing DNS Access To Your Firewall

Allowing WWW And SSH Access To Your Firewall

Allowing Your Firewall To Access The Internet

Allow Your Home Network To Access The Firewall

Masquerading (Many to One NAT)

Port Forwarding Type NAT

Static NAT

Logging & Troubleshooting

© Peter Harrison, www.linuxhomenetworking.com

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

You can convert your Linux box into a firewall using the IPtables package. This page shows how to convert your Linux box into:

·     A firewall while simultaneously being your home website's mail, web and DNS server.

·     A router that will use NAT and port forwarding to both protect your home network and have another web server on your home network while sharing the public IP address of your firewall

What Is iptables?

Originally, the most popular firewall / NAT package running on Linux was ipchains. It had a number of limitations, the primary one being that it ran as a separate program and not as part of the kernel. The Netfilter organization decided to create a new product called iptables in order to rectify this shortcoming. As a result of this, iptables is considered a faster and more secure alternative.  iptables has now become the default firewall package installed under RedHat Linux.  

Download And Install The Iptables Package

Most RedHat Linux software products are available in the RPM format. Downloading and installing RPMs isn’t hard. If you need a refresher, the chapter on RPMs covers how to do this in detail. The latest version of the RPM for RedHat 8.0 is iptables-ipv6-1.2.6a-2.i386.rpm. Install the package using the following command:

[root@bigboy tmp]# rpm -Uvh iptables-ipv6-1.2.6a-2.i386.rpm
Preparing...    ########################################### [100%]
1:iptables      ########################################### [100%]

[root@bigboy tmp]#  

How To Get iptables Started

You can start/stop/restart iptables after booting by using the following commands:

[root@bigboy tmp]# /etc/init.d/iptables start
[root@bigboy tmp]# /etc/init.d/iptables stop
[root@bigboy tmp]# /etc/init.d/iptables restart

To get iptables configured to start at boot:

[root@bigboy tmp]# chkconfig --level 345 iptables on  

Packet Processing In iptables

All packets inspected by iptables pass through a sequence of built-in tables (queues) for processing. Each of these queue is dedicated to a particular type of packet activity and is controlled by an associated packet transformation / filtering chain. Don’t worry if this all seems confusing, there’ll be tables and examples of how the concepts are all interlinked.

For example, the chart and graphic below describe the steps taken by iptables when a packet traverses the firewall.

Iptables Packet Flow Diagram

Processing For Packets Routed By The Firewall

Packet Processing For Data Received By The Firewall

Packet Processing For Data Sent By The Firewall

Targets And Jumps

You don't have to rely solely on the built-in chains provided by iptables, you can create your own chains. These can be accessed by making them the targets of "jumps" in the built-in chains. So in summary, the targets/jumps tell the rule what to do with a packet that matches the rule perfectly.

There are a number of built-in targets that most rules may use.

Descriptions Of The Most Commonly Used Targets

Here are some examples of the output of this file:

o        Firewall denying replies to DNS queries (UDP port 53) destined to server 192.168.1.102 on the home network.

Feb 23 20:33:50 bigboy kernel: IN=wlan0 OUT= MAC=00:06:25:09:69:80:00:a0:c5:e1:3e:88:08:00 SRC=192.42.93.30 DST=192.168.1.102 LEN=220 TOS=0x00 PREC=0x00 TTL=54 ID=30485 PROTO=UDP SPT=53 DPT=32820 LEN=200

o        Firewall denying Windows NetBIOS traffic (UDP port 138)

Feb 23 20:43:08 bigboy kernel: IN=wlan0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:06:25:09:6a:b5:08:00 SRC=192.168.1.100 DST=192.168.1.255 LEN=241 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=138 DPT=138 LEN=221

o        Firewall denying Network Time Protocol (NTP UDP port 123)

Feb 23 20:58:48 bigboy kernel: IN= OUT=wlan0 SRC=192.168.1.102 DST=207.200.81.113 LEN=76 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=123 DPT=123 LEN=56

Note: The traffic in all these examples isn’t destined for the firewall. Therefore you should check your INPUT, OUTPUT, FORWARD and NAT related statements. If the firewall’s IP address is involved, then you should focus on the INPUT, OUTPUT statements

If nothing shows up in the logs, then follow the steps in the Network Troubleshooting chapter to determine whether the data is reaching your firewall at all, and if it is not, the location your network that could be causing the problem.

Troubleshooting NAT: As a general rule, you won’t be able to access the public NAT IP addresses from servers on your home network. Basic NAT testing will require you to ask a friend to try to connect to your home network from the Internet.

You can then use the logging output in /var/log/messages to make sure that:

o        the translations are occurring correctly and

o        iptables isn’t dropping the packets after translation occurs

1. Tcpdump란?
 tcpdump는 Lawrence Berkley Nation Lab의 Network Rearch Gruop에서 만든 것으로 네트워크의 패킷
을 출력해주는 프로그램이다. 주어진 조건식을 만족하는 네트워크 인터페이스를 거치는 패킷들의
헤더들을 출력해 주는 프로그램이다. 주로 쓰임은 지정된 상대방 호스트로부터 들어오는 패킷을 체크
하는 데 있다.

2. tcpdump의 사용법
 (1) 사용법
   tcpdump [options] [host]
 (2) tcpdump의 결과보기 
 [root@www root]# tcpdump 01:37:09.744959 203.xxx.xxx.10.4847 > linux.co.kr.ftp: S 2082495566:2082495566(0) win 16384 (DF)
=> (결과설명) 01:37:09.744959 : Timestamp로 보통 '시간.특정한숫자값'형태로 부여된다. 203.xxx.xxx.10.4847 : Source_address.source_port로 소스의 IP주소와 Port번호를 나타낸 다. > : Traffic direction으로 방향을 나타낸다. linux.co.kr.ftp : destination_address.destination_port로 도착지 IP주소와 Port번호를 나타낸다. S : TCP flag로 특정한 표시문자를 일컫는다. 2082495566:2082495566(0) : sequence number이다. win 16384 : window size이다. : various options 이다. (DF) : Don't fragment 3. tcpdump 플래그(flags)


TCP 플래그
TCPDUMP 플래그
플래그의 의미
SYN
S
SYN패킷, 접속요청을 할 때 보내는 패킷을 말한다.
TCP접속시에 가장먼저 보내는 패킷이다.
ACK
ack
ACK패킷, 상대방으로부터 패킷을 받은 뒤에 알려주는
패킷을 말한다. 다른 플래그와 같이 출력되는 경우도
있다.
FIN
F
접속종료를 위한 플래그로 이 패킷을 보내는 곳이
현재 접속하고 있는 곳과 접속을 끊고자 할 때 사용한다.
RESET
R
이 패킷을 보내는 곳이 현재 접속하고 있는 곳과 즉시
연결을끊고자 할 때 사용한다.
PUSH
P
데이터를 즉시 목적지로 보내라는 의미이다. 텔넷과
같이 상호작용이 중요한 프로그램의 경우 빠른 응답
이 중요한다. 이때 사용하는 플래그이다.
UGENT
URG
긴급한 데이터는 다른 데이터에 비해 우선순위가
높아야 한다. 예를 들어 ftp로 자료를 받고 있는 도중
[CTRL]+[C] 를 받으면 즉시 자료받기를 중단해야 하는
것처럼 이 때 사용하는 플래그이다.
Placeholder
.
패킷이 SYN, FINISH, RESET, PUSH등의 플래그가 없는
경우이 플래그가 세팅된다. 이 플래그는 ACK플래그와
함께 사용되는 경우도 있다.

4. tcpdump의 옵션
 -h: 버전 및 기본 사용법을 보여준다.
 -a : Network & Broadcast 주소들을 이름들로 바꾼다.
 -c Number : 제시된 수의 패킷을 받은 후 종료한다.
 -d : compile된 packet-matching code를 사람이 읽을 수 있도록 바꾸어 표준 출력으로 출력하고,
     종료한다.
 -dd : packet-matching code를 C program의 일부로 출력한다.
 -ddd : packet-matching code를 숫자로 출력한다.
 -e : 출력되는 각각의 행에 대해서 link-level 헤더를 출력한다.
 -f : 외부의 internet address를 가급적 심볼이 아닌 숫자로 출력한다.(Sun의 yp server와의 사용은
     가급적 피한다.)
 -F file : filter 표현의 입력으로 파일을 받아들인다. 커맨드라인에 주어진 추가의 표현들은 모두
          무시된다.
 -i device : 어느 인터페이스를 경유하는 패킷들을 잡을지 지정한다. 지저되지 않으면 시스템의
            인터페이스 리스트를 검색하여 가장 낮은 번호를 가진 인터페이스를 선택한다.(이 때
            loopback은 제외된다.) 예) tcpdump -i eth0
 -l : 표준 출력으로 나가는 데이터들을 line buffering한다. 다른 프로그램에서 tcpdump로부터 데이
     터를 받고자 할 때 유용하다. 보통 'tcpdump -l | tee dat' 나 'tcpdump -l < dat & tail -f
     dat' 명령으로 연계해서 사용하면 편리하다.
 -n : 모든 주소들을 번역하지 않는다(port,host address 등등)
 -N : 호스트 이름을 출력할 때, 도메인을 찍지 않는다.
 -O : packet-matching code optimizer를 실행하지 않는다. 이 옵션은 optimizer에 있는 버그를 찾을
     때나 쓰인다.
 -p : 인터페이스를 promiscuous mode로 두지 않는다.
 -q : 프로토콜에 대한 정보를 덜 출력한다. 따라서 출력되는 라인이 좀 더 짧아진다.
 -r file : 패킷들을 '-w'옵션으로 만들어진 파일로 부터 읽어 들인다. 파일에 "-" 가 사용되면 표준
          입력을 통해서 받아들인다.
 -s length: 패킷들로부터 추출하는 샘플을 default값인 68Byte외의 값으로 설정할 때 사용한다.(Sun
           OS의 NIT에서는 최소가 96Byte이다). 68Byte는 IP,ICMP, TCP, UDP등에 적절한 값이지만
           Name Server나 NFS 패킷들의 경우에는 프로토콜의 정보들을 Truncation할 우려가 있다.
           이 옵션을 수정할 때는 신중해야만 한다. 이유는 샘플 사이즈를 크게 잡으면 곧 패킷 하
           나하나를 처리하는데 시간이 더 걸릴 뿐만아니라 패킷 버퍼의 사이즈도 자연히 작아지게
           되어 손실되는 패킷들이 발생할 수 있기 때문이다. 또, 작게 잡으면 그만큼의 정보를 잃
           게되는 것이다. 따라서 가급적 캡춰하고자 하는 프로토콜의 헤더 사이즈에 가깝게 잡아
           주어야 한다.
 -T type : 조건식에 의해 선택된 패킷들을 명시된 형식으로 표시한다. type에는 다음과 같은 것들이
          올 수 있다. rpc(Remote Procedure Call), rtp(Real-Time Applications protocol), rtcp
          (Real-Time Application control protocal), vat(Visual Audio Tool), wb(distributed
           White Board)
 -S : TCP sequence번호를 상대적인 번호가 아닌 절대적인 번호로 출력한다.
 -t : 출력되는 각각의 라인에 시간을 출력하지 않는다.
 -tt : 출력되는 각각의 라인에 형식이 없는 시간들을 출력한다.
 -v : 좀 더 많은 정보들을 출력한다.
 -vv : '-v'보다 좀 더 많은 정보들을 출력한다.
 -vvv : 16진수값 형태로 정보를 보여준다. 보통 -X옵션과 사용되어
 -w : 캡춰한 패킷들을 분석해서 출력하는 대신에 그대로 파일에 저장한다.
 -x : 각각의 패킷을 헥사코드로 출력한다.
 -X : 헥사코드와 ascii형태 모두 출력해준다. 보통 -x 옵션과 같이 사용된다.
 -조건식(expression!)
  옵션의 제일 마지막인 조건식은 어떤 패킷들을 출력할 지를 선택하는 데 쓰인다. 조건식이 주어지지
 않는다면 모든 패킷들이 대상이 된다. 조건식들은 하나 또는 여러 개의 primitive들로 구성되어 있고
 primitive들은 다시 하나 또는 여러 개의 qualifier들 다음에 오는 하나의 값으로 이루어진다.
  *qualifier
   type : 주어진 값의 종류가 무엇인지를 나타낸다. 가능한 type들은 'host', 'net', 'port'가 있다.
         type이 없는 값들은 type을 host라 가정한다.
   dir : id로 부터의 어떤 특정한 전송 방향을 나타낸다. 가능한 방향은 'src', 'dst', 'src or
        dst', 'src and dst'이다. 만약 방향이 정해지지 않았다면, src or dst라 가정한다. "For
        `null' link layers (i.e. point to point protocols such as slip) the inb ound and out
        bound qualifiers can be used to specify a desired direction."
   proto : 매칭을 특정 프로토콜에 한해서 수행한다. 가능한 프로토콜들은 ether, fddi, ip, arp,
          rarp, decnet, lat, sca, moprc, mopdl, tcp, udp이다. 만약 프로토콜이 명시되지 않았다
          면, 해당하는 값의 type에 관련된 모든 프로토콜들이 그 대상이 된다.
  *위의 패턴을 따르지 않는 primitive : gateway, broadcast, less, greater, 산술식이 있으며 참고
                                   로 더 정교한 조건식을 사용하려면 'and(&&)','or(||)','not(!)'
                                   을 사용할 수 있다.
  *사용가능한 primitive들
   dst host HOST
    => packet의 IP destination 항목이 HOST일때 참이 된다.
   src host HOST
    => packet의 IP source 항목이 HOST일때 참이 된다.
   host HOST
    => IP source, IP destination 항목 중 어느 하나라도 HOST이면 참이다.
   ether dst ehost
    => ethernet destination 주소가 ehost일 때 참이다.
   ether src ehost
    => ethernet source 주소가 ehost일 때 참이다.
   ether host ehost
    => ethernet source, destination 항목들 중 어느 하나라도 ehost이면 참이다.
   gateway host
    => 패킷이 host를 게이트웨이로 사용하면 참이다. 이 말의 의미는 ethernet sour ce나 destina
      tion 항목은 host이지만, IP source와 destination은 host가 아닐 때를 말한다.
   dst net NET
    => 패킷의 IP destination 주소가 NET의 network number를 가지고 있을 때 참이다.
   src net NET
    => 패킷의 IP source 주소가 NET의 network number를 가지고 있을 때 참이다.
   net NET
    => 패킷의 IP source 주소 혹은 destination 주소가 NET의 network number를 가 지고 있을 때
      참이다.
   net netmask mask
    => IP 어드레스가 지정된 netmask를 통해서 net과 매칭되면 참이다.
   net net/len
    => IP 어드레스가 netmask와 len 비트만큼 매치되면 참이다.
   dst port PORT
    => 패킷이 ip/tcp, ip/udp 프로토콜의 패킷이고 destination port의 값이 PORT일 때 참이다.
      port는 /etc/services에 명시된 이름일 수도 있고 그냥 숫자일 수도 있다. 만약 이름이 사용
      됐다면 port 번호와 프로토콜이 같이 체크될 것이다. 만약 숫자나 불 확실한 이름이 사용됐을
      경우에는 port 번호만이 체크될 것이다.
   src port PORT
    => 패킷의 source port의 값으로 PORT를 가지면 참이다.
   port PORT
    => 패킷의 source, destination port 중에 하나라도 PORT이면 참이다.
   less length
    => 패킷이 length보다 짧거나 같으면 참이다.(len <= length)
   greater length
    => 패킷이 length보다 짧거나 같으면 참이다.(len >= length)
   ip proto protocol
    => 패킷이 지정된 종류의 프로토콜의 ip패킷이면 참이다. Protocol은 icmp, igrp, udp, nd, tcp 중의 하나 혹은 몇 개가 될 수 있다. 주의할 점은 tcp, udp, icmp들은 '\'로 escape되어야 한다.
   ehter broadcast
    => 패킷이 ethernet broadcast 패킷이라면 참이다. ehter는 생략 가능하다.
   ip broadcast
    => 패킷이 IP broadcast 패킷이라면 참이다.
   ether multicast 
    => 패킷이 IP multicast 패킷이라면 참이다.
   ether proto protocol
    => 패킷이 ether type의 protocol이라면 참이다. protocol은 ip, arp, rarp 중에 하나 혹은 몇 개가 될 수 있다. ip proto protocol에서와 마찬가지로 ip, arp, rarp는 escape 되어야 한다.
   decnet src host
    => 만약 DECNET의 source address가 host이면 참이다. 이 어드레스는 '10.123'이 나 DECNET의 host name일 수 있다. DECNET host name은 DECNET에서 돌아가도록 설정된 Ultrix 시스템에서 만 사용 가능하다.
   decnet dst host
    => DECNET destination address가 host이면 참이다.
   decnet host HOST
    => DECNET source, destination address중의 하나라도 HOST이면 참이다. ip, arp, rarp, decnet => ether proto [ip|arp|rarp|decnet]의 약어 lat, moprc, mopdl => ether proto [lat|moprc|mopdl]의 약어 tcp, udp, icmp => ip proto [tcp|udp|icmp]의 약어 expr relop expr => EXPR proto [expr:size]의 형식을 띤다. proto, expr, size에 올 수 있는 것들은 다음과 같다. proto : ether, fddi, ip, arp, rarp, tcp, udp, icmp expr : indicate Byte offset of packet of proto size : optional. indicate the size of bytes in field of interest default is one, and can be two or four => RELOP !=, =, <=, >=, etc. 이 조건식을 사용하기 위해서는 먼저 해당하는 Protocol(proto)의 헤더에 관련된 것들을 자세 히 알아야만 한다. proto에는 대상이 될 프로토콜을 지정한다. expr에는 프로토콜 헤더의 처음 부터의 Byte Offset을 지정하는 식이 들어가게 된다. Size는 Option이며 지정이 안 되어 있을 경우에는 자동으로 1byte를 지칭한다. 따라서 이 조건식을 사용하게 되면 헤더에 포함된 정보 를 Bitmask를 사용하여 직 접 원하는 패킷인지를 가려낼 수 있기 때문에, 보다 정밀한 사용이 가능하게 된다.

5. tcpdump 기본사용예
(1) [root@www root]# tcpdump => 현재 서버의 모든 패킷을 보여준다.

(2) [root@www root]# tcpdump port 21
tcpdump: listening on eth0 01:25:43.833276 203.xxx.xxx.100.4828 > xxx.com.ftp: S 1910472596:1910472596(0) win 16384 203.247.40.244.4828: S 4108025962:4108025962(0) ack 1910472597 )
=> 21번 포트로 들어온 패킷만 보여준다.

(3) [root@nanajjang /root]# tcpdump -c 2 host neuro.hannam.ac.kr Kernel filter, protocol ALL, datagram packet socket
tcpdump: listening on all devices 11:09:44.576419 eth0 B arp who-has 203.247.40.247 tell neuro.hannam.ac.kr 11:09:44.576507 eth0 > arp reply 203.247.40.247 (0:c0:26:1:11:77) is-at 0:c0:26)
=> neuro.hannam.ac.kr의 패킷 2개만 보여준다.

(4) tcpdump -v -e broadcast
=> broadcast 의 정보를 자세히 출력하고 출력되는 각각의 행에 대해서 link-level헤더를 출력 한다.

6. tcpdump로 패스워드 알아내기

(1) 설명: telnet을 이용하여 서버에 접속을 하면 패킷전송시에 암호화하지 않기 때문에 쉽게 내용 을 볼 수 있다. tcpdump를 이용하여 패스워드를 알아낼 수 있다.

(2) 사용예
  1) 실행명령 [root@www root]# tcpdump port telnet -l -vvv -x -X >dumpdata & tail -f dumpdata
 
  2) 패킷분석: 패킷의 양이 많으므로 잘 찾아야 한다. 먼저 연결된 포트를 확인한다.
 20:38:22.395281 xxx.com.telnet > 210.xxx.xxx.100.35383: P [tcp sum ok] 175:182(7) ack 81 win 5792 (DF) [tos 0x10] (ttl 64, id 59193, len 59) 0x0000 4510 003b e739 4000 4006 ca41 cbf7 28fc E..;.9@.@..A..(. 0x0010 d27b c1c2 0017 8a37 2992 16d8 2041 b972 .{.....7)....A.r 0x0020 8018 16a0 5fec 0000 0101 080a 04d1 f156 ...._..........V 0x0030 0223 7825 6c6f 6769 6e3a 20 .#x%login:.
=> 서버(xxx.com)에서 클라이언트(210.xxx.xxx.100에게 'login'이라는 것을 보여줌을 알 수 있다. 또한 연결된 클라이언트의 포트번호가 35383임을 알 수 있다. 이 포트번호를 확인해 야 패킷을 찾아갈 수 있다.
---중략---
20:38:23.979306 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 81:82(1) ack 182 win 5840 (DF) [tos 0x10] (ttl 56, id 49475, len 53) 0x0000 4510 0035 c143 4000 3806 f83d d27b c1c2 E..5.C@.8..=.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b972 2992 16df ..(..7...A.r)... 0x0020 8018 16d0 5130 0000 0101 080a 0223 78c3 ....Q0.......#x. 0x0030 04d1 f156 70 ...Vp => 클라이언트에서 서버로 전송하는 부분만을 확인하면 된다. P 다음에 존재하는 81:82와 ack 다음에 182라는 번호를 꼭 확인한다. 다음 패킷번호는 82:83이 되고 ack 183번이 되므로 ack 183이라는 부분을 찾아가면 된다. 여기서 클라이언트에서 서버로 전송한 글자는 맨 마지막에 표시된다. 즉 'p'이다.
---중략---
20:38:24.162490 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 82:83(1) ack 183 win 5840 (DF) [tos 0x10] (ttl 56, id 49477, len 53) 0x0000 4510 0035 c145 4000 3806 f83b d27b c1c2 E..5.E@.8..;.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b973 2992 16e0 ..(..7...A.s)... 0x0020 8018 16d0 4ef0 0000 0101 080a 0223 78d6 ....N........#x. 0x0030 04d1 f481 6f ....o
---중략---
20:38:24.322785 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 83:84(1) ack 184 win 5840 (DF) [tos 0x10] (ttl 56, id 49479, len 53) 0x0000 4510 0035 c147 4000 3806 f839 d27b c1c2 E..5.G@.8..9.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b974 2992 16e1 ..(..7...A.t)... 0x0020 8018 16d0 4a80 0000 0101 080a 0223 78e6 ....J........#x. 0x0030 04d1 f4df 73 ....s
---중략---
20:38:24.445142 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 84:85(1) ack 185 win 5840 (DF) [tos 0x10] (ttl 56, id 49481, len 53) 0x0000 4510 0035 c149 4000 3806 f837 d27b c1c2 E..5.I@.8..7.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b975 2992 16e2 ..(..7...A.u)... 0x0020 8018 16d0 5820 0000 0101 080a 0223 78f2 ....X........#x. 0x0030 04d1 f531 65 ...1e
---중략---
20:38:24.532360 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 85:86(1) ack 186 win 5840 (DF) [tos 0x10] (ttl 56, id 49483, len 53) 0x0000 4510 0035 c14b 4000 3806 f835 d27b c1c2 E..5.K@.8..5.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b976 2992 16e3 ..(..7...A.v)... 0x0020 8018 16d0 53d6 0000 0101 080a 0223 78fb ....S........#x. 0x0030 04d1 f570 69 ...pi
---중략---
20:38:24.624350 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 86:87(1) ack 187 win 5840 (DF) [tos 0x10] (ttl 56, id 49485, len 53) 0x0000 4510 0035 c14d 4000 3806 f833 d27b c1c2 E..5.M@.8..3.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b977 2992 16e4 ..(..7...A.w)... 0x0020 8018 16d0 4e9f 0000 0101 080a 0223 7904 ....N........#y. 0x0030 04d1 f59c 6e ....n
---중략---
=> posein이라는 글자를 알 수 있다. 즉 사용자 ID가 posein이다.
20:38:24.932380 xxx.com.telnet > 210.xxx.xxx.100.35383: P [tcp sum ok] 190:200(10) ack 89 win 5792 (DF) [tos 0x10] (ttl 64, id 59201, len 62) 0x0000 4510 003e e741 4000 4006 ca36 cbf7 28fc E..>.A@.@..6..(. 0x0010 d27b c1c2 0017 8a37 2992 16e7 2041 b97a .{.....7)....A.z 0x0020 8018 16a0 d40b 0000 0101 080a 04d1 f669 ...............i 0x0030 0223 7923 5061 7373 776f 7264 3a20 .#y#Password:.
=> 서버에서 클라이언트에서 Password라는 글을 보내어 출력시킨다.
---중략---
20:38:25.300524 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 89:90(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49490, len 53) 0x0000 4510 0035 c152 4000 3806 f82e d27b c1c2 E..5.R@.8....{.. 0x0010 cbf7 28fc 8a37 0017 2041 b97a 2992 16f1 ..(..7...A.z)... 0x0020 8018 16d0 557f 0000 0101 080a 0223 7947 ....U........#yG 0x0030 04d1 f669 66 ...ip
---중략---
20:38:25.454778 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 90:91(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49491, len 53) 0x0000 4510 0035 c153 4000 3806 f82d d27b c1c2 E..5.S@.8..-.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b97b 2992 16f1 ..(..7...A.{)... 0x0020 8018 16d0 489e 0000 0101 080a 0223 7957 ....H........#yW 0x0030 04d1 f739 72 ...9a
---중략---
20:38:25.659340 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 91:92(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49492, len 53) 0x0000 4510 0035 c154 4000 3806 f82c d27b c1c2 E..5.T@.8..,.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b97c 2992 16f1 ..(..7...A.|)... 0x0020 8018 16d0 554e 0000 0101 080a 0223 796b ....UN.......#yk 0x0030 04d1 f774 65 ...ts
---중략---
20:38:25.777475 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 92:93(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49493, len 53) 0x0000 4510 0035 c155 4000 3806 f82b d27b c1c2 E..5.U@.8..+.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b97d 2992 16f1 ..(..7...A.})... 0x0020 8018 16d0 54d8 0000 0101 080a 0223 7977 ....T........#yw 0x0030 04d1 f7dd 65 ....s
---중략---
20:38:26.049208 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 93:94(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49494, len 53) 0x0000 4510 0035 c156 4000 3806 f82a d27b c1c2 E..5.V@.8..*.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b97e 2992 16f1 ..(..7...A.~)... 0x0020 8018 16d0 887f 0000 0101 080a 0223 7992 .............#y. 0x0030 04d1 f81a 31 ....w
---중략---
20:38:26.243634 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 94:95(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49495, len 53) 0x0000 4510 0035 c157 4000 3806 f829 d27b c1c2 E..5.W@.8..).{.. 0x0010 cbf7 28fc 8a37 0017 2041 b97f 2992 16f1 ..(..7...A..)... 0x0020 8018 16d0 86df 0000 0101 080a 0223 79a6 .............#y. 0x0030 04d1 f8a5 32 ....o
---중략---
20:38:26.442302 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 95:96(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49496, len 53) 0x0000 4510 0035 c158 4000 3806 f828 d27b c1c2 E..5.X@.8..(.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b980 2992 16f1 ..(..7...A..)... 0x0020 8018 16d0 8768 0000 0101 080a 0223 79b9 .....h.......#y. 0x0030 04d1 f908 31 ....r
---중략---
20:38:26.540930 210.xxx.xxx.100.35383 > xxx.com.telnet: P [tcp sum ok] 96:97(1) ack 200 win 5840 (DF) [tos 0x10] (ttl 56, id 49497, len 53) 0x0000 4510 0035 c159 4000 3806 f827 d27b c1c2 E..5.Y@.8..'.{.. 0x0010 cbf7 28fc 8a37 0017 2041 b981 2992 16f1 ..(..7...A..)... 0x0020 8018 16d0 80f7 0000 0101 080a 0223 79c3 .............#y. 0x0030 04d1 f96e 37 ...nd
---이하생략---
=> 패스워드가 password라는 것을 알 수 있다.출처 : http://www.attikorea.com/secuutil/tcpdump.html

TCPDUMP User Guide

Date : January 1999

written by Kwon, YongChul

-. 이 문서의 저작권은 저자(권용철:godslord@sparcs.kaist.ac.kr)에게 있습니다. 본문의 무단 인용, 복사는 저자와의 협의 없이 절대 불가능합니다.

Table of Contents

이 문서를 보기 전에 알아야 할 사전 지식

Tcpdump란?

Tcpdump의 패키지 구하기

Tcpdump의 설치

Tcpdump Source의 간략한 설명

Tcpdump의 옵션들

조건식
Tcpdump의 사용 예제들

Tcpdump의 평가

References

이 문서를 보기 전에 알아야 할 사전 지식

이 문서는 네트워크의 패킷들을 잡아내는 Tcpdump라는 프로그램에 대해서 설명하고 있다. 프로그램의 특성상, 네트워크에 관련된 많은 용어들과 특히 TCP/IP에 대한 내용 이 많이 나온다. 따라서 보다 잘 이해하기 위해서는 네트워크 설비 자료나, 네트워크 프로그래밍 가이드, 혹은 네트워크 프로토콜(TCP/IP)에 관련된 책들을 참조하는 것이 좋을 것이다.

Tcpdump란?

Tcpdump는 주어진 조건식을 만족하는 네트워크 인터페이스를 거치는 패킷들의 헤더들 을 출력해 주는 프로그램이다. 프로그램의 특성상, 네트워크 인터페이스를 아주 심도 있게 사용하기 때문에, 실행하는 사람은 반드시 네트워크 인터페이스에 대한 읽기 권 한이 있어야만 한다.
OS dependent)
위에서 말하는 읽기 권한을 가지고 있어야 하는 파일, 혹은 Tcpdump의 퍼미션 이다.

SunOS : /dev/nit, /dev/bpf*

Solaris(SunOS 5.x) : /dev/le 등

HP-UX & IRIX & Linux : root에서 실행하거나, root로 setuid를 설정해야 함

Ultrix & Digital UNIX : root가 pfconfig를 이용하여, promiscuous-mode를 가능하게 설정하면 어떤 유저라도 사용할 수 있다.

BSD : /dev/bpf*

Tcpdump의 패키지 구하기

Tcpdump는 ftp://ftp.ee.lbl.gov/tcpdump.tar.Z에서 최신 버전을 구할 수 있다. 유명 한 프로그램이기 때문에, 시스템소프트웨어를 패키지형태로 제공해 주는 OS들의 경우 Vendor에서 패키징된(컴파일된) 버전으로도 구할 수 있을 것이다.

Tcpdump의 설치

Tcpdump는 libpcap(Protocol Capture Library)라는 것을 사용한다. 이 라이브러리는 Platform에 상관없이 동일한 방법으로 사용자 레벨에서 패킷들을 캡춰할 수 있게 해 준다. 따라서 이 라이브러리가 없다면, ftp://ftp.ee.lbl.gov/libpcap.tar.Z에서 구하 여 설치하도록 한다.

ANSI C 컴파일러는 아마 대부분의 시스템에서 구비하고 있을 것이다. 만약 없다면 ftp://prep.ai.mit.edu/pub/gnu/gcc.tar.gz를 받아서 설치하기 바란다.

libpcap라이브러리가 완벽하게 설치되었다는 가정하에서 다음의 절차에 따라 설치를 시작한다.

Makefile.in의 BINDEST와 MANDEST 항목에 각각, tcpdump 실행파일과 메뉴얼 페이 지가 설치될 디렉토리들을 입력해 준다.
Tcpdump 패키지와 함께 제공되는 ./configure 스크립트를 실행시킨다. 이 스크립트는 현재 시스템의 환경들을 검사하고 이에 맞추어서 Makefile을 생성해 준다.
make를 실행한다.
컴파일이 다 됐으면, make install을 수행하여 실행파일을 설치하고, make install-man을 실행하여 메뉴얼 페이지도 설치한다.
tcpdump의 퍼미션이 제대로 되었는지를 검사한다. setgid가 설정되어 있기 때문에, 원하지 않는 사람이 실행하게 된다면 위험하다. 위에서 설명한 절차를 그대로 옮겨 본다. # vi Makefile.in
# ./configure
# make
# make install
# make install-man
OS dependent)

DEC/OSF and BSD/386, etc : tcpdump가 OS와 함께 제공되는 경우가 있다. 이럴 경우 tcpdump를 업그레이드 하기 전에 반드시 기존의 tcpdump를 백업해 두도록 하자.

Linux : libpcap 라이브러리는 2.0.25 커널에서 테스트 됐다. 2.0.x 대 커널 에서 25이하의 버전에서는 실행이 아마 가능할 것이다. 그러나 1.x대의 커널에서는 동작 여부가 입증되지 않았으므로 주의해야 한다.

OSF 4 : OSF4에서는 stack C 컴파일러에 치명적인 버그가 있다. 이 버그를 피해가려면, ./configure를 실행한 후 컴파일 하기 전에 Makefile에서 다음의 문장을 삭제한 후 컴파일해 주면 된다.
            -DETHER_HEADER_HAS_EA=1 -DETHER_ARP_HAS_EA=1
 

Tcpdump Source의 간략한 설명

Tcpdump의 옵션들

-a : Network & Broadcast 주소들을 이름들로 바꾼다.

-c Number : 제시된 수의 패킷을 받은 후 종료한다.

-d : comile된 packet-matching code를 사람이 읽을 수 있도록 바꾸어 표준 출력으로 출력하고, 종료한다.

-dd : packet-matching code를 C program의 일부로 출력한다.

-ddd : packet-matching code를 숫자로 출력한다.

-e : 출력되는 각각의 행에 대해서 link-level 헤더를 출력한다.

-f : 외부의 internet address를 가급적 심볼로 출력한다(Sun의 yp server와의 사용은 가급적 피하자).

-F file : filter 표현의 입력으로 파일을 받아들인다. 커맨드라인에 주어진 추가의 표현들은 모두 무시된다.

-i device : 어느 인터페이스를 경유하는 패킷들을 잡을지 지정한다. 지저되지 않으면 시스템의 인터페이스 리스트를 뒤져서 가장 낮은 번호를 가진 인터페이스를 선택한다(이 때 loopback은 제외된다).

-l : 표준 출력으로 나가는 데이터들을 line buffering한다. 다른 프로그램에서 tcpdump로부터 데이터를 받고자 할 때, 유용하다.

-n : 모든 주소들을 번역하지 않는다(port,host address 등등)

-N : 호스트 이름을 출력할 때, 도메인을 찍지 않는다.

-O : packet-matching code optimizer를 실행하지 않는다. 이 옵션은 optimizer에 있는 버그를 찾을 때나 쓰인다.

-p : 인터페이스를 promiscuous mode로 두지 않는다.

-q : 프로토콜에 대한 정보를 덜 출력한다. 따라서 출력되는 라인이 좀 더 짧아진다.

-r file : 패킷들을 '-w'옵션으로 만들어진 파일로 부터 읽어 들인다. 파일에 "-" 가 사용되면 표준 입력을 통해서 받아들인다.

-s length: 패킷들로부터 추출하는 샘플을 default값인 68Byte외의 값으로 설정할 때 사용한다(SunOS의 NIT에서는 최소가 96Byte이다). 68Byte는 IP,ICMP, TCP, UDP등에 적절한 값이지만 Name Server나 NFS 패킷들의 경우에는 프로토콜의 정보들을 Truncation할 우려가 있다. 이 옵션을 수정할 때는 신중해야만 한다. 이유는 샘플 사이즈를 크게 잡으면 곧 패킷 하나하나를 처리하는데 시간이 더 걸릴 뿐만아니라 패킷 버퍼의 사이즈도 자연히 작아지게 되어 손실되는 패킷들이 발생할 수 있기 때문이다. 또, 작게 잡으면 그만큼의 정보를 잃게되는 것이다. 따라서 가급적 캡춰하고자 하는 프로토콜의 헤더 사이즈에 가깝게 잡아주어야 한다.

-T type : 조건식에 의해 선택된 패킷들을 명시된 형식으로 표시한다. type에는 다음과 같은 것들이 올 수 있다. rpc(Remote Procedure Call), rtp(Real-Time Applications protocol), rtcp(Real-Time Application control protocal), vat(Visual Audio Tool), wb(distributed White Board)

-S : TCP sequence번호를 상대적인 번호가 아닌 절대적인 번호로 출력한다.

-t : 출력되는 각각의 라인에 시간을 출력하지 않는다.

-tt : 출력되는 각각의 라인에 형식이 없는 시간들을 출력한다.

-v : 좀 더 많은 정보들을 출력한다.

-vv : '-v'보다 좀 더 많은 정보들을 출력한다.

-w : 캡춰한 패킷들을 분석해서 출력하는 대신에 그대로 파일에 저장한다.

-x : 각각의 패킷을 헥사코드로 출력한다.

조건식(expression!)

옵션의 제일 마지막인 조건식은 어떤 패킷들을 출력할지를 선택하는데 쓰인다. 조건식이 주어지지 않는 다면 모든 패킷들이 그 대상이 될 것이다. 일단 주어지면, 아무리 패킷들이 많아도 조 건식에 부합하는 패킷만을 출력한다.

조건식들은 하나 또는 몇 개의 primitive들로 구성되어 있다. primitive들은 보통 하나 혹은 몇개의 qualifier들 다음에 오는 하나의 값으로 이루어진다. Qualifier들은 모두 3 종류이며 다음과 같다.

type : 주어진 값의 종류가 무엇인지를 나타낸다. 가능한 type들은 'host', 'net', 'port'가 있다. type이 없는 값들은 type을 host라 가정한다.

dir : id로 부터의 어떤 특정한 전송 방향을 나타낸다. 가능한 방향은 'src', 'dst', 'src or dst', 'src and dst'이다. 만약 방향이 정해지지 않았다면, src or dst라 가정한다. "For `null' link layers (i.e. point to point protocols such as slip) the inb ound and out bound qualifiers can be used to specify a desired direction."

proto : 매칭을 특정 프로토콜에 한해서 수행한다. 가능한 프로토콜들은 ether, fddi, ip, arp, rarp, decnet, lat, sca, moprc, mopdl, tcp, udp이다. 만약 프로토콜이 명시되지 않았다면, 해당하는 값의 type에 관련된 모든 프로토콜들이 그 대상이 된다.

이 밖에도 위의 패턴을 따르지 않는 Primitive들이 존재한다(gateway, broadcst, less, greater, 산술식).

좀 더 정교한 조건식들을 사용하려면, 'and(&&)', 'or(||)', 'not(!)'들을 사용하여 여러 primitive들을 연결하면 된다. 같은 표현들은 생략될 수 있다.

사용 가능한 Primitive들

dst host HOST
packet의 IP destination 항목이 HOST일때 참이 된다.
src host HOST
packet의 IP source 항목이 HOST일때 참이 된다.
host HOST
IP source, IP destination 항목 중 어느 하나라도 HOST이면 참이다.
ether dst ehost
ethernet destination 주소가 ehost일 때 참이다.
ether src ehost
ethernet source 주소가 ehost일 때 참이다.
ether host ehost
ethernet source, destination 항목들 중 어느 하나라도 ehost이면 참이다.
gateway host
패킷이 host를 게이트웨이로 사용하면 참이다. 이 말의 의미는 ethernet sour ce나 destination 항목은 host이지만, IP source와 destination은 host가 아닐 때를 말한다.
dst net NET
패킷의 IP destination 주소가 NET의 network number를 가지고 있을 때 참이 다.
src net NET
패킷의 IP source 주소가 NET의 network number를 가지고 있을 때 참이다.
net NET
패킷의 IP source 주소 혹은 destination 주소가 NET의 network number를 가 지고 있을 때 참이다.
net netmask mask
IP 어드레스가 지정된 netmask를 통해서 net과 매칭되면 참이다.
net net/len
IP 어드레스가 netmask와 len 비트만큼 매치되면 참이다.
dst port PORT
패킷이 ip/tcp, ip/udp 프로토콜의 패킷이고 destination port의 값이 PORT일 때 참이다. port는 /etc/services에 명시된 이름일 수도 있고 그냥 숫자일 수도 있다. 만약 이름이 사용됐다면 port 번호와 프로토콜이 같이 체크될 것이다. 만약 숫자나 불 확실한 이름이 사용됐을 경우에는 port 번호만이 체크될 것이다.
src port PORT
패킷의 source port의 값으로 PORT를 가지면 참이다.
port PORT
패킷의 source, destination port 중에 하나라도 PORT이면 참이다.
less length
패킷이 length보다 짧거나 같으면 참이다.(len <= length)
greater length
패킷이 length보다 짧거나 같으면 참이다.(len >= length)
ip proto protocol
패킷이 지정된 종류의 프로토콜의 ip패킷이면 참이다. Protocol은 icmp, igrp, udp, nd, tcp 중의 하나 혹은 몇 개가 될 수 있다. 주의할 점은 tcp, udp, icmp들은 '\'로 escape되어야 한다.
ehter broadcast
패킷이 ethernet broadcast 패킷이라면 참이다. ehter는 생략 가능하다.
ip broadcast
패킷이 IP broadcast 패킷이라면 참이다.
ether multicast
패킷이 IP multicast 패킷이라면 참이다.
ether proto protocol
패킷이 ether type의 protocol이라면 참이다. protocol은 ip, arp, rarp 중에 하나 혹은 몇개가 될 수 있다. ip proto protocol에서와 마찬가지로 ip, arp, rarp는 escape 되어야 한다.
decnet src host
만약 DECNET의 source address가 host이면 참이다. 이 어드레스는 '10.123'이 나 DECNET의 host name일 수 있다. DECNET host name은 DECNET에서 돌아가도록 설정된 Ultrix 시스템에서만 사용 가능하다.
decnet dst host
DECNET destination address가 host이면 참이다.
decnet host HOST
DECNET source, destination address중의 하나라도 HOST이면 참이다.
ip, arp, rarp, decnet
ether proto [ip|arp|rarp|decnet]의 약어
lat, moprc, mopdl
ether proto [lat|moprc|mopdl]의 약어
tcp, udp, icmp
ip proto [tcp|udp|icmp]의 약어
expr relop expr

EXPR
proto [expr:size]의 형식을 띤다. proto, expr, size에 올 수 있는 것들은 다음과 같다.
proto : ether, fddi, ip, arp, rarp, tcp, udp, icmp
expr : indicate Byte offset of packet of proto
size : optional. indicate the size of bytes in field of interest
default is one, and can be two or four
RELOP
!=, =, <=, >=, etc.

이 조건식을 사용하기 위해서는 먼저 해당하는 Protocol(proto)의 헤더에 관련된 것들을 자세히 알아야만 한다. proto에는 대상이 될 프로토콜을 지정한다. expr에는 프로토콜 헤더의 처음부터의 Byte Offset을 지정하는 식이 들어가게 된다. Size는 Option이며 지정이 안 되어 있을 경우에는 자동으로 1byte를 지칭한다. 따라서 이 조건식을 사용하게 되면 헤더에 포함된 정보를 Bitmask를 사용하여 직 접 원하는 패킷인지를 가려낼 수 있기 때문에, 보다 정밀한 사용이 가능하게 된다.

Tcpdump의 사용 예제들

security라는 호스트로부터 날아오고, 날아가는 패킷들을 출력
# tcpdump host security
security와 mazinga, getarobo 사이에 날아다니고 있는 패킷들을 출력
# tcpdump host security and \( mazinga or getarobo \)
security에서 elgaim을 제외한 모든 호스트로 날아다니는 IP 패킷들을 출력
# tcpdump ip host security and not elgaim
gateway amurorei를 거치는 ftp에 관련된 패킷들을 출력
# tcpdump 'gateway amurorei and ( port ftp or ftp-data )'
local호스트가 아닌 호스트와 로컬호스트가 맺는 TCP 커넥션의 시작과 마지막 패 킷들을 출력한다(SYN, FIN 패킷).
# tcpdump 'tcp[13] & 3 != 0 and not src and dst net non-local'
gateway amurorei를 지나는 576Byte보다 큰 패킷들을 출력한다
# tcpdump 'gateway amurorei and ip[2:2] > 576'
Ethernet boradcast 혹은 multicast를 통해서 보내진 것이 아닌, IP broadcast 혹 은 multicast 패킷들을 출력한다.
# tcpdump 'ehter[0] & 1 = 0 and ip[16] >= 224'
Echo request/reply가 아닌 ICMP 패킷들을 모두 출력한다.
# tcpdump 'icmp[0] != 8 and icmp[0] != 0'

Tcpdump의 평가

TCPDUMP는 여러모로 좋은 툴이다. libpcap을 거의 100% 활용한 프로그램의 예이며, 실제로 많은 툴들이 TCPDUMP와 병행하여 돌아가거나, TCPDUMP를 기반으로 제작되었다. TCPDUMP의 막강한 packet filter는 현재 로컬 네트워크 상에서 날아다니고 있는 특정한 패킷들을 실시간으로 기록해 줄 수 있으며, 이를 이용하여 네트워크에서 벌어지는 일들을 네트워크 관리자가 원하는 대로 뽑아 볼 수 있게 해 준다. 또한, 시스템 관리자들에게는 로컬 유저의 외부로의 커넥션들을 감시하고, 또 특정 침입자가 침투 경로로 자주 이용하는 호스트, 혹은 원하지 않는 호스트로부터의 커넥션을 실시간으로 감시할 수 있게 해 준다. libpcap을 이용하여 비슷한 툴을 제작하고자 하는 사람들에게도 TCPDUMP는 가장 훌륭한 예제가 될 것이다.

Introduction
Prerequisites
      Requirements
      Components Used
      Conventions
Background Information
Understand the Catalyst 4500 CPU Packet-Handling Architecture
Identify the Reason for High CPU Utilization on Catalyst 4500
      Baseline the CPU Usage
      Understand the show processes cpu Command on the Catalyst 4500 Switches
      Understand the show platform health Command on the Catalyst 4500 Switches
Troubleshoot Common High CPU Utilization Problems
      High CPU Utilization Due to Process-Switched Packets
      Other Causes of High CPU Utilization
Troubleshooting Tools to Analyze the Traffic Destined to the CPU
      Tool 1: Monitor the CPU Traffic with SPAN—Cisco IOS Software Release 12.1(19)EW and Later
      Tool 2: In-Built CPU Sniffer—Cisco IOS Software Release 12.2(20)EW and Later
      Tool 3: Identify the Interface That Sends Traffic to the CPU—Cisco IOS Software Release 12.2(20)EW and Later
Summary
NetPro Discussion Forums - Featured Conversations
Related Information

Introduction

The Catalyst 4500 series switches, which includes the Catalyst 4948 switches, has a sophisticated packet-handling methodology for CPU-bound traffic. A commonly perceived problem is high CPU utilization on these switches. This document provides details about the CPU packet-handling architecture and shows you how to identify the causes of high CPU utilization on these switches. The document also lists some common network or configuration scenarios that cause high CPU utilization on the Catalyst 4500 series.

Note: If you run Catalyst OS (CatOS)-based Catalyst 4500/4000 series switches, refer to the document CPU Utilization on Catalyst 4500/4000, 2948G, 2980G, and 4912G Switches That Run CatOS Software.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• Catalyst 4500 series switches

• Catalyst 4948 series switches

Note: This document applies only to Cisco IOS® Software-based switches and not CatOS-based switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

Before you look at the CPU packet-handling architecture and troubleshoot high CPU utilization, you must understand the different ways in which hardware-based forwarding switches and Cisco IOS® Software-based routers use the CPU. The common misconception is that high CPU utilization indicates the depletion of resources on a device and the threat of a crash. A capacity issue is one of the symptoms of high CPU utilization on Cisco IOS routers. However, a capacity issue is almost never a symptom of high CPU utilization with hardware-based forwarding switches like the Catalyst 4500. The Catalyst 4500 is designed to forward packets in the hardware application-specific integrated circuit (ASIC) and reach traffic-forwarding speeds of up to 102 million packets per second (Mpps).

The Catalyst 4500 CPU performs these functions:

• Manages configured software protocols, for example:

  • Spanning Tree Protocol (STP)

  • Routing protocol

  • Cisco Discovery Protocol (CDP)

  • Port Aggregation Protocol (PAgP)

  • VLAN Trunk Protocol (VTP)

  • Dynamic Trunking Protocol (DTP)

• Programs configuration/dynamic entries to the hardware ASICs, for example:

  • Access control lists (ACLs)

  • CEF entries

• Internally manages various components, for example:

  • Power over Ethernet (PoE) line cards

  • Power supplies

  • Fan tray

• Manages access to the switch, for example:

  • Telnet

  • Console

  • Simple Network Management Protocol (SNMP)

• Forwards packets via the software path, for example:

  • Internetwork Packet Exchange (IPX)-routed packets, which are only supported in the software path

  • Maximum transmission unit (MTU) fragmentation

According to this list, high CPU utilization can result from the receipt or process of packets by the CPU. Some of the packets that are sent for process can be essential for the network operation. An example of these essential packets are bridge protocol data unit (BPDUs) for spanning-tree topology configurations. However, other packets can be software-forwarded data traffic. These scenarios require the switching ASICs to send packets to the CPU for processing:

• Packets that are copied to the CPU, but the original packets are switched in hardware

  An example is host MAC address learning.

• Packets that are sent to the CPU for processing

  Examples include:

  • Routing protocol updates

  • BPDUs

  • An intentional or unintentional flood of traffic

• Packets that are sent to the CPU for forwarding

  An example is packets that need IPX or AppleTalk routing.

Understand the Catalyst 4500 CPU Packet-Handling Architecture

The Catalyst 4500 has an in-built quality of service (QoS) mechanism in order to differentiate between types of traffic that are destined to the CPU. The mechanism makes the differentiation on the basis of the Layer 2 (L2)/Layer 3 (L3)/ Layer 4 (L4) packet information. The Supervisor packet Engine has 16 queues in order to handle various types of packets or events. Figure 1 shows these queues. Table 1 lists the queues and the packet types that queue in each. The 16 queues allow the Catalyst 4500 to queue the packets on the basis of the packet type or priority.

The Catalyst 4500 CPU assigns weights to the various queues that Table 1 lists. The CPU assigns the weights on the basis of importance, or type, and on the basis of traffic priority, or DSCP. The CPU services the queue on the basis of the relative weights of the queue. For example, if both a control packet, such as a BPDU, and an ICMP echo request are pending, the CPU services the control packet first. An excessive amount of low-priority or less-important traffic does not starve the CPU of the ability to process or manage the system. This mechanism guarantees that the network is stable even under high utilization of the CPU. This ability of the network to remain stable is critical information that you must understand.

There is another very important implementation detail of Catalyst 4500 CPU packet handling. If the CPU has already serviced high-priority packets or processes but has more spare CPU cycles for a particular time period, the CPU services the low-priority queue packets or performs background processes of lower priority. High CPU utilization as a result of low-priority packet processing or background processes is considered normal because the CPU constantly tries to use all the time available. In this way, the CPU strives for maximum performance of the switch and network without a compromise of the stability of the switch. The Catalyst 4500 considers the CPU underutilized unless the CPU is used at 100 percent for a single time slot.

Cisco IOS Software Release 12.2(25)EWA2 and later have enhanced the CPU packet- and process-handling mechanism and accounting. Therefore, use these releases on your Catalyst 4500 deployments.

Identify the Reason for High CPU Utilization on Catalyst 4500

Now that you understand the Catalyst 4500 CPU packet-handling architecture and design, you may still wish to identify why your Catalyst 4500 CPU utilization is high. The Catalyst 4500 has the commands and tools that are necessary to identify the root cause of the high CPU utilization. After you identify the reason, the administrators can perform either of these actions:

• Corrective Action—This can include configuration or network changes, or the creation of a Cisco Technical Support service request for further analysis.

• No action—The Catalyst 4500 performs according to the expectation. The CPU exhibits high CPU utilization because the Supervisor Engine maximizes the CPU cycles in order to perform all the necessary software packet forwarding and background jobs.

Be sure to identify the reason for high CPU utilization even though corrective action is not necessary in all cases. High CPU utilization can be just a symptom of an issue in the network. A resolution of the root cause of that problem can be necessary in order to lower the CPU utilization.

Figure 2 shows the troubleshooting methodology to use in order to identify the root cause of the Catalyst 4500 high CPU utilization.

1. Issue the show processes cpu command in order to identify the Cisco IOS processes that consume CPU cycles.

2. Issue the show platform health command in order to further identify the platform-specific processes.

3. If the highly active process is K2CpuMan Review , issue the show platform cpu packet statistics command in order to identity the type of traffic that hits the CPU.

   If the activity is not due to the K2CpuMan Review process, skip Step 4 and go on to Step 5.

4. Identify the packets that hit the CPU with use of the Troubleshooting Tools to Analyze the Traffic Destined to the CPU, if necessary.

   An example of the troubleshooting tools to use is the CPU Switched Port Analyzer (SPAN).

5. Review this document and the section Troubleshoot Common High CPU Utilization Problems for common causes.

   If you still cannot identify the root cause, contact Cisco Technical Support.

Baseline the CPU Usage

The important first step is to know the CPU utilization of your switch for your configuration and network setup. Use the show processes cpu command in order to identify the CPU utilization on the Catalyst 4500 switch. The continual update of baseline CPU utilization can be necessary as you add more configuration to the network setup or as your network traffic pattern changes. Figure 2 indicates this requirement.

This output is from a fully loaded Catalyst 4507R. The steady-state CPU is about 32 to 38 percent, which is necessary in order to perform the management functions for this switch:

Switch#show processes cpu 
CPU utilization for five seconds: 38%/1%; one minute: 32%; five minutes: 32%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           0        63          0  0.00%  0.00%  0.00%   0 Chunk Manager    
   2          60     50074          1  0.00%  0.00%  0.00%   0 Load Meter       
   3           0         1          0  0.00%  0.00%  0.00%   0 Deferred Events  

!--- Output suppressed.

  27         524    250268          2  0.00%  0.00%  0.00%   0 TTY Background   
  28         816    254843          3  0.00%  0.00%  0.00%   0 Per-Second Jobs  
  29      101100      5053      20007  0.00%  0.01%  0.00%   0 Per-minute Jobs  
  30    26057260  26720902        975 12.07% 11.41% 11.36%   0 Cat4k Mgmt HiPri 
  31    19482908  29413060        662 24.07% 19.32% 19.20%   0 Cat4k Mgmt LoPri 
  32        4468    162748         27  0.00%  0.00%  0.00%   0 Galios Reschedul 
  33           0         1          0  0.00%  0.00%  0.00%   0 IOS ACL Helper   
  34           0         2          0  0.00%  0.00%  0.00%   0 NAM Manager

Five-second CPU utilization is expressed as:

x%/y%

The x% represents total CPU utilization, and y% represents the CPU that is spent at the interrupt level. When you troubleshoot Catalyst 4500 switches, focus only on the total CPU utilization.

Understand the show processes cpu Command on the Catalyst 4500 Switches

This show processes cpu output shows that there are two processes that use the CPU— Cat4k Mgmt HiPri and Cat4k Mgmt LoPri . These two processes aggregate multiple platform-specific processes which perform the essential management functions on the Catalyst 4500. These processes process control plane as well as data packets that need to be software-switched or processed.

In order to see which of the platform-specific processes use the CPU under the context of Cat4k Mgmt HiPri and Cat4k Mgmt LoPri , issue the show platform health command.

Each of the platform-specific processes has a target/expected utilization of the CPU. When that process is within the target, the CPU executes the process in the high-priority context. The show processes cpu command output counts that utilization under Cat4k Mgmt HiPri .  If a process exceeds the target/expected utilization, that process runs under the low-priority context. The show processes cpu command output counts that additional utilization under Cat4k Mgmt LoPri . This Cat4k Mgmt LoPri is also used to run background and other low-priority processes, such as consistency check and reading interface counters. This mechanism allows the CPU to run high-priority processes when necessary, and the idle CPU cycles that remain are used for the low-priority processes. To exceed the target CPU utilization by a small amount, or a momentary spike in utilization, is not an indication of a problem that needs investigation.

Switch#show platform health 
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.02      2      1  100  500    0   0    0  1:09
GalChassisVp-review    3.00   0.29     10      3  100  500    0   0    0  11:15
S2w-JobEventSchedule  10.00   0.32     10      7  100  500    0   0    0  10:14
Stub-JobEventSchedul  10.00  12.09     10      6  100  500   14  13    9  396:35
StatValueMan Update    1.00   0.22      1      0  100  500    0   0    0  6:28
Pim-review             0.10   0.00      1      0  100  500    0   0    0  0:22
Ebm-host-review        1.00   0.00      8      0  100  500    0   0    0  0:05
Ebm-port-review        0.10   0.00      1      0  100  500    0   0    0  0:01
Protocol-aging-revie   0.20   0.00      2      0  100  500    0   0    0  0:00
Acl-Flattener  e       1.00   0.00     10      0  100  500    0   0    0  0:00
KxAclPathMan create/   1.00   0.00     10      5  100  500    0   0    0  0:39
KxAclPathMan update    2.00   0.00     10      0  100  500    0   0    0  0:00
KxAclPathMan reprogr   1.00   0.00      2      0  100  500    0   0    0  0:00
TagMan-RecreateMtegR   1.00   0.00     10      0  100  500    0   0    0  0:00
K2CpuMan Review       30.00  10.19     30     28  100  500   14  13    9  397:11
K2AccelPacketMan: Tx  10.00   2.20     20      0  100  500    2   2    1  82:06
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00
K2AclMan-taggedFlatA   1.00   0.00     10      0  100  500    0   0    0  0:00
K2AclCamMan stale en   1.00   0.00     10      0  100  500    0   0    0  0:00
K2AclCamMan hw stats   3.00   1.04     10      5  100  500    1   1    0  39:36
K2AclCamMan kx stats   1.00   0.00     10      5  100  500    0   0    0  13:40
K2AclCamMan Audit re   1.00   0.00     10      5  100  500    0   0    0  13:10
K2AclPolicerTableMan   1.00   0.00     10      1  100  500    0   0    0  0:38
K2L2 Address Table R   2.00   0.00     12      5  100  500    0   0    0  0:00
K2L2 New Static Addr   2.00   0.00     10      1  100  500    0   0    0  0:00
K2L2 New Multicast A   2.00   0.00     10      5  100  500    0   0    0  0:01
K2L2 Dynamic Address   2.00   0.00     10      0  100  500    0   0    0  0:00
K2L2 Vlan Table Revi   2.00   0.00     12      9  100  500    0   0    0  0:01
K2 L2 Destination Ca   2.00   0.00     10      0  100  500    0   0    0  0:00
K2PortMan Review       2.00   0.72     15     11  100  500    1   1    0  37:22
Gigaport65535 Review   0.40   0.07      4      2  100  500    0   0    0  3:38
Gigaport65535 Review   0.40   0.08      4      2  100  500    0   0    0  3:39
K2Fib cam usage revi   2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib IrmFib Review    2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib Vrf Default Ro   2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib AdjRepop Revie   2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib Vrf Unpunt Rev   2.00   0.01     15      0  100  500    0   0    0  0:23
K2Fib Consistency Ch   1.00   0.00      5      2  100  500    0   0    0  29:25
K2FibAdjMan Stats Re   2.00   0.30     10      4  100  500    0   0    0  6:21
K2FibAdjMan Host Mov   2.00   0.00     10      4  100  500    0   0    0  0:00
K2FibAdjMan Adj Chan   2.00   0.00     10      0  100  500    0   0    0  0:00
K2FibMulticast Signa   2.00   0.01     10      2  100  500    0   0    0  2:04
K2FibMulticast Entry   2.00   0.00     10      7  100  500    0   0    0  0:00
K2FibMulticast Irm M   2.00   0.00     10      7  100  500    0   0    0  0:00
K2FibFastDropMan Rev   2.00   0.00      7      0  100  500    0   0    0  0:00
K2FibPbr route map r   2.00   0.06     20      5  100  500    0   0    0  16:42
K2FibPbr flat acl pr   2.00   0.07     20      2  100  500    0   0    0  3:24
K2FibPbr consolidati   2.00   0.01     10      0  100  500    0   0    0  0:24
K2FibPerVlanPuntMan    2.00   0.00     15      4  100  500    0   0    0  0:00
K2FibFlowCache flow    2.00   0.01     10      0  100  500    0   0    0  0:23
K2FibFlowCache flow    2.00   0.00     10      0  100  500    0   0    0  0:00
K2FibFlowCache adj r   2.00   0.01     10      0  100  500    0   0    0  0:20
K2FibFlowCache flow    2.00   0.00     10      0  100  500    0   0    0  0:06
K2MetStatsMan Review   2.00   0.14      5      2  100  500    0   0    0  23:40
K2FibMulticast MET S   2.00   0.00     10      0  100  500    0   0    0  0:00
K2QosDblMan Rate DBL   2.00   0.12      7      0  100  500    0   0    0  4:52
IrmFibThrottler Thro   2.00   0.01      7      0  100  500    0   0    0  0:21
K2 VlanStatsMan Revi   2.00   1.46     15      7  100  500    2   2    1  64:44
K2 Packet Memory Dia   2.00   0.00     15      8  100  500    0   1    1  45:46
K2 L2 Aging Table Re   2.00   0.12     20      3  100  500    0   0    0  7:22
RkiosPortMan Port Re   2.00   0.73     12      7  100  500    1   1    1  52:36
Rkios Module State R   4.00   0.02     40      1  100  500    0   0    0  1:28
Rkios Online Diag Re   4.00   0.02     40      0  100  500    0   0    0  1:15
RkiosIpPbr IrmPort R   2.00   0.02     10      3  100  500    0   0    0  2:44
RkiosAclMan Review     3.00   0.06     30      0  100  500    0   0    0  2:35
MatMan Review          0.50   0.00      4      0  100  500    0   0    0  0:00
Slot 3 ILC Manager R   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 3 ILC S2wMan Re   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 4 ILC Manager R   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 4 ILC S2wMan Re   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 5 ILC Manager R   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 5 ILC S2wMan Re   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 6 ILC Manager R   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 6 ILC S2wMan Re   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 7 ILC Manager R   3.00   0.00     10      0  100  500    0   0    0  0:00
Slot 7 ILC S2wMan Re   3.00   0.00     10      0  100  500    0   0    0  0:00
EthHoleLinecardMan(1   1.66   0.04     10      0  100  500    0   0    0  1:18
EthHoleLinecardMan(2   1.66   0.02     10      0  100  500    0   0    0  1:18
EthHoleLinecardMan(6   1.66   0.17     10      6  100  500    0   0    0  6:38
                     -------------
%CPU Totals          212.80  35.63

Understand the show platform health Command on the Catalyst 4500 Switches

The show platform health command provides a lot of information that is relevant only for a development engineer. In order to troubleshoot high CPU utilization, look for a higher number in the %CPU actual column in the output. Also, be sure to glance at the right side of that row in order to verify the CPU usage of that process in the 1 minute and 1 hour average %CPU columns. Sometimes, processes momentarily peak but do not hold the CPU for a long time. Some of the momentarily high CPU utilization happens during hardware programming or optimization of the programming. For example, a spike of CPU utilization is normal during the hardware programming of a large ACL in the TCAM.

In the show platform health command output in the section Understand the show processes cpu Command on the Catalyst 4500 Switches, the Stub-JobEventSchedul and the K2CpuMan Review processes use a higher number of CPU cycles. Table 2 provides some basic information about the common platform-specific processes that appear in the output of the show platform health command.

Troubleshoot Common High CPU Utilization Problems

This section covers some of the common high CPU utilization problems on the Catalyst 4500 switches.

High CPU Utilization Due to Process-Switched Packets

One of the common reasons for high CPU utilization is that the Catalyst 4500 CPU is busy with the process of packets for software-forwarded packets or control packets. Examples of software-forwarded packets are IPX or control packets, such as BPDUs. A small number of these packets is typically sent to the CPU. However, a consistently large number of packets can indicate a configuration error or a network event. You must identify the cause of events that lead to the forward of packets to the CPU for processing. This identification enables you to debug the high CPU utilization problems.

Some of the common reasons for high CPU utilization due to process-switched packets are:

• A high number of spanning-tree port instances

• ICMP redirects; routing packets on the same interface

• IPX or AppleTalk routing

• Host learning

• Out of hardware resources (TCAM) for security ACL

• The log keyword in ACL

• Layer 2 forwarding loops

Other reasons for the switch of packets to the CPU are:

• MTU fragmentation—Be sure that all interfaces along the path of the packet have the same MTU.

• ACL with TCP flags other than established

• IP version 6 (IPv6) routing—This is supported only via the software-switching path.

• GRE—This is supported only via the software-switching path.

• Denial of traffic in the input or output router ACL (RACL)

  Note: This is rate-limited in Cisco IOS Software Release 12.1(13)EW1 and later.

  Issue the no ip unreachables command under the interface of the ACL.

• Excessive ARP and DHCP traffic hits the CPU for processing due to a large number of directly connected hosts

  If you suspect a DHCP attack, use DCHP snooping to rate-limit DHCP traffic from any specific host port.

• Excessive SNMP polling by a legitimate or misbehaving end station

A High Number of Spanning-Tree Port Instances

The Catalyst 4500 supports 3000 spanning-tree port instances or active ports in the Per VLAN Spanning Tree+ (PVST+) mode. The support is on all Supervisor Engines, except the Supervisor Engine II+ and II+TS, and the Catalyst 4948. The Supervisor Engine II+ and II+TS and the Catalyst 4948 support up to 1500 port instances. If you exceed these STP-instance recommendations, the switch exhibits high CPU utilization.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

This section reviews the commands that an administrator uses in order to narrow down the problem of high CPU utilization. If you issue the show processes cpu command, you can see that two main processes, Cat4k Mgmt LoPri and Spanning Tree , primarily use the CPU. With only this information, you know that the spanning tree process consumes a sizable portion of the CPU cycles.

Switch#show processes cpu 
CPU utilization for five seconds: 74%/1%; one minute: 73%; five minutes: 50%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           4       198         20  0.00%  0.00%  0.00%   0 Chunk Manager    
   2           4       290         13  0.00%  0.00%  0.00%   0 Load Meter       

!--- Output suppressed.

  25         488        33      14787  0.00%  0.02%  0.00%   0 Per-minute Jobs  
  26       90656    223674        405  6.79%  6.90%  7.22%   0 Cat4k Mgmt HiPri 
  27      158796     59219       2681 32.55% 33.80% 21.43%   0 Cat4k Mgmt LoPri 
  28          20      1693         11  0.00%  0.00%  0.00%   0 Galios Reschedul 
  29           0         1          0  0.00%  0.00%  0.00%   0 IOS ACL Helper   
  30           0         2          0  0.00%  0.00%  0.00%   0 NAM Manager      

!--- Output suppressed.

  41           0         1          0  0.00%  0.00%  0.00%   0 SFF8472          
  42           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R 
  43       78564     20723       3791 32.63% 30.03% 17.35%   0 Spanning Tree    
  44         112       999        112  0.00%  0.00%  0.00%   0 DTP Protocol     
  45           0       147          0  0.00%  0.00%  0.00%   0 Ethchnl

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

In order to understand which platform-specific process consumes the CPU, issue the show platform health command. From this output, you can see that the K2CpuMan Review process, a job to handle CPU-bound packets, uses up the CPU:

Switch#show platform health  
%CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

!--- Output suppressed.

TagMan-RecreateMtegR   1.00   0.00     10      0  100  500    0   0    0  0:00
K2CpuMan Review       30.00  37.62     30     53  100  500   41  33    1  2:12
K2AccelPacketMan: Tx  10.00   4.95     20      0  100  500    5   4    0  0:36
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00
K2AclMan-taggedFlatA   1.00   0.00     10      0  100  500    0   0    0  0:00

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic.

Issue the show platform cpu packet statistics command in order to check which CPU queue receives the CPU-bound packet. The output in this section shows that the control queue receives a lot of packets. Use the information in Table 1 and the conclusion that you drew in Step 1. You can determine that the packets that the CPU processes and the reason for the high CPU utilization is BPDU processing.

Switch#show platform cpu packet statistics

!--- Output suppressed.

Total packet queues 16
Packets Received by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp                            202760       196       173       128         28
Control                         388623      2121      1740       598         16 

Packets Dropped by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Control                          17918         0        19        24          3

Step 4: Identify the root cause.

Issue the show spanning-tree summary command. You can check if the receipt of BPDUs is because of a high number of spanning-tree port instances. The output clearly identifies the root cause:

Switch#show spanning-tree summary
Switch is in pvst mode
Root bridge for: none
Extended system ID           is enabled
Portfast Default             is disabled
PortFast BPDU Guard Default  is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default            is disabled
EtherChannel misconfig guard is enabled
UplinkFast                   is disabled
BackboneFast                 is disabled
Configured Pathcost method used is short 

!--- Output suppressed. 

Name                   Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ----------
2994 vlans                   0         0        0       5999       5999

There are a large number of VLANs with the PVST+ mode configuration. In order to resolve the issue, change the STP mode to Multiple Spanning Tree (MST). In some cases, the number of STP instances is high because a high number of VLANs are forwarded on all trunk ports. In this case, manually prune the VLANs that are not necessary from the trunk in order to drop the number of STP active ports to well below the recommended value.

Tip: Be sure that you do not configure IP phone ports as trunk ports. This is a common misconfiguration. Configure IP phone ports with a voice VLAN configuration. This configuration creates a pseudo trunk, but does not require you to manually prune the unnecessary VLANs. For more information on how to configure voice ports, refer to the Configuring Voice Interfaces software configuration guide. Non-Cisco IP phones do not support this voice VLAN or auxiliary VLAN configuration. You must manually prune the ports with non-Cisco IP phones.

ICMP Redirects; Routing Packets on the Same Interface

Routing packets on the same interface, or traffic ingress and egress on the same L3 interface, can result in an ICMP redirect by the switch. If the switch knows that the next hop device to the ultimate destination is in the same subnet as the sending device, the switch generates ICMP redirect to the source. The redirect messages indicate to the source to send the packet directly to the next hop device. The messages indicate that the next hop device has a better route to the destination, a route of one less hop than this switch.

In the diagram in this section, PC A communicates with the web server. The default gateway of PC A points to the VLAN 100 interface IP address. However, the next hop router that enables the Catalyst 4500 to reach the destination is in the same subnet as PC A. The best path in this case is to send directly to "Router". Catalyst 4500 sends an ICMP redirect message to PC A. The message instructs PC A to send the packets destined to the web server via Router, instead of via Catalyst 4500. However, in most cases, the end devices do not respond to the ICMP redirect. The lack of response causes the Catalyst 4500 to spend a lot of CPU cycles on the generation of these ICMP redirects for all the packets that the Catalyst forwards via the same interface as the ingress packets.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

Issue the show processes cpu command. You can see that two main processes, Cat4k Mgmt LoPri and IP Input , primarily use the CPU. With only this information, you know that the process of IP packets expends a sizable portion of the CPU.

Switch#show processes cpu 
CPU utilization for five seconds: 38%/1%; one minute: 32%; five minutes: 32%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           0        63          0  0.00%  0.00%  0.00%   0 Chunk Manager    
   2          60     50074          1  0.00%  0.00%  0.00%   0 Load Meter      
   3           0         1          0  0.00%  0.00%  0.00%   0 Deferred Events  

!--- Output suppressed.

  27         524    250268          2  0.00%  0.00%  0.00%   0 TTY Background   
  28         816    254843          3  0.00%  0.00%  0.00%   0 Per-Second Jobs  
  29      101100      5053      20007  0.00%  0.01%  0.00%   0 Per-minute Jobs  
  30    26057260  26720902        975  5.81%  6.78%  5.76%   0 Cat4k Mgmt HiPri 
  31    19482908  29413060        662 19.64% 18.20% 20.48%   0 Cat4k Mgmt LoPri 

!--- Output suppressed.

  35          60       902          0  0.00%  0.00%  0.00%   0 DHCP Snooping 
  36    504625304 645491491       781 72.40% 72.63% 73.82%   0 IP Input

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

The output of the show platform health command confirms the use of the CPU in order to process CPU-bound packets.

Switch#show platform health  
%CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

--- Output suppressed.

TagMan-RecreateMtegR   1.00   0.00     10      0  100  500    0   0    0  0:00
K2CpuMan Review       330.00 19.18    150     79   25  500   20  19   18 5794:08
K2AccelPacketMan: Tx  10.00   4.95     20      0  100  500    5   4    0  0:36
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00
K2AclMan-taggedFlatA   1.00   0.00     10      0  100  500    0   0    0  0:00

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic.

Issue the show platform cpu packet statistics command in order to check which CPU queue receives the CPU-bound packet. You can see that the L3 Fwd Low queue receives quite a lot of traffic.

Switch#show platform cpu packet statistics

!--- Output suppressed.

Packets Received by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp                          48613268        38        39        38         39
Control                      142166648        74        74        73         73
Host Learning                  1845568         2         2         2          2
L3 Fwd High                         17         0         0         0          0
L3 Fwd Medium                     2626         0         0         0          0
L3 Fwd Low                  4717094264      3841      3879      3873       3547
L2 Fwd Medium                        1         0         0         0          0
L3 Rx High                      257147         0         0         0          0
L3 Rx Low                      5325772        10        19        13          7
RPF Failure                        155         0         0         0          0
ACL fwd(snooping)             65604591        53        54        54         53
ACL log, unreach              11013420         9         8         8          8

Step 4: Identify the root cause.

In this case, use the CPU SPAN in order to determine the traffic that hits the CPU. For information about the CPU SPAN, see the Tool 1: Monitor the CPU Traffic with SPAN—Cisco IOS Software Release 12.1(19)EW and Later section of this document. Complete an analysis of the traffic and a configuration with use of the show running-configuration command. In this case, a packet is routed through the same interface, which leads to the issue of an ICMP redirect for each packet. This root cause is one of the common reasons for high CPU utilization on the Catalyst 4500.

You may expect the sourcing device to act on the ICMP redirect that the Catalyst 4500 sends and change the next hop for the destination. However, not all devices respond to an ICMP redirect. If the device does not respond, the Catalyst 4500 must send redirects for every packet that the switch receives from the sending device. These redirects can consume a great deal of CPU resources. The solution is to disable ICMP redirect. Issue the no ip redirects command under the interfaces.

This scenario can occur when you also have configured secondary IP addresses. When you enable the secondary IP addresses, IP redirect is automatically disabled. Be sure you do not manually enable the IP redirects.

As this ICMP Redirects; Routing Packets on the Same Interface section has indicated, most end devices do not respond to ICMP redirects. Therefore, as a general practice, disable this feature.

IPX or AppleTalk Routing

The Catalyst 4500 supports IPX and AppleTalk routing via software-forwarding path only. With the configuration of such protocols, a higher CPU utilization is normal.

Note: The switching of IPX and AppleTalk traffic in the same VLAN does not require process switching. Only packets that need to be routed require software path forwarding.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

Issue the show processes cpu command in order to check which Cisco IOS process consumes the CPU. In this command output, notice that the top process is the Cat4k Mgmt LoPri :

witch#show processes cpu 
CPU utilization for five seconds: 87%/10%; one minute: 86%; five minutes: 87%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           4        53         75  0.00%  0.00%  0.00%   0 Chunk Manager    

!--- Output suppressed.

  25        8008   1329154          6  0.00%  0.00%  0.00%   0 Per-Second Jobs  
  26      413128     38493      10732  0.00%  0.02%  0.00%   0 Per-minute Jobs  
  27   148288424 354390017        418  2.60%  2.42%  2.77%   0 Cat4k Mgmt HiPri 
  28   285796820 720618753        396 50.15% 59.72% 61.31%   0 Cat4k Mgmt LoPri

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

The output of the show platform health command confirms the use of the CPU in order to process CPU-bound packets.

Switch#show platform health
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

!--- Output suppressed.

TagMan-RecreateMtegR   1.00   0.00     10      4  100  500    0   0    0  0:00
K2CpuMan Review       30.00  27.39     30     53  100  500   42  47   42  4841:
K2AccelPacketMan: Tx  10.00   8.03     20      0  100  500   21  29   26  270:4

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic.

In order to determine the type of traffic that hits the CPU, issue the show platform cpu packet statistics command.

Switch#show platform cpu packet statistics

!--- Output suppressed.

Packets Received by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp                          48613268        38        39        38         39
Control                      142166648        74        74        73         73
Host Learning                  1845568         2         2         2          2
L3 Fwd High                         17         0         0         0          0
L3 Fwd Medium                     2626         0         0         0          0
L3 Fwd Low                     1582414         1         1         1          1
L2 Fwd Medium                        1         0         0         0          0
L2 Fwd Low                   576905398      1837      1697      1938       1515
L3 Rx High                      257147         0         0         0          0
L3 Rx Low                      5325772        10        19        13          7
RPF Failure                        155         0         0         0          0
ACL fwd(snooping)             65604591        53        54        54         53
ACL log, unreach              11013420         9         8         8          8

Step 4: Identify the root cause.

Since the administrator has configured IPX or AppleTalk routing, identification of the root cause should be straightforward. But in order to confirm, SPAN the CPU traffic and be sure that the traffic that you see is the expected traffic. For information about the CPU SPAN, see the Tool 1: Monitor the CPU Traffic with SPAN—Cisco IOS Software Release 12.1(19)EW and Later section of this document.

In this case, the administrator must update the baseline CPU to the current value. The Catalyst 4500 CPU behaves as expected when the CPU processes software-switched packets.

Host Learning

The Catalyst 4500 learns the MAC addresses of various hosts, if the MAC address is not already in the MAC address table. The switching engine forwards a copy of the packet with the new MAC address to the CPU.

All the VLAN interfaces (layer 3) use the chassis base hardware address as their MAC address. As a result, there is not an entry in the MAC address table, and the packets destined to these VLAN interfaces are not sent to the CPU for processing.

If there is an excessive number of new MAC addresses for the switch to learn, high CPU utilization can result.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

Issue the show processes cpu command in order to check which Cisco IOS process consumes the CPU. In this command output, notice that the top process is the Cat4k Mgmt LoPri :

Switch#show processes cpu 
CPU utilization for five seconds: 89%/1%; one minute: 74%; five minutes: 71%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           4        53         75  0.00%  0.00%  0.00%   0 Chunk Manager    

!--- Output suppressed.

  25        8008   1329154          6  0.00%  0.00%  0.00%   0 Per-Second Jobs  
  26      413128     38493      10732  0.00%  0.02%  0.00%   0 Per-minute Jobs  
  27   148288424 354390017        418  26.47% 10.28% 10.11%  0 Cat4k Mgmt HiPri 
  28   285796820 720618753        396 52.71% 56.79% 55.70%   0 Cat4k Mgmt LoPri

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

The output of the show platform health command confirms the use of the CPU in order to process CPU-bound packets.

Switch#show platform health
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

!--- Output suppressed.

TagMan-RecreateMtegR   1.00   0.00     10      4  100  500    0   0    0  0:00
K2CpuMan Review       30.00  46.88     30     47  100  500   30  29   21  265:01
K2AccelPacketMan: Tx  10.00   8.03     20      0  100  500   21  29   26  270:4

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic.

In order to determine the type of traffic that hits the CPU, issue the show platform cpu packet statistics command.

Switch#show platform cpu packet statistics

!--- Output suppressed.

Packets Received by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp                          48613268        38        39        38         39
Control                      142166648        74        74        73         73
Host Learning                  1845568      1328      1808      1393       1309
L3 Fwd High                         17         0         0         0          0
L3 Fwd Medium                     2626         0         0         0          0
L3 Fwd Low                     1582414         1         1         1          1
L2 Fwd Medium                        1         0         0         0          0
L2 Fwd Low                   576905398        37         7         8          5
L3 Rx High                      257147         0         0         0          0
L3 Rx Low                      5325772        10        19        13          7
RPF Failure                        155         0         0         0          0
ACL fwd(snooping)             65604591        53        54        54         53
ACL log, unreach              11013420         9         8         8          8

Step 4: Identify the root cause.

The output of the show platform health command shows you that the CPU sees a lot of new MAC addresses. This situation is often the result of network topology instability. For example, if the spanning-tree topology changes, the switch generates Topology Change Notifications (TCNs). The issue of TCNs reduces the aging time to 15 seconds in PVST+ mode. MAC address entries are flushed if the addresses are not learned back within the time period. In the case of Rapid STP (RSTP) (IEEE 802.1w) or MST (IEEE 802.1s), the entries immediately age out if the TCN comes from another switch. This age out causes MAC addresses to be learned anew. This is not a major issue if the topology changes are rare. But there can be an excessive number of topology changes because of a flapping link, faulty switch, or host ports that are not enabled for PortFast. A large number of MAC table flushes and subsequent relearning can result. The next step in root cause identification is to troubleshoot the network. The switch works as expected and sends the packets to the CPU for host address learning. Identify and fix the faulty device that results in excessive TCNs.

Your network can have a lot of devices that send traffic in bursts, which causes MAC addresses to be aged out and subsequently relearned on the switch. In this case, increase the MAC address table aging time in order to provide some relief. With a longer aging time, the switches retain the device MAC addresses in the table for a longer period of time before the age out.

Caution: Make this age-out change only after careful consideration. The change can lead to a traffic black hole if you have devices in your network which are mobile.

Out of Hardware Resources (TCAM) for Security ACL

The Catalyst 4500 programs the configured ACLs with use of the Cisco TCAM. TCAM allows for the application of the ACLs in the hardware-forwarding path. There is no impact on performance of the switch, with or without ACLs in the forwarding path. Performance is constant despite the size of the ACL because performance of the ACL lookups is at line rate. However, TCAM is a finite resource. Therefore, if you configure an excessive number of ACL entries, you exceed the TCAM capacity. Table 3 shows the number of TCAM resources available on each of the Catalyst 4500 Supervisor Engines and switches.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

Issue the show processes cpu command. You can see that the CPU utilization is high because the Cat4k Mgmt LoPri process takes up most of the CPU cycles.

Switch#show processes cpu
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1           0        11          0  0.00%  0.00%  0.00%   0 Chunk Manager
   2        9716    632814         15  0.00%  0.00%  0.00%   0 Load Meter
   3         780       302       2582  0.00%  0.00%  0.00%   0 SpanTree Helper

!--- Output suppressed.

  23       18208   3154201          5  0.00%  0.00%  0.00%   0 TTY Background
  24       37208   3942818          9  0.00%  0.00%  0.00%   0 Per-Second Jobs
  25     1046448    110711       9452  0.00%  0.03%  0.00%   0 Per-minute Jobs
  26   175803612 339500656        517  4.12%  4.31%  4.48%   0 Cat4k Mgmt HiPri
  27   835809548 339138782       2464 86.81% 89.20% 89.76%   0 Cat4k Mgmt LoPri
  28       28668   2058810         13  0.00%  0.00%  0.00%   0 Galios Reschedul

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

Issue the show platform health command. You can see that the K2CpuMan Review, a job to handle CPU-bound packets, uses the CPU.

Switch#show platform health  
%CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.01      2      0  100  500    0   0    0  13:45
GalChassisVp-review    3.00   0.20     10     16  100  500    0   0    0  88:44
S2w-JobEventSchedule  10.00   0.57     10      7  100  500    1   0    0  404:22
Stub-JobEventSchedul  10.00   0.00     10      0  100  500    0   0    0  0:00
StatValueMan Update    1.00   0.09      1      0  100  500    0   0    0  91:33
Pim-review             0.10   0.00      1      0  100  500    0   0    0  4:46
Ebm-host-review        1.00   0.00      8      4  100  500    0   0    0  14:01
Ebm-port-review        0.10   0.00      1      0  100  500    0   0    0  0:20
Protocol-aging-revie   0.20   0.00      2      0  100  500    0   0    0  0:01
Acl-Flattener          1.00   0.00     10      5  100  500    0   0    0  0:04
KxAclPathMan create/   1.00   0.00     10      5  100  500    0   0    0  0:21
KxAclPathMan update    2.00   0.00     10      6  100  500    0   0    0  0:05
KxAclPathMan reprogr   1.00   0.00      2      1  100  500    0   0    0  0:00
TagMan-InformMtegRev   1.00   0.00      5      0  100  500    0   0    0  0:00
TagMan-RecreateMtegR   1.00   0.00     10     14  100  500    0   0    0  0:18
K2CpuMan Review       30.00  91.31     30     92  100  500  128 119   84  13039:02
K2AccelPacketMan: Tx  10.00   2.30     20      0  100  500    2   2    2  1345:30
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic.

You need to further understand which CPU queue and, therefore, what type of traffic hits the CPU queue. Issue the show platform cpu packet statistics command. You can see that the ACL sw processing queue receives a high number of packets. Therefore, TCAM overflow is the cause of this high CPU utilization issue.

Switch#show platform cpu packet statistics

!--- Output suppressed.

Packets Received by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Control                       57902635        22        16        12          3
Host Learning                   464678         0         0         0          0
L3 Fwd Low                      623229         0         0         0          0
L2 Fwd Low                    11267182         7         4         6          1
L3 Rx High                         508         0         0         0          0
L3 Rx Low                      1275695        10         1         0          0
ACL fwd(snooping)              2645752         0         0         0          0
ACL log, unreach              51443268         9         4         5          5
ACL sw processing            842889240      1453      1532      1267       1179 

Packets Dropped by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
L2 Fwd Low                        3270         0         0         0          0
ACL sw processing                12636         0         0         0          0

Step 4: Resolve the issue.

In Step 3, you determined the root cause in this scenario. Remove the ACL which caused the overflow or minimize the ACL to avoid overflow. Also, review the Configuring Network Security with ACLs configuration guideline in order to optimize the ACL configuration and programming in the hardware.

The log Keyword in ACL

The Catalyst 4500 supports logging of packets detail that hit any specific ACL entry, but excessive logging can cause high CPU utilization. Avoid the use of log keywords, except during the traffic discovery stage. During the traffic discovery stage, you identify the traffic that flows through your network for which you have not explicitly configured ACEs. Do not use the log keyword in order to gather statistics. In Cisco IOS Software Release 12.1(13)EW and later, the log messages are rate-limited. If you use log messages in order to count the number of packets that match the ACL, the count is not accurate. Instead, use the show access-list command for accurate statistics. Identification of this root cause is easier because a review of the configuration or log messages can indicate the use of the ACL logging feature.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

Issue the show processes cpu in order to check which Cisco IOS process consumes the CPU. In this command output, you find that the top process is the Cat4k Mgmt LoPri :

Switch#show processes cpu  
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1           0        11          0  0.00%  0.00%  0.00%   0 Chunk Manager
   2        9716    632814         15  0.00%  0.00%  0.00%   0 Load Meter

!--- Output suppressed.

  26   175803612 339500656        517  4.12%  4.31%  4.48%   0 Cat4k Mgmt HiPri
  27   835809548 339138782       2464 86.81% 89.20% 89.76%   0 Cat4k Mgmt LoPri
  28       28668   2058810         13  0.00%  0.00%  0.00%   0 Galios Reschedul

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

Check the platform-specific process that uses the CPU. Issue the show platform health command. In the output, notice that the K2CpuMan Review process uses most of the CPU cycles. This activity indicates that the CPU is busy as it processes packets destined to it.

Switch#show platform health 
                      %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.01      2      0  100  500    0   0    0  13:45
GalChassisVp-review    3.00   0.20     10     16  100  500    0   0    0  88:44
S2w-JobEventSchedule  10.00   0.57     10      7  100  500    1   0    0  404:22
Stub-JobEventSchedul  10.00   0.00     10      0  100  500    0   0    0  0:00
StatValueMan Update    1.00   0.09      1      0  100  500    0   0    0  91:33
Pim-review             0.10   0.00      1      0  100  500    0   0    0  4:46
Ebm-host-review        1.00   0.00      8      4  100  500    0   0    0  14:01
Ebm-port-review        0.10   0.00      1      0  100  500    0   0    0  0:20
Protocol-aging-revie   0.20   0.00      2      0  100  500    0   0    0  0:01
Acl-Flattener          1.00   0.00     10      5  100  500    0   0    0  0:04
KxAclPathMan create/   1.00   0.00     10      5  100  500    0   0    0  0:21
KxAclPathMan update    2.00   0.00     10      6  100  500    0   0    0  0:05
KxAclPathMan reprogr   1.00   0.00      2      1  100  500    0   0    0  0:00
TagMan-InformMtegRev   1.00   0.00      5      0  100  500    0   0    0  0:00
TagMan-RecreateMtegR   1.00   0.00     10     14  100  500    0   0    0  0:18
K2CpuMan Review       30.00  91.31     30     92  100  500  128 119   84  13039:02
K2AccelPacketMan: Tx  10.00   2.30     20      0  100  500    2   2    2  1345:30
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic.

In order to determine the type of traffic that hits the CPU, issue the show platform cpu packet statistics command. In this command output, you can see that the receipt of packets is due to the ACL log keyword:

Switch#show platform cpu packet statistics

!--- Output suppressed.

Total packet queues 16
Packets Received by Packet Queue

Queue             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
Control                     1198701435        35        35        34         35
Host Learning                   874391         0         0         0          0
L3 Fwd High                        428         0         0         0          0
L3 Fwd Medium                    12745         0         0         0          0
L3 Fwd Low                     2420401         0         0         0          0
L2 Fwd High                      26855         0         0         0          0
L2 Fwd Medium                   116587         0         0         0          0
L2 Fwd Low                   317829151        53        41        31         31
L3 Rx High                        2371         0         0         0          0
L3 Rx Low                     32333361         7         1         2          0
RPF Failure                       4127         0         0         0          0
ACL fwd (snooping)            107743299        4         4         4          4
ACL log, unreach            1209056404      1987      2125      2139       2089 

Packets Dropped by Packet Queue 

Queue             Total                5 sec avg 1 min avg 5 min avg 1 hour avg
----------------- -------------------- --------- --------- --------- ----------
ACL log, unreach             193094788       509       362       437        394

Step 4: Resolve the issue.

In Step 3, you determined the root cause in this scenario. In order to prevent this problem, remove the log keyword from the ACLs. In Cisco IOS Software Release 12.1(13)EW1 and later, the packets are rate-limited so that CPU utilization does not get too high. Use the access list counters as a way to keep track of ACL hits. You can see the access list counters in the show access-list acl_id command output.

Layer 2 forwarding loops

Layer 2 forwarding loops can be caused by poor implementation of Spanning Tree Protocol (STP) and various issues that can affect STP.

Step 1: Check for the Cisco IOS process with the show processes cpu command

This section reviews the commands that an administrator uses in order to narrow down the problem of high CPU utilization. If you issue the show processes cpu command, you can see that two main processes, Cat4k Mgmt LoPri and Spanning Tree , primarily use the CPU. With only this information, you know that the spanning tree process consumes a sizable portion of the CPU cycles.

Switch#show processes cpu 
CPU utilization for five seconds: 74%/1%; one minute: 73%; five minutes: 50%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           4       198         20  0.00%  0.00%  0.00%   0 Chunk Manager    
   2           4       290         13  0.00%  0.00%  0.00%   0 Load Meter       

!--- Output suppressed.

  25         488        33      14787  0.00%  0.02%  0.00%   0 Per-minute Jobs  
  26       90656    223674        405  6.79%  6.90%  7.22%   0 Cat4k Mgmt HiPri 
  27      158796     59219       2681 32.55% 33.80% 21.43%   0 Cat4k Mgmt LoPri 
  28          20      1693         11  0.00%  0.00%  0.00%   0 Galios Reschedul 
  29           0         1          0  0.00%  0.00%  0.00%   0 IOS ACL Helper   
  30           0         2          0  0.00%  0.00%  0.00%   0 NAM Manager      

!--- Output suppressed.

  41           0         1          0  0.00%  0.00%  0.00%   0 SFF8472          
  42           0         2          0  0.00%  0.00%  0.00%   0 AAA Dictionary R 
  43       78564     20723       3791 32.63% 30.03% 17.35%   0 Spanning Tree    
  44         112       999        112  0.00%  0.00%  0.00%   0 DTP Protocol     
  45           0       147          0  0.00%  0.00%  0.00%   0 Ethchnl

Step 2: Check for the Catalyst 4500-specific process with the show platform health command

In order to understand which platform-specific process consumes the CPU, issue the show platform health command. From this output, you can see that the K2CpuMan Review process, a job to handle CPU-bound packets, uses up the CPU:

Switch#show platform health  
%CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU

!--- Output suppressed.

TagMan-RecreateMtegR   1.00   0.00     10      0  100  500    0   0    0  0:00
K2CpuMan Review       30.00  37.62     30     53  100  500   41  33    1  2:12
K2AccelPacketMan: Tx  10.00   4.95     20      0  100  500    5   4    0  0:36
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00
K2AclMan-taggedFlatA   1.00   0.00     10      0  100  500    0   0    0  0:00

Step 3: Check the CPU queue that receives traffic in order to identify the type of CPU-bound traffic

Issue the show platform cpu packet statistics command in order to check which CPU queue receives the CPU-bound packet. The output in this section shows that the control queue receives a lot of packets. Use the information in Table 1 and the conclusion that you drew in Step 1. You can determine that the packets that the CPU processes and the reason for the high CPU utilization is BPDU processing.

Switch#show platform cpu packet statistics

!--- Output suppressed.

Total packet queues 16
Packets Received by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Esmp                            202760       196       173       128         28
Control                         388623      2121      1740       598         16 

Packets Dropped by Packet Queue 

Queue                  Total           5 sec avg 1 min avg 5 min avg 1 hour avg
---------------------- --------------- --------- --------- --------- ----------
Control                          17918         0        19        24          3

Step 4: Identify the root cause and fix the issue

Generally, you can complete these steps in order to troubleshoot (depending on the situation, some steps are not be necessary):

1. Identify the loop.

2. Discover the scope of the loop.

3. Break the loop.

4. Fix the cause for the loop.

5. Restore redunancy.

Each of the steps are explained in detail at Troubleshooting Forwarding Loops - Troubleshooting STP on Catalyst Switches Running Cisco IOS System Software.

Step 5: Implement advanced STP features

• BDPU Guard—Secures STP from unauthorized network devices connected to portfast enabled ports. Refer to Spanning Tree PortFast BPDU Guard Enhancement for more information.

• Loop Guard—Increases the stability of layer 2 networks. Refer to Spanning-Tree Protocol Enhancements using Loop Guard and BPDU Skew Detection Features for more information.

• Root Guard—Enforces root bridge placement in the network. Refer to Spanning Tree Protocol Root Guard Enhancement for more information.

• UDLD—Detects unidirectional links and prevents forwarding loops. Refer to Understanding and Configuring the Unidirectional Link Detection Protocol Feature for more information.

Other Causes of High CPU Utilization

These are some other known causes of high CPU utilization:

• Excessive link flaps

• Spikes in CPU utilization due to FIB consistency check

• High CPU utilization in the K2FibAdjMan Host Move process

• High CPU utilization in the RkiosPortMan Port Review process

• High CPU utilization when connected to an IP phone with the use of trunk ports

• Spike during large ACL programming

  The spike in CPU utilization occurs during application or removal of a large ACL from an interface.

Excessive Link Flaps

The Catalyst 4500 exhibits high CPU utilization when one or more of the attached links starts to flap excessively. This situation occurs in Cisco IOS Software releases earlier than Cisco IOS Software Release 12.2(20)EWA.

Step 1: Check for the Cisco IOS process with the show processes cpu command.

Issue the show processes cpu command in order to check which Cisco IOS process consumes the CPU. In this command output, notice that the top process is the Cat4k Mgmt LoPri :

Switch#show processes cpu  
CPU utilization for five seconds: 96%/0%; one minute: 76%; five minutes: 68%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           0         4          0  0.00%  0.00%  0.00%   0 Chunk Manager    
   2        9840    463370         21  0.00%  0.00%  0.00%   0 Load Meter       
   3           0         2          0  0.00%  0.00%  0.00%   0 SNMP Timers   

!--- Output suppressed.

  27   232385144 530644966        437 13.98% 12.65% 12.16%   0 Cat4k Mgmt HiPri
  28   564756724 156627753       3605 64.74% 60.71% 54.75%   0 Cat4k Mgmt LoPri
  29        9716   1806301          5  0.00%  0.00%  0.00%   0 Galios Reschedul 

Step 2: Check for the Catalyst 4500-specific process with the show platform health command.

The output of the show platform health command indicates that the KxAclPathMan create process uses up the CPU. This process is for internal path creation.

Switch#show platform health 
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.03      2      0  100  500    0   0    0  9:49
GalChassisVp-review    3.00   1.11     10     62  100  500    0   0    0  37:39
S2w-JobEventSchedule  10.00   2.85     10      8  100  500    2   2    2  90:00
Stub-JobEventSchedul  10.00   5.27     10      9  100  500    4   4    4  186:2
Pim-review             0.10   0.00      1      0  100  500    0   0    0  2:51
Ebm-host-review        1.00   0.00      8      4  100  500    0   0    0  8:06
Ebm-port-review        0.10   0.00      1      0  100  500    0   0    0  0:14
Protocol-aging-revie   0.20   0.00      2      0  100  500    0   0    0  0:00
Acl-Flattener          1.00   0.00     10      5  100  500    0   0    0  0:00
KxAclPathMan create/   1.00  69.11     10      5  100  500   42  53   22  715:0
KxAclPathMan update    2.00   0.76     10      6  100  500    0   0    0  86:00
KxAclPathMan reprogr   1.00   0.00      2      1  100  500    0   0    0  0:00
TagMan-InformMtegRev   1.00   0.00      5      0  100  500    0   0    0  0:00
TagMan-RecreateMtegR   1.00   0.00     10    227  100  500    0   0    0  0:00
K2CpuMan Review       30.00   8.05     30     57  100  500    6   5    5  215:0
K2AccelPacketMan: Tx  10.00   6.86     20      0  100  500    5   5    4  78:42

Step 3: Identify the root cause.

Enable logging for link up/down messages. This logging is not enabled by default. The enablement helps you to narrow down the offending links very quickly. Issue the logging event link-status command under all the interfaces. You can use the interface range command in order to conveniently enable on a range of interfaces, as this example shows:

Switch#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Switch(config)#interface range gigabitethernet 5/1 - 48
Switch(config-if-range)#logging event link-status
Switch(config--if-range)#end

Switch#show logging

!--- Output suppressed.

3w5d: %LINK-3-UPDOWN: Interface GigabitEthernet5/24, changed state to down
3w5d: %LINK-3-UPDOWN: Interface GigabitEthernet5/24, changed state to up
3w5d: %LINK-3-UPDOWN: Interface GigabitEthernet5/24, changed state to down
3w5d: %LINK-3-UPDOWN: Interface GigabitEthernet5/24, changed state to up
3w5d: %LINK-3-UPDOWN: Interface GigabitEthernet5/24, changed state to down
3w5d: %LINK-3-UPDOWN: Interface GigabitEthernet5/24, changed state to up

After you have identified the faulty or flapping interface, shut down the interface in order to resolve the high CPU utilization issue. Cisco IOS Software Release 12.2(20)EWA and later have improved the Catalyst 4500 behavior for this flapping-links condition. Therefore, the impact on the CPU is not as great as before the improvement. Remember that this process is a background process. High CPU utilization because of this issue does not cause adverse effects on the Catalyst 4500 switches.

Spikes in CPU Utilization Due to FIB Consistency Check

The Catalyst 4500 can show momentary spikes in the CPU utilization during a FIB table consistency check. The FIB table is the L3 forwarding table that the CEF process creates. The consistency check maintains consistency between the Cisco IOS Software FIB table and the hardware entries. This consistency ensures that packets are not misrouted. The check occurs every 2 seconds and runs as a low-priority background process. This process is normal behavior and does not interfere with other high-priority processes or packets.

The output of the show platform health command shows that K2Fib Consistency Ch consumes most of the CPU.

Note: The average CPU utilization for this process is insignificant over a minute or an hour, which confirms that the check is a short periodic review. This background process only uses the idle CPU cycles.

Switch#show platform health 
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.02      2      1  100  500    0   0    0  1:09
GalChassisVp-review    3.00   0.29     10      3  100  500    0   0    0  11:15

!--- Output suppressed.

K2Fib cam usage revi   2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib IrmFib Review    2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib Vrf Default Ro   2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib AdjRepop Revie   2.00   0.00     15      0  100  500    0   0    0  0:00
K2Fib Vrf Unpunt Rev   2.00   0.01     15      0  100  500    0   0    0  0:23
K2Fib Consistency Ch   1.00  60.40      5      2  100  500    0   0    0 100:23
K2FibAdjMan Stats Re   2.00   0.30     10      4  100  500    0   0    0  6:21
K2FibAdjMan Host Mov   2.00   0.00     10      4  100  500    0   0    0  0:00
K2FibAdjMan Adj Chan   2.00   0.00     10      0  100  500    0   0    0  0:00
K2FibMulticast Signa   2.00   0.01     10      2  100  500    0   0    0  2:04

High CPU Utilization in the K2FibAdjMan Host Move Process

The Catalyst 4500 can display high CPU utilization in the K2FibAdjMan Host Move process. This high utilization appears in the output of the show platform health command. Many MAC addresses frequently expire or are learned on new ports, which causes this high CPU utilization. The workaround for this issue is to increase the MAC address aging time. Or, you can engineer the network in order to avoid the high number of MAC address moves. Cisco IOS Software Release 12.2(18)EW and later have enhanced this process behavior in order to consume less CPU. Refer to Cisco bug ID CSCed15021 ( registered customers only) .

Switch#show platform health 
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.02      2      1  100  500    0   0    0  1:09
GalChassisVp-review    3.00   0.29     10      3  100  500    0   0    0  11:15
S2w-JobEventSchedule  10.00   0.32     10      7  100  500    0   0    0  10:14

!--- Output suppressed.

K2FibAdjMan Stats Re   2.00   0.30     10      4  100  500    0   0    0  6:21
K2FibAdjMan Host Mov   2.00  18.68     10      4  100  500   25  29   28  2134:39
K2FibAdjMan Adj Chan   2.00   0.00     10      0  100  500    0   0    0  0:00
K2FibMulticast Signa   2.00   0.01     10      2  100  500    0   0    0  2:04
K2FibMulticast Entry   2.00   0.00     10      7  100  500    0   0    0  0:00

High CPU Utilization in the RkiosPortMan Port Review Process

The Catalyst 4500 can display high CPU utilization in the RkiosPortMan Port Review process in the output of the show platform health command in Cisco IOS Software Release 12.2(25)EWA and 12.2(25)EWA1. Cisco bug ID CSCeh08768 ( registered customers only) causes the high utilization, which Cisco IOS Software Release 12.2(25)EWA2 resolves. This process is a background process and does not affect the stability of the Catalyst 4500 switches.

Switch#show platform health
                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
                     Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Lj-poll                1.00   0.02      2      1  100  500    0   0    0  1:09
GalChassisVp-review    3.00   0.29     10      3  100  500    0   0    0  11:15
S2w-JobEventSchedule  10.00   0.32     10      7  100  500    0   0    0  10:14

!--- Output suppressed.

K2 Packet Memory Dia   2.00   0.00     15      8  100  500    0   1    1  45:46
K2 L2 Aging Table Re   2.00   0.12     20      3  100  500    0   0    0  7:22
RkiosPortMan Port Re   2.00  87.92     12      7  100  500   99  99   89  1052:36
Rkios Module State R   4.00   0.02     40      1  100  500    0   0    0  1:28
Rkios Online Diag Re   4.00   0.02     40      0  100  500    0   0    0  1:15

High CPU Utilization When Connected to an IP Phone with the Use of Trunk Ports

If a port is configured for both the voice VLAN option and the access VLAN option, the port acts as a multi-VLAN access port. The advantage is that only those VLANs that are configured for the voice and access VLAN options are trunked.

The VLANs that are trunked to the phone increase the number of STP instances. The switch manages the STP instances. Management of the increase in STP instances also increases the STP CPU utilization.

The trunking of all the VLANs also causes unnecessary broadcast, multicast, and unknown unicast traffic to hit the phone link.

Switch#show processes cpu
CPU utilization for five seconds: 69%/0%; one minute: 72%; five minutes: 73%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           4       165         24  0.00%  0.00%  0.00%   0 Chunk Manager    
   2       29012    739091         39  0.00%  0.00%  0.00%   0 Load Meter       
   3       67080     13762       4874  0.00%  0.00%  0.00%   0 SpanTree Helper  
   4           0         1          0  0.00%  0.00%  0.00%   0 Deferred Events  
   5           0         2          0  0.00%  0.00%  0.00%   0 IpSecMibTopN     
   6     4980144    570766       8725  0.00%  0.09%  0.11%   0 Check heaps      
  26   539173952 530982442       1015 13.09% 13.05% 13.20%   0 Cat4k Mgmt HiPri 
  27   716335120 180543127       3967 17.61% 18.19% 18.41%   0 Cat4k Mgmt LoPri 
  33     1073728     61623      17424  0.00%  0.03%  0.00%   0 Per-minute Jobs  
  34  1366717824 231584970       5901 38.99% 38.90% 38.92%   0 Spanning Tree    
  35     2218424  18349158        120  0.00%  0.03%  0.02%   0 DTP Protocol     
  36        5160    369525         13  0.00%  0.00%  0.00%   0 Ethchnl          
  37      271016   2308022        117  0.00%  0.00%  0.00%   0 VLAN Manager     
  38      958084   3965585        241  0.00%  0.01%  0.01%   0 UDLD             
  39        1436     51011         28  0.00%  0.00%  0.00%   0 DHCP Snooping    
  40         780     61658         12  0.00%  0.00%  0.00%   0 Port-Security    
  41     1355308  12210934        110  0.00%  0.01%  0.00%   0 IP Input         

Troubleshooting Tools to Analyze the Traffic Destined to the CPU

As this document has shown, traffic that is destined to the CPU is one of the major causes of high CPU utilization on the Catalyst 4500. The CPU-destined traffic can be either intentional because of the configuration, or unintentional because of misconfiguration or a denial-of-service attack. The CPU has an in-built QoS mechanism to prevent any adverse network effects because of this traffic. However, identify the root cause of CPU-bound traffic and eliminate the traffic if it is undesirable.

Tool 1: Monitor the CPU Traffic with SPAN—Cisco IOS Software Release 12.1(19)EW and Later

The Catalyst 4500 allows for the monitor of the CPU-bound traffic, either ingress or egress, with the use of the standard SPAN function. The destination interface connects to a packet monitor or an administrator laptop that runs packet sniffer software. This tool helps to quickly and accurately analyze the traffic that the CPU processes. The tool provides the ability to monitor individual queues that are bound to the CPU packet engine.

Note: The switching engine has 32 queues for the CPU traffic, and the CPU packet engine has 16 queues.

Switch(config)#monitor session 1 source cpu ? 
  both   Monitor received and transmitted traffic 
  queue  SPAN source CPU queue 
  rx     Monitor received traffic only 
  tx     Monitor transmitted traffic only 
  <cr> 
Switch(config)#monitor session 1 source cpu queue ? 
  <1-32>          SPAN source CPU queue numbers 
  acl             Input and output ACL [13-20] 
  adj-same-if     Packets routed to the incoming interface [7] 
  all             All queues [1-32] 
  bridged         L2/bridged packets [29-32] 
  control-packet  Layer 2 Control Packets [5] 
  mtu-exceeded    Output interface MTU exceeded [9] 
  nfl             Packets sent to CPU by netflow (unused) [8] 
  routed          L3/routed packets [21-28] 
  rpf-failure     Multicast RPF Failures [6] 
  span            SPAN to CPU (unused) [11] 
  unknown-sa      Packets with missing source address [10] 
Switch(config)#monitor session 1 source cpu queue all rx 
Switch(config)#monitor session 1 destination interface gigabitethernet 1/3 
Switch(config)#end 
4w6d: %SYS-5-CONFIG_I: Configured from console by console 

Switch#show monitor session 1 
Session 1 
--------- 
Type              : Local Session 
Source Ports      : 
    RX Only       : CPU 
Destination Ports : Gi1/3 
    Encapsulation : Native 
          Ingress : Disabled 
         Learning : Disabled 

If you connect a PC that runs a sniffer program, you can quickly analyze the traffic. In the output that appears in the window in this section, you can see that the cause of the high CPU utilization is an excessive number of STP BPDUs.

Note: STP BPDUs in the CPU sniffer is normal. But if you see more than you expect, you may have exceeded the recommended limits for your Supervisor Engine. See the A High Number of Spanning-Tree Port Instances section of this document for more information.

Tool 2: In-Built CPU Sniffer—Cisco IOS Software Release 12.2(20)EW and Later

The Catalyst 4500 provides an in-built CPU sniffer and decoder to quickly identify the traffic that hits the CPU. You can enable this facility with the debug command, as the example in this section shows. This features implements a circular buffer that can retain 1024 packets at a time. As new packets arrive, they overwrite the older packets. This feature is safe to use when you troubleshoot high CPU utilization issues.

Switch#debug platform packet all receive buffer 
platform packet debugging is on 
Switch#show platform cpu packet buffered 
Total Received Packets Buffered: 36 
------------------------------------- 
Index 0: 
7 days 23:6:32:37214 - RxVlan: 99, RxPort: Gi4/48 
Priority: Crucial, Tag: Dot1Q Tag, Event: Control Packet, Flags: 0x40, Size: 68 
Eth: Src 00-0F-F7-AC-EE-4F Dst 01-00-0C-CC-CC-CD Type/Len 0x0032 
Remaining data: 
 0: 0xAA 0xAA 0x3  0x0  0x0  0xC  0x1  0xB  0x0  0x0  
10: 0x0  0x0  0x0  0x80 0x0  0x0  0x2  0x16 0x63 0x28 
20: 0x62 0x0  0x0  0x0  0x0  0x80 0x0  0x0  0x2  0x16 
30: 0x63 0x28 0x62 0x80 0xF0 0x0  0x0  0x14 0x0  0x2  
40: 0x0  0xF  0x0  0x0  0x0  0x0  0x0  0x2  0x0  0x63 
Index 1: 
7 days 23:6:33:180863 - RxVlan: 1, RxPort: Gi4/48 
Priority: Crucial, Tag: Dot1Q Tag, Event: Control Packet, Flags: 0x40, Size: 68 
Eth: Src 00-0F-F7-AC-EE-4F Dst 01-00-0C-CC-CC-CD Type/Len 0x0032 
Remaining data: 
 0: 0xAA 0xAA 0x3  0x0  0x0  0xC  0x1  0xB  0x0  0x0  
10: 0x0  0x0  0x0  0x80 0x0  0x0  0x2  0x16 0x63 0x28 
20: 0x62 0x0  0x0  0x0  0x0  0x80 0x0  0x0  0x2  0x16 
30: 0x63 0x28 0x62 0x80 0xF0 0x0  0x0  0x14 0x0  0x2  
40: 0x0  0xF  0x0  0x0  0x0  0x0  0x0  0x2  0x0  0x63

Note: The CPU utilization when you issue a debug command is always almost 100%. It is normal to have high CPU utilization when you issue a debug command.

Tool 3: Identify the Interface That Sends Traffic to the CPU—Cisco IOS Software Release 12.2(20)EW and Later

Catalyst 4500 provides another useful tool to identify the top interfaces that send traffic/packets for CPU processing. This tool helps you quickly identify an errand device that sends a high number of broadcast or other denial-of-service attacks to the CPU. This feature is also safe to use when you troubleshoot high CPU utilization issues.

Switch#debug platform packet all count
platform packet debugging is on 
Switch#show platform cpu packet statistics

!--- Output suppressed.

Packets Transmitted from CPU per Output Interface 

Interface              Total           5 sec avg 1 min avg 5 min avg 1 hour avg 
---------------------- --------------- --------- --------- --------- ---------- 
Gi4/47                            1150         1         5        10          0 
Gi4/48                              50         1         0         0          0

Packets Received at CPU per Input Interface 

Interface              Total           5 sec avg 1 min avg 5 min avg 1 hour avg 
---------------------- --------------- --------- --------- --------- ---------- 
Gi4/47                           23130         5        10        50         20
Gi4/48                              50         1         0         0          0

Note: The CPU utilization when you issue a debug command is always almost 100%. It is normal to have high CPU utilization when you issue a debug command.

Summary

The Catalyst 4500 switches handle a high rate of IP version 4 (IPv4) packet forwarding in hardware. Some of the features or exceptions can cause the forward of some packets via the CPU process path. The Catalyst 4500 uses a sophisticated QoS mechanism to handle CPU-bound packets. This mechanism ensures reliability and stability of the switches and, at the same time, maximizes the CPU for the software forwarding of packets. Cisco IOS Software Release 12.2(25)EWA2 and later provide additional enhancements for packet/process handling as well as accounting. The Catalyst 4500 also has sufficient commands and powerful tools to aid in the identification of the root cause of high CPU-utilization scenarios. But, in most cases, high CPU utilization on the Catalyst 4500 is not a cause of network instability nor a cause for concern.