How to Investigate SNMP Issues on a DXi (DRAFT)

Overview

The Simple Network Management Protocol (SNMP) was designed to work over many types of networks. SNMP started with version 1 (SNMPv1) and has been replaced by SNMPv2, which has been extended to SNMPv2c. SNMP is a protocol used to do primitive messaging between computer systems. SNMP was designed by committee, for the management of equipment in a network operations center. The interface is between the Network layer protocol and the physical network layer. Structures called MIB-II contains the most wanted, and generic configuration, status, and other information for any type of interface.

This topic will not deep dive into Internet Control Message Protocol (ICMP), but it expects the user community to understand the basics, being that ICMP messages are flowing across the network. Transmission Control Protocol (TCP) is responsible for reliable construction of connections, between partners, retransmission, and termination of connections. User Datagram Protocol (UDP) is used to send stand alone messages between applications. SNMP uses UDP to send “get”, “set”,” response”, and “traps” between systems. The DXi uses the SNMP to support standard and custom SNMP data. The investigation to resolve snmp issues on the DXi, we will start with a top down debug of the environment. Customer use Network Monitor program that targets the Management IP address on the DXi. Note: The DXi is SNMPv2c compliant for the protocol. Not SNMPv1. SNMP uses two unix ports to communicate, port 161 and port 162. When referencing this topic, you will need basic network background and basic UNIX skill to complete the Investigation.

Topic Conventions

Commentary is in black text, commands are in blue BOLD text, and data is in light blue text. Do not attempt this investigation if uncomfortable with vi or Linux command line interface. The DXi in question shall be refered to as “DXi” and will be noted in the UNIX command prompt “ [root@DXi /]# ”.

The "How To" procedures for this investigation is separated into 6 steps. Each Step below is targeting the next level of functionality, starting at network path, to snmp environment, to snmpd internal routing, to snmpd responses, and finishing with snmpd traps.

Step 1: Find the IP path for the DXi customer Network Monitor to target the Management Segment IP, and confirm this in the output of Step 3 and 5.
Step 2: Check the route and confirmed a valid IP route from DXi to the customer Network Monitor. Trace route and netcat command utilities, probe down the route path to the Network Monitor.
Step 3: Check the local SNMP environment on the DXi; looking for processes, IP addresses, ports. Confirm snmpd is running, and talking to the ports 161 and ports 162. Confirm snmp has a open file handle on port 161, check the route tables, check netstat for port 161 tied to snmpd process.
Step 4: Sanity check: Verify the DXi firewall is not blocking SNMP protocol traffic. Use the iptables status from Step 3, to review firewall status. Test Network Monitor with firewall enabled and disabled.
Step 5: Modify snmp config files for testing, check snmpd response and review environment. Adding “public” declaration to the SNMP configuration. This opens snmp to respond to almost any query. The goal of this is to touch snmpd and check its configuration in the system is matching data in Step 3.
Step 6: Probe DXi_B to DXi. The Goal is testing the DXi snmp response and connectivity across the network, DXi to DXi_B. Generate a snmptrapd on the “DXi” system from both localhost and DXi_B.

Step 1

The SNMP is designed to work across the Management IP. Locate the management IP address and make sure the SNMP monitor is targeting the Management IP. This may be the bond0 in a bond all configuration.

Is the customer using the Management IP to address the DXi with the Network Monitor? Make sure the Network Monitor is using the IP address, not the DNS name.

[logsystem]$ grep ManagementSegmentIPAddress systemconfig.xml

Step 2

Begin with the basics by collect the network environment. The network starts with the Management Segment IP Address and routes across the customer environment to the customer SNMP Network Monitor. Ask the customer for the IP address of the customer SNMP Network Monitor system. The Data IPs below have been modified to protect customer information. So for the DXi below, 198.30.218.46:161 and 198.30.218.46:162.

# tracert 198.27.99.252

traceroute to 198.27.99.252 (198.27.99.252), 30 hops max, 40 byte packets

1 198.30.218.1 (198.30.218.1) 2.201 ms 2.726 ms 2.727 ms

2 195.24.244.65 (195.24.244.65) 1.024 ms 1.573 ms 1.573 ms

3 88.35.195.37 (88.35.195.37) 6.351 ms 6.884 ms 6.882 ms

4 scsce1-wan.net.americas.nobody.com (30.212.46.226) 85.362 ms 85.876 ms 85.936 ms

5 scscsr01-vl951.scs.nobody.com (198.28.23.62) 84.817 ms 85.352 ms 85.350 ms

6 scssr1-vl957.net.americas.nobody.com (141.183.12.11) 84.792 ms 84.862 ms 84.375 ms

7 scsnet01.scs.nobody.com (198.27.99.252) 84.784 ms 85.129 ms 84.565 ms

This customer has two paths, which uses quite a bit of data output, looking at just one of the traceroutes below, and now using netcat command (nc) to look at each hop in the path. Using netcat to probe back down tracerout output, note we are check the two ports 161 and 162. The netcat utility succeeds when it can receive a ICMP response. Starting on the DXi system, work back on the route path to the Network Monitor.

[root@DXi snmp]# nc -zu 198.30.218.46 162

Connection to 130.30.218.45 162 port [udp/snmptrap] succeeded!

[root@DXi snmp]# nc -zu 198.30.218.46 161

Connection to 130.30.218.45 161 port [udp/snmp] succeeded!

Ok, so the DXi itself succeed in a ICMP reply. Let's continue working down the trace route list above.

[root@DXi snmp]# nc -zu 198.30.218.1 161

Connection to 130.30.218.1 161 port [udp/snmp] succeeded!

[root@DXi snmp]# nc -zu 195.24.244.65 161

Connection to 195.24.244.65 161 port [udp/snmp] succeeded!

[root@DXi snmp]# nc -zu 88.35.195.37 161

Connection to 88.35.195.37 161 port [udp/snmp] succeeded!

[root@DXi snmp]# nc -zu 30.212.46.226 161

Connection to 30.212.46.226 161 port [udp/snmp] succeeded!

[root@DXi snmp]# nc -zu 198.28.23.62 161

Connection to 198.28.23.62 161 port [udp/snmp] succeeded!

Hmm, what happened below? Some one in the route is not responding to their port. This is not the root cause, but something for the customer to investigate.

[root@DXi snmp]# nc -zu 141.183.12.11 161

[root@DXi snmp]# nc -zu 198.27.99.252 161

Connection to 198.27.99.252 161 port [udp/snmp] succeeded!

Step 3:

Look at the local DXi environment for SNMP. Start with the lsof command to check on open file handles. Snmpd has a open on port 161 below. Then lets look at the iptables status, we are looking specifly for the Management network ip 198.30.218.46 and then the ports tcp dpt:162 Below is the standard response to assure the snmp service is running and avail on port 161 and port 162. If any of the ports do not appear in the local DXi probing then you may want to restart the snmp service. Look for the processes

[root@DXi snmp]# ps -ef | grep snmp

root 442 29705 0 11:52 pts/3 00:00:00 grep snmp

root 18987 1 0 Sep23 ? 00:00:00 /usr/adic/perl/bin/perl /opt/DXi/sl_noti_snmptrap_monitor

root 31584 1 0 11:52 ? 00:00:00 /usr/local/sbin/snmpd

root 31941 1 1 11:52 pts/3 00:00:00 /usr/local/sbin/qtm_snmp_subagent

root 31956 1 0 11:52 ? 00:00:00 /usr/local/sbin/snmptrapd

To restart the service, use the following command.

[root@DXi snmp]# service snmpd restart

Shutting down snmpd: [ OK ]

OKarting snmpd: [ OK ]

No log handling enabled - turning on stderr logging

NET-SNMP version 5.4 AgentX subagent connected

OK [ OK ]

NET-SNMP version 5.4 AgentX subagent connected

[root@DXi snmp]# lsof -i :161

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

snmpd 16817 root 7u IPv4 57571440 UDP *:snmp

The next command is looking at a route issue in the iptables. This looks all open, which probably means the firewall on the DXi is in a enabled state, more later on Step 4. Look for the ports 161 and 162 and the management network ip.

[root@DXi snmp]# service iptables status

Table: nat

Chain PREROUTING (policy ACCEPT)

num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)

num target prot opt source destination

Chain OUTPUT (policy ACCEPT)

num target prot opt source destination

Table: mangle

Chain PREROUTING (policy ACCEPT)

num target prot opt source destination

Chain INPUT (policy ACCEPT)

num target prot opt source destination

Chain FORWARD (policy ACCEPT)

num target prot opt source destination

Chain OUTPUT (policy ACCEPT)

num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)

num target prot opt source destination

Table: filter

Chain INPUT (policy DROP)

num target prot opt source destination

1 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 255

2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED

3 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:53

4 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:53

5 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0

6 ACCEPT tcp -- 0.0.0.0/0 10.17.21.1 tcp

7 ACCEPT udp -- 0.0.0.0/0 10.17.21.1 udp

8 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:22

9 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:80

10 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:443

11 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:162

12 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:161

13 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:8406

14 ACCEPT tcp -- 0.0.0.0/0 198.30.218.46 tcp dpt:8407

Now look at the process snmpd is in the network tables.

[snmp]# netstat -ulpn|grep snmpd

udp 0 0 0.0.0.0:161 0.0.0.0:* 13829/snmpd

What does netstat think about the system environment with regards to the snmp processes? (TS: Do we need to provide an answer to this? Or explain what they are seeing in the example below?)

[snmp]# netstat -anpo | grep snmp

tcp 0 0 127.0.0.1:51665 127.0.0.1:9998 ESTABLISHED 2072/qtm_snmp_subag off (0.00/0/0)

tcp 0 0 127.0.0.1:51666 127.0.0.1:9998 ESTABLISHED 2072/qtm_snmp_subag off (0.00/0/0)

tcp 0 0 127.0.0.1:51667 127.0.0.1:9998 ESTABLISHED 2072/qtm_snmp_subag off (0.00/0/0)

tcp 0 0 127.0.0.1:51668 127.0.0.1:9998 ESTABLISHED 2072/qtm_snmp_subag off (0.00/0/0)

udp 0 0 0.0.0.0:161 0.0.0.0:* 2061/snmpd off (0.00/0/0)

udp 0 0 0.0.0.0:162 0.0.0.0:* 2083/snmptrapd off (0.00/0/0)

unix 2 [ ACC ] STREAM LISTENING 99390822 2061/snmpd /var/agentx/master

unix 3 [ ] STREAM CONNECTED 99390881 2061/snmpd /var/agentx/master

unix 3 [ ] STREAM CONNECTED 99390880 2072/qtm_snmp_subag

unix 3 [ ] STREAM CONNECTED 99390870 2061/snmpd /var/agentx/master

unix 3 [ ] STREAM CONNECTED 99390869 2083/snmptrapd

Step 4

Check if the DXi firewall is causing problems with the SNMP route. Disable the firewall temporarily to test the DXi side. This will need to have the customer attempting a discovery of SNMP to the DXi with the Network Monitor tool, both with firewall enabled and disabled. Don’t forget to re-enable the firewall.

First check the iptables status. In the example below, the firewall service is already disabled. Remember this and compare with the outputs of the iptables status. If the fire wall was off, then make sure it's off when you are finished. If the firewall was enabled, them make sure its running when you are finished.

[root@DXi snmp]# service iptables status

Table: nat

Chain PREROUTING (policy ACCEPT)

num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)

num target prot opt source destination

Chain OUTPUT (policy ACCEPT)

num target prot opt source destination

Table: filter

Chain INPUT (policy ACCEPT)

num target prot opt source destination

Chain FORWARD (policy DROP)

num target prot opt source destination

Chain OUTPUT (policy ACCEPT)

num target prot opt source destination

Lets make sure its off. Disable the firewall. Then test the Customer Network Monitor ability to Discover SNMP service on the DXi.

[root@DXi snmp]# /opt/DXi/scripts/netcfg.sh --disablefw

INFO: *** /opt/DXi/scripts/netcfg.sh script started with arguments: --disablefw ***

INFO: *** /opt/DXi/scripts/netcfg.sh script completed successfully ***

If the Firewall was on, remember to re-enable the service when you are finished.

[root@DXi snmp]# /opt/DXi/scripts/netcfg.sh --enablefw

INFO: *** /opt/DXi/scripts/netcfg.sh script started with arguments: --enablefw ***

INFO: *** /opt/DXi/scripts/netcfg.sh script completed successfully ***

Step 5

Now we need to begin adding our own snmp public environment, so that we can test the response of the snmp activity. To begin part of the investigation, make a backup copy of any files about to be modified. Modify the files and restart the snmpd service for the testing duration. The addition of the public declarations to the snmpd.conf allows us to probe the DXi snmpd using the public declaration. When complete, copy back the saved snmp config files and restart the snmpd service. The output below is from LAB systems, again to protect customer data. The goal in this step is to probe the snmp environment using snmp commands. The “DXi” here is IP address 10.20.234.51 and used below in some UNIX commands.

[root@DXi /] # cd /usr/local/share/snmp

[root@DXi snmp]# cp snmpd.conf snmpd.conf.orig

[root@DXi snmp]# cp snmptrapd.conf snmptrapd.conf.orig

[root@DXi snmp]# vi /usr/local/share/snmp/snmpd.conf

# two lines below added for debug after the master agentx line

rocommunity public .1

rwcommunity public default

[root@DXi snmp]# vi /usr/local/share/snmp/snmptrapd.conf

#Un-comment (remove the #) from the line below already in snmptrapd.conf

#logoption f /var/log/snmptrapd-direct.log

Now restart the services to make the changes above take effect.

[root@DXi snmp]# service snmpd restart

Shutting down snmpd: [ OK ]

OKarting snmpd: [ OK ]

OK [ OK ]

NET-SNMP version 5.4 AgentX subagent connected

Now its time to start testing the snmpd daemons on the system, the output from some commands may vary depending on the galaxy software and DXi system.

[root@DXi snmp]# snmpwalk -v2c -c public localhost system

SNMPv2-MIB::sysDescr.0 = STRING: DXi8500

SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (28287) 0:04:42.87

SNMPv2-MIB::sysContact.0 = STRING:

SNMPv2-MIB::sysName.0 = STRING: SES8500DXi51

SNMPv2-MIB::sysLocation.0 = STRING:

SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORID.1 = OID: SNMP-FRAMEWORK-MIB::snmpFrameworkMIBCompliance

SNMPv2-MIB::sysORID.2 = OID: SNMP-MPD-MIB::snmpMPDCompliance

SNMPv2-MIB::sysORID.3 = OID: SNMP-USER-BASED-SM-MIB::usmMIBCompliance

SNMPv2-MIB::sysORID.4 = OID: SNMPv2-MIB::snmpMIB

SNMPv2-MIB::sysORID.5 = OID: TCP-MIB::tcpMIB

SNMPv2-MIB::sysORID.6 = OID: IP-MIB::ip

SNMPv2-MIB::sysORID.7 = OID: UDP-MIB::udpMIB

SNMPv2-MIB::sysORID.8 = OID: SNMP-VIEW-BASED-ACM-MIB::vacmBasicGroup

SNMPv2-MIB::sysORDescr.1 = STRING: The SNMP Management Architecture MIB.

SNMPv2-MIB::sysORDescr.2 = STRING: The MIB for Message Processing and Dispatching.

SNMPv2-MIB::sysORDescr.3 = STRING: The management information definitions for the SNMP User-based Security Model.

SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module for SNMPv2 entities

SNMPv2-MIB::sysORDescr.5 = STRING: The MIB module for managing TCP implementations

SNMPv2-MIB::sysORDescr.6 = STRING: The MIB module for managing IP and ICMP implementations

SNMPv2-MIB::sysORDescr.7 = STRING: The MIB module for managing UDP implementations

SNMPv2-MIB::sysORDescr.8 = STRING: View-based Access Control Model for SNMP.

SNMPv2-MIB::sysORUpTime.1 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.2 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.3 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.4 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.5 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.6 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.7 = Timeticks: (0) 0:00:00.00

SNMPv2-MIB::sysORUpTime.8 = Timeticks: (0) 0:00:00.00

Now look at the output of snmp netstat from the public declaration. Does this match the DXi route table, “ ip route show table main”?

[root@DXi snmp]# snmpnetstat -v2c -cpublic -Crs 10.20.234.51

Routing tables

Destination Gateway Flags Interface

default 10.20.232.1 <UG> bond0

10.17.21/24 * <U> bond9

10.20.232/21 * <U> bond0

Does the output match the DXi output from the UNIX command “ifconfig”?

[root@DXi snmp]# snmpnetstat -v2c -cpublic -Ci 10.20.234.51

Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Queue

lo 16436 127/8 127.0.0.1 1072633692 0 1072633692 0 0

bond0 1500 10.20.234/21 10.20.234.51 66736448 0 29286944 0 0

eth0 1500 64253378 0 58408143 0 0

eth1 1500 308857 0 0 0 0

eth2 1500 33275911 0 14643472 0 0

eth3 1500 33460537 0 14643472 0 0

eth4 1500 0 0 0 0 0

eth5 1500 0 0 0 0 0

eth6 1500 0 0 0 0 0

eth7 1500 0 0 0 0 0

bond9 1500 10.17.21/24 10.17.21.1 64562235 0 58408143 0 0

Does this match the output from UNIX command “netstat –anpo”?

[root@DXi snmp]# snmpnetstat -v2c -cpublic -Cp tcp 10.20.234.51

tcp SES8500DXi51.loc.60980 SES8500DXi51.loc.distiESTABLISHED

tcp SES8500DXi51.loc.60981 SES8500DXi51.loc.distiESTABLISHED

[root@DXi snmp]# snmpnetstat -v2c -cpublic -Cp udp 10.20.234.51

Active Internet (udp) Connections

Proto Local Address

udp *.tftp

udp *.sunrpc

udp *.ntp

udp *.netbios-

udp *.snmp

udp *.snmptrap

udp *.qmqp

udp *.xfr

udp *.device

udp *.880

udp *.nfs

udp *.32000

udp *.45154

udp *.56618

udp *.58607

udp Qnode1.ntp

udp ses8500dxi51.dnw.ntp

udp ses8500dxi51.dnw.netbi

udp 10.20.239.255.netbios-

udp SES8500DXi51.loc.ntp

Now we are changing gears and testing not only resolution of localhost, but doing a snmp walk. This is the type of data the Network Monitor might extract.

[root@DXi snmp]# snmpwalk -v2c -c public localhost memory

UCD-SNMP-MIB::memIndex.0 = INTEGER: 0

UCD-SNMP-MIB::memErrorName.0 = STRING: swap

UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 16779852 kB

UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 16779852 kB

UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 65714136 kB

UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 15952200 kB

UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 32732052 kB

UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB

UCD-SNMP-MIB::memShared.0 = INTEGER: 0 kB

UCD-SNMP-MIB::memBuffer.0 = INTEGER: 503820 kB

UCD-SNMP-MIB::memCached.0 = INTEGER: 9408868 kB

UCD-SNMP-MIB::memSwapError.0 = INTEGER: noError(0)

UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING:

[root@DXi snmp]# snmptable -v2c -c public localhost IF-MIB::ifTable

SNMP table: IF-MIB::ifTable

Now below test the three paths to check resolution of IP address(127.0.0.1, localhost, 10.20.234.51)

[root@DXi snmp]# snmpwalk -v2c -c public 127.0.0.1 QUANTUM-SNMP-MIB::quantumsnmp

QUANTUM-SNMP-MIB::qNoInterventionNeededEnable.0 = INTEGER: trapnotavailable(3)

QUANTUM-SNMP-MIB::qRequestedInterventionEnable.0 = INTEGER: trapnotavailable(3)

QUANTUM-SNMP-MIB::qRequiredInterventionEnable.0 = INTEGER: trapnotavailable(3)

[root@DXi snmp]# snmpwalk -v2c -c public localhost QUANTUM-SNMP-MIB::quantumsnmp

QUANTUM-SNMP-MIB::qNoInterventionNeededEnable.0 = INTEGER: trapnotavailable(3)

QUANTUM-SNMP-MIB::qRequestedInterventionEnable.0 = INTEGER: trapnotavailable(3)

QUANTUM-SNMP-MIB::qRequiredInterventionEnable.0 = INTEGER: trapnotavailable(3)

[root@DXi snmp]# snmpwalk -v2c -c public 10.20.234.51 QUANTUM-SNMP-MIB::quantumsnmp

QUANTUM-SNMP-MIB::qNoInterventionNeededEnable.0 = INTEGER: trapnotavailable(3)

QUANTUM-SNMP-MIB::qRequestedInterventionEnable.0 = INTEGER: trapnotavailable(3)

QUANTUM-SNMP-MIB::qRequiredInterventionEnable.0 = INTEGER: trapnotavailable(3)

Step 6

Next check the access from one DXi to another DXi in the environment. Locate a second DXi_B in the Customer environment and send snmp query to the first DXi system snmp service. Start by attempting to snmp walk query from DXi_B to DXi, in this case a DXi8500. Results may not exactly match, but will should match the output from just above in Step 4. Note: you will need to make the same modification to DXi_B /usr/local/share/snmp/snmpd.conf as before and restart the services snmpd to access the public declaration. In the example below “DXi_B” is IP address 10.20.234.55

Just to confirm the large system information is avail, from DXi_B probe itself (DXi_B ) then system DXi.

Remember system DXi is the target system in question. .

[root@DXi_B /]# snmpwalk -v2c -c public 10.20.234.55 system

SNMPv2-MIB::sysDescr.0 = STRING: DXi6500

SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (27619) 0:04:36.19

SNMPv2-MIB::sysContact.0 = STRING:

SNMPv2-MIB::sysName.0 = STRING: SES6702DXi55