Verify Network Settings

 For starters verify that all expected interfaces are present on the system, and that the configuration appears as expected. Each system has its own tools for inspecting network interfaces.

 

• Linux

Run "ifconfig -a" and verify that all interfaces are up, and have the appropriate IP address. In addition, inspect the error counts in the RX and TX line. These should be 0, or there should be a reason behind any errors.

 

    

 

• Windows

                        Run "ipconfig /all". Generally you will be looking at the Ethernet Connections for Local Area Connections. Verify all settings are expected.

 

    

 

• Solaris

    

 

• HPUX - Add description here
• AIX - Add description here

---

 

connectalot and friends

 

These utilities are maintained by StorNext Development to test the ability to connect and stay connected. Why would one use these utilities instead of an industry standard, like netperf? For one thing, connectalot works a little differently in that it tries to make as many connections as possible, sending little or no data before disconnecting. We have used this tool to debug a bad ethernet slave interface which was causing a mover startup to fail intermittently when the connection was being set up over the bad NIC.

 

 

Another reason is that these are very simple and straightforward - one page of c code - and you can easily modify them to do exactly what you want. They also build and run on all of our supported platforms. 

 

Connectalot usage:

    

Stayinalive usage:

    

Simpleserver usage:

    

 

 

Network Statistics

Running "netstat -p" will show a number of statistics for the network. 

 

Some statistics to pay particular attention to are

• Tcp -> failed connection attempts
• Tcp -> connection resets received
• Tcp -> segments retransmitted
• Udp -> packet receive errors

 

    

---

DLC Network performance and connectivity tests

Netperf - benchmark used for measuring various aspects of network performance

Netperf can be found at the following location. http://www.netperf.org/netperf/

Netperf is especially useful for determining whether the network itself is capable of providing the bandwidth required. When diagnosing DLC performance issues, it is very important to run netperf as one of the initial step to confirm there is sufficient bandwidth. If there isn't, there is little point in performing end-to-end testing with DLC until this is resolved. Here is an example of simulating the network load from 3 DLC clients.

Start 3 copies of netserver on the gateway.
gateway# netserver –p 9990 & 
gateway# netserver –p 9991 &
gateway# netserver –p 9992 &
Then, to simulate streaming write throughput, use the TCP_STREAM test by running these command concurrently from the 3 clients. In the example below, "10.65.181.127" is the IP address of the interface on the gateway used for DLC traffic.
 
client1# netperf -p 9990 -t TCP_STREAM -fM -H 10.65.181.127 -- -D -s 1048576 -S 1048576
client2# netperf -p 9991 -t TCP_STREAM -fM -H 10.65.181.127 -- -D -s 1048576 -S 1048576
client3# netperf -p 9992 -t TCP_STREAM -fM -H 10.65.181.127 -- -D -s 1048576 -S 1048576
To test streaming read throughput, the TCP_MAERTS test should be run:
 
client1# netperf -p 9990 -t TCP_MAERTS -fM -H 10.65.181.127 -- -D -s 1048576 -S 1048576
client2# netperf -p 9991 -t TCP_MAERTS -fM -H 10.65.181.127 -- -D -s 1048576 -S 1048576
client3# netperf -p 9992 -t TCP_MAERTS -fM -H 10.65.181.127 -- -D -s 1048576 -S 1048576
In the above example, if the test is run in isolation, 10.65.181.127 is the IP address of a 10GbE interface, and the three clients are all connected at 1GbE, each of the tests should report around 100 MB/s of throughput if the network is configured and functioning properly since the server network bandwidth is only 30% utilized when the 3 clients transfer data as fast as they can. It's usually important to run tests with multiple clients because increased load can expose certain kinds of ecosystem problems such as switches with poor flow control.
It is recommended that "netstat -s" be run before and after netperf tests are run. Comparing error counts from the netstat output can show whether or not there are stability issues inside the network infrastructure.
 
DLC Client Statistics - measuring DLC performance

The cvdb utility can be used on the DLC clients and servers to retrieve the kernel statics for each DLC connection. Here is an example run on the server:

 
[root@jk-rh6sp3-2 foo]# cvdb -x
 Proxy # Listen/Backlog Sockets: 8
 Proxy # Active Sockets:         1
 Proxy # Server Address Entries: 0
 Proxy # Active Servers:         1
 Proxy: Connection # 1 (Server) for 
   Client System ID  169.254.83.165
   Client   IP Addr  10.65.184.80 (52261)
   Server   IP Addr  10.65.178.217 (57733)
     Read  Bytes/Sec 17154456 (16MB + 368KB + 408 bytes)
     Write Bytes/Sec 0
  FS Read  Bytes/Sec 17154456 (16MB + 368KB + 408 bytes)
  FS Write Bytes/Sec 0
     Queued I/O      0 
     Reads 2649 (2KB + 601 bytes)
     Writes 207 (207 bytes)
     Read  Bytes 171544576 (163MB + 612KB)
     Write Bytes 13197312 (12MB + 600KB)
 
[root@jk-rh6sp3-2 foo]# 
     
A few things to note here. There will be a set of statistics for each client connection. If a server has two interfaces for clients to connect, there will be stats for each connection, i.e., two times the number of connected clients.
 
When run on the DLC client, these stats can give an indication of how effective the load balancing algorithm is working.
In all cases, the useful counters are the Read/Write Bytes, not the /Sec counters. The /Sec counters are used in the load balancing algorithm on the client, but quickly decay to zero. The Bytes counters can be compared in successive snaps to get an idea of how much traffic is going over a connection over a period of time.

 Additional Network Diagnostics info

• Slide Deck put together by Alain Renaud
• Using Wireshark to analyze the network traffic