Overview

CT: Imagine a situation in which the customer is complaining about slow backup speeds between the media server and the DXi. In addition, the customer says that their opimized duplication jobs are failing.

 

Unfortunately, in order to resolve NAS related performance issues we have to take on the role of the customer’s network administrator.  Some customers have great network admins and others are not as fortunate.  It is the latter who present the most challenge for our service teams when determining “poor performance” when it comes to their backup solution.  

Professionals make a living at only troubleshooting networks and troubleshooting NAS related performance issues from a Quantum service perspective is one of the most challenging things we encounter. The purpose of this performance troubleshooting page is to help us quickly identify bottlenecks without becoming an expert in all things networking.

 

CT: Do you want to include something like this for team members to use or will there be too much variance to provide any sort of template? You can use the template (MAKE THIS A DOWNLOADABLE DOC) to help you organize the output from your analysis.

 

Remember, you can use the Export as PDF option in the left-hand pane if you want to save a copy to your desktop for use at the customer site.

 

---

A Little Background

Network administrators use something called the OSI model to help them understand network issues and identify bottlenecks. From a Quantum perspective we’re really only interested in layers 1-3.  If you want to fully understand the layers in the OSI model , read more about it on the internet by reviewing the following information:

• Layer 1 = physical .  (Cable, NIC, NIC ports, Switch ports, etc…)
• Layer 2 = mac address 
• Layer 3 = IP address 

---

Start at Layer 1: The Physical Layer

First the physical layer should be verified.  This can be done using the following steps:

1. Remove any windows teaming that is involved.  (Customer will have to delete the team then configure a single interface with the desired IP info.)
2. Disable ports or remove cables so that only a single interface is connected to backup server.
3. Connect only a single cable to the DXi.
4. Zero counters on the switch for respective ports. (Hopefully customer will know how to do this for their own switch.  If not, then search the internet for a product manual to understand how to zero counters.)
5. Enable netperf from the GUI.
6. Copy attached netperf windows version to the backup server.
7. Run the following command to test throughput between backup server and DXi. Note, the -l 30 option means the test will run for 30 seconds.  30 can be changed to any value.
   • netclient.exe -H 10.20.230.25 -f M -l 30 -- -s 64000,64000
     
8. The output from netclient should be 80-120 MB/s.  Rate can vary depending on what else is happening on the network.
9. The switch ports that were zeroed out should not have any errors from the file transfer.
10. If the output is below 80MB/s or there are errors on the switch ports, then use process of elimination to determine which piece of hardware is bad.  Do this by simply trying different ports, cables and NICs/ports.
11. Repeat tests for any ports and cables to be used for aggregation.

---

Continue with Layer 2 and Layer 3

For layers 2 and 3, look at aggregation.  Testing layer 1 actually involved your typical layer 2 and 3, MAC and IP, communication.  We only care about these layers in the customer’s network when it comes to aggregation.

 

Cisco aggregation is referred to as Etherchannel Technology . Reading about Etherchannel Technology  from Cisco will provide an understanding for most aggregation solutions from other vendors as well.  For the purposes of this document and solutions with a DXi system, keep these things in mind:

• Aggregation only works when throughput can exceed a single channel (IP connection).
• Out of the box, this means that a customer with a single backup server will never see more than 120MB/s and the average is 85 MB/s total aggregate throughput.

• There is a common misunderstanding that aggregating 2 ports from the backup server then aggregating 2 ports from the DXi would double or even increase throughput. While this may be possible with specific switches it is not the norm and should not be communicated to the customer.
• We have successfully increased throughput between a single backup server and a DXi6520 by creating multiple virtual IPs on the DXi, using LACP on both the customer’s backup server and the DXi and then configuring the backup application to use the multiple IP addresses.  his created multiple IP connections which resulted in multiple channels to be split by LACP. While this is possible it is not yet supported or widely known. The example is in this document to make aware that creativity can increase throughput when necessary. This will be available out of the box with mininet in 2.x but for now requires changes to system files that do not survive FW upgrades.
• Instead of discussing the approach from the last bullet, an additional backup server should be recommended.  Unfortunately, we haven’t done a good job of communicating numbers on a per backup server basis and the previous bullet may become necessary for political reasons.

---

Troubleshooting

CT: Should these be listed as steps? In other words, is there an order here?

 

• Now that all physical issues have been verified, configure aggregation on the DXi.
• LACP  was discouraged from engineering but most of our customers use it as their aggregation standard.
• Start with LACP  unless customer states otherwise. 
• Hopefully customer knows how to create an aggregation group (it is called an etherchannel on Cisco systems)
• Make sure the hash algorithm, load-balance mode , for the switch is set properly.  See examples in the next bullet.
• If not then search the vendor’s website for documentation on how to create an LACP configuration.  Click here for a tutorial on finding the right guide and creating an LACP etherchannel. 
• Use a connection in the group for each endpoint (backup server).
• Once an aggregation group is created on the DXi use the netperf testing sequence from the Layer 1 testing.
• If you have multiple endpoints (multiple backup servers) test using netperf from all of them at the same time. They should all be able to still get their 80-120MB/s while they are all running netperf concurrently.
• If total aggregate throughput from all endpoints isn’t 80-120MB/s per endpoint then there is probably a problem with the LACP  configuration on the switch.
• The only problem that could be possible at this point on the DXi is a corrupted driver or an OS problem. Both of which should show up in the messages file.  Appropriate action is apparent if this is the case.
• Attempt different aggregation configurations on the switch or ask the customer to engage someone more specialized to resolve the issue.

 

---

Common Concerns

The 80-120 MB/s range

There will always be customers who think 80MB/s is not adequate on a single 1 GbE connection but 85 is about our average when it comes to Windows.  Even when going direct there are a lot of things that play into this that need to be changed in the registry to be optimized.  In my experience I’ve always been able to find a system that can show higher throughput to the DXi somewhere on the network proving that it is an issue local to the backup server.

Customer will sometimes claim that DART is inaccurate

I have yet to see DART Ethernet statistics to be inaccurate.  With one customer I had to capture packets on the DXi using tcpdump then open the capture file in wireshark.  Wireshark numbers were identical to DART numbers.

 

---

Exit Strategy

It is easy to get caught up in a NAS related performance issue for weeks or months with a customer who usually thinks there is no problem with their network.  Our focus in service should always be helping the customer to identify the problem then hand off to them if we don’t have the resources to resolve it.

 

Using the processes above should make it easier to identify where the problem is. Showing adequate throughput between the backup server/s and the DXi would prove that the bottleneck is mostly like due to the network between the backup clients or is due to configuration of the backup software.

 

If the customer is not in agreement with the findings from above where adequate performance between the DXi and the backup server has been proven, then communicate this to management and an exit strategy will be discussed.

---

Conclusion

Using the troubleshooting methods in this guide will hopefully help us all save time when faced with challenging NAS performance issues. This guide lives on qwikipedia so that it can be contributed to by all service members. There is still a lot to be added such as Linux specific testing, CIFS vs NFS and more educational links.  For now, please send any ideas, comments or suggestions to me at ryan.davies@quantum.com

 

---