Thanks to Andres Moreno/SPS for input into this.
Symptom:
FC ABORT MESSAGES IN /var/log/messages LOG:
2017-09-05-13:01:38.053758 MDC1 kernel: qla2xxx [0000:0a:00.1]-801c:9: Abort command issued nexus=9:3:1 -- 1 2002.
2017-09-05-13:18:30.053758 MDC1 kernel: qla2xxx [0000:0a:00.1]-801c:9: Abort command issued nexus=9:0:1 -- 1 2002.
2017-09-05-13:38:21.053754 MDC1 kernel: qla2xxx [0000:0a:00.1]-801c:9: Abort command issued nexus=9:6:1 -- 1 2002.
2017-09-05-13:40:45.053758 MDC1 kernel: qla2xxx [0000:0a:00.1]-801c:9: Abort command issued nexus=9:1:3 -- 1 2002.
Solution:
In the Stornext snapshot (capture_state):
1. open up lsscsi.txt , cvgather_multipath, cvlavbel_ls files.
2. in /var/log/messages file, see which nexus number Aborts are coming from:
2017-09-05-13:40:45.053758 MDC1 kernel: qla2xxx [0000:0a:00.1]-801c:9: Abort command issued nexus=9:1:3 -- 1 2002.
nexus=9:1:3
3. Take the nexus number, add a 0 in 2nd field, so nexus=9:1:3 becomes: 9:0:1:3.
4. Look for 9:0:1:3 in multipath.conf file in the snapshot:
mpathi (321370022c1653733) dm-27 Active,AC16SFC02DC
size=11T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 8:0:1:3 sds 65:32 active ready running
|- 9:0:1:3 sdag 66:0 active ready running
|- 8:0:5:3 sdz 65:144 active ready running
`- 9:0:5:3 sdan 66:112 active ready running
5. Look for WWWN node name in the stanza on the "mpath" line : 321370022c1653733.
6. Remove the first digit 3 and then take the rest of that number 21370022c165373.
7. Look for 21370022c1653733 in the cvlabel output to see which LUN the Aborts are coming from.
8. You'll be able to determine which LUN and which filesystem that pertains to.
9. You also might want to look at the "fc-statistics" file in the snapshot to see if you see errors in it:
10. Replace the FC HBA if you see these errors.
/sys/class/fc_host/host9/statistics/seconds_since_last_reset:0x4df7
/sys/class/fc_host/host9/statistics/tx_frames:0x7556
/sys/class/fc_host/host9/statistics/tx_words:0x0
/sys/class/fc_host/host9/statistics/rx_frames:0xbbed
/sys/class/fc_host/host9/statistics/rx_words:0x17d3f1a
/sys/class/fc_host/host9/statistics/lip_count:0x1
/sys/class/fc_host/host9/statistics/nos_count:0x1
/sys/class/fc_host/host9/statistics/error_frames:0x135
/sys/class/fc_host/host9/statistics/dumped_frames:0x0
/sys/class/fc_host/host9/statistics/link_failure_count:0x1
/sys/class/fc_host/host9/statistics/loss_of_sync_count:0x0
/sys/class/fc_host/host9/statistics/loss_of_signal_count:0x0
/sys/class/fc_host/host9/statistics/prim_seq_protocol_err_count:0x0
/sys/class/fc_host/host9/statistics/invalid_tx_word_count:0x61214
/sys/class/fc_host/host9/statistics/invalid_crc_count:0xa6
/sys/class/fc_host/host9/statistics/fcp_input_requests:0x3201
/sys/class/fc_host/host9/statistics/fcp_output_requests:0x0
/sys/class/fc_host/host9/statistics/fcp_control_requests:0x9be
/sys/class/fc_host/host9/statistics/fcp_input_megabytes:0x17
/sys/class/fc_host/host9/statistics/fcp_output_megabytes:0x0