F1000 Installation and Configuration

Caution: Before installing or configuring the system, read the safety information.

To avoid personal injury or property damage, before you begin installing the product, read, observe, and adhere to all of the following safety instructions and information. The following safety symbols are used throughout the documentation:

Caution: Indicates the presence of a hazard that may cause minor personal injury, property damage, or data loss if ignored.

WARNING: Indicates the presence of a hazard that may result in serious personal injury if ignored.

Caution: Electrostatic discharge can harm delicate components inside the F1000.

Electrostatic discharge (ESD) is a discharge of stored static electricity that can damage equipment and impair electrical circuitry. It occurs when electronic components are improperly handled and can result in complete or intermittent failures.

Wear an ESD wrist strap for installation, service, and maintenance to prevent damage to components in the product. Ensure the antistatic wrist strap is attached to a chassis ground (any unpainted metal surface). If possible, keep one hand on the frame when you install or remove an ESD-sensitive part.

Before moving ESD-sensitive parts, place them in ESD static-protective bags until you are ready to install the part.

When choosing an installation site for the F1000, make sure the location meets the requirements listed in F1000 Planning, which includes detailed physical and environmental requirements for the installation site.

Complete the steps in this section to unpack the system and verify that all the components were included in the F1000 box.

Note: Retain all packing materials, as well as the documentation and other items included in the shipping box. The packaging materials must be used if you move the system.

Caution: Two people are needed to lift and move the F1000 box. Use care to avoid injury.

To unpack the hardware:

  1. After you carefully open the box, ensure the following items are included, as shown in the diagram.

    Item

    Name

    Description

    1

    F1000 chassis

    Chassis with pre-installed PSUs, SSDs, and Fiber Channel or Ethernet cards

    2

    F1000 Getting Started Guide

    Provides an overview about how to install and set up the system

    F1000 Customer Site Configuration Data Sheet

    Contains F1000 MAC addresses and World Wide Port Names (WWPNs)

    3

    Cables

    CAT 6A RJ45 (7 feet) (Two cables)

    LC/LC optical (2 meters) (Eight cables, Fibre Channel units only)

    4

    Quantum bezel

    The front bezel

    5

    Accessory Kit

    Includes:

    • C13 to C14 power cord (2 cords)
    • Screws to secure the rails to the F1000
    • L-brackets for installing the F1000 in a Telco rack
    6 Rails The outer rail and part of the inner rails are not attached, but are included in the box.
  2. Place the F1000 on an electrostatic discharge (ESD)-protected surface, and remove the plastic bag.

The F1000 is attached to a rack using two rack-mounted rails that allow the system to be pulled out, making the back of the chassis available for servicing. You can install the F1000 in a rack with square holes or Telco racks.

This section describes how to install the F1000 into a rack.

Required Tools

A standard, Phillips screwdriver for flat-head, countersunk screws

The F1000 rack rails should be installed into the rack at the lowest possible U height to keep the load on the rack balanced.

To install the rack rails into a rack:

To install the chassis into a Telco or post-style rack, use two L-shaped brackets on either side of the chassis (four total).

  1. Determine how far the server will extend from the front of the rack.
  2. Attach the two front brackets to each side of the chassis.
  3. Attach the two rear brackets, positioned with just enough space to accommodate the width of the Telco rack.
  4. Slide the chassis into the rack.
  5. Tighten the brackets to the rack.

This section describes how to cable the F1000. Power cables and the an Ethernet cable are provided with the F1000. Other cables, such as the 100 GbE Copper Direct Attach Cables (DAC) and Fibre Channel cables, need to be available for the installation. These can be purchased separately, or from Quantum.

For a diagram of the ports in the back of the F1000, see Ports, Switch, and LEDs.

Cable the system

From the rear of the chassis, cable the F1000 by doing the following:

  1. Connect the 1 GbE management port (M1P1) (see the diagram below) on the system to the management network. This provides remote access to F-Series UI GUI.

  2. Connect the end of the CAT6A Ethernet cable to a network switch. This connection is for management of the system. It is not used for data traffic.

    Note: Note: Do not connect the configuration port (M1P2) until you are ready to configure the system, which occurs in a later step.

  3. Connect FC or Ethernet cables.

    1. Connect up to four FC cables to the ports on each quad-port, 32 Gb FC card located in top and bottom full-height PCIe slots. (You can connect up to eight cables, total.)

      Item

      Description

      1

      Quad-port 32 Gb FC card in top, full-height PCIe slot

      2

      Quad-port 32 Gb FC card in bottom, full-height PCIe slot

    2. Connect the other end of up to four FC cables (for each card) to a 32 Gb FC switch.

      Note: For maximum performance, connect eight FC cables from the FC switch to the FC ports on the F1000. If you are connecting only two FC cables to the F1000, make sure to connect one cable to each FC card.

    1. Connect two customer-provided 100 GbE Copper Direct Attach cables to the two Ethernet ports in one or both of the full-height PCIe slots, as follows:

      • If you will configure your storage network to have one subnet, connect one cable to port S2P2 (item 1 in the diagram below).
      • If you will configure your storage network to have dual subnets, connect one cable to port S2P2 (item 1 in the diagram below), and another cable to port S1P2 (item 3 in the diagram below).

      Item

      Description

      1

      Port S2P2 in 100 GbE card in top, full-height PCI slot

      2

      Port S2P1 in 100 GbE card in top, full-height PCI slot

      3

      Port S1P2 in 100 GbE card in bottom, full-height PCI slot

      4

      Port S1P1 in 100 GbE card in bottom, full-height PCI slot

    2. Connect the other end of the 100 GbE Copper Direct Attach cables to a 100 GbE network switch.

  4. Connect the power cables to the PSUs. Use the attached straps to secure the power cords to the chassis.

    Item

    Description

    1

    Power supply unit (PSU) A

    2

    Power supply unit (PSU) B

  5. Connect the other end of the power cables to a primary and secondary AC power source.

    Quantum recommends connecting each power cord to a separate AC circuit to ensure system availability in case of a power failure.

Ensure that your FC network meets the network requirements, and that your FC switch is configured per the information in the following sections:

Ensure that your Ethernet network meets the network requirements for the F1000. See the following:

Before you configure the F1000, power on the system by pressing the Power button in the control panel on the front of the system.

Then, attach the Quantum bezel to the front of the chassis.
  1. Connect to the configuration port. On the F1000, this port is called "M1P2."

    Using an Ethernet cable, connect the Ethernet port on a notebook computer to M1P2 on the F1000.

  2. Enter the IP address and subnet values for the connection using one of the following methods:

    Depending on your macOS version, the following steps might vary. Watch a video on how to set the IPv4 settings on Mac systems.

    1. Select the Apple menu > System Preferences> Network > Configure IPv4, and select the Ethernet connection.
    2. In the Configure IPv4 list, select Manually.

    3. Enter the following IP address and subnet mask values:

      • TCP/IPv4 address: 192.168.17.10
      • Subnet mask: 255.255.255.0
    4. Click Apply.

    5. Close the Network dialog.

      The computer might take up to two minutes to enable the network connection.

    This procedure is for Windows 10. The steps might be different if you are using another operating system. Watch a video on how to set the IPv4 settings on Windows 10 systems.

    1. On your computer, access the Network and Sharing Center control panel. To do this, press the Windows key on the keyboard, type “view network status,” and press Enter.
    2. On the left side of the Network and Sharing Center control panel, click Change adapter settings.
    3. Right-click the network connection that corresponds to the Ethernet port on your computer (for example, Ethernet), and click Properties. The Ethernet Properties window displays.
    4. Select Internet Protocol Version 4 (TCP/IPv4) in the list of connections, and then click Properties. The Internet Protocol Version 4 (TCP/IPv4) Properties window displays.

    5. Select Use the following IP address, then enter the following information:

      • IP address: 192.168.17.10
      • Subnet mask: 255.255.255.0

    6. Click OK, then click Close.

      The computer might take up to two minutes to enable the network connection.

Watch a video on how to use the F1000 Setup Wizard

The networks are configured through the graphical user interface (GUI) and through command line interface (CLI), depending upon the chosen boot protocol network type and the model.

Overall, this part of the process has two steps:

  1. Configure the network through the GUI and CLI, depending upon the model.
  2. If you configured the management IP address with DHCP, determine the assigned IP address through the CLI.

After you complete those steps, you can verify the IP address is functional. These steps are outlined in the flowchart.

FC models have four configuration steps, which are enumerated at the top of the GUI. You do not have to configure a storage network on an FC model because that is supported by the Fibre Channel switch and zone configuration used in your FC network.

Watch a video on how to launch the F1000 Setup Wizard using your browser

  1. Open a browser, and navigate to the initial system configuration interface URL:

    http://192.168.17.21/

    The EULA displays.

    (click image to enlarge)

  2. Click AGREE to accept the EULA.
  1. Enter the same password in the New Password and Confirm Password fields.

    2. (click image to enlarge)

    Note: This password will be used for the root user account to access the F-Series UI.

  2. Click NEXT to accept the password changes.

Configure the management IP address, depending on the desired boot protocol network type.

(click image to enlarge)

  1. In the Management Controller section, select Static in the Boot Protocol field.

    2. (click image to enlarge)

  2. Enter the information in the required fields:

    • IPv4 Address
    • Netmask
    • Default Gateway
  3. Change the MTU value to 9000, if needed.
  4. In the DNS section, enter the following server IP addresses:
    • Primary DNS (required)
    • Alternate DNS (if needed)
  5. Click NEXT to continue.

Note: After you configure the system to use DHCP, you must determine the IP address that was assigned for the management network.

  1. Select DHCP in the Boot Protocol field. DHCP is the default for the management network interfaces for the system.

    2. (click image to enlarge)

  2. Change the MTU value to 9000, if needed (the default MTU value is 1500).

  3. Click NEXT to continue.

Note: After you configure the system to use DHCP, you must determine the IP address that was assigned for the management network.

  1. Select DHCP in the Boot Protocol field. DHCP is the default that is set for the management network interfaces for the system.

    2. (click image to enlarge)

  2. Copy the MAC address for the system for the management port (M1P1) for use in configuring your switch.

    3. Note: The MAC address for the system is pre-populated in the MAC Address field. It is also in the F1000 Customer Site Configuration Data document.

  3. If the port is connected to a network segment with jumbo frames enabled, change the MTU value to 9000. If the network segment is not configured for jumbo frames, the MTU field should remain at 1500.
  4. Click NEXT to continue.
  5. On the DHCP server, register the MAC address for the system and the IP addresses for the management network that you previously defined.
  6. Launch a command line on the server node, and ping the management IP address you just assigned.
  7. Disconnect the network cable from the system, and close the SSH sessions to the system.
  8. Click NEXT to confirm your settings.

Click Apply to confirm settings.

If you did not use DHCP to configure the management network IP, continue to the next step. However, if you configured the management IP address with DHCP, confirm the IP address that DHCP assigned after you confirm your settings in the F-Series UI:

  1. Open an SSH session, such as PuTTY, to the following IP address: 192.168.17.21.

  2. Log into the F1000's command line by entering the following:

    1. At the login as prompt, enter root; then press Enter.
    2. At the root@192.168.17.21's password prompt, enter the current password, then press Enter.

  3. At the [root@node-b8a7d1-1 ~] # prompt, enter ifconfig. Output similar to below displays.

  4. Look for the inet address that is assigned to M1P1. In the example below, look for the m1p1 section; then, find inet addr.

    In this case, the IP address assigned by DHCP is 10.20.216.172.

Continue to: Step 3c: Verify the IP address is functional.

100GbE models have five configuration steps, which are enumerated at the top of the GUI.

  1. Open a browser, and navigate to the initial system configuration interface URL:

    http://192.168.17.21/

    The EULA displays.

    (click image to enlarge)

  2. Click AGREE to accept the EULA.
  1. Enter the same password in the New Password and Confirm Password fields.

    2. (click image to enlarge)

    Note: This password will be used for the root user account to access the F-Series UI.

  2. Click NEXT to accept the password changes.

Configure the management IP address, depending on the desired boot protocol network type.

(click image to enlarge)

  1. In the Management Controller section, select Static in the Boot Protocol field.

    2. (click image to enlarge)

  2. Enter the information in the required fields:

    • IPv4 Address
    • Netmask
    • Default Gateway
  3. Change the MTU value to 9000, if needed.
  4. In the DNS section, enter the following server IP addresses:
    • Primary DNS (required)
    • Alternate DNS (if needed)
  5. Click NEXT to continue.

Note: After you configure the system to use DHCP, you must determine the IP address that was assigned for the management network.

  1. Select DHCP in the Boot Protocol field. DHCP is the default for the management network interfaces for the system.

    2. (click image to enlarge)

  2. Change the MTU value to 9000, if needed (the default MTU value is 1500).

  3. Click NEXT to continue.

Note: After you configure the system to use DHCP, you must determine the IP address that was assigned for the management network.

  1. Select DHCP in the Boot Protocol field. DHCP is the default that is set for the management network interfaces for the system.

    2. (click image to enlarge)

  2. Copy the MAC address for the system for the management port (M1P1) for use in configuring your switch.

    3. Note: The MAC address for the system is pre-populated in the MAC Address field. It is also in the F1000 Customer Site Configuration Data document.

  3. If the port is connected to a network segment with jumbo frames enabled, change the MTU value to 9000. If the network segment is not configured for jumbo frames, the MTU field should remain at 1500.
  4. Click NEXT to continue.
  5. On the DHCP server, register the MAC address for the system and the IP addresses for the management network that you previously defined.
  6. Launch a command line on the server node, and ping the management IP address you just assigned.
  7. Disconnect the network cable from the system, and close the SSH sessions to the system.
  8. Click NEXT to confirm your settings.

The F1000 100 GbE model comes with two 100GbE ports on the system. To use all ports, Quantum supports establishing a dual-subnet storage network. If a single subnet is used, only one interface should be connected and configured.

The two ports on the F1000 100 GbE card use a PCIe gen 3 x 16 connection to the host server. This connection limits the aggregate throughput of those two ports to a maximum of 16 GB/second. A single port (i.e., on a single-subnet configuration) is limited (by 100 Gbps Ethernet) to a maximum of 12.5 GB/second.

If you connected two cables to the 100 GbE card on the system when cabling the F1000, configure the system with a dual subnet.

(click to enlarge)

  1. Select Dual-subnet.
  2. On the right side of the screen, enter the IPv4 and Netmask network settings for the first subnet to be used for the storage network.
  3. If needed, change the MTU setting to 1500.

  4. On the left side of the screen, enter the IPv4 and Netmask network settings for the second subnet used for the storage network.

    (click to enlarge)

  5. If needed, change the MTU setting to 1500.

  6. Click Next.

If you connected one cable to the 100 GbE card when cabling the F1000, configure the system with a single subnet.

(click to enlarge)

  1. Select Single-subnet.
  2. Enter the IPv4 Address and Netmask.
  3. If needed, change the MTU setting to 1500.

  4. Click Next.

After you configure the storage settings in the setup wizard, a confirmation screen displays your settings. See below for an example.

(click to enlarge)

  1. Click Apply to confirm your changes.

    The F-Series Manager log in page appears. However, you are not required to log in to the F-Series Manager at the point.

  2. Disconnect from the configuration port on rear of the F1000, and connect to the StorNext network, if desired.

    Note: You can continue system configuration from another location, provided the system is on a network to which you can connect. Otherwise, you can continue configuration attached to the rear of the system.

    To change your computer's network settings to access the system from within the storage network:

    Depending on your macOS version, the following steps might vary. Watch a video on how to set the IPv4 settings on Mac systems.

    1. Select the Apple menu > System Preferences> Network > Configure IPv4, and select the Ethernet connection.
    2. In the Configure IPv4 list, select Manually.

    3. Enter the following IP address and subnet mask values:

      • TCP/IPv4 address: 192.168.17.10
      • Subnet mask: 255.255.255.0
    4. Click Apply.

    5. Close the Network dialog.

      The computer might take up to two minutes to enable the network connection.

    This procedure is for Windows 10. The steps might be different if you are using another operating system. Watch a video on how to set the IPv4 settings on Windows 10 systems.

    1. On your computer, access the Network and Sharing Center control panel. To do this, press the Windows key on the keyboard, type “view network status,” and press Enter.
    2. On the left side of the Network and Sharing Center control panel, click Change adapter settings.
    3. Right-click the network connection that corresponds to the Ethernet port on your computer (for example, Ethernet), and click Properties. The Ethernet Properties window displays.
    4. Select Internet Protocol Version 4 (TCP/IPv4) in the list of connections, and then click Properties. The Internet Protocol Version 4 (TCP/IPv4) Properties window displays.

    5. Select Use the following IP address, then enter the following information:

      • IP address: 192.168.17.10
      • Subnet mask: 255.255.255.0

    6. Click OK, then click Close.

      The computer might take up to two minutes to enable the network connection.

If you did not use DHCP to configure the management network IP, continue to the next step. However, if you configured the management IP address with DHCP, confirm the IP address that DHCP assigned after you confirm your settings in the F-Series UI:

  1. Open an SSH session, such as PuTTY, to the following IP address: 192.168.17.21.

  2. Log into the F1000's command line by entering the following:

    1. At the login as prompt, enter root; then press Enter.
    2. At the root@192.168.17.21's password prompt, enter the current password, then press Enter.

  3. At the [root@node-b8a7d1-1 ~] # prompt, enter ifconfig. Output similar to below displays.

  4. Look for the inet address that is assigned to M1P1. In the example below, look for the m1p1 section; then, find inet addr.

    In this case, the IP address assigned by DHCP is 10.20.216.172.

Continue to Step 3c: Verify the IP address is functional.

100GbE models have five configuration steps in the GUI, which are enumerated at the top of the screen. After completing setup in the GUI, you can complete setting up the converged/VLAN storage network through a command line interface.

  1. Open a browser, and navigate to the initial system configuration interface URL:

    http://192.168.17.21/

    The EULA displays.

    (click image to enlarge)

  2. Click AGREE to accept the EULA.
  1. Enter the same password in the New Password and Confirm Password fields.

    2. (click image to enlarge)

    Note: This password will be used for the root user account to access the F-Series UI.

  2. Click NEXT to accept the password changes.

Configure the management IP address, depending on the desired boot protocol network type.

(click image to enlarge)

  1. In the Management Controller section, select Static in the Boot Protocol field.

    2. (click image to enlarge)

  2. Enter the information in the required fields:

    • IPv4 Address
    • Netmask
    • Default Gateway
  3. Change the MTU value to 9000, if needed.
  4. In the DNS section, enter the following server IP addresses:
    • Primary DNS (required)
    • Alternate DNS (if needed)
  5. Click NEXT to continue.

Note: After you configure the system to use DHCP, you must determine the IP address that was assigned for the management network.

  1. Select DHCP in the Boot Protocol field. DHCP is the default for the management network interfaces for the system.

    2. (click image to enlarge)

  2. Change the MTU value to 9000, if needed (the default MTU value is 1500).

  3. Click NEXT to continue.

Note: After you configure the system to use DHCP, you must determine the IP address that was assigned for the management network.

  1. Select DHCP in the Boot Protocol field. DHCP is the default that is set for the management network interfaces for the system.

    2. (click image to enlarge)

  2. Copy the MAC address for the system for the management port (M1P1) for use in configuring your switch.

    3. Note: The MAC address for the system is pre-populated in the MAC Address field. It is also in the F1000 Customer Site Configuration Data document.

  3. If the port is connected to a network segment with jumbo frames enabled, change the MTU value to 9000. If the network segment is not configured for jumbo frames, the MTU field should remain at 1500.
  4. Click NEXT to continue.
  5. On the DHCP server, register the MAC address for the system and the IP addresses for the management network that you previously defined.
  6. Launch a command line on the server node, and ping the management IP address you just assigned.
  7. Disconnect the network cable from the system, and close the SSH sessions to the system.
  8. Click NEXT to confirm your settings.

Configuring the storage network in this instance involves both the F-Series UI and the command line interface (CLI).

  1. Select Disable.

  2. Click Next.

    A confirmation screen displays your settings.

Click Apply to confirm settings. The F-Series UI log in page appears. You do not need to log into the F-Series UI, yet.

If you did not use DHCP to configure the management network IP, continue to the next step. However, if you configured the management IP address with DHCP, confirm the IP address that DHCP assigned after you confirm your settings in the F-Series UI:

  1. Open an SSH session, such as PuTTY, to the following IP address: 192.168.17.21.

  2. Log into the F1000's command line by entering the following:

    1. At the login as prompt, enter root; then press Enter.
    2. At the root@192.168.17.21's password prompt, enter the current password, then press Enter.

  3. At the [root@node-b8a7d1-1 ~] # prompt, enter ifconfig. Output similar to below displays.

  4. Look for the inet address that is assigned to M1P1. In the example below, look for the m1p1 section; then, find inet addr.

    In this case, the IP address assigned by DHCP is 10.20.216.172.

The F1000 100 GbE model comes with two 100 GbE ports on the system. Quantum recommends configuring all ports by establishing a dual-subnet storage network. If a single subnet is used, connect and configure only one interface.

Note: The two ports on the system's 100 GbE card use a PCIe gen 3 x16 connection to the host server. This connection limits the aggregate throughput of those two ports to a maximum of 16 GB/second. A single port (i.e., on a single-subnet configuration) is limited (by 100 Gbps Ethernet) to a maximum of 12.5 GB/second.

This section includes steps to configure for networks that will not be a completely dedicated storage network. This configuration is completed through the command line interface (CLI), rather than through the F-Series UI.

  • An SSH client like the macOS terminal, Windows 10 command prompt, or PuTTY
  • A list of the network connections you want to configure
  1. Make sure you have access to the pre-defined network settings before configuring the converged/VLAN network.
  2. Open multiple SSH sessions at the same time. Since you need SSH access for multiple systems—one for the F1000 system, and one for each host system server node—opening multiple SSH client sessions will help complete configuration quicker, rather than using a single SSH session.
  3. When server nodes are rebooted, restart the SSH session for each node.

Once the IPv4 address on a lossless channel are established, iSCSI access to the F1000 must be enabled.

  • As manufactured, three IQNs are predefined on the system: one for SCSI command forwarding (i.e., non-optimal path), and two for array access (one for each port). The former must not be changed; the latter must be bound to the chosen IP addresses.
  • Configure the IP address from the command line of the system.

Change the configuration of the iSCSI target driver:

  1. Log in to the command line of the F1000:
    1. On a laptop, login using the following syntax:
      ssh <user account >@<system IP address>
    2. Open an SSH to the command line of the system using the management IP address of the system.

      3. Example (macOS ):

      #ssh root@10.20.85.72

      Example (Windows 10):

      c:\>ssh root@10.20.85.72

    3. Log in to the command line:

      • Username: root
      • Password: <password>

  2. Back up the /etc/scst.conf file by entering the following:

    # cp /etc/scst.conf /etc/scst.conf.bak
  3. Edit the /etc/scst.conf file using vi:
  4. Use vi to open the /etc/scst.conf file. Enter:

    # vi /etc/scst.conf

  5. Search for allowed_portal. Enter:

    /allowed_portal

    6. The section will look similar to the following. The IP address you will change in the next step is marked in bold green text:

    ...
TARGET_DRIVER iscsi {
	enabled 1

	TARGET iqn.2020-03.esos.node-6261c4-1:123ee {
		allowed_portal 192.168.1.100
		enabled 1
		rel_tgt_id 2

		GROUP default {
			LUN 0 F1000_6261c4_01
			LUN 1 F1000_6261c4_02


			INITIATOR*
		}
	}

	TARGET iqn.2020-03.esos.node-6261c4-1:43c2b {
		allowed_portal 192.168.2.100
		enabled 1
		rel_tgt_id 1

		GROUP default {
			LUN 0 F1000_6261c4_01
			LUN 1 F1000_6261c4_02

			INITIATOR *
		}
	}
  6. Type i to enter the vi INSERT mode.
  7. Position your cursor, and change the IP to the IPv4 address to be used for the storage network. For example: 192.168.1.100.

    8. Note: Use the Up Arrow, Down Arrow, Right Arrow, and Left Arrow keys to move the cursor as needed.

  8. Type ESC to exit the vi INSERT mode.
  9. Search for allowed_portal again by entering the following:

    /allowed_portal

    10. The section will look similar to the following. The IP address you will change in the next step is marked in bold green text: 

    ...
	TARGET iqn.2020-03.esos.node-6261c4-1:43c2b {
		allowed_portal 192.168.2.100
		enabled 1
		rel_tgt_id 1

		GROUP default {
			LUN 0 F1000_6261c4_01
			LUN 1 F1000_6261c4_02

			INITIATOR *
		}
	}
}

...
  10. Type i to enter the vi INSERT mode.

  11. Position your cursor, and change the IP to the IPv4 address to be used for the storage network. For example: 192.168.2.100.

    Note: Use the Up Arrow, Down Arrow, Right Arrow, and Left Arrow keys to move the cursor as needed.

  12. Press ESC to exit INSERT mode.
  13. Enter the following to save the file:

    :wq!

  14. Update the F1000 configuration. Enter:

    # conf_sync.sh

  15. To make the new F1000 configuration active, enter:

    # scstadmin -config /etc/scst.conf

DCBX (Data Center Bridging capability eXchange protocol) must be enabled. This allows much of the DCB configuration (PFC and ETS) to be performed once on the Ethernet storage network, which can be received and used by the F1000 and its clients. To enable DCBX on the Mellanox cards:

  1. Log in to the command line of the F1000:
    1. On a laptop, login using the following syntax:
      ssh <user account >@<system IP address>
    2. Open an SSH to the command line of the system using the management IP address of the system.

      3. Example (macOS ):

      #ssh root@10.20.85.72

      Example (Windows 10):

      c:\>ssh root@10.20.85.72

    3. Log in to the command line:

      • Username: root
      • Password: <password>

  2. Run the following command to find the Ethernet interfaces (ports) and devices (NICs). PCI device names ("/sys/bus/pci/devices/...") associated with the Mellanox NICs are shown, as well as how they are set up for passing DCBX. In this example, the first NIC (interfaces S1P1 and S1P2) is set correctly. The second NIC (interfaces S2P1 and S2P2) is not: 

    # echo; echo "======== ConnectX-5 Interfaces (Ports) ========";\
  for ei in `lshw -class network -businfo | tr -s ' ' ' ' | grep "ConnectX-5" | cut -d ' ' -f 2`; do \
  echo "$ei:"; ethtool $ei | grep -e "Speed:" -e "Link detected:"; done; \
  echo; echo "======== ConnectX-5 Devices (NICs) ========";\
  mstconfig query | grep -i -e lldp -e device; echo

======== ConnectX-5 Interfaces (Ports) ========
s1p1:
	Speed: 100000Mb/s
	Link detected: yes
s1p2:
	Speed: Unknown!
	Link detected: no
s2p1:
	Speed: 100000Mb/s
	Link detected: yes
s2p2:
	Speed: Unknown!
	Link detected: no

======== ConnectX-5 Devices (NICs) ========
Device #1:
Device type:    ConnectX5
Device:         /sys/bus/pci/devices/0000:65:00.0/config
	LLDP_NB_DCBX_P1                     True(1)
	LLDP_NB_RX_MODE_P1                  ALL(2)
	LLDP_NB_TX_MODE_P1                  ALL(2)
	LLDP_NB_DCBX_P2                     True(1)
	LLDP_NB_RX_MODE_P2                  ALL(2)
	LLDP_NB_TX_MODE_P2                  ALL(2)
Device #2:
Device type:    ConnectX5
Device:         /sys/bus/pci/devices/0000:21:00.0/config
	LLDP_NB_DCBX_P1                     False(0)
	LLDP_NB_RX_MODE_P1                  OFF(0)
	LLDP_NB_TX_MODE_P1                  OFF(0)
	LLDP_NB_DCBX_P2                     True(1)
	LLDP_NB_RX_MODE_P2                  OFF(0)
	LLDP_NB_TX_MODE_P2                  OFF(0)

  3. Run the following command to configure the second NIC to pass DCBX configuration data:

    # mstconfig -d /sys/bus/pci/devices/0000:21:00.0/config set \
		LLDP_NB_DCBX_P1=TRUE LLDP_NB_TX_MODE_P1=2 LLDP_NB_RX_MODE_P1=2 \
		LLDP_NB_DCBX_P2=TRUE LLDP_NB_TX_MODE_P2=2 LLDP_NB_RX_MODE_P2=2

Device #1:
----------

Device type:    ConnectX5
Name:           N/A
Description:    N/A
Device:         /sys/bus/pci/devices/0000:21:00.0/config

Configurations:                              Next Boot       New
	LLDP_NB_DCBX_P1                     False(0)        True(1)
	LLDP_NB_TX_MODE_P1                  OFF(0)          ALL(2)
	LLDP_NB_RX_MODE_P1                  OFF(0)          ALL(2)
	LLDP_NB_DCBX_P2                     False(0)        True(1)
	LLDP_NB_TX_MODE_P2                  OFF(0)          ALL(2)
	LLDP_NB_RX_MODE_P2                  OFF(0)          ALL(2)

 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.
  4. Repeat this for each Mellanox NIC (not each port) that is not configured correctly on the F1000, then reboot the system before continuing. Enter the following:

    # reboot

  5. After the reboot, confirm specific configuration from the network (assuming the network was configured prior to the F1000 array) using the following command:

    # mlnx_qos -i S1P1
****** WARNING: lldpad service is running and may overwrite your settings ******

DCBX mode: OS controlled
Priority trust state: pcp
Receive buffer size (bytes): 130944,130944,0,0,0,0,0,0,
Cable len: 7
PFC configuration:
	priority    0   1   2   3   4   5   6   7
	enabled     0   0   0   0   0   1   0   0
	buffer      0   0   0   0   0   1   0   0
tc: 0 ratelimit: unlimited, tsa: strict
	priority:  0
	priority:  1
	priority:  2
	priority:  3
	priority:  4
	priority:  5
	priority:  6
	priority:  7
  6. Depending on the exact configuration of the network, the output may vary. From this output, verify:
    • The lldpad service is running. By default, the F1000 uses 'lldpad', a daemon that supports LLDP and its extension, DCBX.
    • The DCBX mode is OS controlled. This says that the Mellanox driver receives its configuration information from the OS (actually from 'lldpad').
    • PFC priority 5 is both enabled and buffered. This means that the Ethernet storage network (and switch supporting this NIC) is configured for PFC priority 5 (presumably on VLAN 100).

In this example, we will define and configure VLANs S1P2.100 and S2P2.100. This involves accessing a text-based user interface (TUI), then configuring both physical interfaces with a DHCP IP address, and the both virtual interfaces to a static IP address.

  1. Add S1P2.100 and S2P2.100 VLAN interfaces to the system. To add the first VLAN interface, enter:

    2. # vconfig add S1P2 100
  2. To add the second VLAN interface, enter:

    3. # vconfig add S2P2 100
  3. Open the QCSP TUI of the system. At the /usr/local/bin/ prompt, enter esos_tui.

  4. Configure both physical interfaces to a DHCP address by completing the following steps for S1P2 and S2P2:

    1. In the QCSP TUI root screen, press the <s> key to display the System menu:

    2. Use the Up and Down arrow keys to select Network Settings, then press Enter.

    3. In the Choose a Network Configuration Option box, select the parent interface that you want to configure, then press Enter.

      The Configuring Interface [Name] box appears.

    4. In the Interface MTU field of the Configuring Interface [Name] box, enter 9000.

    5. In the IP Settings area, select DHCP.
    6. Click OK.
    7. A dialog box asks if you want to restart the network now; click No. The QCSP TUI returns the root screen.

  5. Configure both virtual interfaces to a static IP address by completing the following steps for S1P2.100 and S2P2.100:

    1. In the QCSP TUI root screen, press the <s> key to display the System menu:

    2. Use the Up and Down arrow keys to select Network Settings, then press Enter.

    3. In the Choose a Network Configuration Option box, select the virtual interface that you want to configure, then press Enter.

      The Configuring Interface [Name] box appears.

    4. In the Interface MTU field of the Configuring Interface [Name] box, enter 9000.

    5. In the IP Settings area, select Static.

    6. In the Static IPv4 Settings area, enter the following values:

      • In the IP address field, enter 192.168.1.100.
      • In the Netmask field, enter 255.255.255.0.
      • In the Broadcast field, enter 192.168.1.255.
    7. Click OK.
    8. A dialog box asks if you want to restart the network now; click No. The QCSP TUI returns the root screen.
  6. In the QCSP TUI root screen, press the <s> key to display the System menu.
  7. Use the Up and Down arrow keys to select Sync Configuration, then press Enter. This synchronizes the configuration settings.

  8. In the QCSP TUI root screen, press the <i> key to display the Interface menu.

  9. Use the Up and Down arrow keys to select Quit, then press Enter. The command prompt appears.

  10. Go back to the QCSP/TUI session, and reboot the system. Enter:

    # reboot
  11. Wait at least 30 seconds F1000 to reboot.

When the reboot completes, enable DCB features for each VLAN by setting up the egress mapping on that VLAN.

  1. Open an SSH session to the F1000.
  2. Enter:
    3. # set_egress_map -v 100 -m 0:5
skprio2up is available only for RoCE in kernels that don't
support set_egress_map
Traffic classes are set to 8
UP  0
UP  1
UP  2
UP  3
UP  4
UP  5
	skprio: 0
	skprio: 0 (vlan 100)
UP  6
UP  7
skprio2up is available only for RoCE in kernels that don't
support set_egress_map
Traffic classes are set to 8
UP  0
UP  1
UP  2
UP  3
UP  4
UP  5
	skprio: 0
	skprio: 0 (vlan 100)
UP  6
UP  7

    At this point, VLAN 100 is enabled for lossless transmission using PFC priority 5.

  3. When the F1000 reboots, these settings will be lost—along with the performance they enable—unless you save the settings. To save the settings, add the information to the /etc/rc.local file. For example:
    4. # cat /etc/rc.local
#! /bin/sh

# Send a system startup email
#/usr/local/sbin/startup.sh > /dev/null 2>&1 &

/opt/sbin/set_egress_map -v 100 -m 0:5
  4. Make the /etc/rc.local file executable by root. Run the following commands:
    5. # ls -l /etc/rc.local
-rw-r--r--    1 root     root           129 Nov  8 18:20 /etc/rc.local

# chmod 755 /etc/rc.local

# ls -l /etc/rc.local
-rwxr-xr-x    1 root     root           129 Nov  8 18:20 /etc/rc.local

    The settings will now persist across F1000 reboots.

  5. Close the SSH session to exit the command line.

These steps, in conjunction with correct configuration of the storage network, should result in a VLAN capable of lossless transmission, suitable for RDMA traffic. If you encounter issues, use a diagnostic script, netdiag, to attempt to locate the problem.

  • netdiag does not support arguments.
  • netdiag iterates through major elements of network configuration, pausing strategically after sections of related information that should be carefully examined for accuracy. The user is prompted to move from section to section.

Here is the output of netdiag, by section:

The first section provides the ability to determine whether the NICs are configured correctly to pass DCBX protocol. The system under examination has two NICs. Both are configured correctly.

     1  «screen is cleared»
     2  ======================================================================
     3  == Diagnostic data for 5e:00
     4  ======================================================================
     5  Device type:    ConnectX5
     6           LLDP_NB_DCBX_P1                     True(1)
     7           LLDP_NB_RX_MODE_P1                  ALL(2)
     8           LLDP_NB_TX_MODE_P1                  ALL(2)
     9           LLDP_NB_DCBX_P2                     True(1)
    10           LLDP_NB_RX_MODE_P2                  ALL(2)
    11           LLDP_NB_TX_MODE_P2                  ALL(2)
    12
    13  ======================================================================
    14  == Diagnostic data for d8:00
    15  ======================================================================
    16  Device type:    ConnectX5
    17           LLDP_NB_DCBX_P1                     True(1)
    18           LLDP_NB_RX_MODE_P1                  ALL(2)
    19           LLDP_NB_TX_MODE_P1                  ALL(2)
    20           LLDP_NB_DCBX_P2                     True(1)
    21           LLDP_NB_RX_MODE_P2                  ALL(2)
    22           LLDP_NB_TX_MODE_P2                  ALL(2)

This section provides a view of a network interface, any VLANs defined on it, and all egress mappings. The examined interface has a VLAN (100), with an egress mapping (0:5). Items to look for include:

  • Both parent interface and VLAN are "UP BROADCAST RUNNING MULTICAST" (lines 28 & 39) -- necessary for an active network interface, (one that is plugged into a correctly-configured network switch).
  • MTU on the parent interface (line 28) is greater than or equal to the largest MTU (line 39) of any VLAN defined on this interface.
  • The "EGRESS priority mappings" (line 45) are as expected
  • User priority (UP) 5 (from 0:5) is associated with skprio 0 (again, from 0:5) for VLAN 100 (lines 52 & 53).
    23  «screen is cleared»
24  ======================================================================
25  == Diagnostic data for S1P1
26  ======================================================================
27  S1P1 Link encap:Ethernet  HWaddr B8:59:9F:13:A6:4C
28            UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
29            RX packets:82184 errors:0 dropped:0 overruns:0 frame:0
30            TX packets:13843 errors:0 dropped:0 overruns:0 carrier:0
31            collisions:0 txqueuelen:1000
32            RX bytes:14882707 (14.1 MiB)  TX bytes:3241331 (3.0 MiB)
33
34  <<< VLAN S1P1.100 >>>
35
36  S1P1.100 Link encap:Ethernet  HWaddr B8:59:9F:13:A6:4C
37            inet addr:192.168.1.104  Bcast:192.168.1.255  Mask:255.255.255.0
38            inet6 addr: fe80::ba59:9fff:fe13:a64c/64 Scope:Link
39            UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
40            RX packets:3995 errors:0 dropped:0 overruns:0 frame:0
41            TX packets:1368 errors:0 dropped:0 overruns:0 carrier:0
42            collisions:0 txqueuelen:1000
43            RX bytes:167790 (163.8 KiB)  TX bytes:45318 (44.2 KiB)
44
45   EGRESS priority mappings: 0:5
46  Traffic classes are set to 8
47  UP  0
48  UP  1
49  UP  2
50  UP  3
51  UP  4
52  UP  5
53          skprio: 0 (vlan 100)
54  UP  6

55  UP  7

This section provides details about the link established for this network interface. Critical items include:

  • Link speed (line 58) is as expected.
  • Link-layer flow control (lines 62 & 63) is as expected. Link layer flow control must be 'off' for converged storage networks (i.e., if DCB features are used to create a lossless network). By contrast, link-layer flow control must be 'on' for dedicated storage networks.
    56  <<< Link Characteristics >>>
57
58          Speed: 100000Mb/s
59          Link detected: yes
60  Pause parameters for S1P1:
61  Autonegotiate:  off
62  RX:             off
63  TX:             off
64

This section provides some statistics for the network interface. Zero values are suppressed. Critical items include:

  • If you are using PFC priority 5 (from 0:5), you should see some non-zero statistics with "prio5" in their name; and they should increase over time in proportion to how busy the channel is.
    65  <<< RDMA & Flow Control Statistics >>>
    66
    67       rx_vport_rdma_unicast_packets: 104051668
    68       rx_vport_rdma_unicast_bytes: 425987567612
    69       tx_vport_rdma_unicast_packets: 5858739
    70       tx_vport_rdma_unicast_bytes: 507634526
    71       tx_pause_ctrl_phy: 7503738
    72       rx_prio0_bytes: 15227095
    73       rx_prio0_packets: 81969
    74       tx_prio0_bytes: 3279949
    75       tx_prio0_packets: 13580
    76       rx_prio5_bytes: 426403824212
    77       rx_prio5_packets: 104052442
    78       tx_prio5_bytes: 531086054
    79       tx_prio5_packets: 5859001
    80       tx_prio5_pause: 7503738
    81       tx_prio5_pause_duration: 25771160
    82

This section provides a view of the DCB features imposed by the storage network. Critical items include:

  • The F1000 should always run the "lldpad service" (line 85), and the DCBX mode should always be "OS controlled" (line 87).

  • If you think you are using PFC priority 5, and it's not enabled (lines 92 & 93) somebody, somewhere made a mistake.

  • If the storage network you are using has explicit QoS bandwidth reservation policies set, make sure you see them reflected in this section (lines 96 - 103). In this example, none are set.

    83  <<< DCB Configuration >>>
    84
    85  ****** WARNING: lldpad service is running and may
overwrite your settings ******
    86
    87  DCBX mode: OS controlled
    88  Priority trust state: pcp
    89  Receive buffer size (bytes): 130944,130944,0,0,0,0,0,0,
    90  Cable len: 7
    91  PFC configuration:
    92          priority    0   1   2   3   4   5   6   7
    93          enabled     0   0   0   0   0   1   0   0
    94          buffer      0   0   0   0   0   1   0   0
    95  tc: 0 ratelimit: unlimited, tsa: strict
    96           priority:  0
    97           priority:  1
    98           priority:  2
    99           priority:  3
    100           priority:  4
    101           priority:  5
    102           priority:  6
    103           priority:  7
    104

This section provides information about to what switch, and which port on that switch, this interface is connected.

	105  <<< Connected To >>>
	106
	107  System Name DCS-7280CR2A-60-F
	108  System Description Arista Networks EOS version
4.22.1FX-CLI running on an Arista Networks DCS-7280CR2A-60
	109  Management Address IPv4: 10.65.189.58
	110  Port ID Ifname: Ethernet32/1

The remaining three sections provide similar detail for network interfaces S1P2 (which is not connected, in this case), S2P1 (which is connected), and S2P2 (which is not connected).

   111  «screen is cleared»
   112  ======================================================================
   113  == Diagnostic data for S1P2
   114  ======================================================================
   115  S1P2 Link encap:Ethernet  HWaddr B8:59:9F:13:A6:4D
   116            UP BROADCAST MULTICAST  MTU:1500  Metric:1
   117            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
   118            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
   119            collisions:0 txqueuelen:1000
   120            RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
   121
   122  Traffic classes are set to 8
   123  UP  0
   124  UP  1
   125  UP  2
   126  UP  3
   127  UP  4
   128  UP  5
   129  UP  6
   130  UP  7
   131  <<< Link Characteristics >>>
   132
   133          Speed: Unknown!
   134          Link detected: no
   135  Pause parameters for S1P2:
   136  Autonegotiate:  off
   137  RX:             on
   138  TX:             on
   139
   140  <<< RDMA & Flow Control Statistics >>>
   141
   142
   143  <<< DCB Configuration >>>
   144
   145  ****** WARNING: lldpad service is running and may overwrite your settings ******
   146
   147  DCBX mode: OS controlled
   148  Priority trust state: pcp
   149  Receive buffer size (bytes): 262016,0,0,0,0,0,0,0,
   150  Cable len: 7
   151  PFC configuration:
   152          priority    0   1   2   3   4   5   6   7
   153          enabled     0   0   0   0   0   0   0   0
   154          buffer      0   0   0   0   0   0   0   0
   155  tc: 1 ratelimit: unlimited, tsa: vendor
   156           priority:  0
   157  tc: 0 ratelimit: unlimited, tsa: vendor
   158           priority:  1
   159  tc: 2 ratelimit: unlimited, tsa: vendor
   160           priority:  2
   161  tc: 3 ratelimit: unlimited, tsa: vendor
   162           priority:  3
   163  tc: 4 ratelimit: unlimited, tsa: vendor
   164           priority:  4
   165  tc: 5 ratelimit: unlimited, tsa: vendor
   166           priority:  5
   167  tc: 6 ratelimit: unlimited, tsa: vendor
   168           priority:  6
   169  tc: 7 ratelimit: unlimited, tsa: vendor
   170           priority:  7
   171
   172  <<< Connected To >>>
   173
   174
   175
   176
   177
   178  «screen is cleared»
   179  ======================================================================										180  == Diagnostic data for enp216s0f0
   181  ======================================================================
   182  S2P1 Link encap:Ethernet  HWaddr 50:6B:4B:B6:C9:70
   183            UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
   184            RX packets:82187 errors:0 dropped:0 overruns:0 frame:0
   185            TX packets:13843 errors:0 dropped:0 overruns:0 carrier:0
   186            collisions:0 txqueuelen:1000
   187            RX bytes:14884082 (14.1 MiB)  TX bytes:3240525 (3.0 MiB)
   188
   189  <<< VLAN S2P1.100 >>>
   190
   191  S2P1.100 Link encap:Ethernet  HWaddr 50:6B:4B:B6:C9:70
   192            inet addr:192.168.2.104  Bcast:192.168.2.255  Mask:255.255.255.0
   193            inet6 addr: fe80::526b:4bff:feb6:c970/64 Scope:Link
   194            UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
   195            RX packets:3993 errors:0 dropped:0 overruns:0 frame:0
   196            TX packets:1368 errors:0 dropped:0 overruns:0 carrier:0
   197            collisions:0 txqueuelen:1000
   198            RX bytes:167706 (163.7 KiB)  TX bytes:45318 (44.2 KiB)
   199
   200   EGRESS priority mappings: 0:5
   201  Traffic classes are set to 8
   202  UP  0
   203  UP  1
   204  UP  2
   205  UP  3
   206  UP  4
   207  UP  5
   208          skprio: 0 (vlan 100)
   209  UP  6
   210  UP  7
   211  <<< Link Characteristics >>>
   212
   213          Speed: 100000Mb/s
   214          Link detected: yes
   215  Pause parameters for S2P1:
   216  Autonegotiate:  off
   217  RX:             off
   218  TX:             off
   219
   220  <<< RDMA & Flow Control Statistics >>>
   221
   222       rx_vport_rdma_unicast_packets: 104052697
   223       rx_vport_rdma_unicast_bytes: 425987658466
   224       tx_vport_rdma_unicast_packets: 5856107
   225       tx_vport_rdma_unicast_bytes: 507486282
   226       tx_pause_ctrl_phy: 8687797
   227       rx_prio0_bytes: 15227823
   228       rx_prio0_packets: 81968
   229       tx_prio0_bytes: 3279221
   230       tx_prio0_packets: 13581
   231       rx_prio5_bytes: 426403919260
   232       rx_prio5_packets: 104053472
   233       tx_prio5_bytes: 530927204
   234       tx_prio5_packets: 5856368
   235       tx_prio5_pause: 8687797
   236       tx_prio5_pause_duration: 29746538
   237
   238  <<< DCB Configuration >>>
   239
   240  ****** WARNING: lldpad service is running and may overwrite your settings ******
   241
   242  DCBX mode: OS controlled
   243  Priority trust state: pcp
   244  Receive buffer size (bytes): 130944,130944,0,0,0,0,0,0,
   245  Cable len: 7
   246  PFC configuration:
   247          priority    0   1   2   3   4   5   6   7
   248          enabled     0   0   0   0   0   1   0   0
   249          buffer      0   0   0   0   0   1   0   0
   250  tc: 0 ratelimit: unlimited, tsa: strict
   251           priority:  0
   252           priority:  1
   253           priority:  2
   254           priority:  3
   255           priority:  4
   256           priority:  5
   257           priority:  6
   258           priority:  7
   259
   260  <<< Connected To >>>
   261
   262  System Name DCS-7280CR2A-60-F
   263  System Description Arista Networks EOS version 4.22.1FX-CLI running on an 
Arista Networks DCS-7280CR2A-60
   264  Management Address IPv4: 10.65.189.58
   265  Port ID Ifname: Ethernet31/1
   266  «screen is cleared»
267  ======================================================================
268  == Diagnostic data for S2P2
269  ======================================================================
270  S2P2 Link encap:Ethernet  HWaddr 50:6B:4B:B6:C9:71
271            UP BROADCAST MULTICAST  MTU:1500  Metric:1
272            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
273            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
274            collisions:0 txqueuelen:1000
275            RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
276
277  Traffic classes are set to 8
278  UP  0
279  UP  1
280  UP  2
281  UP  3
282  UP  4
283  UP  5
284  UP  6
285  UP  7
286  <<< Link Characteristics >>>
287
288          Speed: Unknown!
289          Link detected: no
290  Pause parameters for S2P2:
291  Autonegotiate:  off
292  RX:             on
293  TX:             on
294
295  <<< RDMA & Flow Control Statistics >>>
296
297
298  <<< DCB Configuration >>>
299
300  ****** WARNING: lldpad service is running and may overwrite your settings ******
301
302  DCBX mode: OS controlled
303  Priority trust state: pcp
304  Receive buffer size (bytes): 262016,0,0,0,0,0,0,0,
305  Cable len: 7
306  PFC configuration:
307          priority    0   1   2   3   4   5   6   7
308          enabled     0   0   0   0   0   0   0   0
309          buffer      0   0   0   0   0   0   0   0
310  tc: 1 ratelimit: unlimited, tsa: vendor
311           priority:  0
312  tc: 0 ratelimit: unlimited, tsa: vendor
313           priority:  1
314  tc: 2 ratelimit: unlimited, tsa: vendor
315           priority:  2
316  tc: 3 ratelimit: unlimited, tsa: vendor
317           priority:  3
318  tc: 4 ratelimit: unlimited, tsa: vendor
319           priority:  4
320  tc: 5 ratelimit: unlimited, tsa: vendor
321           priority:  5
322  tc: 6 ratelimit: unlimited, tsa: vendor
323           priority:  6
324  tc: 7 ratelimit: unlimited, tsa: vendor
325           priority:  7
326
327  <<< Connected To >>>
328
329
330
331
332

This section provides high-level guidance for configuring 100 GbE connections.

The iSCSI plugin in Appliance Controller 2.3.3 contains iscsiadm command support. The Appliance Controller software runs on the host StorNext Appliance. The updates include support for configuration of converged storage networks using iSER. Upgrade your system to Appliance Controller 2.3.3 or later before continuing. See Appliance Controller Upgrades for instructions for upgrading the Appliance Controller software on your host StorNext Appliance.

For this procedure, configure the following, using the Appliance Controller on the host StorNext Appliance for the F1000:

  • Configure ports on the host StorNext Appliance as iSCSI initiators.
  • F1000 ports are configured as iSCSI targets for the host StorNext Appliance.

Choose a unique VLAN ID to use for the configuration, including:

  • Initiator node interfaces must be configured with the selected VLAN ID.
  • Target node interfaces must be configured with the selected VLAN ID, if supported. Otherwise, configure the target interfaces as usual, without use of VLAN.

Complete additional, required switch configuration steps:

  1. Create a VLAN with the selected VLAN ID.
  2. Add the switch ports connecting to the initiator and the targets interfaces to the VLAN.

  3. Turn on Priority Flow Control (PFC).

  4. Activate PFC for the ports of the VLAN group.

  1. Open an SSH connection to the host StorNext Appliance.
  2. Log in as the sysadmin user account.
  3. At the password prompt, enter the sysadmin user's password (the default is sysadmin) to log in to the Appliance Controller.

    4. Example

    login as: sysadmin

    sysadmin@quantum's password:

    Last login: Fri Dec 7 13:37:11 2018 from eng.example.com

    Welcome to Quantum Xcellis Appliance Controller Console

    ---------------------------------------------------

    *** Type 'help' for a list of commands.

    Xcellis:xcellis1>

    For additional options for logging in to the Appliance Controller console, see Access the Appliance Controller Console in the Appliance Controller Documentation Center.

  1. Configure the required iSER initiators. See Create a TCP or iSER iSCSI Initiator in the Appliance Controller Doc Center.
  2. Configure the required targets. See Create TCP or iSER iSCSI Targets in the Appliance Controller Doc Center.

    3. At this point, you can interact with iSCSI in the following ways:

Verify that you properly attached the storage in the following ways:

Verify iSER sessions

See Show iSCSI Configurations in the Appliance Controller Doc Center.

You should see either two (single subnet) or four (dual subnet) of these on the F1000. If not, follow the instructions below.

Verify block devices

You will see the LUNs for the F1000 system. Enter the following to verify the path to them:

lsscsi -s

[...]

[1:0:0:0] disk QUANTUM F1000_138600_01 200 /dev/sdi 9.55TB

[1:0:0:1] disk QUANTUM F1000_138600_02 200 /dev/sdm 9.55TB

Verify Multipath and ALUA

You should see two multipaths (for each F1000 system)—one for each LUN—not in any particular order. Each multipath should have two active paths and two non-optimal paths for a dual-subnet configuration (one and one for a single-subnet configuration). Enter:

multipath -ll -v2

mpathr (26661393137643465) dm-3 QUANTUM ,F1000_138600_02

size=8.7T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| |- 11:0:0:1 sdk 8:160 active ready running

| `- 12:0:0:1 sdl 8:176 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

|- 9:0:0:1 sdm 8:192 active ready running

`- 10:0:0:1 sdn 8:208 active ready running

mpathq (26365373664356437) dm-4 QUANTUM ,F1000_138600_01

size=8.7T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| |- 9:0:0:0 sdg 8:96 active ready running

| `- 10:0:0:0 sdj 8:144 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

|- 11:0:0:0 sdh 8:112 active ready running

`- 12:0:0:0 sdi 8:128 active ready running

Verify SNFS labels

Query the SNFS labels. The LUNs are not labeled, yet. Enter:

cvlabel -l | grep F1000

/dev/mapper/mpathcf [QUANTUM F1000_6261c4_01 210 ] SNFS-EFI "F1000_6261c4_01"Sectors: 9358193915. Sector Size: 4096. Maximum sectors: 9358193915. Stripebreadth: 0.

/dev/mapper/mpathck [QUANTUM F1000_6261c4_02 210 ] SNFS-EFI "F1000_6261c4_02"Sectors: 9358193915. Sector Size: 4096. Maximum sectors: 9358193915. Stripebreadth: 0.

Continue to: Step 3c: Verify the IP address is functional.

To confirm the IP address is functional, enter it into your browser, and ensure the F-Series Manager displays.

Multiple paths run from a StorNext client to an F1000. These paths must be organized into a multipath to provide optimal performance and availability. To organize this multipath, the multipath.conf file on both host server nodes must be updated with F1000 device settings.

To complete this procedure, access the command line on the host server node.

Note: For additional information about the multipath.conf file, refer to Modify the multipath.conf File for External Storage Arrays on www.quantum.com.

Note: Make sure to complete this process on both host server nodes.

To update the multipath.conf file:

1. Open an SSH connection on the host server node.
2. Log in to the command line using the following credentials:

User name: stornext

Password: <stornext user account password>

Note: Unless the password changed, password is the default password for the StorNext user account.

3. Enter sudo rootsh to gain root user access.
4. Enter the password for the root user account.
5. Make a backup copy of the multipath.conf file:

# cp /etc/multipath.conf /etc/multipath.conf.bak
6. Use vi to open the multipath.conf file by entering the following command:

# vi /etc/multipath.conf
7. Search for the term “blacklist_exceptions” by entering the following command:

/blacklist_exceptions

The location of the “blacklist_exceptions” text depends on the operating system (OS). In RedHat Linux, it looks similar to the following example:

# Exceptions to the default blacklist of all devices

#

blacklist_exceptions {
8. Use the Down Arrow key to move the cursor to the end of last device in the “blacklist_exceptions” section.
9. Type i to enter the vi INSERT mode.
10. Position your cursor by using the arrow keys, then add the following new F1000 device information (shown in bold below) to the “blacklist_exceptions” section.

Note: Before adding new F1000 device information, scroll through to see if it already exists.

# Exceptions to the default blacklist of all devices
#
blacklist_exceptions {
	. . .
	device {
	     vendor "QUANTUM"
	     product "^(F|P|H).*"
	}

11. Type ESC to exit INSERT mode.
12. Search for the term “devices” by entering the following command:

/devices

The beginning of the “devices” section resembles the following example:

devices {
        device {
                vendor...
13. Use the Down Arrow key to move the cursor to the end of the “devices” section.
14. Type i to enter the vi INSERT mode.
15. Position your cursor using the arrow keys, then add the following new device information (shown in bold below) to the “devices” section:

Note: Before adding new F1000 device information, scroll through to see if it already exists.

devices {

. . .

 

	  device {
	       vendor			"QUANTUM"
	       product			"^(F)1.*"
	       path_grouping_policy	group_by_serial
	       features			"0"
	       path_checker		tur
	       path_selector		"round-robin 0"
	       hardware_handler		"0"
	       rr_weight		uniform
	       no_path_retry		30
	       failback			immediate
	       rr_min_io_rq		1
	  }

16. Press ESC to exit INSERT mode.
17. Enter the following to save the file:

:wq!
18. Reload the multipath.conf file:

# systemctl reload multipathd
19. Enter the following command to verify that each F1000 volume has only one path:

# /usr/cvfs/bin/cvlabel -l | grep F1000
20. Enter the following command to view additional information about the number and type of paths that make up the multipath configuration:

# multipath -ll -v2

Paths that show “status=active” are optimal (and in use). Paths that show “status=enabled” are non-optimal (failover) paths.

21. Open another SSH session, and complete this procedure on the other node.

Appliance Controller 2.3.3 and later includes the qstorage plugin that monitors the health of the F1000. Before the qstorage plugin can monitor the F1000, you must run a command that assigns the management IP addresses of the F1000 system to both host server nodes.

Access the Appliance Controller on the host system server to complete this process.

Configure health monitoring for the storage array

To configure health monitoring for the F1000 storage, you must use the qstorage assign command.

Do the following on both host server nodes (watch a video on how to assign QXS storage to the F1000):

  1. Open an SSH connection to the host StorNext Appliance.
  2. Log in as the sysadmin user account.
  3. At the password prompt, enter the sysadmin user's password (the default is sysadmin) to log in to the Appliance Controller.

    4. Example

    login as: sysadmin

    sysadmin@quantum's password:

    Last login: Fri Dec 7 13:37:11 2018 from eng.example.com

    Welcome to Quantum Xcellis Appliance Controller Console

    ---------------------------------------------------

    *** Type 'help' for a list of commands.

    Xcellis:xcellis1>

    For additional options for logging in to the Appliance Controller console, see Access the Appliance Controller Console in the Appliance Controller Documentation Center.

  1. Find the management IP address that was assigned to system during initial system configuration.
  2. In the Appliance Controller console, enter the following command to assign the F1000 system IP address to the host server node, where <mgmt_ipaddr> is the management IP address for the system. The default user name is root.

    qstorage assign <mgmt_ipaddr> <user name>

    Example:

    qstorage assign 10.65.168.182 root

  3. Enter the password that you created when you configured your F1000.

    At this point, the system returns output similar to the following example, This indicates that the F1000 system management IP address was assigned to the host system server node.

    Stored Qstorage controller system IP 10.65.168.182
  4. Complete Step 1 through Step 5 for the second host server node.

This section provides instructions for adding the F1000 storage to a new StorNext file system.

Required Tools

An SSH client, such as PuTTY, is required to complete the procedures in this section.

Overview

The F1000 is pre-configured with two LUNs and is optimized for high-resolution workflows.

Caution: Do not change the 2 pre-configured LUNs on the F1000.

Note: Quantum recommends creating one data stripe group for an F1000. Assign LUNs 1 and 2 to that data stripe group.

Note: Quantum also recommends not using the F1000 system’s LUNs for the metadata and journal stripe group. Use an available LUN from a metadata array in the StorNext shared storage environment, if available.

This section provides instructions for labeling the two F1000 LUNs in StorNext with Quantum-recommended LUN labels to make it easier to identify StorNext configurations with F1000 systems.

Note: For additional information about labeling disks, see Administer > File System Tasks > Label Disks in the StorNext Documentation Center. Complete this procedure for both LUNs.

To label the F1000 LUNs in StorNext:

1. Open an SSH connection to the host StorNext Appliance.
2. Log in to the command line using the following credentials:

User name: stornext

Password: <stornext user account password>

Note: Unless the password changed, password is the default password for the StorNext user account.

3. Enter sudo rootsh to gain root user access:

sudo rootsh
4. Enter the following command to display the volume names:

/usr/cvfs/bin/cvlabel -l | grep F1000 | sort

The volume names display in path order as shown below. For example, /dev/mapper/mpathh displays first, and /dev/mapper/mpathi displays second. Leave this SSH session window open. You will use this information to label the LUNs in the StorNext GUI.

/dev/mapper/mpathh [QUANTUM F1000_b8a7d1_01 210 ] unknown Sectors: 4665816315. Sector Size: 4096. Stripebreadth: 0.

/dev/mapper/mpathi [QUANTUM F1000_b8a7d1_02 210 ] unknown Sectors: 4665816315. Sector Size: 4096. Stripebreadth: 0.
5. Log in to the StorNext GUI.
6. Navigate to Tools > File Systems > Label Disks.
7. Select the Show Details check box at the lower right corner of the page.
8. Select Show Details at the bottom right of the table.

9. Click the Device Path column heading to sort the device paths so they appear in the same order as they do in the SSH session window.

10. Identify the first device path for an F1000 LUN, and select the check box in the left column. In the example below, the first device path is /dev/mapper/mpathh.

11. Go back to the SSH session window, and copy the label for the device path that corresponds with the device path that you selected in the previous step. In the following example, the label for /dev/mapper/mpathh is F1000_b8a7d1_01 (highlighted in blue).

/dev/mapper/mpathh [QUANTUM F1000_b8a7d1_01 210 ] unknown Sectors: 4665816315. Sector Size: 4096. Stripebreadth: 0.

/dev/mapper/mpathi [QUANTUM F1000_b8a7d1_02 210 ] unknown Sectors: 4665816315. Sector Size: 4096. Stripebreadth: 0.
12. Return to the StorNext GUI, and paste the label into the LUNLabel field at the bottom right of the screen.

13. Click Label to the right of LUN Label field to apply the new label to the LUN.

14. Click Yes on the Label Disk Drives dialog window to confirm the label.

The system displays the new label for the LUN in the Label column.

15. Complete this process for the other LUN.

This procedure describes the high-level steps needed to create a file system using the StorNext GUI. In this procedure, you will create two stripe groups: one metadata and journal stripe group, and one user data stripe group.

Note: For additional information about creating a new file system, see Administer > File System Tasks > Add a File System in the StorNext Doc Center.

To create a new file system in StorNext:

1. In the StorNext GUI, navigate to Configuration > File Systems.
2. Click New to open the New page.

3. Enter a name for the file system in the File System Name field—for example, F1000_1 — then press the Tab key.

The Mount Point text box automatically populates with /stornext/ and your file system name.

4. Determine if the Storage Manager check box and the Replication / Deduplication check boxes should be selected, and configure these items as appropriate.
5. Select Manual for Stripe Group Configuration:

6. Click Continue to display the [File System Name] page.

7. Expand the Configuration Parameters section, and click the Allocation tab.
8. In the Allocation Session Reservation Size field, enter 0 to disable the setting.

Note: Quantum recommends disabling this setting for the F1000 storage to improve performance in high resolution workflows.

9. Configure the remaining settings in the Advanced Parameters section as needed.
10. In the Stripe Group / Disk Management section, add a strip group for the metadata and journal content by completing the steps below.

Note: It is not recommended to use the F1000 system’s LUNs for the metadata and journal stripe group. Use an available LUN from a metadata array in the StorNext shared storage environment, if available.

a. Under the Stripe Groups section, click Add to open the Stripe Group / Disk Management section.

b. In the Name field, leave the default name of sg0, or change the name to something more descriptive—for example, meta_jrnl.

Note: You cannot name the stripe group metadata or journal because those names are internally reserved.

c. Configure the stripe Breadth setting as needed.
d. In the Content section, select both Metadata and Journal.

e. In the Disk Assignment section, select a labeled disk to assign to the metadata and journal stripe group.

Note: It is recommended to not assign any of the F1000 LUNs to the metadata and journal stripe group.

f. Click Assign to assign the disk to the stripe group.

The assignment appears in the Assigned To column for the disk:

11. Under Disk Assignment, in the Order column, enter numbers for the LUNs that correspond to the LUN label numbers. For example, enter 1 in the Order column for F1000_b8a7d1_01. Enter 2 in the Order column for F1000_b8a7d1_02.

12. Add one additional stripe group for user data content by doing the following:
a. Under the Stripe Groups section, click Add.
b. In the Name text box, leave the default name of sg1 to indicate the first user data stripe group, or change it to something like data_1.
c. From the Breadth list, select 2.500 MiB. This setting is the optimal breadth setting for the F1000.

d. Select User Data in the Content section.

e. In the Disk Assignment section, select LUN1 and LUN2:

Note: Press the Ctrl key to select multiple disks.

f. Click Assign to assign LUN1 and LUN2 to the first user data stripe group.

The assignment appears in the Assigned To column for the disk.

13. Click Apply to save the new file system settings.
14. Click OK to confirm the creation of the new file system.

StorNext displays a summary of the new file system.

It is necessary to update the StorNext /usr/cvfs/config/deviceparams file on the host system server with device parameters specifically for SSD drives.

To set StorNext device parameters:

1. Open an SSH connection to the host system server.
2. Log in to the command line using the following credentials:

User name: stornext

Password: <stornext user account password>

Note: Unless the password changed, password is the default password for the StorNext user account.

3. Enter sudo rootsh to gain root user access:

# sudo rootsh
4. Cat the /usr/cvfs/config/deviceparams file as shown below:

# cat >> /usr/cvfs/config/deviceparams
5. Enter the following SSD device parameters to append them to the file.

Use the following syntax as of RHEL 8:

ssd=1 scheduler=none

ssd=1 nr_requests=4096

ssd=1 max_sectors_kb=640

Use the following syntax for RHEL 7:

ssd=1 scheduler=noop

ssd=1 nr_requests=4096

ssd=1 max_sectors_kb=640
6. Press the Ctrl+D keys.

The system saves the file with the new device parameter information.

7. View the file to verify that the SSD device parameters were appended to the file:

# cat /usr/cvfs/config/deviceparams

The /usr/cvfs/config/deviceparams file appears, as shown in the examples below.

RHEL 8 /usr/cvfs/config/deviceparams file:

scheduler=deadline

nr_requests=4096

ssd=1 scheduler=none

ssd=1 nr_requests=4096

ssd=1 max_sectors_kb=640

RHEL 7 /usr/cvfs/config/deviceparams file:

scheduler=deadline

nr_requests=4096

ssd=1 scheduler=noop

ssd=1 nr_requests=4096

ssd=1 max_sectors_kb=640
8. Repeat Step 1 through Step 7 for the other StorNext host server node.

StorNext v6.3.1.1 or later is required for Windows or Linux environments where 4K or larger video frames are read at 60 frames per second (FPS) or higher.

Complete the procedures in the following sections as needed for the environment.

Required Tools

An SSH client, such as PuTTY, is required for this procedure.

Complete the procedures in this section for each Linux client workstation that will access the new file system with the F1000 storage.

Note the following requirements for Linux systems:

  • StorNext v6.3.1.1 or later is required for Windows or Linux environments where 4K or larger video frames are read at 60 frames per second (FPS) or higher.

  • macOS 10.14.6 is required for StorNext Mac clients connecting to an F1000 over FC using Xsan, and for StorNext Mac clients connecting via DLC.

To install/upgrade StorNext client software on a Linux client workstation:

1. If you have not done so already, download the appropriate StorNext 6.3.1.1 or later client software for Linux.
2. Do one of the following, depending on whether you need to install software on a new client workstation or upgrade an existing client workstation:

New install: complete the procedures on the Install the StorNext Client Software on Linux page of the StorNext Documentation Center.

Upgrade: complete the procedures on the Upgrade a Client page of the StorNext Documentation Center.

3. After installing or upgrading the StorNext client software on the Linux client workstation, continue with the next section to configure the client.

To configure client software on Linux platforms:

1. Using an SSH client, such as PuTTY, open an SSH connection to the Linux client workstation and log in to the client with the client credentials.
2. Enter the following command to temporarily disable Transparent Huge Pages (THPs):

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

This command temporarily disables THPs until the client system is rebooted.

Note: After disabling THPs, applications that make use of the StorNext file system should also be restarted.

3. Use vi to open the /etc/fstab file by running the following command:

# vi /etc/fstab

The /etc/fstab file appears as shown in the following example:

#

# /etc/fstab

# Created by anaconda on Wed Sep 26 15:49:52 2018

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/rhel-root / xfs defaults 0 0

UUID=9f1a2605-a19c-49c7-8a9d-680c801580aa /boot xfs defaults 0 0

/dev/mapper/rhel-home /home xfs defaults 0 0

/dev/mapper/rhel-swap swap swap defaults 0 0

<fsname> /stornext/<fsname> cvfs rw 0 0 0 0

Note: Use the Up Arrow, Down Arrow, Right Arrow, and Left Arrow keys to move the cursor as needed.

4. Use the Down Arrow key to move the cursor to the file system line.
5. Use the Right Arrow key to move the cursor to the “r” in “rw”.
6. Type x six times to delete “rw 0 0”. Use the Left Arrow key to move the cursor to the left as needed.
7. Type Ito enter INSERT mode.
8. Add the “parallel_submit” information to the file as shown below:

#

# /etc/fstab

# Created by anaconda on Wed Sep 26 15:49:52 2018

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/rhel-root / xfs defaults 0 0

UUID=9f1a2605-a19c-49c7-8a9d-680c801580aa /boot xfs defaults 0 0

/dev/mapper/rhel-home /home xfs defaults 0 0

/dev/mapper/rhel-swap swap swap defaults 0 0

<fsname> /stornext/<fsname> cvfs parallel_submit 0 0
9. Press ESC to exit INSERT mode.
10. Enter the following command to save the file:

:wq!

11. Start the file system service by entering the following command:

# service cvfs start
12. Continue with the next procedure to test the configuration of the client.

After disabling THPs and configuring the parallel_submit mount option for the client, verify that the client is correctly configured by performing I/O tests.

To test the client configuration:

1. If necessary, open an SSH connection to the Linux client and Log in to the client with the client credentials.
2. Change directories to your F1000 file system:

# cd /stornext/<fsname>
3. Create a large file (approximately 200 MB) by entering the following command:

# dd bs=209715200 of=foo count=1 oflag=direct if=/dev/zero
4. Clear qustats by entering the following commands:

# qustat -M -m client

# qustat -R -m client
5. Read the file in one shot by entering the following command:

# dd if=foo bs=209715200 count=1 iflag=direct of=/dev/null
6. Get the Load page time by entering the following command:

# qustat -m client | awk '/Load pages \(Rd\)/{print$NF}'
7. Record the value displayed.
8. Repeat Step 4 through Step 7 five times to identify the average load page time.
9. Do one of the following:

If the average load page time is 50 and under, then the parallel_submit optimization is configured and working correctly. You can now permanently disable THPs by continuing with Step 10.

If the average load page time is over 50, then do not permanently disable THPs.

10. To permanently disable THPs, use vi to open the /etc/default/grub file by entering the following command:

# vi /etc/default/grub

The /etc/default/grub file appears as shown in the following example:

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT="console"

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=1"

GRUB_DISABLE_RECOVERY="true"

 
11. Use the Down Arrow key to move the cursor to the GRUB_CMDLINE_LINUX line.
12. Use the Right Arrow key to move the cursor to the quotation marks (“) at the end of the line.
13. Type i to enter INSERT mode.
14. Add transparent_hugepage=never to the end of the GRUB_CMDLINE_LINUX line as shown in the following example:

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT="console"

GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_idle.max_cstate=0 processor.max_cstate=1 transparent_hugepage=never"

GRUB_DISABLE_RECOVERY="true"

 
15. Press ESC to exit INSERT mode.
16. Enter the following to save the file:

:wq!

17. Enter the following command:

# grub2-mkconfig -o /boot/grub2/grub.cfg
18. Update the multipath.conf file on the client workstation.

Note: For instructions on updating the multipath.conf file, see Step 4a: Update the Multipath configuration.

19. Reboot the client for the changes to take effect:

# reboot
20. Do one of the following:

If you need to install or upgrade client software on additional Linux client workstations, repeat the steps in Install/upgrade and configure StorNext client software on macOS/Linux for each additional client.

If you are done installing or upgrading client software on Linux client workstations, and need to install client software on Windows client workstations, continue with Install/upgrade and configure StorNext client software on Windows.

If you do not need to install or upgrade StorNext client software on additional client workstations, continue with Step 4e: Configure 100 GbE iSER for RHEL, CentOS, and Fedora client systems.

Complete the procedures in this section for each Windows client workstation that will access the new file system with the F1000 storage.

Note: StorNext v6.3.1.1 or later is required for Windows or Linux environments where 4K or larger video frames are read at 60 frames per second (FPS) or higher.

To install/upgrade software on a Windows client workstation:

1. If you have not done so already, download the StorNext 6.3.1.1 or later client software for Windows.
2. Do one of the following, depending on whether you need to install software on a new client workstation, or upgrade an existing client workstation:

New install: complete the procedures in Install the StorNext Client Software on Windows in the StorNext Documentation Center (www.quantum.com/snsdocs).

Upgrade: complete the procedures in Upgrade a Windows Client or Non-HA Server in the StorNext Documentation Center (www.quantum.com/snsdocs).

3. Continue with the next procedure to configure the StorNext Windows client.

To configure software on a Windows client workstation:

1. Configure the StorNext 6.3.1.1 or later client software on Windows clients using the instructions in the Configure the StorNext Client Software on Windows page of the StorNext 6 Documentation Center.
2. After configuring the software to access the new file system, select your mounted file system, and click the Properties icon:

The Properties window appears:

3. Click the Advanced Mount Options tab:

4. Do the following:

Select the Enable Parallel I/O Submission check box if the F1000 will perform large I/Os.

Make sure the Max In-Flight Data Per I/O field is set to the default 512M. The value of 512 MB supports raw frame sizes of 8K and beyond. This value can be increased to 4GB.

Note: The Max In-Flight Data Per I/O field may also provide noticeable benefits with somewhat less-demanding workflows, such as 4K 60 frames per second (FPS) (potentially supporting more concurrent streams without dropping frames).

5. Click Apply.

A message informs you that the changes require you to remount the client:

6. Adhere to the guidelines in the message above as needed, then click Yes to remount the client.
7. Do one of the following:

If you need to install or upgrade client software on additional Windows client workstations, repeat the steps in Install/upgrade and configure StorNext client software on Windows for each additional client.

If you are done installing or upgrading client software on Windows client workstations, and need to install client software on Linux client workstations, continue with Install/upgrade and configure StorNext client software on macOS/Linux.

If you do not need to install or upgrade StorNext client software on Linux client workstations, continue with Step 4e: Configure 100 GbE iSER for RHEL, CentOS, and Fedora client systems.

100 GbE supports only Linux RHEL, CentOS, and Fedora client systems, and they must be prepared to operate with iSCSI/iSER storage. Other Linux OS variants, macOS, and Windows clients are not supported. This section provides high-level guidance for configuring Linux clients.

The main steps are as follows:

Flow control is not needed for converged storage networks. Enter:

ethtool -A «device» tx on rx off

Repeat this process for each network interface device (like "eth0”) that participates in your storage network.

Verify flow control is disabled using ethtool.

Example:

(For one of your storage network devices: s1p1).

Enter:

ethtool -a s1p1

Output:

Pause parameters for s1p1:

Autonegotiate: off

RX: off

TX: on

Example parameters (for using dual subnet - two active interfaces the system):

The F1000 uses 192.168.1.100/24 and 192.168.2.100/24

Choose a unique VLAN ID to use for the configuration, including:

  • Initiator node interfaces must be configured with the selected VLAN ID.
  • Target node interfaces must be configured with the selected VLAN ID, if supported. Otherwise, configure the target interfaces as usual, without use of VLAN.

Complete additional, required switch configuration steps:

  1. Create a VLAN with the selected VLAN ID.
  2. Add the switch ports connecting to the initiator and the targets interfaces to the VLAN.

  3. Turn on Priority Flow Control (PFC).

  4. Activate PFC for the ports of the VLAN group.

Configure and load the Linux iSER kernel module by adding or changing /etc/modprobe.d/ib_iser.conf.

  1. Enter:

    # cat /etc/modprobe.d/ib_iser.conf options ib_iser max_sectors=5120

  2. Next, enter:

    # modprobe ib_iser

  1. Enter:

    2. # iscsiadm -m discovery -t st -p 192.168.1.100 192.168.1.100:3260,1 iqn.2019-04.esos.node-ee90dd-1:f5b20
  2. Next, enter:

    3. # iscsiadm -m discovery -t st -p 192.168.2.100 192.168.2.100:3260,1 iqn.2019-04.esos.node-ee90dd-1:58e5d

  3. Then, enter:

    4. # iscsiadm -m discovery -t st -p 192.168.1.101 192.168.1.101:3260,1 iqn.2019-04.esos.node-ee90dd-2:0dc00

     
  4. Finally, enter:

    5. # iscsiadm -m discovery -t st -p 192.168.2.101 192.168.2.101:3260,1 iqn.2019-04.esos.node-ee90dd-2:5a43b
  1. Enter:

    # iscsiadm -m node 192.168.1.100:3260,1 iqn.2019-04.esos.node-138600-1:f5b20 192.168.2.100:3260,1 iqn.2019-04.esos.node-138600-1:58e5d 192.168.1.101:3260,1 iqn.2019-04.esos.node-138600-2:0dc00 192.168.2.101:3260,1 iqn.2019-04.esos.node-138600-2:5a43b

  2. Next, enter:

    3. # iscsiadm -m node -T iqn.2019-04.esos.node-138600-1:f5b20 -o update -n iface.transport_name -v iser
  3. Then, enter:

    4. # iscsiadm -m node -T iqn.2019-04.esos.node-138600-1:58e5d -o update -n iface.transport_name -v iser
  4. Next, enter:

    5. # iscsiadm -m node -T iqn.2019-04.esos.node-138600-2:0dc00 -o update -n iface.transport_name -v iser
  5. Finally, enter:

    6. # iscsiadm -m node -T iqn.2019-04.esos.node-138600-2:5a43b -o update -n iface.transport_name -v iser

Enter:

Note: Word wrap is used for easier viewing.

iscsiadm -m node -L all

Logging in to [iface: default, target: iqn.2019-04.esos.node-138600-1:f5b20,portal: 192.168.1.100,3260] (multiple)

Logging in to [iface: default, target: iqn.2019-04.esos.node-138600-1:58e5d,portal: 192.168.2.100,3260] (multiple)

Logging in to [iface: default, target: iqn.2019-04.esos.node-138600-2:0dc00, portal: 192.168.1.101,3260] (multiple)

Logging in to [iface: default, target: iqn.2019-04.esos.node-138600-2:5a43b,portal: 192.168.2.101,3260] (multiple)

Login to [iface: default, target: iqn.2019-04.esos.node-138600-1:f5b20,portal: 192.168.1.100,3260] successful.

Login to [iface: default, target: iqn.2019-04.esos.node-138600-1:58e5d,portal: 192.168.2.100,3260] successful.

Login to [iface: default, target: iqn.2019-04.esos.node-138600-2:0dc00,portal: 192.168.1.101,3260] successful.

Login to [iface: default, target: iqn.2019-04.esos.node-138600-2:5a43b,portal: 192.168.2.101,3260] successful.

Verify that you properly attached the storage in the following ways:

Verify iSER sessions

See Show iSCSI Configurations in the Appliance Controller Doc Center.

You should see either two (single subnet) or four (dual subnet) of these on the F1000. If not, follow the instructions below.

Verify block devices

You will see the LUNs for the F1000 system. Enter the following to verify the path to them:

lsscsi -s

[...]

[1:0:0:0] disk QUANTUM F1000_138600_01 200 /dev/sdi 9.55TB

[1:0:0:1] disk QUANTUM F1000_138600_02 200 /dev/sdm 9.55TB

Verify Multipath and ALUA

You should see two multipaths (for each F1000 system)—one for each LUN—not in any particular order. Each multipath should have two active paths and two non-optimal paths for a dual-subnet configuration (one and one for a single-subnet configuration). Enter:

multipath -ll -v2

mpathr (26661393137643465) dm-3 QUANTUM ,F1000_138600_02

size=8.7T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| |- 11:0:0:1 sdk 8:160 active ready running

| `- 12:0:0:1 sdl 8:176 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

|- 9:0:0:1 sdm 8:192 active ready running

`- 10:0:0:1 sdn 8:208 active ready running

mpathq (26365373664356437) dm-4 QUANTUM ,F1000_138600_01

size=8.7T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| |- 9:0:0:0 sdg 8:96 active ready running

| `- 10:0:0:0 sdj 8:144 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

|- 11:0:0:0 sdh 8:112 active ready running

`- 12:0:0:0 sdi 8:128 active ready running

Verify SNFS labels

Query the SNFS labels. The LUNs are not labeled, yet. Enter:

cvlabel -l | grep F1000

/dev/mapper/mpathcf [QUANTUM F1000_6261c4_01 210 ] SNFS-EFI "F1000_6261c4_01"Sectors: 9358193915. Sector Size: 4096. Maximum sectors: 9358193915. Stripebreadth: 0.

/dev/mapper/mpathck [QUANTUM F1000_6261c4_02 210 ] SNFS-EFI "F1000_6261c4_02"Sectors: 9358193915. Sector Size: 4096. Maximum sectors: 9358193915. Stripebreadth: 0.

The diagnostic commands below help troubleshoot client configuration issues.

Storage network health

RDMA traffic bypasses the CPU, interrupts, and so on. This means that you won't generally see byte and packet counts increasing in all of the normal places. However, the NIC sees them, and you can extract the data from the NIC using ethtool. If your storage network NIC is "s1p1", enter:

# ethtool -S s1p1 | grep -i -e rdma -e pause

rx_vport_rdma_unicast_packets: 4

rx_vport_rdma_unicast_bytes: 1288

tx_vport_rdma_unicast_packets: 0

tx_vport_rdma_unicast_bytes: 0

rx_vport_rdma_multicast_packets: 0

rx_vport_rdma_multicast_bytes: 0

tx_vport_rdma_multicast_packets: 0

tx_vport_rdma_multicast_bytes: 0

rx_pause_ctrl_phy: 0

tx_pause_ctrl_phy: 0

rx_global_pause: 0

rx_global_pause_duration: 0

tx_global_pause: 0

tx_global_pause_duration: 0

rx_global_pause_transition: 0

tx_pause_storm_warning_events : 0

tx_pause_storm_error_events: 0

Network statistics accumulate until the card is reset (e.g., by booting, or reloading the driver stack), at which point they start from zero.

Important statistics:

Normal RDMA traffic shows up as "vport_rdma_unicast_packets" (and _bytes); "tx_" are packets (bytes) are transmitting from this host, and "rx_" are packets (bytes) being received by this host.

Flow control is a normal part of RDMA traffic over a properly configured network. However, large numbers or rapid increases in "global_pause" (and corresponding increases in "global_pause_duration") probably point to a problematic situation; again here, "tx_" are pauses sent by this host (indicating that it is being overrun by data), "rx_" are being received by this host (indicating that it is producing data too fast).

After you log into the F-Series UI, you can configure the items listed below.

To log into the F-Series UI:

  1. Access the F-Series UI log in page using the management IP address assigned to the F1000 during initial system configuration on the log in screen.

    2. (click image to enlarge)

  2. Enter root in the Login field, and enter the password you created during initial system configuration in the Password field.

Configure time and date settings by completing one of the following procedures.

  1. In the F-Series UI, select Miscellaneous in the left navigation menu.
  2. Click the NTP tab.
  3. Click the Enable NTP toggle. It turns blue, indicating it is enabled.
  4. Enter an NTP Server address.

    5. Example (NTP server name): time.nist.gov

    Example (NTP server IP address): 129.6.15.28

  5. Click Apply to confirm the change.

Update the date and time settings on the F1000 to match the local time at the installation location using the QCSP (Quantum Cloud Storage Platform) TUI.

Required tools

An SSH client, such as PuTTY, is required to complete this procedure.

Update the date and time

To update the date and time settings:

  1. Access the QCSP TUI:
    1. On a laptop, open an SSH session to the command line of the system using the management IP address of the F1000.
    2. Log in to the command line:

      Username: root

      Password: <password>

    3. Launch the Quantum Cloud Storage Platform TUI. Enter:

      4. /usr/local/bin/esos_tui

  2. Press the S key to display the System menu:

  3. Press the Down arrow to select Date/Time Settings and press Enter.

    The Edit date and time settings... screen appears:

  4. Press the Up and Down arrow keys to find the time zone of the installation site.
  5. Press the Space Bar to select the time zone.
  6. Press the Tab key to go to the Current Date section and select the correct month and day.
  7. Press the Tab key to go to the Hour, Minute, and Second fields and set those to the current time.
  8. Press the Tab key to go to the OK button and press Enter to save the settings.

You must have an account to enable Cloud-Based Analytics (CBA). To request a CBA account, click the Request an account link in the Cloud-Based Analytics site at insight.quantum.com.

  1. In the F-Series UI, select Miscellaneous in the left navigation menu.
  2. Click the CLOUD-BASED ANALYTICS tab.
  3. Click the Enable Cloud-Based Analytics toggle. It turns blue, indicating it is enabled.
  4. Enter the Proxy Server name, if you are using an SSL proxy.
  5. Enter the Port number for that proxy server, if you are using an SSL proxy.
  6. For now, leave the Device Registration ID blank. You must have an account before you populate this ID. To request a CBA account, click the Request an account link in the Cloud-Based Analytics site at insight.quantum.com. The system can be found on the CBA portal for your particular Solution.
  7. Click Apply to confirm the change.

  1. In the F-Series UI, select Miscellaneous in the left navigation menu.

  2. Click the SNMP tab.

  3. Click the Enable SNMP toggle. It turns blue, indicating it is enabled.

  4. Enter the SNMP Community String. The default is public.

    Note: To avoid security issues, change this value.

  5. Enter the SNMP Trap Receiver, if needed. The default is localhost.

    6. Note: The SNMP Trap Reciever is the hostname of the system that collects SNMP traps. If set to "localhost", SNMP data won't go anywhere. When enabling SNMP, change this to the hostname of the trap receiver used for SNMP. If this is left as "localhost", SNMP data will not be collected.

  6. Enter the Port for the SNMP trap receiver. There is no default port value.
  7. Click Apply to confirm the change.