Quality of Service Bandwidth Management (QBM)

The QoS Bandwidth Management feature allows administrators to configure I/O bandwidth on a data stripe group/client basis within a particular file system.

  • A data stripe group is configured with a specified bandwidth capacity.
  • That capacity is then divided among connected clients, based on the configuration.
  • Each stripe group may be configured and managed independently.
  • Clients are assigned bandwidth based on the configuration of classes and bandwidth.

  • Three classes are defined, with different behaviors.

  • Alternatively, you can define a mover configuration to throttle a client running storage manager Distributed Data Mover (DDM). Other clients are not throttled and must not be configured.

An example would be a file system with a single data stripe group and five clients using that file system. If the bandwidth capacity of the stripe group is 1000MB/s, configuring the default client class as fair share would result in each client using 200MB/s if all 5 clients were actively doing I/O and consuming at least 200MB/s. If only 2 clients were actively doing I/O, they could each be allocated 500MB/s.

The section discusses the terminology and concepts of QoS Bandwidth Management. Quality of Service in this context applies only to bandwidth management. QoS bandwidth Management will hereafter be referred to as QBM. 

QBM is used to manage bandwidth on data stripe groups. Any stripe group containing metadata is not eligible for management, which includes mixed metadata/data stripe groups. Thus a file system must have a minimum of two stripe groups to use this feature where one stripe group contains metadata and the other contains user data.

QBM is used to limit or throttle client bandwidth usage. But it is not a guarantee of bandwidth availability, as there is no way to accelerate I/O. However, configuration of this feature will allow particular clients to obtain specified levels of bandwidth regardless of competing client I/O.

The assignment of bandwidth to a client is dynamic, and is based on both the QBM configuration and the current I/O usage of clients. The goal is to use the bandwidth without compromising configured client bandwidth.

QBM is used to limit clients' bandwidth usage based on configuration. The total bandwidth of a stripe group is specified in a configuration file. That total is then allocated to clients using information in the configuration file. If a file system consists of a single data stripe group, allocating bandwidth to stripe groups is synonymous with allocating bandwidth to the file system.

The configuration file consists of three sections: general, stripe groups, and clients. The general section is required, while the stripe group and client sections are optional.

Consider a file system containing a metadata stripe group and a single data stripe group. The general section can be used to specify the total bandwidth capacity, the class used for clients, and default settings for clients.

Note: Any SAN or LAN client that has a file system mounted which is managed by a QoS configuration, is influenced by QoS regardless if the client is configured within the QoS configuration.

Note: If you do not leave bandwidth for movers, tape shoe-shining might occur. See Mover for more information.

Use the qbmanage command, along with the following command options, to create and modify the configuration file:

  • To start a configuration, use the “new” option.
  • To modify the general section, use the “modify” option.
  • To add and remove stripe group entries, use the “rmsg” and “addsg” options. 
  • To add and remove client entries, use the “addclient” and “rmclient” options. Multiple client entries can be specified if they are associated with different stripe groups.

Three classes are defined with different behaviors. A fourth class called mover is implicit for throttling storage manager Distributed Data Movers (DDMs) when mover is specified for the configuration.

The allocation classes are as follows. See Configure QBM for information on how to configure stripe groups with these classes.

  • First Come: This is the highest priority of allocation. Clients configured for this class of service will either get all the bandwidth they are assigned, or will be denied allocations in the First Come class. . See First Come (FC) for additional details.
  • Fair Share: This is the second highest priority of allocation. Clients configured for this class of service will either get their configured bandwidth or will share the bandwidth that has not been allocated to the First Come class of clients, in proportion to their configured amounts. You might put clients that are involved in production work in this class. Clients in this class are dynamically changing and need to share a limited resource. See Fair Share (FS) for additional details.
  • Low Share: Clients configured for this class of service get their configured bandwidth, or share the bandwidth not allocated to the higher priority clients, in proportion to their configured amounts. You might put clients that mostly perform background work in this class, to do that work as resources permit. See Low Share (LS) for additional details.
  • Mover Class: This is the lowest priority. Whatever bandwidth is left is proportionally distributed to the clients in the class. This class is similar to Low Share, but is a special case reserved for the DDM application. See Mover for additional details.

    Mover Class Considerations for Implementations that use NAS and QBM

    • If you use the Mover Class, you must consider the MDC. The MDC can be (and is, by default) a data mover to tertiary storage.

    • You can put the MDC into the Mover Class to control its bandwidth usage for moving data to and from tertiary storage. However, this might be undesirable in certain situations, for example if the MDC is also a NAS cluster node. In that case, NAS clients are also throttled, which is not the intent.

    • In an environment with DDMs, you can run the fsddmconfig -M command to set the maximum number of simultaneous mover copy requests allowed on the host when it is running as the MDC server. The command allows you to prevent (in most cases) the MDC from being a data mover, instead leaving all that work to the DDM nodes. Run the command if you want to limit/restrict the copy requests performed on the MDC that are throttled by the Mover Class settings.. See the fsddmconfig(1) command in the StorNext Man Pages Reference Guide for details.

These classes are the configuration building blocks. In addition to class of service, each client can be configured to have a minimum bandwidth allocation. The combination of class, bandwidth requested, and amount of bandwidth requested in each class will determine what bandwidth each client will be allocated.

This class has priority over all other classes. A client in this class that is granted bandwidth is guaranteed to retain its minimum bandwidth.

The First Come class is an all or nothing class. That is, either all of the client's configured bandwidth is granted, or the client is rejected from the class.

  • If a new client mounts the file system and this new client is configured as a First Come class client. with more configured bandwidth than is currently available in the First Come class, that client's First Come bandwidth allocation request is rejected.
  • The new client is then assigned to the next lowest priority class, the Fair Share class.
  • The total First Come bandwidth is the configured capacity of the configured stripe group, minus any bandwidth reserved exclusively for other classes of service.
  • Each client accepted to the First Come class reduces the available bandwidth for that class by its configured amount.

If the First Come class is over-subscribed and some clients are being rejected from the class, consider:

  • Changing the configuration file and running qbmanage --newconfig. Running qbmanage --newconfig causes QBM to re-evaluate all client allocations according to the current QBM configuration file.
  • Reducing the number of clients configured as First Come clients that could mount the file system at the same time.

Consider the case where you need to keep a First Come class client that is running a movie playback jitter free, and a large amount of bandwidth is allocated to that client. If this leaves QBM with no more bandwidth available, a subsequent First Come class client's request will be rejected, and the second client's bandwidth will be fulfilled from the bandwidth available to the Fair Share class. This guarantees that when a First Come client performs properly, it is not subject to future over-subscription problems when subsequent clients are activated. The premise is that it is better to keep some number of client applications running at adequate levels than it is to have all applications running, but in a state where none are able to run sustainably at adequate levels.

This class is second in priority for obtaining bandwidth. QBM shares allocation across clients in proportion to their configuration. For example, with a capacity of 900 MB/s and 4 clients configured with a minimum allocation of 200 and a maximum of 900, each client receives at least 200 MB/s. The 100 MB/s left can be shared by those 4 clients. If only two clients are active, they could both use 450 MB/s. Over-subscription will cause all clients to run with less than the preferred minimum bandwidth, where each client expects to have bandwidth equivalent to other clients with identical configuration. If 9 clients are active and are configured identically, each receives 100 MB/s.

Consider an office where there is a set of Fair Share clients that run applications that need a minimum bandwidth to operate effectively, while there are other clients performing other activity, such as file movement. You could configure the clients running these applications as Fair Share and the other clients as Low Share. This would let the applications run at an adequate level even when a copy activity on a LS client its using considerable bandwidth.

This class is third in priority for obtaining bandwidth. A desired minimum and maximum can be configured to inform QBM of expected activity. Sharing among clients is not guaranteed. This class has no explicit requirements.

The mover configuration is for throttling clients running Storage Manager Distributed Data Movers (DDM). A mover configuration is created by setting mover=true|1. In a mover configuration, all clients added to the QBM configuration are automatically put in a fourth class called mover and are referred to as mover clients. Only clients added to the mover configuration are throttled. All other StorNext clients are not throttled. When you add client information into a mover configuration, do not set the class keyword. The class keyword cannot be explicitly set. The client’s class is implicitly set to mover.

QBM monitors clients not defined to the Mover class by running a daemon on those clients. The reported bandwidth usage is used to determine allocation to clients configured as movers.

The Mover class is targeted at server applications that can consume considerable bandwidth. It allows other clients to run without interference by the mover clients. A mover can also be configured with a minimum bandwidth setting, which allows it to use that bandwidth regardless of non-mover client activity. No reduction in bandwidth occurs on any clients other than the configured mover clients.

Note: The mover clients are limited to an I/O impact of the configuration set for the reserved_mover parameter.

It is recommended that any clients configured to be a mover client and running DDM are configured such that a minimum bandwidth can be allocated to them that is sufficient for tape usage.

You must allocate a minimum amount of QOS bandwidth to a tape drive to prevent "tape shoe-shining" (otherwise known as "back-hitching"). See Minimum Transfer Rates to Prevent Tape Shoe-shining.

If a tape drive does not receive a minimum amount of data per seconds, then the drive stops and waits for data to fill its buffers again. This causes a back-and-forth motion on the tape head. This is known as "tape shoe-shining". The minimum tape drive streaming rate is different for each type of tape drive. Use the reserved_mover parameter to specify this minimum amount to prevent tape shoe-shining. The amount to enter on the reserved_mover parameter is the highest minimum tape drive transfer rate for all the tape drives on the system times the number of tape drives. For example:

reserved_mover = MAX[min transfer rate(tapedrive1), min transfer rate(tapedrive2), (...)] * number_of_tape_drives

All clients added to the QBM mover configuration should be configured with a minimum bandwidth equal to or greater than the highest maximum tape drive transfer rate that a mover client can access. For example:

min_bw = MAX[maximum transfer rate(tapedrive1), maximum transfer rate(tapedrive2), ..]

If the client is configured to read/write to multiple tape drives at the same time, the minimum bandwidth should be set to the sum of the maximum tape drive transfer rates:

min_bw = [maximum transfer rate(tapedrive1) + maximum transfer rate(tapedrive2) + ..]

The configuration uses JSON formatting with a configuration file for each file system. The name of the configuration file is <fsname>_qbm.conf. You can use the qbmanage command to create the configuration for a file system by executing a sequence of commands. You can use these commands to create and modify the general section of the file, add a stripe group, remove a stripe group, add a client, and remove a client.

The general configuration includes the on/off status of QBM at file system start, along with whether the mover is being used, and whether all stripe groups are considered managed by default. 

Among the configurations you can specify are:

  • A default value for stripe group capacity. 
  • A default class and minimum/maximum bandwidth for clients.
  • A default value may be overridden by explicit stripe group or client configuration entries.
  • Stripe groups by their name, with a total bandwidth capacity. 
  • A default value for clients.
  • Clients by their IP address. The values are the minimum and maximum bandwidth as well as the class.

There are four classes that affect the behavior of the FSM when allocating bandwidth.

You can use the qbmanage command to create and modify configurations. See the StorNext MAN Pages Reference Guide for complete details.

The following example creates a QBM configuration with one high priority client:

qbmanage --new --fsname snfs1 --all_sg=true --sgcapacity=1000MB

qbmanage --addclient --fsname snfs1 --clientname 10.20.72.128 --min_bw=100MB --max_bw=400MB --class=first_come

The example above creates a configuration with all data stripe groups having a capacity of 1000 MB. The client 10.20.72.128 is configured to have top priority in the first_come class with a minimum of 100 MB and a maximum of 400 MB.

This next example is for Distributed Data Movers (DDMs) associated with two tape drives.

  • Tape drive 1 is an LTO-6 drive and needs a minimum of 54 MB/sec to prevent tape shoe-shining.

    • An LTO-6 tape drive has a max, uncompressed, transfer rate of 160 MB/sec.

  • Tape drive 2 is an LTO-7 and needs 101 MB/sec to prevent tape shoe-shining.

    • An LTO-7 tape drive has a max, uncompressed, transfer rate of 300 MB/sec.

As mentioned in the section Mover, the minimum bandwidth amount that should be available to the mover class should be set to the highest minimum drive rates for both drives, which is 101 MB/sec, times the number_of_tape_drives, which is 2, or 202 MB/sec.

Note: To be on the safe side, add 5 MB for a total of 207 MB for the reserved_mover value.

In this example, assume both clients can read and write to either tape drive.

qbmanage --new --fsname snfs1 --all_sg=true --reserved_mover=207M --sgcapacity=1000M

qbmanage --addclient --clientname 10.20.70.90 --fsname snfs1 --min_bw=300M --max_bw=1000M

qbmanage --addclient --clientname 10.20.80.91 --fsname snfs1 --min_bw=300M --max_bw=1000M

Note: You must set the reserved_mover and min_bw values to be higher if tape compression is enabled. Most tape drives are capable of around a 3:1 compression ratio.When tape compression is enabled, multiply the derived reserved_mover value and the min_bw value by 3 to obtain appropriate values. In the above example, the --reserved_mover parameter would be set to 621M.

The following example creates a QBM configuration with multiple stripe groups and clients as fair_share:

qbmanage --new --fsname snfs1 --class fair_share

qbmanage --addsg --fsname snfs1 --sgname sg1 --sgcapacity 500M --reserved_fs 400M

qbmanage --addsg --fsname snfs1 --sgname sg2 --sgcapacity 1000M

qbmanage --addsg --fsname snfs1 --sgname sg3 --sgcapacity 800M

qbmanage --addclient --fsname snfs1 --clientname 10.65.178.185  --min_bw 50M --max_bw 500M

qbmanage --addclient --fsname snfs1 --clientname 10.65.177.184 --sgname sg2 --min_bw 100M --max_bw 200M

qbmanage --addclient --fsname snfs1 --clientname 10.65.178.189 --min_bw 25M --max_bw 500M

qbmanage --addclient --fsname snfs1 --clientname 10.65.178.189 --sgname sg3 --min_bw 200M --max_bw 800M

The table below provides the recommended minimum transfer rates to prevent tape shoe-shining.

LTO Ultrium Tape Media Minimum Transfer Rates
LTO-1 8 MB/sec
LTO-2 18 MB/sec
LTO-5 47 MB/sec
LTO-6 54 MB/sec
LTO-7 101 MB/sec
LTO-8 112 MB/sec

Running a bandwidth capacity test can help you make QBM decisions. There are two scripts provided for running these tests. The qbm-ladder test runs a set of tests using various buffer sizes, numbers of I/O streams, clients, and I/O queue depths. The I/O queue depth is the number of I/O requests outstanding at any given time. Specifying a queue depth of 1means that the I/O is single threaded. The qbm-ladder test uses the qbm-mio test to run I/O tests on the specified clients.

The qbm-ladder test submits sets of qbm-mio tests simultaneously on different clients. Both tests require the file system name and client name. Multiple clients are specified as a comma separated list. For example, the command below uses qbm-mio to run a single test using the default values specified in the StorNext MAN Pages Reference Guide.

qbm-mio --fsname snfs1 --clients cl1,cl2

The command below uses qbm-ladder to run sets of tests. 

qbm-ladder --fsname snfs1 --clients cl1,cl2

The number of streams will start at 1 and progress by 1 up to 16. The queue depth will start at 1 and progress by 1 to 8. The buffer size starts at 64K and progresses to 32M. The tests start with one client, then add one client at a time until all clients have been tested. The example above would run the set of tests using client clients cl1 and cl2. You can specify the --short option to run a shorter set of tests than the defaults, as describe on the man page.

The qbm-ladder test saves results in an sqlite database, which is located in the directory /usr/cvfs/data/<fsname>/qbm/qbm-db. Each test execution is assigned an identification number, labeled Test id, in which with the time the group of tests started is used to identify a set of test results.

After you run the qbm-ladder tests, you should run the qbm-analyze script to analyze qbm-ladder test results to determine the maximum bandwidth for a stripe group for a file system using multiple clients, multiple streams, variable queue depth, and variable buffer sizes.

  • You can use the qbm-analyze command option --action list to display the test numbers found in the database with their associated parameters and time when qbm-ladder was started. For example:

    qbm-analyze --action list --fsname|-f FsName | --db | -f name [--testid test_number] [--verbose]

If the database is removed, the next qbm-ladder command creates a new qbm-db for that file system.

  • You can use the qbm-analyze command option --testid to display the unique test numbers available for analysis. For example:

    qbm-analyze --action testid --fsname|-f FsName | --db | -f name [--verbose]

Use the eval option of the --action command to show the maximum bandwidth and the number of tests that achieved the maximum bandwidth. For example:

qbm-analyze --fsname perf1 --action eval --testid 20925
Test id 20925
Largest bandwidth is 117MB/s which occurred in 901 out of 1210 tests.