StorNext File System Pooling

A storage pool is a set of stripe groups with an associated name that represent the targets for data movement. Pools are generally associated with stripe groups using specific types of storage; for example, a fast pool for SSD or NVMe and a slow pool for spinning disk. However, there is nothing in the product which requires this be so. You can configure several pools out of the same type of storage.

A storage pool can be exclusive or non-exclusive. You can only use an exclusive pool by content specifically targeted to that pool, and you can only use a non-exclusive pool by both targeted and untargeted content.

You can place both files and directories in a pool. A file in a pool means its contents are placed on the devices in the pool. A directory in a pool means that new file content within the directory will also use the pool. The only limit on the number of pools is the number of data stripe groups in a file system.

A pooling job consists of instructions for the system to perform an action on specified content; the most common job is to move the data content to a specific storage pool. If you are granted access to the pooling system, you can initiate a job, or the job can also be triggered by a pooling policy.

Jobs have different controls to modify its behavior; details are described in a different section. A job can run immediately or at a specified time in the future. When a job is finished, a completion report is held in an internal database allowing the report to be emailed or viewed later.

A pooling policy consists of a request to run a job on a regular basis looking for work to do and executing the work.

Both jobs and policies define the content to be acted on using the same mechanisms, a set of content can either be explicitly referenced and filtered down by a set of rules, or you can use an internal query mechanism of the file system to look up candidates for processing.

All the execution of work for pooling is handled by a daemon; jobs are submitted to the daemon and policies are stored and run by the daemon. You can use the command line to access the service using a REST API.

You must run the daemon, sntierd, on a Linux-based StorNext client running either the RedHat 7 (or later) operating system or the CentOS 7 (or later) operating system. The system can be a customer supplied system, an MDC, or an Xcellis Workflow Extender.

Note: You must have sufficient storage bandwidth from the system to support the workload.

Multiple instances of the service may be run at the same time. The services are all independent systems and are not aware of each other. Each service can support running policies and jobs, and is capable of managing content in all the StorNext file systems configured with storage pools and mounted on the client host.

Each pooling service maintains a persistent state in a small internal database, which contains configuration information, jobs, and policies. If you restart the service while a job is running, those jobs are re-run after the service is restarted.

Each job runs on a single pooling service. Illustrated below, the CLI and Web UI can access the pooling service on multiple hosts; there are three pooling services, two of the services operate on FS1 and one service operates on FS2.

All command line access is via the sntier command that is installed under /usr/cvfs/bin; it has a hierarchical argument structure.

To run a pooling service, your system must meet the following prerequisites.

• You must be running StorNext 7.0.1 (or later) on both the client system and the appliance.

• You must enable Metadata Archive on the file system(s). See Edit a File System.

  Note: If you do not enable Metadata Archive, then you run the risk of limiting the function of the File System Pooling feature. The primary example is the use of the search term in a policy operation; in order to find a small number of files in a large set which meet a policy criteria, then you must enable Metadata Archive. However, explicitly moving content between pools does not require you enable Metadata Archive.

• You must enable Web Services. See Web Services.
• Pooling can inter-operate with Storage Manager but does not require a managed file system to operate, or use any Storage Manager code.
• If you are using Storage Manager, you must not have Disk-to-Disk Relocation configured or enabled. See Storage Manager Policy.

• Quantum recommends you have good bandwidth to the data stripe groups of the file system, since the host that runs a pooling service acts as a data mover for bulk content. It is not relevant what type of connectivity is used. Workloads that perform a checksum against content when it is copied require a significant amount of CPU power to perform efficiently. You can possibly use MD5 to utilize several CPU cores in parallel when you move data.

• You must have some optional Python components that might not be installed on your system by default, since the pooling software uses Python. Verify you have installed the following RPM files:

  • pyxattr
  • python-requests
  • python-dateutils

As the root user on your host system, do the following to enable and configure the pooling service for the first time. After you enable and configure the service, you can use the CLI or the GUI to perform all operation and administration functions from a remote host.

Note: You can create, queue, or delete a pool without enabling the pooling service, as it is processed on the file system stripe management level. All other pooling operations require that you enable the pooling service.

Use the StorNext user interface to discover your pooling node.

1. Click Discover Node.

2. Enter the following in the fields:

   1. DNS name, host name, or IP address of your system.

   2. Username.

   3. Password.

3. Click Discover.

After the initial configuration, the daemon (sntierd) is running and is only accessible to the root user on the local system.

The pooling daemon is connected with the Linux pluggable authentication module (or PAM) system which can use any authentication system and users are configured for this. For example, if the host is connected to an active directory using PAM, then the pool of users and groups who can be granted access is those available to the active directory.

The pooling service supports role-based access control; there are two types of roles:

• User. A user can submit jobs which move any content between pools, control their own jobs and view pools and policies.

  Note: The user name for submitting a job does not relate to the permissions on the file system.

• Administrator. An administrator can perform any action, and can perform any configuration from any host where the sntier command is available.

Use the StorNext user interface to define who is a user or an administrator in terms of a user name or a group membership.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Configuration.

   1. Click Authorization.

      

   2. Under the User access or Admin access section, enter a user name and/or group associated with each role (user or administrator).

4. Click Add.

For example, the CLI command below configures user1, user2 and members of group1 to act as users and the user root to act as an administrator:

Use the StorNext user interface to configure a pool on any StorNext file system that is mounted on the host.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Pools and then click Create Pool.

   

   1. Click a file system.

   2. Enter a name for the pool (for example, fast or slow).

      Note: A pool name must be lower case up to eight characters in length.

   3. (Optional) Click to define if the pool is exclusive; a storage pool can be exclusive or non-exclusive. You can only use an exclusive pool by content specifically targeted to that pool, and you can only use a non-exclusive pool by both targeted and untargeted content.

   4. Click the stripe group.

   5. Click Submit.

You can also use the CLI to configure a pool on any StorNext file system that is mounted on the host, in an interactive manner, by using the sntier pool command.

When you execute the command, the output displays a list of mounted file systems and their data stripe groups (for file systems which have data stripe groups that are not allocated to a pool). Press the Up arrow key or the Down arrow key to highlight a file system; press the A key to add a pool to a file system or press the D key to delete a pool from a file system.

You are prompted to define a file system pool. The example below illustrates the output, with user input in bold letters:

Note: A pool name must be lower case up to eight characters in length.

If you press Enter, the process assigns the pool and displays the current configuration:

Mount: /stornext/tiered

PoolName    Capacity     Used   Exclusive   Stripe Groups
slow        4.91 TiB     2.16%  No          sg1
fast        372.60 GiB   0.00%  Yes         sg2

You can also execute the entire sequence from the command line. For example:

Repeat this for other file systems; the configuration of pools is persistent in the file system and visible from all hosts running the pooling service.

Note: If you use the CLI, you must define a complete pooling configuration in one step. You cannot add a new pool without deleting the old configuration and starting over. You cannot perform a stripe group offload on a stripe group which is part of a pool; you must redefine the a pool to not include the stripe group first.

You can optionally configure the following.

Email Delivery

The pooling service can deliver an email to a single email address when a job is complete, as long as the service has access to an email server. If you do not configure email delivery, the service uses the mailx command, which attempts to find an open relay; in most modern deployments, an open relay does not exist and you must define an SMTP server.

A pooling email message is delivered from this account.

Note: The password must be an application password configured for the email account.

Use the StorNext user interface to configure email notifications.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Configuration.

   1. Click Advanced.

      

   2. Under the SMTP section, enter a host, port, user name and password for the SMTP server.

      

   3. (Optional) Click to enable or disable TLS. By default, TLS is enabled.

4. Click Apply.

The CLI example below illustrates a configuration for a Microsoft Office 365 service:

sntier config --smtp_server smtp.office365.com --smtp_port 587 \

--smtp_tls --smtp_user user@company --smtp_passwd xxxx

Storage Manager Web Service Access

You can use Storage Manager web services by pooling jobs to retrieve truncated content, as long as the pooling service has access to the account information for the web services. You must enable the Storage Manager web services using the StorNext GUI, and you must create a web service account along with a password.

Use the StorNext user interface to configure the web service account and password in the pooling service.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Configuration.

   1. Click Advanced.

      

   2. Under the Storage Manager section, enter the user name and password for the web service account.

      

   3. (Optional) In the Batch field, enter the number of files to submit to Storage Manager in one request.

4. Click Apply.

You can also use the CLI to execute the following command to configure the web service account and password in the pooling service:

sntier config --snsm_user webuser --snsm_passwd webpasswd

You do not need to configure the location of the web service address, as the pooling service locates it automatically.

Note: If you are accessing multiple managed MDCs from a single pooling service, the web service information must be identical amongst the MDCs as there is only one configuration for this information.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Jobs.

   1. Click + New Job.

   2. Click a file system.

   3. Click the action to perform.

   4. Specify the content directory, and then click Next.

   5. (Optional) Specify the selection criteria, and then click Next.

   6. Define the schedule; you can configure a job to run at a later date (at a specified time), or now.

4. Click Submit.

You can also use the CLI to run a job; use the following command:

See The sntier CLI Command Reference for additional details. You can use short form commands to specify the action, pool and paths in a less verbose form. The example below illustrates how to submit a move job and specify a directory without a criteria specified:

The example below illustrates how to use the short form syntax to achieve the same result:

Note: Jobs that request to perform an action without an effect will check files, but do not act on them if they are already in the required state. In other words, if you attempt to move content to a pool which it is already on, then the action is not processed.

By default, a job runs as soon as possible; however, you can configure the job to run at a later time. You can also configure the service to deliver an email report when a job is complete.

Note: The email report and job completion information provide a summary of the amount of content that is processed, the amount of time required to process the job, and any errors that occurred during the job.

Do the following to create a policy for the pooling service.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Policies.

   1. Click + New Policy.

   2. Enter a name for the policy, and then click Next.

   3. Click a file system.

   4. Click the action to perform.

   5. Specify the content directory, and then click Next.

   6. Specify the selection criteria, and then click Next.

   7. Define the schedule; you can configure a policy to run daily or weekly at a specified time, or based on a time interval (for example, every 10 minutes or hours).

4. Click Submit.

Do the following to create a policy for the pooling service.

1. Click Services.

   

2. Click the service name associated with the pooling service type.

   

3. On the left navigation menu, click Templates.

   1. Click + New Template.

   2. Enter a name for the template, and then click Next.

   3. Click the action to perform, and then click Next.

   4. Specify the selection criteria, and then click Next.

4. Click Submit.

By default, the sntier command attempts to communicate with the pooling service on a local host and utilizes the identity of a local user for authentication.

Note: For the root user on the same host as the pooling service, you do not require authentication as private authentication mechanism is used.

In all other cases, you must enable remote access. You can use the --host and the --user options to access a remote host and a different user account than the current user. If you are logged in remotely, or as a non-root user, you are prompted for a password (unless you provide the password using the command line). The name and password are used to obtain an authentication token. The token is stored locally and you can use it for a certain period of time (by default, you can use the token for one hour) to access the service using the same credentials.

If you want to use the service remotely, then you can define the default account and the host you want to communicate with:

You can also use optional arguments:

You can use the optional arguments in place of the local account and the host for this user, unless you use the --host and or the --user argument on an invocation.

Use the sntier jobs command to monitor and control a job. Without the extra arguments, the process enters a refreshing display:

Jobs (host: localhost)
jobId   User    Action   Target State     Files   Finish Time
1326    policy  move     slow   completed  8      Sep 10 11:08:25
1323    admin   move     fast   completed  8      Sep 10 11:02:32
1112    policy  move     slow   completed  40779  Sep 09 14:54:19
1111    policy  move     slow   completed  287    Sep 09 14:51:57
1108    admin   move     fast   completed  41066  Sep 09 14:46:56
1104    policy  move     slow   completed  8      Sep 09 14:38:44
1100    admin   move     fast   completed  8      Sep 09 14:32:15
1099    admin   validate  --    completed  41235  Sep 09 14:31:27
1098    admin   inventory --    completed  41235  Sep 09 14:31:04
1097    admin   checksum  --    completed  41235  Sep 09 14:30:44
1095    admin   move     slow   completed  8      Sep 09 14:30:07
1094    admin   move     fast   completed  0      Sep 09 14:29:44
1092    policy  move     slow   completed  0      Sep 09 14:26:11
1091    admin   move     fast   completed  0      Sep 09 14:26:10
1005    jdoe    inventory --    completed  42801  Sep 09 11:10:16
1002    policy  move     slow   completed  41066  Sep 09 11:05:44
1000    jdoe    move     fast   completed  41066  Sep 09 10:59:25

• To navigate highlight a job, press the Up arrow key or the Down arrow key.
• To view the detail about a job, press the space bar.
• To delete, press the D key.
• To suspend, press the S key.
• To resume, press the R key.
• To quit, press the Q key.

Host: localhost Job: 1112       completed

      Action:   move to pool slow
   Submitter:   policy slow
       Email:   user@quantum.com
      Search:   /snfs/flexsync/
    Criteria:
         From pool: fast
        Older than: 5 minutes
      Process only: Files

      Queued:   Mon Sep  9 14:53:56 2019
     Started:   Mon Sep  9 14:53:56 2019
    Finished:   Mon Sep  9 14:54:19 2019
     Elapsed:    23.32 seconds

     Scanned:    40779 files

       Moved:    40779 files          2.94 GiBytes
                                    128.91 MiBytes/sec

• To delete, press the D key.
• To suspend, press the S key.
• To resume, press the R key.
• To exit, press the Q key or the space bar.

You can do the following to a running job belonging to a user (or any job for an administrator):

• Press the D key to delete a job. This process stops the job.
• Press the S key to suspend (pause) a job.
• Press the R key to resume a job.

You can configure the amount of time that a job history is retained after the job is complete. If you delete a job sooner than the configured time, the record is not retained. You can configure what jobs are allowed to be deleted when you press the D key.

Below is the complete set of options for the sntier jobs command:

You can also use optional arguments:

If you do not use the options, then this enters the curses-based full screen display of jobs:

• The --list option displays the same information; one line per job on stdout.
• The --show jobid option reports the complete job status in json.
• The --delete jobid option deletes a completed, suspended or pending job from the system. If a job is running, then it stops the job from executing and places it in a cancelled state.
• The --suspend jobid option stops a job from executing and places it in a suspended state.
• The --resume jobid option resumes a job that was placed in a suspended state.
• The --wait jobid option blocks the CLI until a job is complete. The CLI polls the pooling service to determine the state of a job.
• The --watch jobid option enters the full screen refreshing job details display.

Note: You can view any job, but can only control your own job. An administrator can view and control any job.

Use the sntier submit command to execute a job. You can also use shortcut commands to common operations for different job types by using the move, promote, demote, inventory, checksum and validate commands.

The following are options for the sntier submit command:

You can also use optional arguments:

You can specify the following content options:

You can specify the following task options:

You can configure the following selection criteria:

You can configure the following reporting options:

You can configure the following scheduling options:

If a job submission is successful, then the command reports a job sequence number that you can use to control or query the job status. You can use the --wait or the --watch option to silently wait for a job to complete. You can also enter a job's details display to view activity about a job.

Many parameters of a job are checked upon submission. If an error is returned, then it is printed out and the command aborts with non-zero exit status.

You can use shortcut commands to specify the action and paths to scan directly without extra command line options. In the case of a movement operation, you specify the target pool inline. In the case of a checksum or a validate job, you specify the checksum algorithm inline.

The following two examples execute the same job:

You can use all other optional arguments on these commands.

• For a movement operation, you specify the target pool after one or more paths.
• For an inventory operation, you specify a path immediately.
• For a checksum or a validate operation, you specify the checksum algorithm before a path.

• You can include or exclude a file name from consideration in a job by using the --match or the --exclude option. A regular expression syntax is similar to the syntax in a Unix shell.

  Note: When you run a command from a shell, you must wrap a regular expression in quotes to prevent the shell from evaluating it. The quotes are not part of the pattern itself.

  For example, the following expression matches all files with the extension mxf:

  "*.mxf"

  You can also match a set of characters using square brackets.

  For example, the following expression matches all files ending with .c or .h:

  "*.[ch]"

  You can define a complex expression for multiple patterns.

  For example, the following expression matches all files ending with .mxf, .dpx and .xml:

  "@(*.mxf|*.xml|*.dpx)"

Use the sntier policy command to view and control a policy. If you do not specify an argument, the command enters a display of all the configured policies. You can select a policy and view its details. An administrator can delete a policy from this view. A policy detail view displays the policy configuration and information about its schedule:

The time values display when the policy last performed a scan, when the policy last executed a run, and when the policy is scheduled to perform the next scan.

You can specify the following options:

You can also use optional arguments:

You can use content options:

You can use task options:

You can use selection criteria:

You can use reporting options:

You can use scheduling options:

Creating a policy is very similar to submitting a job and you must configure a schedule of when the policy must run. You can use the sntier policy --add name command to define a policy with a name; however, the name is convenient for reporting displays. Internally, a policy contains a unique ID that you can use to delete a policy.

Note: There is no check on a supplied policy name being unique.

To delete a policy by its ID, use the --delete option. To create a complete dump of the information about a policy, use the --list option.

Note: The output is only available in JSON format.

A template is a partial definition of a job or a policy that is stored by the system with a unique name. You can use a template to fill in a subset of the details of a job and combine it with other content that you provide. The options for the template command are the same as for creating a policy. Usually, a template does not include a content definition - you provide the content definition when you submit a job or if you use a policy command that specifies the template.

Below are the main controls for the template command:

Use the sntier location command to examine files in the file system and provides information about their location in storage pools.

Note: The location command does not interact with the pooling service.

The location command retrieves a list of files or directories as input and provides information about their pool location:

Use the sntier pool command to view the current status of a pool, to define a pool, and to remove a pool from a file system. Any authorized user can view the state of a pool but only an administrator can modify the pool. If you do not specify an argument, the sntier pool command enters a refreshing display that provides information about the current mounted file systems on the pooling service host and their pool status:

The complete command line syntax is:

You can also use optional arguments:

You can configure pool specifications:

Use the command line to define a pool or use the full screen display.

To configure pooling, you must enable the service and then configure the run time for the service (see Install and Configure). On the pooling host, use the sntier service command to enable or disable the pooling service.

You can also use optional arguments:

If you disable the pooling service, then the service stops and it does not restart on a service restart. It does not remove the internal database for configuration information that resides under /usr/cvfs/data/tier_db.dat.

After it is enabled, the sntier config command covers all other aspects.

Note: You must locally enable authentication.

After you enable remote access, you can run a command remotely.

Below is the set of options:

You can use optional arguments:

You can configure authorization settings:

You can configure a job processing setting:

You can configure SMTP email settings:

You can configure Storage Manager web service settings:

You can configure pool settings:

You can configure an individual setting or a group of settings in a single command. You must configure email settings, Storage Manager, or authentication settings as a group. If a change is successful, the current pooling configuration is displayed and you can also see it by using the sntier config command without any options:

You can use the sntier configreset command to reset parts of, or all of the configuration to the default setting:

You can specify optional arguments:

Note: If you change or reset an authentication setting, then you also impact the remote access to the service. You can change an authentication setting remotely, but if you inadvertently remove an administrator account, then you must locally execute the config command.