Tools > Storage Manager > Alternate Store and Retrieval Location

The StorNext Alternate Store Location feature provides an automatic system for copying files from a main instance of StorNext to a remote instance of StorNext at the same time as copies are made to tertiary storage at the main site. The remote copies can serve as a copy-of-last resort with the Alternate Retrieval Location feature in StorNext. The feature also supports background copying of files that existed before the deployment of the feature. Background copy activity can be limited to avoid overwhelming the use of the StorNext system for new files.

Note: This feature applies only to managed file systems that have at least one configured policy class.

The feature can be enabled per Storage Manager Class Policy per file system. After it is enabled, new files are automatically copied to the remote site. Pre-existing files can also be enabled for automatic copying by providing them as input to the altstoreadd command. On completion of the copy, the completion is recorded per file so that the completion status can be displayed with the fsfileinfo command. Prior to completion of the copy action, the fsfileinfo command reports if a file is enabled for copying, and reports that the copy action has not been performed.

After the copy action has been performed, the main-site instance of StorNext does not maintain a status of the remote-site copies. Ensuring that the remote-site copies are maintained as a read-only image of the main site is an administrative responsibility. Necessary actions to remedy any discrepancies that can accumulate at the remote site as a result of deletions or modifications caused by either user activities or equipment failure are also administrative responsibilities.

Note: Zero-length files are not stored by Storage Manager class policies; since the Alternate Store Location feature depends on policy processing of files, zero-length files are not copied to the remote location.

In situations where file retrieval fails because the normal file copies cannot be retrieved from the machine on which StorNext Storage Manager resides, the Alternate Retrieval Location feature enables you to retrieve a copy of the truncated file from a different machine. (Both machines must be using the same operating system.

For example, if StorNext creates two copies of each file, when retrieving a truncated file StorNext tries to retrieve Copy One and then Copy Two. If neither of these copies can be retrieved and this feature is not enabled, the retrieval fails. However, if this feature is enabled for the file system, after retrieving Copy Two fails Storage Manger tries to retrieve the file from the alternate machine you specified during feature setup. Because the file already exists in the StorNext file system, it retains the permissions it already has. No permissions are changed based on the file on the alternate machine.

Note: This feature applies only to managed file systems that have at least one configured policy class.

For this feature to work correctly, it is your responsibility to make sure all files you might want to retrieve are copied to the alternate machine. Otherwise retrieval will fail when StorNext attempts to retrieve the file from the alternate location and cannot find the file.

1. Review and perform the procedures in the following topics:

   • Installation, Configuration and Upgrade
   • User Customizable Transfer Script
2. On the Tools menu, click Storage Manager, and then click Alternate Store Location or Alternate Retrieval Location.

3. At the field Remote Node IP/Host Name field, enter either the IP address or the host name of the remote server from which you would like to retrieve data.

4. Select Enable to activate the Alternate Store Location or Alternate Retrieval Location feature.

5. At the field under the Remote Path heading, enter the directory path for the remote node (server).

6. Click Apply to save your changes, or Cancel to exit without saving.

7. When the confirmation message appears, click Yes to proceed or No to abort.

8. After a message informs you that the Alternate Store Location or Alternate Retrieval Location was successfully added, click OK.

The StorNext Alternate Store Location feature runs an agent on both the main site and the remote site, which must be at the same StorNext release level. Each site can serve as an Alternate Store Location main site, remote site, or both at the same time. One StorNext instance can serve as the remote site for multiple main-site StorNext instances. Each main-site file system must have a distinct remote-site file system that cannot be shared with any other main site. A main site can only send copies to a single remote site.

The fs_altstore resident process of TSM provides both the main site and the remote site processing. It starts and stops with TSM, runs in an idle mode when it is not configured as a main site, and is always ready to serve as a remote site. Communication between the resident processes on the main and remote sites is with a TCP/IP socket connection to a designated port. The altstore service can be controlled with the fsschedlock utility.

Transferring of files between the main site and the remote site is done with a user-modifiable script. The default script provided with StorNext uses the Secure Shell scp command for maximum security. Other transfer schemes, such as, FTP or NFS can be substituted according to the security and performance requirements at customer sites. Files are transferred to a staging location at the mount point for each remote file system that stores remote copies. The name of the staging folder is .AltStoreStagedir under the mount point of the file system. Files are moved or copied from there to each file’s target pathname. The per-file owner, group and rwx permissions are set to match their values at the main site. Files are read on the main site with root permission. They are sent to the staging directory with the altstore user’s permissions. The remote agent does the final actions with root permission.

When a file is created under a policy with Alternate Store Location enabled, the file is added to a database table as a demand type request for remote copying. Files that were created before the feature was configured can be added to the table as a background type request for remote copying. All of the demand-type requests have priority and are processed before any background-type request is processed. There is a configurable limit on the number of simultaneous demand-copy transfer processes that can be started. There is a separate configurable limit on the subset of those transfer processes that can be used for background copies when they are not being used for demand copies. Experimenting with these limits can help to tune your system so that throughput is maximized while keeping system responsiveness at an appropriate level for other activities.

Truncated files that are enabled for copying and are in the database table of requests are automatically retrieved. The truncation status is checked at the point of calling the transfer script for the file. These files are changed to the background request type and placed on an internal queue for retrieval. This queue is allowed to grow to a maximum size for a limited amount of time to allow the retrieve operation to optimize the order of retrievals within a single request. The queue-size parameter is described in ALTSTORE_RTRV_WAIT_COUNT, and the parameter for time is described in ALTSTORE_RTRV_WAIT_TIME. When a large number of new demand-type requests get created, these can cause the retrieve-queue to be emptied to give priority to the demand requests. Eventually, the truncated background-type requests will be reloaded and retrieved. When they have been retrieved, they are converted to demand-type requests for expedient copying to avoid being re-truncated before that can be completed.

When a file is enabled for copying, but has not been copied yet, manual requests for truncation with the fsrmdiskcopy command will state that “not all copies…are stored”, and will reject the truncation request. The retention policy for immediate file cleanup will not apply when the remote copy has not been made. However, truncation-policy processing that depends on file-system fill levels can truncate files that do not have a remote copy.

The intent of the Alternate Store Location feature is to create copies of file systems on a remote StorNext system with matching pathnames, user and group owners, and simple permissions (rwx). However, the feature is enabled in policy classes, which are applied to folders within file systems, and the mount points for the top-level directories of file systems can have different pathnames on the main and remote StorNext systems. Displayed file and directory owner and group names will be identical if the mappings of user and group names to UID and GID values are the same on both systems.

The indication in fsfileinfo output that a file copy exists is simply a record that a file copy completed successfully. Once a copy has been made, the Alternate Store Location feature does not perform any further tracking of the remote copy at the main site.

It is possible for a file to be marked for copying, but not to have a row in the database table to cause the copy to be performed. In that event, periodic rebuild processing will discover the discrepancy and will add a row to the table for the file to be copied.

Quantum provides a default transfer script that uses SCP for the highest level of security. The choice of file-transfer technologies is dependent on local considerations for performance and security.

When the Alternate Store Location feature is configured, the same script is used for the Alternate Retrieval Location feature. Its two modes (store and retrieve) are distinguished by the value of the fourth parameter, which is set to ALTSTORE for the Alternate Store Location feature.

When the transfer script is in ALTSTORE mode, it uses the root user ID at the main site for sending files since it must be able to read all users’ files, and uses the altstore user ID to write files into the remote site’s staging directory. When the transfer script is also used for the Alternate Retrieval Location feature, it must use root privileges on the remote site as well to be able to read any file.

Instructions are provided inside the script for creating the altstore user and setting up Secure Shell (SSH) credentials for running the script. To test the credentials, while running as root on the main site, you should be able to use the scp command to copy a file to the altstore user’s home directory on the remote site.

When both the Alternate Store Location feature and the Alternate Retrieval Location feature are configured, do not create the altstore user on the remote site. Instead, set up SSH credentials for the root user to access the remote site as root, and change the value of ALTSTORE_TXFR_USERNAME to root in the /usr/adic/TSM/config/fs_sysparm_override file.

If you prefer to use a different script, create a new executable file under a new name so your custom script is not overwritten when StorNext is upgraded. Set the ALTSTORE_TXFR_SCRIPT parameter in the /usr/adic/TSM/config/fs_sysparm_override file to the pathname of the executable. The Alternate Store Location parameters for the executable are as follows:

1. Main-site full pathname of the file to transfer.
2. Remote site IP address or Host Name (as configured with fsaltnode).
3. Remote-site full pathname of the file in the staging area.
4. The word “ALTSTORE”, which indicates that the executable is being invoked for stores. When the parameter is missing, the executable is being invoked for retrieves.

The following commands are for operating and administering the Alternate Store Location feature. These are described in greater detail in their respective man pages.

altstoreadd

Enables the Alternate Store Location remote-copy feature on files and adds them to the alternate store candidate list. It can also be used to verify that a remote copy has been made for a file or list of files.

altstoremod

Display or manipulate the Alternate Store Location feature’s alternate store candidate list.

fsaltnode

Add, modify or delete Alternate Retrieval Location information in the Tertiary Manager database. The settings made with this command also apply to Alternate Store Location. The information from this command can also be set and displayed by the StorNext GUI as demonstrated in Alternate Retrieval Location Configuration.

fsfileinfo

Generate a report about files known to the Tertiary Manager. A field in the output describes if the Alternate Store Location feature is enabled and whether the copy action is pending or has been completed. Information from this command can also be displayed by the StorNext GUI as demonstrated in Sample File Report.

fschfiat

Modify the class attributes of a file. This can be used to disable or enable copying on a per-file basis.

fsclassinfo

Report policy-class processing parameters, associated directory paths, and affinity lists. This command can display the alternate store location state per policy class. Information from this command can also be displayed by the StorNext GUI.

fsschedlock

Command used for locking/unlocking some automated features. The altstore feature type controls the alternate store location feature. There may be a delay of a few minutes after the lockout schedule is reached before the fs_altstore receives the command, completes in-process transfers, and stops processing new transfers.

fsmodclass

Modify the processing parameters of a policy class. This command can change the alternate store location state per class policy. This information from this command can also be set and displayed by the StorNext GUI.

fsrmcopy

The -r option to fsrmcopy invalidates the remote copy, if it exists, and a new copy is created.

Log, trace, and history information from the fs_altstore resident process is recorded under the /usr/adic/TSM/logs directory in the trace/trace_06, tac/tac_00 files, and in the /usr/adic/TSM/history/hist_06 file.

Operation of the Alternate Store Location feature requires that the main site, the remote site, and the network between them are up and running. Whenever all of these are brought on-line, file transfers will resume automatically after a brief pause, typically 20 seconds.

Operation can be suspended during periods as needed for system performance or other reasons by use of the fsschedlock command as described in section fsschedlock. When the lockout period ends, there may be a few minutes delay before file transfers resume. The delay is affected by the setting described in ALTSTORE_POLL_TIME.

The backlog of pending file copies can be viewed as described in section altstoremod.

The status of copy operations per file can be viewed as described in section altstoreadd.

Files that existed before their policies had the Alternate Store Location feature enabled are not copied without taking additional steps. It is recommended that the set of these files be added in subsets of the total list, rather than all at once. This allows for optimizing the rate of copy completions by organizing the retrieval of files for a minimum amount of tape handling. When the Alternate Store Location feature encounters a file that is truncated, it will retrieve the file automatically with a small amount of retrieval optimization, but it can end up bottle-necking file copies behind inefficient retrievals when there is a large set of truncated files in the list of files to be copied.

Best practices for the copying of pre-existing files is to:

1. Identify the set of all files you wish to copy that are not already enabled for Alternate Store Location copying.
2. Organize the full set into subsets that can be retrieved efficiently by subset.
3. Retrieve one complete subset of files to disk.
4. After the subset retrieval has completed, submit the subset to the input of the altstoreadd command as described in section altstoreadd, and start the retrieval of another subset of files.
5. Monitor the completion of background copies as described in section altstoremod. When a subset has been copied, truncate the files listed in the subset.

Log messages are written as described in section Log Files. These may provide clues for the diagnosis of operational problems.

The most common problems preventing copies from being made are in the transfer operation. The user-customizable transfer script, as described in section User Customizable Transfer Script, depends on the proper setup of authentication credentials and the ability to make a connection through firewalls etc. Communication between the fs_altstore resident processes at the main and remote sites is also dependent on communication by TCP through firewalls, etc. as described in section ALTSTORE_AGENT_PORT_NUMBER.