Add a Storage Manager Policy

The sections below provide information about how Storage Manager Disk to Disk relocation works and when data is relocated.

Storage Manager persists data archived to object or cloud storage as key:value objects, each keyed on a globally unique identifier generated by the Linux library uuid_generate(). There is at least one distinct object for each archived version of each file, but may be more, since large files might be segmented.

In order to provide additional information about the object, the Metadata feature can be enabled to store user-defined metadata along with each stored object. The metadata is stored as key-value pairs and includes the copy number, last modification time, file offset, file key, path, segment number, and version. The path is URL encoded if it contains non-ASCII characters.

The metadata allows the file path for the object to be identified and provides the information needed to restore the object.

Each object storage system supports user-defined metadata, prefixed to distinguish them from system headers:

REST API	Metadata Prefix
S3	x-amz-meta-
Azure	x-ms-meta-
Google	x-goog-meta-

If you enable Metadata for a policy, headers are added on upload requests, allowing the object to be restored, if needed. The following data, represented as key-value pairs, are added for each object:

Name	Description	S3 User Metadata	Azure User Metadata	Google User Metadata
cpynum	Copy number.	x-amz-meta-cpynum	x-ms-meta-cpynum	x-goog-meta-cpynum
mtime	Last modification time.	x-amz-meta-mtime	x-ms-meta-mtime	x-goog-meta-mtime
offset	Byte offset in the file where the object begins.	x-amz-meta-offset	x-ms-meta-offset	x-goog-meta-offset
oneup	Database file key.	x-amz-meta-oneup	x-ms-meta-oneup	x-goog-meta-oneup
path	Absolute file pathname.	x-amz-meta-path	x-ms-meta-path	x-goog-meta-path
segnum	File segment number of object.	x-amz-meta-segnum	x-ms-meta-segnum	x-goog-meta-segnum
version	Version number of the file.	x-amz-meta-version	x-ms-meta-version	x-goog-meta-version

You can enable user-defined metadata with each stored object either by using the Metadata feature on the Steering tab or when you enable object metadata using the policy class commands, fsaddclass and fsmodclass.

For example, to enable object metadata on all object storage copies for policy class class1:

To modify a policy class to enable object metadata on copy 2:

You can manage the order in which copies are used for retrieval either by using the Retrieve Order feature on the Steering tab or by defining the order using the policy class commands, fsaddclass and fsmodclass. If the retrieve order is not explicitly defined, the default retrieve order is used, which retrieves the copies in the order of their copy number, with the exception that copy 2 is always retrieved last.

For example, to create a policy class class1 with three copies enabled, using the first available copy in the order copy 2, copy 1, and then copy 3:

To modify a policy class to retrieve copy 2 first, followed by copy 3 and then copy 1:

Overview

The Copy Expiration feature provides for automatic removal of copies based on the time since the last access of a file, which allows file copies to be stored on mid-range-performance storage without requiring excessive capacity or excessive administration. This simplifies management of high-performance low-capacity storage tiers.

Old data can be purged automatically to make room for new data. When a file copy has expired and needs to be retrieved from a lower-performance tier, the copy can be stored again and then deleted again after a configurable amount of inactivity. For example, consider a class policy that calls for the storage of two copies: one on sdisk and one on tape. The class copy configuration could state that the sdisk copy should be expired after a file has not been accessed for 30 days. This would free sdisk for use by more recently accessed files, which could improve retrieve-latency performance for data that has been removed from disk.

Operating Characteristics

The Copy Expiration feature is configurable per class per copy. Individual copy numbers may be configured to be expired when a file has not been referenced for a specified amount of time, and optionally to be restored after the file is referenced, which updates the file's access time. All copy numbers may be configured for expiration, in which case the file is automatically removed after all copies have expired.

Note: Copy expiration configuration changes to the class policy apply immediately to all files in the class.

The automatic aging and deletion process is driven by an expiration-time value that is added to the access-time value of each file, to be compared to the current time to determine the eligibility of a copy for deletion.

For example, you could have two policies which are schedule to perform the following:

1. One policy identifies the eligible copies.
2. One policy deletes the eligible copies.

• When a new file or a new version of an existing file is created, Copy Expiration does not apply until after all of the file's copies have been made.
• When a copy number is expired, all versions of that copy number are removed.
• When a file that has expired copies is accessed, expired copies that have been configured for restore-on-reference are recreated. The effect is delayed when the file is on disk as compared to when the file is truncated. Accessing the on-disk file does not need to initiate a retrieval, so the Storage Manager subsystem is not notified of the file activity. The next notification and opportunity for action occurs when the file is being truncated, at which point the truncation is delayed while the copies have been restored.

Note: You can also manually delete copies. The results are the same as with automatic deletions, except that the expiration-time value is irrelevant.

Example

In the image below, the StorNext GUI is used to configure three copies to sdisk (1), tape (2), and tape (3), which expire in 15 days, 1 year, and 3 years after the last access respectively. The Restore On Reference option is enabled for the first two copies, which means that the copies will be recreated for the current version if the file is accessed after one or both of the copies has expired. The third copy may have more than one version, but only the current version will be recreated. Because all copies are configured for expiration, the file automatically will be removed from disk after the last copy expires.

Note: The policy may also be configured to immediately truncate the file on disk after the last copy expires.

Note: If you configured a policy that contains copies that have all been configured for Copy Expiration, the following confirmation dialog appears:

"If you select to expire all copies, and when the last copy of a file expires, the file and all stored copies are removed. The file is DELETED from the file system and it is DELETED from all storage destinations. This operation cannot be undone; you are no longer able to retrieve the file and the file is no longer accessible at all from any location."

Configuration Using the Command Line Interface

You can use any of the following commands to configure the Copy Expiration feature:

• Execute the fsmodclass command to configure copies to expire per class.

• Execute the fsschedule command to configure scheduled processing.

  Note: You can use the StorNext GUI to schedule a copy expiration event to run automatically based on a specified schedule. See Tools > Storage Manager > Scheduler.

• Execute the fsclassinfo command to display, for each class, the expiration and restore-on-reference settings per copy and the expire-all-copies setting.
• Execute the fsexpcopy command for manual expirations and file deletions (when all copies are expired).

• Execute the fsfileinfo command to display the expiration status for a file in the Expired Copies field with a value for the number of expired copies followed by a parenthesized comma-separated list of copy numbers that are expired.

• Execute the fsschedule command to schedule processing of all Storage Manager features.

For additional information on all the commands listed above, refer to the StorNext Man Pages Reference Guide.

Log Messages

The following are examples of the /usr/adic/TSM/logs/tac/tac_00 log messages from the scheduled running of the fsexpcopy feature. In the example below, both copies can be expired, which results in the deletion of the file from the file system.

In the following example, a file with copies 1 and 2 is in a policy configured for 3 copies with expiration for copies 1 and 2, but not for copy 3. An error is logged for the file because the policy is not configured to expire all copies, which is what would happen if copies 1 and 2 expire. A file can be in this state if it was created when the policy was configured to make 2 copies and then the policy was changed to 3 copies. The expiration feature is designed to prevent expiration of the last copy unless policy is explicitly configured to allow that. In this example, either copy 1 or copy 2 could be the last copy remaining, depending on configured expiration intervals and on manual expirations that were executed before automatically scheduled ones.