Storage Policies Overview

• A Storage Manager storage policy defines how the Storage Manager feature manages files in a directory and its sub-directories. Specifically, these are the available Storage Manager storage policy settings:

  • Number of copies to create

  • Media type to use when storing data

  • Amount of time to store data after data is modified

  • If disk-to-disk relocation is enabled, the amount of time (in days) before relocating a file

  • Amount of time before truncating a file after a file is modified

• A Storage Manager storage policy is used to make copies of files to tape or to storage disk, and also to relocate files between stripe groups within a file system.

• Number of copies to create
• Media type to use when storing data
• Amount of time to store data after data is modified
• If disk-to-disk relocation is enabled, the amount of time (in days) before relocating a file
• Amount of time before truncating a file after a file is modified

Storage policies can be related to one or more directories. In this situation, all files in that directory and sub-directories are governed by the storage policy.

Note: The connection between a storage policy and a directory is called the relation point.

• A directory in which to store backups every night is created. This directory is seldom accessed after the files are copied over. A storage policy could be set up to create two tape copies of the files, store one copy of the files to LTO media after residing on disk for 10 minutes, and then truncate the other set of files immediately after storing the other set to tape in order to free up disk space. This policy can be associated with a directory such as: /sandsm/dsm1/backup.
• A directory has been created to store all documents that are accessed frequently, and if truncated, need to be retrieved quickly. In this cas, the policy could be configured to create a single tape copy, store the files to LTO media 15 minutes after being on disk, and then truncate after 60 days of non-use. This policy can be associated with a directory such as: /sandsm/dsm1/docs.

For large files, a file may be broken up into multiple segments, and each segment is stored independently. In order to decide whether to break the file into multiple segments, Storage Manager uses the MED_SEG_OVER_XXX system parameter(s) to determine the segment size for each file. The default size varies by media type. By default, all the segment sizes are a multiple of 2 MiB. Quantum recommends that any overrides also be a multiple of 2 MiB.

There is a trade-off to selecting a larger or smaller segment size. Below are some of the advantages and disadvantages for different segment sizes.

• A large segment size reduces the number of segments that are tracked in the metadata and database for a large file. Each segment requires an entry in the metadata and the database, increasing the overall metadata and database content. Therefore, a larger segment size reduces the number of metadata and database entries, thus reducing the size of metadata consumed. Fewer number of resources are required when storing the segments. When the storage media is tape, less tape drives could be used when storing the file.
• A small segment size has less I/O and therefore quicker retries of failed transfers, which might be of benefit if the network or storage system is unreliable. A smaller segment size could also reduce the overall storage and retrieval time, because Storage Manager can store and retrieve multiple segments concurrently.

If you change segment sizes and decide on smaller sizes, you must be aware of the MAX_SEGMENTS_PER_FILE system parameter. As mentioned, having more segments introduces more metadata and database content. Having too many segments might results in too much overhead so there is a system limit defined by MAX_SEGMENTS_PER_FILE, which defaults to 50. See the /usr/adic/TSM/config/fs_sysparm.README file for more information on this parameter.

You can update the fs_sysparm_override file to set and modify StorNext system parameters for segments. See Manually Modify the Storage Manager Configuration for more information and Object Storage Segment Size for additional information on Object Storage segment sizes.

Checksum Generation

All files are stored as segments and each file consists of one or more segments. The following text refers to segments instead of files.

If a policy class is configured to generate checksums, a checksum is generated for the entire file if it only has one segment. If the file is broken into multiple segments based upon the MED_SEG_OVER_XXX parameter, then a checksum is generated for each stored segment of the file, so there is no overall checksum for the file. The checksums are generated on a segment basis since the segments are stored independently of one another and can be processed concurrently. The data for the entire file segment is not read into memory to do the checksum processing. The checksums are computed inline as data is being read from the file and written to the storage media.

After the data is written to the storage media, it is not read back from that media to verify the checksum, as this would have significant impact on store performance. The checksum value for each segment is stored in the database and inode.

If there are multiple copies being generated, the checksum is generated for each copy since the copies could be on different media types which means the segment sizes could be different.

The fsmedcopy and fsfilecopy operations also allow checksums to be generated for file segments. Both commands have an option to generate checksums for the segments being copied if a checksum does not already exist. The checksum is generated as data is read from the source media. Similar to store processing, there is no checksum validation performed during these operations.

Checksum Validation

If a policy class is configured to validate checksums, and a checksum was generated when the file segment(s) were stored, then a checksum is generated for each segment as they are read from the storage media. These values are compared to the values that were generated at store time for each segment. If any of the values do not match, then the retrieve of the file fails.

Topic	Description
Add a Storage Manager Policy	Create a StorNext Storage Manager storage policy.
View, Run, Edit, Delete or Test a Storage Policy	View, edit or delete an existing storage policy.