Truncation and Deduplication / Replication (with and without SM)

We have already mentioned how deduplication allows files to be truncated. “Truncated” in this case means that the extents have been partially or completely removed from disk, and that the snpolicyd daemon must reconstitute the missing extents when a process wants to access them.

We also discussed how we can use the "ls -ls" command to identify truncated files. We looked for files with "0" in the first column of the output of "ls -ls". The 0 means there are no blocks associated with the file. The file size field in the "ls -l" or "ls -ls" output reflects the real size of the file, and is not changed when the file is truncated.

Example

In the earlier example we this saw this output (for a truncated file) after running "ls -ls":

For an untruncated file, the "ls -ls" output might look something like the following:

The 1360 blocks in this file are enough to contain a file of size 1388936 (since 1360 * 1024 = 1392640). However, we might also see a blocks value that was non-zero but not enough to contain the entire file size. This might indicate the following:

• A sparse file (this will not be discussed here)
• A file with a stub left on the disk
• A file that had been partially retrieved

Both Storage Manager and snpolicyd (replication / deduplication) can truncate files and can retrieve truncated files.

Both Storage Manager and snpolicyd can be configured to leave a stub file on disk when a file is truncated. Using the StorNext GUI you can configure the deduplication stub size on the Deduplication tab when creating or editing a replication / deduplication policy. A non-zero stub size must be a multiple of the file system block size.

Both Storage Manager and snpolicyd will retrieve a file when a portion of the file is read that is not already on disk. For Storage Manager there are really three different possibilities for a file’s truncation state:

• File is totally truncated. The file has no block in the file system. Reading any byte of the file causes Storage Manager to retrieve the entire file.
• File is truncated, but there is a stub. Reading within the stub causes no retrieval. Reading anything not in the stub causes Storage Manager to retrieve the entire file.
• File is completely on disk.

For a truncated file that was deduplicated by snpolicyd, there can be partial file retrieval from the blockpool. In this situation there is one more possibility in addition to the three previous possibilities:

• Partially retrieved. The file has some data on disk (besides the stub) but the entire file is not on disk.

Example

Suppose you have a 100 GB file that is truncated. If a process reads a few bytes (at the front or even in the middle of the file), several megabytes of file data are retrieved from the blockpool and the process continues. There is no need for the entire file to be retrieved. If more of the file is read, even larger chunks of the file are retrieved.

You can see the snpolicyd state of a file by using the command "snpolicy -report".

Example

Running the command snpolicy -report /stornext/sn1/dd1/kcm2 gives us output similar to the following:

/stornext/sn1/dd1/kcm2 policy: 18 inode: 1704267 flags: TRUNC TAG mtime: 2010-02-05 11:01:25.960012768 CST ingest: 2010-02-05 11:01:25.960012768 CST size: 1388936 disk blocks: 0 seqno: 16 blk seqno: 3 offset: 0 length: 1388936 tag: 0D4093057370DB4FA7EF8162C23AE416

The line beginning with "flags:" contains the keyword TRUNC. This tells us that at least part of the file is not on disk in the file system and must be retrieved to be used.

If only snpolicyd is managing a directory, snpolicyd can truncate files when the snpolicyd rules are satisfied. This means that the deduplication has happened and the file is bug enough and perhaps old enough. “Large enough” and “old enough” are determined by the deduplication policy parameters.

If only Storage Manager is managing a directory, the Storage Manager truncation rules determine whether and when a file can be truncated. This usually means that all Storage Manager copies have been made and that the file is large enough and old enough. “Large enough” and “old enough” are determined by the Storage Manager policy parameters.

If both Storage Manager and snpolicyd are managing a directory, Storage Manager must do the truncation. Storage Manager can only truncate a file when the Storage Manager rules are satisfied and any snpolicyd data copies have been completed.

You will know that both Storage Manager and snpolicyd are managing a directory if:

• The directory is a deduplicated directory and/or a replication source directory, and the directory is a Storage Manager relation point or is within a Storage Manager relation point.
• The directory is a replication target directory within a Storage manager relation point.

The table below summarizes some of the possibilities for snpolicyd managed directories and when truncation is allowed.

Snpolocyd State of the Directory	Directory is in an SM Relation Point	Directory is Not in an SM Relation Point
Non-deduplication Replication Source	SM can truncate when replications are complete	No truncation
Deduplication Replication Source	SM can truncate when deduplication has happened - even before replication	snpolicyd can truncate after deduplication
Deduplication Without Replication	SM can truncate when deduplication has happened	snpolicyd can truncate after deduplication
Target of Deduplication Source	Files are replicated as truncated (0 blocks). However, SM will eventually store each replicated file, causing it to be retrieved by snpolicyd on the target. Retrieved files must be truncated by SM and can only be truncated after all SM copies are made.	Files are replicated as truncated (0 blocks)
Target of Deduplication Source with "Replicate Deduplicated Content" Off	Files are replicated untruncated and are not tagged (deduplicated). Files can be truncated by SM after all SM copies are made (normal SM rules).	Files are replicated untruncated and are not tagged (deduplicated). Not truncatable.
Target of Dedup source with "Replicate Deduplicated Content" off but deduplication is on in the target policy.	Files are replicated untruncated and are not tagged (deduplicated). Files can be truncated by SM when deduplicated by snpolicyd and stored by SM.	Files are replicated untruncated and are not tagged (deduplicated). Files can be truncated by snpolicyd after deduplication.

The following sections summarize some of the facts above (and add some more information) in a "usage case" or “scenario” format.

Replicating From an SM Relation Point and/or Deduplicating the Relation Point

For a new configuration, create the relation point first. Then make it a replication source by applying an snpolicyd policy with outbound replication enabled.

From the command line, you could use the following commands:

Note: These commands assume that the Storage Manager relation point and replication policy have already been configured.

faddrelation directory_path -c sm_policy_name spolicy -assignpolicy directory_path -inherit replication_policy_name

Remember that the directory should be empty before using fsaddrelation, or else the command will try to unmount the file system (which is often hard to do).

When a file is both an SM relation point and a replication source, the files cannot be truncated by SM until:

1. Either all replications have been completed (non-deduplicated replication)

OR

2. All files in the directory have been deduplicated (deduplicated replication)

If a truncated file is both deduplicated and stored by SM, it can be retrieved by either service. By default we retrieve using snpolicyd (from the blockpool) and only use the SM copy if there is an error retrieving from the blockpool.

You can use the fsretrieve command to force retrieval from Storage Manager instead of from snpolicyd.

Adding Source Replication or Deduplication to an Existing SM Relation Point

The following table summarizes the key points you should consider:

When You Are Making a Directory with Existing SM Managed Files Into This	Then Expect This
Snpolicyd deduplication policy (no replication)	Untruncated files are deduplicated per snpolicyd policy. SM truncated files will not be deduplicated until SM retrieval occurs. (snpolicyd will not retrieve the files from SM.) Once retrieved from SM, files cannot be re-truncated by SM until deduplication is complete. Therefore files may not all be deduplicated.
Snpolicyd deduplication policy that is also a replication source	About the same as above. SM truncated files are not deduplicated or replicated until something causes SM retrieval of the file. Thus there may be some files not deduplicated and not replicated.
Snpolicyd policy that is a replication source with no deduplication	Similar to above. Files are not replicated until something causes SM retrieval. Once retrieved SM will not truncate the file again until each target of the replication policy has its copy. Not all files will be deduplicated unless retrieved.

Adding Target Replication to an SM Relation Point (New or Existing)

When adding targets within an existing SM relation point, the concepts are a little simpler because a new directory is created each time a replication occurs. There are no existing files other than previously replicated files.

Remember that you must specify a directory within a Storage Manager relation point when you want replicated files to be stored by Storage Manager.

When replication occurs into a directory in a Storage Manager relation point, the replicated files become SM store candidates (unless they are links to previously replicated files). Storage Manager can then store the files based on age and size. Age is determined by the file's modification time in the source directory because the access and modification times are replicated when a file is replicated.

Storage Manager can store replicated files after they have passed the minimum time, regardless of whether or not they have been truncated by snpolicyd. Storage Manager retrieves a truncated file from snpolicyd in order to store it to SM tape. Deduplicated replicated files are replicated as truncated files, but they are retrieved by snpolicyd when the replication is into a Storage Manager relation point.

Note the following implications:

1. This means that more file system space will be used when replicating deduplicated files into an SM relation point than is used when replicating deduplicated files into a directory that is not a relation point. In the latter case there is no StorNext process that will cause the file to be retrieved from the blockpool.
2. When the file is retrieved it can be re-truncated after all SM copies have been made. Storage Manager will do the truncation. You can configure the SM policy so that it truncates the file immediately after all SM copies have been made.
3. This behavior is different than in the case where we add replication / deduplication to a SM relation point. Truncated files are not automatically retrieved from SM tape so that they can be replicated or deduplicated, but deduplicated files from the blockpool are retrieved so that they can be stored by SM.

Also note that when Storage Manager processes each store candidate, it needs to obtain its parent directory information. If the parent directory is removed, the candidate is discarded. A store candidate event is generated in the first namespace replication after the file has been changed on the source side. As a result, if the store candidate is discarded during store processing, unless the file on the source is changed again, all subsequent namespace replications will not generate a store candidate event for this file. This could lead to file candidates not being stored promptly. The file needs to become a store candidate again by rebuilding store candidate list before it can be stored to media by Storage Manager.

A store candidate parent directory could be removed on the target if the interval between two consecutive namespace replications is too short. For each namespace replication, the target retains a maximum number of rep copies designated by replication parameter rep_copies or Copies to Keep on Target labeled by the GUI. By default, it is 1. When the number of rep copies exceeds the number designated by rep_copies, the oldest namespace is removed. So if rep_copies is 1, then only 1 namespace is maintained, the next namespace replication will cause the prior namespace to be removed. Thus, if the store candidates are generated from the first namespace replication, when rep_copies is 1 and the store operation has not been finished yet, its parent directory can be destroyed by the next namespace replication.

As a result, to ensure that legitimate store candidates are not discarded, configure the interval of scheduled namespace replication to be long enough or configure enough replication copies to allow the newly generated store candidates to finish storing before its namespace is destroyed by subsequent replication.

If, for example, a store candidate has been discarded due to the parent directory missing, then you can detect the files that were discarded and add them back to the store candidates list by performing the following:

• Run fspolicy -b -y mnt_pnt to add the discarded files back as store candidates

To detect whether store candidates were discarded, check the TSM tac log file for log messages, such as:

Mar 11 20:54:47 sjoshi-rh62-2 sntsm fspolicy[13776]: E1201(8)<1107031798>:msa2dmi1222: dm_get_fileattr failed, ino: 3429418 gen: 0 errno: (2) No such file or directory Mar 11 20:54:47 sjoshi-rh62-2 sntsm fspolicy[13776]: E1200(7)<1107031798>:mda4str1954: parent stat failed ino: 3429418 gen: 0 errno: 2

Alternatively, check whether the file was stored long after the namespace replication was finished by running /usr/adic/TSM/bin/fsfileinfo -o file. Running the command will display whether the file was stored to tape or object IDs, if it was stored to WAStorage.

Adding Storage Manager to an Existing snpolicyd Directory

You cannot add a Storage Manager relation point to an existing replication target directory. You would have to create a new directory, add the SM relation point to that directory, and then create or edit a snpolicyd policy to realize to a directory or a set of directories inside that relation point.

When adding a Storage Manager relation point to any existing directory, one of the following must be true:

1. The directory must be empty.
2. You must be able to temporarily unmount the file system. (It gets remounted as part of the add relation point process.)

If the directory is empty there is nothing to worry about. If it is not empty you must make sure no process is working in the directory and no files are open. The directory should not be NFS exported or Samba shared.

Once the relation point has been added, Storage Manager makes copies of the files according to the Storage Manager policy settings. As mentioned earlier, Storage Manager retrieves a file from the blockpool if it needs to in order to store the file.