Considerations When Scheduling Truncation and Relocation Policies

Because StorNext allows you to set the minimum truncation and relocation times for a class in hours or minutes (as well as days), it may be desirable for sites to schedule policies to run more frequently. By default when StorNext is first installed, a class truncation policy is run for each installed class starting at midnight. Relocation policies are also run at this time for classes with relocation enabled.

Note: These policies are not visible in the output from fsschedule, but are run automatically by the daemons that monitor fill levels in managed file systems. If desired, these policies can be locked out by using the fsschedlock command.

If there is a need to run these policies more frequently because you cannot wait until midnight to have files truncated (and you do not want to truncate the files immediately upon storing them) then you will need to schedule more of these policies at the desired times. That will currently have to be done by setting up cron jobs to invoke the fspolicy commands. (Note the section above on creating the policy scripts and scheduling the cron jobs.)

When scheduling the extra policies take these considerations into account. Mainly these considerations are the time required to run and the effect they have on other processing on the machine. Note that the test machine where these metrics were gathered had 8 CPUs (3 GHz) and 16 GB memory. Also note that the number of candidates referred to below are for files whose blocks are disk resident and do not include files that are already truncated.

• A truncation policy that scans 1 million candidates looking for files to truncate can take over 25 minutes even when no files are actually truncated. The process uses 15% of a CPU and .3 GB of memory. (These numbers will be the same for a relocation policy in that the candidate processing is done in the same way.)
• To scan and truncate these same 1 million files takes 45 minutes, uses 15% of a CPU and use .4 GB of memory. (For relocation the time to relocate is totally dependent on the file sizes as data is actually copied from one location to another.)
• While truncation was running on a file system, writes to that file system were observed to be up to 70% slower and reads were 5% slower.
• The storing of other managed files was observed to be up to 25% slower than normal while truncation was running.

Because of the impacts of candidate processing and truncation only one truncation process is allowed to run simultaneously. Note again the section on cron setup that mentions how to run these policies sequentially. It is easy to see as well that there is an expense to running these policies even if nothing is actually truncated. Care should be taken to only run the policies as often as is absolutely needed so unnecessary impact on other system activity can be avoided. If you have a system in which it is desired that truncation run every 5 minutes then there can't be any more than 111,000 files to truncate every 5 minutes or the system will not keep up. Note that this is assuming no other activity so the real number is probably going to be lower. If the desire is to truncate hourly then the files to truncate maxes out at around 1.33 million files. The recommendation here is to run these policies as infrequently as possible to meet your space requirements.