General Operating Guidelines and Limitations

Below is some guidance on how a hard link is processed in a StorNext managed file system:

• For a file that crosses a file system, the hard link fails and is not allowed.

• For a file that crosses a policy class, the hard link fails and is not allowed.

• For a file in the same policy class, it is a true hard link. StorNext does not double dip, it will "follow". When you inspect the hard link, it is actually the same file. If you retrieve the hard link, it retrieves what it is pointing to.

• You cannot move a directory crossing policy border.

To successfully remove a relation point from a storage manager policy, the directory relation point must also be empty.

For additional information, refer to the fsrmclass(1) command in the StorNext 6 Man Pages Reference Guide.

Caution: If you stop snpolicyd on a replication source, then you might encounter performance issues. Quantum recommends you do not run the following command on a replication source:

During a TSM shut down, the following operations are blocked:

• If a managed file system is full, then writes that allocate new space are blocked due to the NOSPACE event (most writes allocate new space unless you are re-writing a file).

• If a rename is attempted, it might be blocked due to the RENAME event. This is more common when renameTracking is enabled in the .cfgx file but can also occur when it is not.

  The renameTracking Feature

  Some Microsoft applications end up making temporary copies of operational files while they are being updated. For example while a file myDocument.doc is being updated with Word it may be renamed to a random sequence of characters like ABC123DYXab. When Word completes the updates it will save the new myDocument.doc file and remove ABC123DYXab. When Storage Manager recognizes that file ABC123DYXab has been removed it makes it recoverable but with the current temporary name. This makes tracking old instances of myDocument.doc difficult if not impossible to accomplish. To get around this scenario the renameTracking feature is provided. You can enable this feature on in the file system (see the snfs_config(5) man page) and makes tracking of old instances of these temporary files possible.

  If you enable the renameTracking feature, as a file is renamed it is made recoverable at that time. For example, in the scenario described above as Word renames myDocument.doc to ABC123DYXab, Storage Manager makes myDocument.doc recoverable with its original name before the rename proceeds; that renamed instance becomes inactive. Then after Word is done, the new myDocument.doc becomes the new instance of the file and the previous instance becomes available for recovery using fsrecover(1) with the correct name.

  Note: The renameTracking functionality is NOT a general purpose feature for keeping the Storage Manager database up to date with rename activity. If you enable the feature, then the files that are being renamed are re-stored. This is not, in general, a desirable behavior except in the specific case described.

• If a managed file is opened O_TRUNC, it is blocked due to the TRUNCATE event.
• If a truncate(2) call is performed on a managed file, it is blocked due to the TRUNCATE event.
• If you attempt to read an OFFLINE file, the operation is blocked due to the READ event.

During a SRVCLOG shut down, the following operations are impacted:

• New RAS events do not appear in the StorNext GUI and are not sent.
• New Admin alerts do not appear in the StorNext GUI and are not sent.

During a mysql shut down, the following operations are impacted:

• New RAS events do not appear in the StorNext GUI and are not sent.
• New Admin alerts do not appear in the StorNext GUI and are not sent.
• The async Web Services using wsar_agent are not available (the Web Services do not collect StorNext metrics statistics).
• Most operations in the StorNext GUI do not work.

Solaris hosts may need to rescan disk devices after StorNext labels have been applied.

In particular, when a StorNext label is put on a LUN less than 1TB in size, Solaris hosts will not be able to use that LUN until they have done a device rescan. A device rescan is accomplished with a boot flag:

In the meantime, work around this issue by rescanning devices using the boot flag reboot -- -r

If the labeling operation was performed on a Solaris host, that host does not need to do the rescan. However, some intermediate versions of the Solaris 10 Kernel Jumbo Patch break the necessary functionality to support this; please be sure you have applied the latest Solaris 10 Kernel Jumbo Patch before labeling any StorNext LUNs.

Current versions of the Linux DM Multipath driver assign a default value of 1000 for rr_min_io which is too high for most configurations having multiple active paths to storage. Using a smaller value such as 32 will typically result in significantly improved performance. Refer to the RedHat or SUSE documentation provided with your version of Linux for details on how to apply this setting.

Note: Experimentation may be required to determine the optimal value.

StorNext File System does not support the Linux sendfile() system call.

This issue causes Apache web servers to deliver blank pages when content resides on StorNext file systems.

This issue also affects Samba servers running on Linux.

The workaround is to disable sendfile usage by adding the following entry into the Apache configuration file httpd.conf:

The workaround for Samba servers is to add the following line into the configuration file:

Many versions of Unix run the command, updatedb, using a nightly cron script to build a database used by the commands, locate, slocate and mlocate.

If you mount StorNext file systems, then they are traversed by this cron job, which might have a dramatic impact on the performance of other applications currently using these file systems and may appear to hang.

Perform the following steps (based on Linux version) to prevent the cron script from traversing StorNext file systems.

RedHat Enterprise Linux 4, 5, and 6

Add cvfs to the list of file system types to be skipped. This is usually done by modifying the PRUNEFS line in the /etc/updatedb.conf file to read:

PRUNEFS="cvfs sysfs selinuxfs usbdevfs devpts NFS nfs nfs4 afs sfs proc smbfs cifs autofs auto iso9660 udf"

SUSE Linux Enterprise Server 10, and 11

The optional findutils-locate package is used to build the slocate database. The default behavior is to disable building the database. If enabled, to prevent cvfs file systems from being scanned, add “cvfs” to the list of file system types to be skipped. This is usually done by modifying the “UPDATEDB_PRUNEFS” line in the /etc/sysconfig/locate file to read:

UPDATEDB_PRUNEFS="cvfs"

StorNext users migrating their metadata controllers from Apple Xsan to Linux should be aware of the following upgrade considerations:

• If the file system is running Xsan 2.1.1 or earlier, it should be a simple upgrade: just replace the MDC.
• If the file system is running Xsan 2.2 or later with “NamedStreams No” (which is the default for Xsan 2.2,) it should also be a simple upgrade: just replace the MDC.
• If the file system is running Xsan 2.2 or later with “NamedStreams Yes,” you must completely remake (reformat) the file system. For obvious reasons, you should do a complete backup before migrating.

Due to the way Linux handles errors, the appearance of SCSI “No Sense” messages in system logs can indicate possible data corruption on disk devices.

This affects StorNext users on Red Hat 4, Red Hat 5, Red Hat 6, SuSe 10 and SuSe 11.

This issue is not caused by StorNext, and is described in detail in StorNext Product Alert 20.

For additional information, see Red Hat 5 CR 468088 and SUSE 10 CR 10440734121.

Software firewalls such as “iptables” on Linux and Windows Firewall can interfere with the proper functioning of StorNext and result in unexpected errors unless specifically configured for use with StorNext.

Quantum strongly recommends that all software firewalls be disabled on systems used as StorNext clients and servers. If required, StorNext can be configured for use with hardware firewalls.

For more information, refer to the fsports man-page or help file and the section Port Configuration.

Changing the haFsType parameter in a file system configuration file to one of the HA types, and then (re)starting its FSM enables HA-specific features that change the functionality of StorNext.

When the HaShared or HaManaged types are configured, other changes must be completed by successfully running the cnvt2ha.sh script, which is indicated by the creation of the

/usr/adic/install/.snsm_ha_configured touch file ($SNSM_HA_CONFIGURED environment variable). No conversion is done or necessary for SNFS only (HaUnmanaged) configurations.

If the conversion is not successfully completed, the HaManaged FSMs will not start, and the HaShared FSM will cause an HA Reset when it is stopped.

To remedy this situation, edit every FSM configuration file to set its haFsType parameter to HaUnmonitored, then run the following commands to avoid the HA Reset in this special case only:

Subtree Check Option in NFS No Longer Supported

Although supported in previous StorNext releases, the subtree_check option (which controls NFS checks on a file handle being within an exported subdirectory of a file system) is no longer supported beginning with StorNext 4.0.

SuSe Linux distributions automatically associate the FQDN of the local machine with the address 127.0.0.2 in the /etc/hosts file. There is no benefit from doing this when the machine is connected to a network that can resolve its name to an IP address.

However, the existence of this entry can sometimes cause a failure of configuration synchronization within and between the server computers in an HA configuration. For this reason, the 127.0.0.2 entry should be deleted from the /etc/hosts file.

cpuspeed, an external Linux service on recent Intel processors, is not correctly tuned to allow StorNext to take advantage of processor speed. Suse systems may also be impacted, as may AMD processors with similar capabilities.

On processors with a variable clockspeed (turboboost), the cpuspeed service on Redhat controls the actual running speed of the processors based on system load.

A workload such as a heavily used FSM and probably Storage Manager does not register as something which needs a faster cpu. Turning off the cpuspeed service has been shown to double metadata performance on affected hardware.

Looking at the reported CPU clock speed by doing cat /proc/ cpuinfo while the system is under load displays if a system is impacted by this issue.

During testing a Quantum PX502 library running Red Hat 6.1 did not finish power-on diagnostics. When the same test was run on a PX502 library running either Red Hat 5.X or SUSE 10 / 11, power-on diagnostics completed and the system initialized without any issues.

The workaround for this issue is to disconnect the SAN from the library running Red Hat 6.1. If the library powers on while the SAN is disconnected from the library controller, the library finishes its power-on diagnostics and performs an audit of the library. Subsequently reconnecting the Red Hat 6.1 server to the SAN (library ready) causes the library to perform a new physical audit of the library.

Note: Testing was performed on the Red Hat 6.1 system which did not have StorNext loaded or running.

If you create a symbolic (soft) link in Linux to a directory on a StorNext file system, the link cannot be used by Windows. Windows also cannot process a symbolic link which contains a path to a file in another file system.

When a StorNext file system is mounted to a drive letter or a directory, configure the Windows backup utility to NOT include the StorNext file system.

Virtual Hard Disk (VHD) files are not supported on a StorNext file system.

In StorNext releases prior to 3.5, the StorNext Windows client attempted to keep the UNIX uid, gid and mode bits synchronized with similar fields in the Windows security descriptor. However, these Windows and UNIX fields were often not synchronized correctly due to mapping and other problems. One consequence of this problem was that changing the owner in Windows incorrectly changed the UNIX uid and file permissions and propagated these errors into sub-directories.

In release 3.5, the StorNext Windows client sets the UNIX uid, gid and mode bits only when Windows creates a file. The StorNext Windows client will no longer change the Unix uid, gid or mode bits when a Windows user changes the Windows security descriptor or Read-Only file attribute.

If you change the UNIX mode bits and the file is accessible from Windows, you must change the Windows security descriptor (if Windows Security is configured On) or Read-Only file attribute to ensure the change is reflected on both Windows and UNIX.

StorNext upgrades on Vista machines can fail in the middle of installation. This problem is caused by the way Windows Vista handles software upgrades. A related error is described in Microsoft article 263253 (see http://support.microsoft.com/kb/263253).

To work around this issue, follow these steps:

1. Click Start, and then click Run.
2. In the Open box, type Regedit and then click OK.
3. On the Edit menu, click Find.
4. In the Find what box, type Snfs_XXX.dat and then click Find Next.
5. If the search result selects a string value called PackageName, continue with these steps. Otherwise, repeat steps 3-4.
6. Double-click the PackageName string value.
7. In the Value data box, change the installation directory path to the new pathname. For example if the old installation directory path contained OCT10, change that to the current path (for example, NOV12.)
8. On the Registry menu, click Exit.

If you are using the StorNext client software with the Windows Operating System turn off the Recycle Bin in the StorNext file systems mapped on the Windows machine.

You must disable the Recycle Bin for the drive on which a StorNext file system is mounted. Also, each occurrence of file system remapping (unmounting/mounting) will require disabling the Recycle Bin. For example, if you mount a file system on E: (and disable the Recycle Bin for that drive) and then remap the file system to F:, you must then disable the Recycle Bin on the F: drive.

Beginning with StorNext 3.5, StorNext supports mounting file systems to a directory. For Windows Server 2003 and Windows XP you must disable the Recycle Bin for the root drive letter of the directory-mounted file system.

For example: For C:\MOUNT\File_System you would disable the Recycle Bin for the C: drive.

For Windows Server 2003 or Windows XP:

1. On the Windows client machine, right-click the Recycle Bin icon on the desktop and then click Properties.
2. Click Global.
3. Click Configure drives independently.
4. Click the Local Disk tab that corresponds to the mapped or directory-mounted file system.
5. Click the checkbox Do not move files to the Recycle Bin. Remove files immediately when deleted.
6. Click Apply, and then click OK.

(Disabling the Recycle Bin, Continued)

For Windows Server 2008, Windows Server 2012, Windows Vista, Windows 7 and Windows 8 systems, you must disable the Recycle Bin on C: and the File system name:

1. On the Windows client machine, right-click the Recycle Bin icon on the desktop and then click Properties.
2. Click the General tab.
3. Select the mapped drive that corresponds to the StorNext mapped file system. For directory-mounted file systems, select the file system from the list.
4. Choose the option Do not move files to the Recycle Bin. Remove files immediately when deleted.
5. Click Apply.
6. Repeat Step 3 through Step 5 for each remaining directory-mounted file system.
7. When finished, click OK.

Before attempting to upgrade from a previous StorNext release, make sure you have free space on the file system. If the file system is nearly full when you begin the upgrade, serious errors may occur or the upgrade could fail. Best practice is to maintain an area on the file system which is not used for data or system files, but is reserved as an empty buffer to ensure that upgrades and other operations complete successfully.

StorNext's configuration of the tape drives within an ACSLS library can get out of sync with the ACSLS server's configuration of the tape drives. If this occurs, then when StorNext sends a request(s) to the ACSLS server for a specific drive it could be sending the wrong location to the drive.

The problem occurs if location modification occurs with the tape drives in the ACSLS library such that what StorNext thinks are the correct ACSLS locations no longer match what the ACSLS server knows about.

This can happen by one of the following library maintenance type activities:

1. If a tape drive is replaced with a new tape drive.
2. If two tape drives are swapped within the library.
3. If new tape drive(s) are added into location(s) that causes ACSLS to assign different locations to the previously existing drives.
4. If a tape drive(s) are removed from location(s) that causes ACSLS to assign different locations to the remaining drives.

Workaround

Contact Quantum Technical Support.

There is currently no automated way to update StorNext to re-sync itself with the tape drive changes done within the library. It requires knowledge of what library scenarios were done in order to make the correct changes within StorNext. The necessary changes can be done to update StorNext's library/tape drive configuration to match what ACSLS knows of, but this is a manual process, and requires detailed knowledge of what needs to be updated in order to accomplish the procedure.

StorNext does not support hot-swapping tape drives. When replacing or adding new tape drives you must first stop StorNext before installing the new drive.

Tools outside of StorNext that issue a st command to rewind tapes may result in data loss.

Workaround

1. Do not run any tools that could possibly issue a rewind.
   • It may not be possible to determine this ahead of time.
   • Gathering hardware info is imperative.
2. Rename the /dev/st* devices per “Product Alert 16”. Product Alerts are available on the Quantum Service and Support website at www.quantum.com/ServiceandSupport.
   • They get created on the next reboot.
3. Stop SNSM before running any tools.
   • Stopping TSM could take 10-15 minutes if very busy.
   • Not optimal to stop/start TSM all the time.

The StorNext Cluster-Wide Central Control file (nss_cctl.xml) is used to enforce the cluster-wide security control on StorNext nodes (client nodes, fsm nodes, and nodes running cvadmin). This file resides under /usr/cvfs/config on an nss coordinator server.

Currently the nss coordinator server capable of parsing this xml file must be on the Linux platform.

Quantum recommends disabling sleep mode on the Xsan client to prevent the client from going into a sleep state while files are open in the StorNext file system or data is in the file system cache.

Note: If the client is in a sleep state while a file is open, then it will prevent other clients from opening the same file.

It is not possible to delete data within a StorNext policy relation point from an Xsan client via the Finder. Rather, data must be deleted using the shell.

Disks with existing non-StorNext labels may not show up in the StorNext GUI in order to protect non-StorNext disks from being accidentally overwritten. If you need to label a disk that is not visible in the StorNext GUI, use the cvlabel command to label the disk or use cvlabel -U to remove the existing label from the disks. (Refer to the cvlabel man pages for instructions on labeling and unlabeling drives.)

Caution: Modifying the label on an active non-StorNext disk can make the disk unusable. Proceed with caution.

On HA systems only:

The /usr/cvfs/config/ha_peer file supports some essential HA features by providing an address for HA administrative communications between the MDCs in an HA Cluster. If CVFS is started without this file having correct information, the probability of an HA Reset increases. To correct this condition, restore the ha_peer file to the IP address of the peer MDC, and restart StorNext by running the following command: service cvfs restart

Note: The peer will be Primary after running this command.

If the ha_peer file is removed for any length of time while StorNext is running, the snhamgr(1) HA Manager subsystem could stop functioning, which impacts the GUI HA Manage status page and the starting and stopping of CVFS, as well as any command line use of snhamgr itself. If this occurs, restore the ha_peer file to the IP address of the peer MDC, and then restart the HA Manager service by running the following command: service snhamgr restart

On HA systems only:

You may receive the following incorrect error message when scanning for a secondary MDC from the StorNext Convert to HA page:

This message is generated if /usr/adic/util/cnvt2ha.sh fails for any reason (for example, if the file system exists on the secondary, if a shared file system cannot mount, etc). Upon secondary conversion failures, StorNext resets the ha_peer file to 255.255.255.255 on the secondary. Since the conversion fails, the primary ha_peer file is not updated and faulty comparison logic causes the erroneous error message (255.255.255.255 == 255.255.255.255).

The workaround consists of two steps:

1. Remove the /usr/cvfs/config/ha_peer file from the secondary system.
2. Reset the StorNext processes on the secondary system by running /etc/init.d/stornext_web restart.

On HA systems only:

When a non-managed file system is converted to a managed file system in an HA pair, it is possible for the FSMPM on the secondary MDC to continue to operate this FSM as non-managed, which incorrectly allows the FSM to start on the secondary MDC.

Restarting the CVFS service corrects the problem. Quantum recommends taking the following steps as a temporary workaround after converting any non-managed file systems to managed file systems:

1. Complete the configuration changes
2. Make sure that CVFS is running on the secondary MDC, and wait 120 seconds to be sure that the configuration-file changes have been synchronized to the secondary MDC
3. Restart CVFS on the secondary by issuing "service cvfs restart"
4. Issue the command "cvadmin -e fsmlist" on the secondary MDC, and make sure that the output displays the FSM as "State: Blocked (waiting for MDC to become HA primary)"

Use caution when configuring the netmask for the HA Virtual Interface (VIP).

The VIP is an alias IP address that is associated with a real interface. For example, if the VIP is based on eth0, eth0:ha will be created as the VIP.

The netmask you associate with the VIP should generally be the same as that of the base interface, but in no case should it be more specific. For example, if the netmask on eth0 is 255.255.224.0 (a /19), then configuring the VIP netmask as anything more than a /19, such as a /24 (255.255.255.0) would be incorrect.

Using the same /19 mask on both eth0 and eth0:ha is the correct approach.

Note: The above applies only when the IP address of the VIP falls into the subnet defined by the base interface’s IP address and mask.

Quotas can only be enabled or disabled by modifying the Quotas parameter of the file system configuration file. The CLI snquota -L -Ffile-system command informs you whether the file system has quotas enabled.

Note: For a given file system, you can enable quotas in the StorNext GUI within the Configuration Parameters > Features tab. Click Quotas to toggle the quotas function. For additional information on how to manage quotas, see Manage Quotas.

If you run multiple fsretrieve commands simultaneously to find files (for example, find -type -f | xargs fsretrieve), you might receive error messages because doing this taxes system resources.

Instead, use the recursive retrieve command. When you use this command the files under a directory are retrieved in batches, and more sorting is done to put files in tape order for increased performance. Run recursive retrieve by entering % fsretrieve -R.

As of StorNext 5, stripe group expansion, also known as bandwidth expansion, is not supported. The functionality works but has been deprecated and should not be used.

For example, if you create a file system that has two different-sized disks in a userdata only stripe group and then attempt to add a new disk to that stripe group and expand it, the expansion will fail.

In some cases the physical IP address must be included in the dpserver file in addition to the interface name. Note these conditions:

• When there is one IP address associated with a NIC interface, the interface name alone is a sufficient identifier
• If there are multiple IP addresses associated with a NIC interface, one IP address is required in addition to the interface name
• On HA systems, the physical IP address is required if virtual IP is configured for the NIC interface. For additional information, see StorNext LAN Clients in HA Environments.

By design, replication or deduplication must be completed before data files can be truncated if these files are associated with both a replication/dedup policy and a Storage Manager policy. Even if the Storage Manager policy is configured with the “Truncate Immediately” option, the truncation may not occur at store time unless the file has been replicated or deduplicated.

Note the following recommendations and limitations for using DXi as a virtual tape library for StorNext:

• Recommended library emulation: “ADIC Scalar i2000”
• Recommended tape drive emulation: “IBM LTO-x” or “HP LTO-x”
• DDM (Distributed Data Mover): This feature is currently not supported due to lack of full SCSI3 support in DXi.

When a file system with two affinities is to be managed by the Storage Manager, the GUI forces those affinities to be named tier1 and tier2. This will cause an issue if a site has an existing unmanaged file system with two affinities with different names and wants to change that file system to be managed. There is a process for converting a file system so it can be managed but it is non-trivial and time consuming. Please contact Quantum Support if this is desired.

Note: The restriction is in the StorNext GUI because of a current system limitation where affinity names must match between one managed file system and another. If a site was upgraded from a pre-4.0 version to post-4.0, the affinity names get passed along during the upgrade. For example, if prior to StorNext 4.0 the affinity names were aff1 and aff2, the GUI would restrict any new file systems to have those affinity names as opposed to tier1 and tier2.

StorNext does not currently support converting from a managed file system to an unmanaged file system.

For more information about JournalSize, refer to FSBlockSize, Metadata Disk Size, and JournalSize Settings.

StorNext does not currently support running two truncation policies for different policy classes at the same time.

Workaround:

To avoid errors, do not run two truncation policies for different classes at the same time.

Caution: As the File System Manager (FSM) supports over 1000 clients (with more than 1000 file requests per client), the resource limits of your MDC may be exhausted with additional load from other processes. Exceeding the file descriptor limit will cause errors to your system. Quantum recommends you not run additional applications on the MDC.