Tips and FAQs

The following network ports have been configured as defaults.

In general, FlexSync runs fast enough to saturate a network connection if the following are true:

• The network is configured properly.
• The file systems being used as the source host and destination target can respond quickly.

FlexSync only supports a single IP address per host, but it can use multiple sockets over multiple TCP ports for data transfer.

This allows Quantum systems the ability to saturate a network link with the default settings. If you want to configure load balancing across multiple network interfaces, then you must use aggregation.

Use the StorNext GUI (see Configuration > System) to create a bonded interface and configure the switch to use the bonded interface on your Quantum appliance.

Yes, you can throttle the bandwidth. See If needed, select Bandwidth Throttling..

With FlexSync, you should be able to achieve full line speed for small configurations without tuning. However, if you are scaling to larger configurations, you may need to configure tuning for maximum performance.

Depending on your configuration's ecosystem details and application behavior, you may need to configure the following Transmission Control Protocol (TCP) tuning parameters. Keep in mind that you might need to experiment to find optimal values for your particular network configuration.

Changing the values are universal across all TCP connections and might impact other applications running on your host.

TCP Window Size

High-speed and high-latency networks require large TCP windows to achieve full bandwidth. Although you typically use the TCP window size parameter to set a range for connections, you can lower the range's high-end value to adjust for system-imposed maximums.

Adjust the TCP Window Size

1. Check the values of the rmem_max and wmem_max parameters.

   Example

   net.core.wmem_max = 268435456

   net.core.rmem_max = 268435456

   net.ipv4.tcp_rmem = 4096 65536 268435456

   net.ipv4.tcp_wmem = 4096 65536 268435456

2. Execute the following commands to adjust the rmem_max and wmem_max parameters:

   sysctl -w net.core.rmem_max=<number>

   sysctl -w net.core.wmem_max=<number>

   Example

   sysctl -w net.core.rmem_max=4194304

   sysctl -w net.core.wmem_max=4194304

   The exact syntax may vary by Linux version. For details, refer to documentation for your version of Linux.

TCP Window Scaling

When window scaling is enabled, the advertised TCP receive window can be larger than 64K. This value is required to achieve maximum per-connection bandwidth when high-speed or high-latency networks are used.

Although newer versions of Linux have TCP window scaling enabled by default, we recommend that you verify that this parameter is enabled. If it is not enable, you can explicitly set the value.

Enable TCP Window Scaling

1. Verify whether window scaling is enabled.

   Example

   net.ipv4.tcp_window_scaling = 1

2. Execute the following command to enable window scaling:

   sysctl –w net.ipv4.tcp_window_scaling=<number>

   • To determine the correct kernel parameter to adjust, refer to the documentation for your version of Linux.
   • In order for it to be effective, make sure to enable window scaling on both ends of a connection.

Be default, FlexSync uses 8 streams for the network transport stream count. You can define a different value for the number of network transport streams, as needed.

Configure Network Transport Stream Count for FlexSync

1. Log on to an SSH client, and connect to your system.
2. At the prompt, enter the following command to configure a different stream count value:

   /opt/quantum/flexsync/sbin/flexsyncd --max-connections=<number | -M number;

   Example

   /opt/quantum/flexsync/sbin/flexsyncd --max-connections=6

   /opt/quantum/flexsync/sbin/flexsyncd -M 6

   Note: Either parameter can be used to enter the new value. The -M <number> parameter is shorthand for the --max-connections=<number> parameter.

If identical data exists on both your source and destination file systems, then you must verify the following to avoid resynchronization of your data:

• The modification time of a file must be identical on both your source and destination.

• The directory structure must be identical on both your source and destination.

• The metadata attribute of a file and directory must be identical on both your source and destination. For example, you must preserve ACLs, extended attributes, set the mtime accordingly, and so on, to verify the metadata is identical.

  Note: If the metadata is not identical, then FlexSync must read the entire file on your source and destination, exchange hashes in order to verify that the data is identical, and then update the metadata. This operation does not transmit the actual file data over a network pipe; however, the operation requires work and data to transmit.

• If you enable the Versioning feature, FlexSync might duplicate data if the metadata is not identical. For example, if you use rsync to replicate your data and you do not preserve ACLs, then you must configure a FlexSync replication task that indicates versions and preserve ACLs. FlexSync might duplicate your data since the original copy does not contain ACLs and it cannot be changed if it is part of a version "snapshot".

  Note: To avoid this issue, you must initially synchronize without the Versioning feature and then enable the Versioning feature after your replication tasks are operating successfully.

• If Storage Manager contains a truncated file, then your configuration becomes complicated. This causes your file to be re-duplicated since delta compression from tape is not possible. In this case FlexSync transmits the entire file.

• If your source and destination are both local to the same host, then your data is duplicated again since FlexSync does not perform delta compression.

If you receive an error message stating the following during your FlexSync installation, you might be experiencing issues related to an incorrect platform or version of the binary file.

error: Failed dependencies

Resolution

If you are experiencing failed dependencies, confirm that you are installing FlexSync using the correct platform and version of the binary file. See Install the FlexSync Application.

No. Mixed versions of FlexSync daemons is not supported and results in a communication error.

You must install the same version of FlexSync, or upgrade to the same version of FlexSync on all of the hosts or systems using Flexsync. A newer version of the Flexsync daemon cannot communicate with an older version within a configuration, or on another host or system. See Considerations for additional information.

FlexSync can be installed on node 2 of an Xcellis Workflow Director, Xcellis Workflow Extender, or StorNext client. However, you should consider the existence of a DDM on the same system. If the amount of data to be moved by both processes exceeds the system availability, we recommend installing FlexSync at an alternate location.

FlexSync does not support HA. In a non-HA environment, FlexSync does not interact with StorNext, beyond having StorNext read the license.dat file [This ONLY applies if your system is running StorNext 6.x. Beginning with StorNext 7.0, you must purchase and install a Node license on your system (contact your Quantum Sales representative to obtain a Node license)]. However, in HA environments, you must run the FlexSync management functions on the passive node, as well as install the FlexSync license on that node.

If you assign the same name — differentiated by case or suffix only — to FlexSync users, such as Admin, admin, or admin1, and then delete one of the users, neither the flexsyncd nor flexsyncadmin will respect the case or suffix difference. All users sharing the name — differentiated by case or suffix only — will be deleted. This issue does not occur if a user name has a unique prefix.

This issue also affects host user names and the effected host's task configurations.

Resolution

1. Re-add the original user name. See Step 4: Manage FlexSync Users.
2. Add new unique user names for the effected hosts and task configurations. See Step 4: Manage FlexSync Users.
3. Suspend and delete the original hosts and task configurations. See Configure File Replication.
4. Create new host and task configurations, assigning them a new unique user name. See Configure File Replication.

Your host could be experiencing issues related to a task, or tasks, that experienced an error. Otherwise, the host status is the usual laptop icon.

Note: The error status for all of a host's tasks are rolled up and displayed as a red warning triangle icon in the host status. If at least one task experienced an error, then the host status displays the red warning triangle icon that is being used for the task.

Resolution

You can view the contents of FlexSync logs using the FlexSync Dashboard. See Configure File Replication.

Your host could be experiencing connectivity issues related to any of the following circumstances:

• Invalid username and password credentials.
• DNS or network issues between the node and the FlexSync administrative server.
• Unresponsive or down host or daemon.

Resolution

• If the credentials are invalid, verify the user name and password. See Step 4: Manage FlexSync Users.
• Delete the host and re-create the host. See Configure File Replication.
• Check the status of your flexsyncd or flexsyncadmind daemon to verify that it is running:
  RHEL 6

  Execute the following command:

  service <daemon> status

  RHEL 7

  Execute the following command:

  systemctl status <daemon>

You can use Flexsync with StorNext Storage Manager managed file systems (source or destination), if the FlexSync version is 2.0 or greater and the StorNext version is 6.3.0 or greater. See FlexSync and Managed Files for configuration considerations when using StorNext Storage Manager.

Note: FlexSync version 2.2.0 or greater is ONLY supported on a system running StorNext 7.0.0 or greater.

Note: You cannot use FlexSync version 3.0 S3 object-based replication with StorNext Storage Manager managed file systems (source or destination).

Your task is suspended. Otherwise, the task status is the usual laptop icon. See Configure File Replication.

The minimum possible replication interval for FlexSync is 1 second. Of course, the interval depends on factors such as file system performance, network performance, and the change rate.

With FlexSync, only 1 mover would be needed to move this amount of data between appliances.

You cannot use FlexSync to modify a given tree at two locations and then synchronize those changes with each other.

You cannot use the Reverse task option to go back to a specific point-in-time. However, you can set up a task to replicate from a specific version on the destination target to the source host.

See Configure File Replication.

FlexSync encrypts all transferred files in transit using SSL. However, FlexSync does not encrypt anything on disk, nor does it support doing so.

If file versioning is not enabled for a replication task, the replication task simply updates the destination copy to match the source. Keep in mind that without file versioning, the destination has only 1 copy of the file.

If file versioning is enabled for a replication task, with propagate deletes enabled at a later time, the versions already on the destination are not affected and are retained. When propagate deletes is enabled, files deleted from the source are deleted from the destination for that version only. Other versions are retained without change.

Case sensitive settings (caseInsensitive=false) must be configured on both the source and destination hosts. Otherwise, the destination host will merge two files/directories that match on the identical spelling of a file name, even when the names do not match on case.

Resolution

Configure your target host with the same case sensitivity setting as your source host.

Caution: If you do not configure your target host with the same case sensitivity setting as your source host, then the replication task might result with an error message, such as a rename error.

Example of a Rename Error

20210727 12:02:04 Starting task... 
20210727 12:02:05 Checking for filter... 
20210727 12:02:05 Filter using last marker 901578. 
20210727 12:02:05 Received filter, new marker: 912030. Found 201 candidate(s) to check. 
20210727 12:02:05 Starting estimation pass... 
20210727 12:02:05 Estimation pass complete. Found 2200 file(s) to copy. 
20210727 12:02:05 Starting replication pass... 
20210727 12:02:05 Error renaming /stornext/murphy1/data/dir78/._flexsync_tmp_file5 to /stornext/murphy1/data/dir78/file5: No such file or directory 
20210727 12:02:05 Replication completed with errors. 39 files/segments copied 16.92 MiB moved. 
20210727 12:02:05 Task completed with errors. 

Under the following instances, FlexSync might start repeatedly replicating the same content:

• When source file systems are replicated with the auditing feature enabled, the auditing feature will make further changes to the source file system. As a result, FlexSync might interpret the files as candidates for additional replication.
• Read activity in the file system might cause FlexSync to scan for content to replicate.

Both instances usually do not result in additional data transfers by Flexsync. However, under certain conditions, such as when extended attributes are replicated, this issue can occur.

Resolution

Do not configure a replication task if the auditing feature is enabled on the source file system, or when replicating extended attributes.

Under the following specific circumstances, reverse tasks can overwrite the original source data:

1. Create a replication task.
2. While the task is estimating, suspend it.
3. Create a corresponding reverse task.
4. Start both the original source and corresponding reverse tasks.

Resolution

Do not simultaneously run the original source task and its corresponding reverse task.

You can use the flexsyncdump command to examine the configuration database. To list the document keys of the database:

For example, to list the document keys for flexsyncadmind:

To list one of the documents:

/opt/quantum/flexsync/bin/flexsyncdump -d flexsyncadmind -s hostgroups/ac27d990-3ef4-49eb-995e-f50cdd6701e1
{
    "hosts": {
        "9e96025f-2abb-48c0-a21d-ffda42f74740": {
            "addresses": [
                {
                    "admin_port": 10453,
                    "data_port": 10454,
                    "data_ssl_port": 10554,
                    "hostname": "candolim.mdh.quantum.com"
                }
            ],
            "bwlimit": null,
            "hostgroup": {
                "name": "hostgroup1",
                "uuid": "ac27d990-3ef4-49eb-995e-f50cdd6701e1"
            },
            "name": "candolim",
            "password": "password",
            "username": "admin",
            "uuid": "9e96025f-2abb-48c0-a21d-ffda42f74740"
        }
    },
    "name": "hostgroup1",
    "uuid": "ac27d990-3ef4-49eb-995e-f50cdd6701e1"
}

FlexSync does not support the ability to collect and forward the contents of the FlexSync log file.

FlexSync log formatting is proprietary information that is only available to Quantum support. You can view the contents of FlexSync logs using the FlexSync Dashboard. See Configure File Replication.

At this time, you may see 2 vertical scroll bars if the Task Report's browser window is not tall enough to display all of the line items.

If you are disconnected from the FlexSync Dashboard, the displayed Reload button may not work correctly.

Workaround

Close the browser tab or window, and initiate a new session with the FlexSync Dashboard.