Advanced Topics

This chapter contains end-to-end workflows for advanced system administration tasks.

When replication is enabled, ActiveScale OS continuously forwards the contents of the source bucket(s) to the destination bucket(s) through designated S3 accounts (replication accounts) on the source and destination systems. Replication accounts (admin controlled) are typically used across accounts for replication and need read and write access to these buckets and should not get abused by any end user. The destination bucket can be on another ActiveScale system or on Amazon Web Services (AWS). Replication between ActiveScale systems can be unidirectional or bidirectional. Replication between an ActiveScale system and AWS is uni-directional, with AWS acting as the replication target.

Caution: Different versions are not always replicated in order, but when replicating to ActiveScale, the final ordering after all versions are replicated will be the same on the destination as on the source.

Additional Information

ActiveScale OS can only access one AWS region, make sure the ActiveScale system can reach the AWS region: US-East-1 when replicating to AWS.
Objects will be replicated with the settings that apply at the time of replication, not at the time of object creation. Updating the bucket replication rule or the system wide replication configuration will apply to all objects that are not yet replicated.

Differences Between Replicating to ActiveScale or AWS

There are differences in replication behavior depending on your replication target as described in the table below:

Table 1: Differences in replication pipelines

	ActiveScale-to-ActiveScale	ActiveScale-to-AWS
Version Ordering	The pipeline preserves version ordering.	The pipeline does not preserve version ordering. Therefore, when retrieving replicated objects from an AWS destination bucket, always identify the correct version of the object by checking its com.wdc.activescale.versionid attribute against the corresponding attribute of its identical object in the source bucket.
Version and Timestamp Metadata	An object in the destination bucket has the same version and timestamp information as its identical object in the source bucket.	An object in the destination bucket will not have the same version and timestamp information as its identical object in the source bucket.
Multiple Copies of the Same Version of an Object	The pipeline may send more than one copy of the same version of an object to the destination system. However, the destination bucket will have only one copy.	The pipeline may send more than one copy of the same version of an object to the destination system. The destination system thus can have multiple copies of same version, all tagged with same value for com.wdc.activescale.versionid.
DELETE Markers	DELETE markers are preserved in correct order.	DELETE markers are replaced with a 0 byte object plus an object tag: com.wdc.activescale.requesttype, value: delete-marker. Therefore, you may notice objects of size 0 bytes.
Lock Retention Information	Lock retention mode and retain until date are preserved. Both for future and past retain until dates.	Lock retention mode and retain until date are preserved for retain until dates in the future. If the object is unlocked the lock retention mode (com.wdc.activescale.lockComplianceMode) and retain until date (com.wdc.activescale.lockRetainUntilDate) are added to a tag.

ActiveScale 6.4 and Above: New Supported Features

The following new features are supported with the release of ActiveScale OS 6.4 and above.

Make sure the source and destination systems can communicate with each other over public interfaces. In other words, all S3 endpoints of each system must be network-reachable. Endpoints can be on the same subnet or routed.

Configure the ActiveScale OS replication pipeline with your settings. You can do this using ascli or ActiveScale SM.
1. To use ascli, refer to the CLI Reference Guide.
2. To use ActiveScale SM, refer to Manage System Settings.

Note: This step is only for setting up your replication pipeline; in subsequent steps you'll configure your source and destination buckets.

To create a versioned bucket, you can use ActiveScale View Managing Buckets (see Create a Bucket and Versioning) or use the S3 API with a front-end client of your choice. The example below uses AWS CLI to create a versioned bucket. Do this for both the source and destination site.

Note: Fill in the correct bucket names ( <*_bucket_name>) and IP or domain ( <ip or domain to * ActiveScale>).

Note: For all these commands, you will need access keys to the system, but we will omit this from the commands for clarity reasons.

Create Buckets:

Copy

aws s3api create-bucket --bucket <*_bucket_name> 
                        --endpoint http://<ip or domain to souce/destination ActiveScale>

Enable Versioning:

Copy

aws s3api put-bucket-versioning --bucket <*_bucket_name> 
                                --versioning-configuration Status=Enabled
                                --endpoint http://<ip or domain to souce/destination ActiveScale>

Quantum advises that the replication account is not the owner of the source and destination bucket, therefore grant it read and write permissions.

Note: Fill in the correct bucket names ( <*_bucket_name>), account email ( <*_account_email>) and account id (<*_account_id>).

Using awscli for this example, on the source bucket perform the following command:

Copy

aws s3api put-bucket-policy --bucket <source_bucket_name> --policy file://SourceReplicationPolicy.json

With SourceReplicationPolicy:

Copy

{
  "Version": "2012-10-17",
  "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
      "AWS": "<source_replication_account_id>"
    },
    "Action": [ "s3:GetObjectVersionForReplication", "s3:ListBucket" ],
    "Resource": [ "arn:aws:s3:::<src_bucket_name>/*", "arn:aws:s3:::<src_bucket_name>" ]
    }
  ]
}

Similarly, perform the following command on the destination bucket:

Copy

aws s3api put-bucket-policy --bucket <destination_bucket_name> --policy file://DestinationReplicationPolicy.json

With DestinationReplicationPolicy.json:

Copy

{

{
  "Version": "2012-10-17",
  "Statement": [
    {
    "Effect": "Allow",
    "Principal": {
      "AWS": "<destination_replication_account_id>"
    },
    "Action": [ "s3:ReplicateObject", "s3:ReplicateDelete", "s3:ListBucket", "s3:ListMultipartUploadParts" ],
    "Resource": ["arn:aws:s3:::<destination_bucket_name>/*", "arn:aws:s3:::<destination_bucket_name>"]
    }
  ]
}

The account id for the account can be obtained using de IAM GetUser call as the root-user without specifying a user-name:

Copy

~ $ aws iam get-user
{
    "User": {
        "UserName": <account_name>,
        "UserId": <account_id>,
        "Arn": "arn:aws:iam::<account_id>:root",
        "CreateDate": "2024-02-22T12:26:18+00:00"
    }
}

To grant permissions on ActiveScale versions prior to ActiveScale 7.0, refer to User Permissions . If required, also perform this step for the replication account on the destination system. Using awscli, the command is the following:

Copy

aws s3api put-bucket-acl --bucket <*_bucket_name> 
                         --grant-read emailaddress=<*_account_email>
                         --grant-write emailaddress=<*_account_email>

Create an S3 replication rule as a JSON file (e.g. replication_rule.json).

Note: This step can be done interactively using the ActiveScale View Managing Buckets Asynchronous Replication to a Preset Destination tool.

Basic Rule for Enabling Replication

Only the destination bucket name is required. The object will be replicated to the destination bucket in the storage class it was originally created in at the source. Other characteristics of the object that are set at time of replication will also be applied to the replica (retention time, custom metadata, creation time,…).

Note: If an OLM storage class transition (e.g. Transition to S3 Glacier Flexible Retrieval after 1 year) takes place on the source bucket, the destination storage class is the initial/original storage class used when uploading the object.
Copy
```
{
  "Role": "Not Applicable",
  "Rules": [
    {
      "Prefix": "",
      "Status": "Enabled",
      "Destination": {
        "Bucket": "arn:aws:s3:::<destination_bucket_name>"
      }
    }
  ]
}
```
Replication rule with an override of the destination storage class

Both the destination bucket name and destination storage class are required. The destination storage class is the same for all objects in the bucket. An example scenario is below:
- At the source objects are written to S3 Standard storage class and to S3 Intelligent-Tiering
- With the following replication rule, all replica’s will be written to S3 Glacier Flexible Retrieval storage class and eventually moved to the tape tier.
Copy
```
{
  "Role": "Not Applicable",
  "Rules": [
    {
      "Prefix": "",
      "Status": "Enabled",
      "Destination": {
        "Bucket": "arn:aws:s3:::<destination_bucket_name>",
        "StorageClass": "GLACIER"
      }
    }
  ]
}
```
Rule to enable existing data replication with an override storage class GLACIER

Both the destination bucket name and destination storage class are required.

Note: The existing object replication feature is not allowed to be used by default in the ActiveScale OS. Contact support if the feature should be enabled for your ActiveScale system. Otherwise, setting this example replication rule will result in an error.
Copy
```
{
  "Role": "Not Applicable",
  "Rules": [
    {
      "Prefix": "",
      "Status": "Enabled",
      "Destination": {
        "Bucket": "arn:aws:s3:::<destination_bucket_name>",
        "StorageClass": "GLACIER"
      },
      "ExistingObjectReplication": {
        "Status": "Enabled"
      }
    }
  ]
}
```

Push the S3 replication rule to the source bucket. Bidirectional replication is possible, create a new replication rule JSON file and push it to the destination bucket.

Refer to ActiveScale View Managing Buckets Asynchronous Replication to a Preset Destination or AWS CLI to put a bucket replication configuration.

Copy

aws s3api put-bucket-replication --bucket <source_bucket_name> 
                                 --replication-configuration file://replication_rule.json
                                 --profile <source_credentials_account> 
                                 --endpoint http://<ip or domain to souce ActiveScale>

(Optional Step): Put an object into the source bucket to validate that the replication configuration is working. Small objects should appear in the destination bucket within one minute. Replication time for a large object is proportional to its size.

Note: To view ActiveScale OS replication pipeline settings, see Enabling Asynchronous Replication.

Configuring Kerberos Authentication with Active Directory for NFS

This section provides the procedure for configuring Kerberos Authentication with the AD for NFS.

Sample Configuration Inputs

The following table provides sample configurations that will be used:

Table 2: Sample AD Configuration Inputs

Configuration Inputs	Sample Values
AD Domain	example.com
AD Server FQDN (same server for all 3 geos)	adserver.example.com
AS NFS Server FQDN	asnfsserver.example.com
NFS Service Principal	nfs/asnfsserver.example.com
NFS Server Keytab file	asnfsserver.keytab
NFS Client FQDN	asnfsclient1.example.com
NFS Client Host Principal	host/asnfsclient1.example.com
NFS Client Keytab file path	asnfsclient1.keytab
Based DN	dc=example,dc=com
AD NFS Group Name	nfsgroup1
AD NFS Group Id	30001
AD NFS User Name	nfsuser1
AD NFS User Id	20001

Create AD NFS Groups and Users for accessing the NFS Share

The following describes how to create AD NSF Groups and Users for accessing the NFS Share feature:

Create NFS groups in AD.
For new or existing groups, make sure to populate the following attributes.
1. gidNumber
Sample group
1. Group Name: nfsgroup1
2. Group Id: 30001
Create NFS Users in AD.
For new or existing users, make sure to populate the following attributes.
1. uidNumber
2. gidNumber
3. loginshell
4. Homedirectory
Sample user
1. User Name: nfsuser1
2. User Id: 20001

Configuring Kerberos Authentication for ActiveScale NFS Server

This section provides the procedure for configuring Kerberos Authentication for the ActiveScale NFS server.

Calculating Capacity Utilization

This section describes how the capacity utilization and the system storage overhead are calculated. The widget provides an accurate look at raw disk statistics in an environment while also providing insight on usable capacity given the Erasure Coding Policy. Starting from ActiveScale OS 6.1, it also provides insight in the capacity usage of the attached tape libraries and the capacity that is still in use on disk by data that needs to move to the tape libraries.

Note: The new widgets are purely statistical accounting and presentation. There are no changes in how data is stored and protected, nor is there any change to the amount of capacity, raw or usable, available from previous releases.

This list defines the content of the images below:

Used: capacity currently in use by live objects.
Used - Tape Staging Area: disk capacity temporarily in use by objects that are moving to tape (only when tape is enabled).
Free - Pending Cleanup: capacity occupied by already deleted objects for which ActiveScale has temporarily postponed space reclaim.
Free: capacity that can be used to store new data.

Figure 1: Capacity Utilization - Tape Disabled

Figure 2: Capacity Utilization - Tape Enabled

Calculating Total Usable Capacity

Total usable capacity is the sum of used usable capacity and the free usable capacity. Both are calculated as follows

Used usable capacity is the capacity currently still in use by objects, based on the used raw capacity and the average overhead of all stored objects. Note that this may still include already deleted objects whose capacity still has to be reclaimed by background garbage collection
Free usable capacity is based on the free raw capacity and the overhead of the large file policy. Due to small object aggregation, ultimately no new objects (written after upgrade to AS 5.7) will be stored with the small file policy. This means we only need to take the large file policy into account for the free usable capacity.

Example

Initial bring up with 20/5 large file policy (1.33x overhead) for a system with 6650 TiB of raw capacity:
- Used usable capacity: 0 bytes
- Free usable capacity: 6650 TiB * / 1.33 = 5000 TiB
- Total usable capacity = 5000 TiB
Upgrade of the same system installed with AS < 5.7 and 3990 TiB of used space with a historical average overhead of 2x (due to many uncontainerized small objects)
- Used usable capacity: 3990 TiB / 2 = 1995 TiB
- Free usable capacity: 2660 TiB / 1.33 = 2000 TiB
- Total usable capacity = 3995 TiB

Enabling DPS/ONS

When DPS is enabled, ActiveScale sends object operation notifications from each of the configured

buckets to the corresponding Kafka topic on the external Kafka broker.

For more information on the notification structure refer to, https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html.

Differences in the notification structure between Active and AWS are listed below.

Functionality	ActiveScale	AWS
OLM object deletes	Notifies automatic object deletes from OLM	Does not notify
POST	Not supported	Supported
Object name prefix/suffixes	Not supported	Supported
COPY	Notifies as PUT object	Notifies as COPY
Event Message Structure	`userIdentity`, `requestParameters` and `responseElements` fields are not populated in the structure	These fields are populated in the structure

Enabling DPS/ONS requires system admin configuration followed by individual bucket configuration to verify the notifications as per the subsequent sections.

System Administration Configuration

Perform the following steps to enable DPS/ONS:

Ensure that the Kafka broker IP/name that will be used for configuration is accessible from the system nodes.
Download a sample DPS configuration file, refer to Object Notification in the CLI Reference Guide for more details.
Make changes to the downloaded file with the following steps:
1. Create a list of all the Kafka brokers.
2. Configure each destination under the topicResource section.
3. For each destination, specify the list of topics that can be used for notifications along with the Kafka broker endpoints.
  
  For example:
  
  "1": {
  
  "topicNames": [
  
  "topic1", "topic2", "topic3"
  
  ],
  
  "notificationConnection": {
  
  "ipPorts": [
  
  {
  
  "ip": "IP1",
  
  "port": port1
  
  },
  
  {
  
  "ip": "IP2",
  
  "port": port2
  
  },
  
  "ip": "IP3",
  
  "port": port3
  
  ]
  
  }
  
  }
4. For each destination, use a unique number: 1, 2, 3, and so on.
  
  Note: Only configured destinationID (configured as perObject Notification in the CLI Reference Guide and topicNamescan be used in the bucket configuration through S3 API.
After entering all destination configurations in JSON, configure DPS/ONS using ascli configure. Refer toObject Notification in the CLI Reference Guide for more information.
View the current configuration using ascli. Refer toObject Notification in the CLI Reference Guide to ensure that the configurations are correct.

This completes the system administration configuration.

NFS Service Principal:	nfs/asnfsserver.example.com
Keytab File :	asnfsserver.keytab

NFS Client Host Principal:	host/asnfsclient1.example.com
Keytab File:	asnfsclient1.keytab