Converged Scale Out Guide

Networking documentation can be found in the Combined Scale Out Networking Guide:

• 1-GEO: Combined Scale Out Networking Guide

• 3-GEO: Combined Scale Out Networking for 3-GEO Deployments

• Switch Configuration: Configure the Switches

The networking guide provides examples and recommendations for installing the scale out system with certain subnets. Of course it is important to respect the customer’s wishes. The recommended subnets might not be an option in the customer’s datacenter due to e.g. conflicts. The installation process makes no assumptions about what subnets are being used. The subnets and IP address allocation is completely customizable so the customer can choose them at their convenience.

There is one restriction: it is important to note that the scale out system that is the subject of this installation has to occupy different private subnets than the private networks of the base system and they should also be different from the private subnets of previously installed scale out systems. It is not possible to just add the nodes of the scale out system to the already defined subnets of the base system or previously installed scale out systems. The new nodes will have to be in their own, separate subnets. In order for the base system and the scale out system to communicate, the customer has to prepare gateways that can route traffic between the base system and the scale out system. Should the customer already have a previously installed scale out system, the customer can choose to use different gateways at the base system, or reuse the same gateways. The requirement here is that the gateway allows traffic to and from the scale out system.

Starting from ActiveScale software version 7.0, it is also possible to send traffic to the scale out nodes and not only to the nodes of the base system. Refer to the section about Unlocking S3 Access to Scale Out Nodes. Should the customer want this, pay attention to the following:

• The customer has to prepare their public network. Make sure the public interfaces of the new scale out nodes are cabled.

• In some cases the customer has to configure private network gateways. Please refer to Configure gateways.

• Make sure when installing the scale out rack that you have defined the public networks and IP addresses in the inventory file. More information about this in the Software installation guide.

If the customer does not want S3 access to their new scale out nodes, pay attention to the following:

• We recommend to pull cables of the public interfaces. The customer is free to follow this recommendation. the software does not depend on this.

• Software Installation talks about the different possibilities to fill in the inventory file when removing these cables.

• In some cases the customer has to configure private network gateways. Please refer to Configure gateways.

Before installing the scale out system it is useful to already prepare a document or note that describes the customer’s networking layout.

• All subnets of the scale out system and also of the base system and all previously installed scale out systems.

• IP addresses of all nodes in the scale out system.

• Which nodes have cabled public interfaces and which nodes don’t.

• Gateways IPs should be known and gateways should be configured:

  • Gateways of scale out networks should be able to route to base system.

  • Gateways of base system networks should be able to route to all scale out systems.

  • Gateways of scale out networks should be able to route to previously installed scale out systems if the customer has Performance Scale Out.

  • Gateways of previously installed scale out systems should be able to route to the new scale out system if the customer has Performance Scale Out.

Determine the selected policy on the existing deployment. We need:

• policy type

• policy category

• policy name

Using sherpa on a scaler management node on the existing deployment:

%cpaste
policy_type = !/opt/ampli/apps/marvinbase/venv/bin/python -c "from installer.common import find_policy_group; print find_policy_group()"
policy_type = policy_type[0]
policy_category = cli.marvin.environment.list()[0]['selectedstoragepolicy']['category']
supportedstoragepolicies = cli.marvin.environment.list()[0]['supportedstoragepolicy']
policy_name = next((policy['name'] for policy in supportedstoragepolicies if policy['category'] == policy_category), None)
if not policy_name:
    from marvin_lib.utils import load_yaml
    dssconfig_path = "/opt/ampli/apps/marvinbase/config/dssconfig.yml"
    dss_config = load_yaml(dssconfig_path)
    policy_name = dss_config.get('policies').get(policy_type).get(policy_category).get('name')
print "policy type: %s\npolicy category: %s\npolicy name: %s" % (policy_type, policy_category, policy_name)
--

Make sure to note these values. The customer can only choose a policy of the same category on the scaleout rack.

The converged scale out procedure depends on the location names that are defined in the model of the existing deployment. Make sure it is known in which datacenter each rack is situated. This information is visible in the GUI and can be provided by the customer. It is also possible to determine it from the model. Execute following snippet in sherpa to get a list of datacenters and corresponding racks:

for dc in cli.marvin.location.list(query="locationtype=DATACENTER"):
    for rack in [l for l in cli.marvin.location.list(query="locationtype=RACK") if l['parent']==dc["guid"]]:
        print "DATACENTER %s with RACK %s (serial: %s)" % (dc["name"], rack["name"], rack["serial"])

example output:

See installer guide: 3.0 Setup the Quantum Installer VM

Follow the different steps in the Quantum VM installer guide to prepare and set up the installer. Make sure that the host machine has the necessary hardware requirements and that the VM installer can actually reach the ActiveScale nodes that are going to be installed. As this is an installation of a scale out rack, make sure that the VM is connected to one of the switches of the scale out rack and that the specific port on that switch is on the same VLAN as the nodes that are being installed.

Then follow the VM installer guide steps to build the inventory and configure the ActiveScale system options for the scale out rack. For the converged scale out it is important to make sure the settings in the following section are correctly specified in the inventory file.

Among the asnodes variables in the inventory file there will be a parameter called scale out_rack. By default its value is set to false, but when installing a capacity only scale out rack, the value has to be set to true:

all:
  children:
    ...
    asnodes:
      vars:
        ...
        scaleout_rack: true
        ...

The purpose of this setting is to skip column configure and scaler account generation on the scale out rack.

AS 6.1 and above leverages the federation concept for scaling out. The federation is stored in a separate database (a dedicated mongodb cluster). During the install of the scale out environment, it will contact the base rack for information. To get the credentials to access the base rack, execute the following sherpa script on the base rack:

%cpaste
from marvin_lib.network import ha
import json
cluster = [c for c in cli.marvin.mongocluster.list() if c['name'] == 'federated_model']
for c in cluster:
    for a in c['application']:
        machine = cli.marvin.machine.get(a['machine'])
        for n in machine['nic']:
            if 'network' not in n or n['network'] is None:
                continue
            nw = cli.marvin.network.get(n['network'])
            if nw['networktype'] == 'PRIVATE':
                print("%s [%s/%s]" % (n['address'], nw['network'], nw['netmask']))
        print a['ports']['general']
    print('')
print('')
print(json.loads(ha.get('/model/environment/?insecure=True'))['environment'][0]['ssl_key'])
print(json.loads(ha.get('/model/environment/?insecure=True'))['environment'][0]['ssl_certificate'])
--

It will list the IP addresses (with network and netmask) and port to connect to the (usually 3) mongodb instances that make up the federated cluster, e.g. when executed on an X100:

This information is then input into the federated_api section of the VM installer inventory. The keys and certificate need to be copied into separate files in the quantum_installer/federated_api_files folder, the filenames can then be input into the VM installer inventory, together with the information for the mongodb endpoints:

    # Below section is not relevant if this is not a scaleout rack
        federated_api:
          ssl_key: federated_ssl.key # File name of file containing the SSL key for the federated api, located in quantum_installer/federated_api_files folder
          ssl_cert: federated_cert.crt # File name of file containing the certificate for the federated api, located in quantum_installer/federated_api_files folder
          endpoints:
            - endpoint: # endpoint 1 for federated api
                ip1: 172.16.110.1
                ip2: 172.16.210.1
                port: 27018 # Port number for federated api on endpoint 1
            - endpoint: # endpoint 2 for federated api
                ip1: 172.16.110.1
                ip2: 172.16.210.1
                port: 27018 # Port number for federated api on endpoint 2
            - endpoint: # endpoint 3 for federated api
                ip1: 172.16.110.1
                ip2: 172.16.210.1
                port: 27018 # Port number for federated api on endpoint 3
        # end scaleout rack section

The inventory file of the VM installer has a datacenter section where the datacenter name can be specified. The converged scale out procedure requires that the datacenter name that is defined at installation of the scale out rack has to be same as the one specified in the model of the existing deployment. Datacenters and rack names of the existing deployment were determined in the prerequisite section (Determine datacenter and rack names of existing deployment). Determine which rack of the existing deployment is in the same physical datacenter, and get the corresponding datacenter name. For a 1 GEO deployment this will be trivial as all racks are in that same datacenter, and there is really only 1 datacenter name. In the 3 GEO case you’ll have to make sure to select the correct datacenter.

In either case, the datacenter should be defined in the model of the existing deployment. This can be verified in the GUI or by entering following command (change <dc name> to the datacenter name you’ve found) in sherpa on one of the scaler management nodes of the existing deployment:

cli.marvin.location.get(<dc name>)

Once you are certain that you’ve found the correct datacenter name, adjust the inventory file of the VM installer as outlined in the installer guide:

all:
  children:
    ...
    asnodes:
      vars:
        ...
        datacenter:
          - name: <dc name>
        ... 

The inventory file of the VM installer will also have a rack section where the rack name and serial can be specified. The converged scale out procedure requires that the rack name has to be unique. All rack names in the existing deployment were already determined in the prerequisite section (Determine datacenter and rack names of existing deployment), so make sure to not pick a name that is already defined. It is also crucial that the rack serial is correctly specified:

all:
  children:
    ...
    asnodes:
      vars:
        ...
        rack:
          - name: <rack name>
              serial: <rack serial>
        ... 

The inventory file of the VM installer will have a network section where we can specify the private networks that have to be modeled on the scale out rack. Using the specific guide on the networking scenario that was determined in the prerequisite section Networking, the network and netmask of the 2 private networks should be determined.

In case the customer wants to have the machines of the scale out rack on the same subnets as the machines of the existing deployment, the startip and endip should be carefully chosen such that there is no overlap with the ip range that is specified for the private networks in the model of the existing deployment. The ip range should be big enough to cover all machines. In case of scale out with X200 there should also be extra ip addresses for IPMI and JBODs.

The scale out rack needs to be able to route to the base rack during install. These are also added to the private network definitions in the installer. The information for the networks that need to be routed to was listed as output of the command to get the credentials to access the federation (federated_api section). Note that when following the recommended network numbering, systems consisting only of X200/P200 can use aggregated routes towards all “private 1” and “private 2” (destination 172.20.128.0/17) networks (See the X200 networking guide).

• Single GEO aggregated routes for a column X:

  • private 1: 172.20.0.0/17 via ToR 1: 172.20.X.200

  • private 2: 172.20.128.0/17 via ToR 2: 172.20.[128+X].200

• 3-GEO systems aggregated routes for column X

  • private 1: destination networks 172.21.0.0/17, 172.22.0.0/17, 172.23.0.0/17 via ToR 1

  • private 2: destination networks 172.21.128.0/17, 172.22.128.0/17, 172.23.128.0/17 via ToR 2

all:
  children:
    ...
    asnodes:
      vars:
        ...
        private1:
          network: <nw 1>
          netmask: <nm 1>
          startip: <startip 1>
          endip: <endip 1>
          routes:
            - name: <route name>
              destination_network: <private network 1 on base rack>
              destination_netmask: <network mask for destination network>
              destination_gateway: <gateway in private network 1> 
        ...
        private2:
          network: <nw 2> 
          netmask: <nm 2>
          startip: <startip 2>
          endip: <endip 2>
          routes:
            - name: <route name>
              destination_network: <private network 2 on base rack>
              destination_netmask: <network mask for destination network>
              destination_gateway: <gateway in private network 2> 

        ...

The inventory file of the VM installer will also contain the public network definitions. In case the customer decides to not cable public nics, these public network definitions can just be omitted from the file.

The inventory file of the VM installer will have a dssconfig section where the customer can specify the storage policy. The converged scale out procedure requires the policy type, category and name to be the same on the scale out rack as for the existing deployment. Please make sure the dss policy values that were determined in the prerequisite section Determine selected policy type, category and name are correctly specified in the inventory file.

all:
  children:
    ...
    asnodes:
      vars:
        ...
        dssconfig:
          policy-type: <policy_type>
          policy-category: <policy_category> 
          policy-name: '<policy_name>'
          spread_width: <spread width>
          blockstore_safety: <blockstore safety>
          max_blocks_per_node: <max blocks>
        ... 

Complete the connectivity from the base system towards the scale out column. To easily configure the network routes, use the support script: /opt/ampli/scripts/capacity_scaleout_routes.py

In this step you need following information to determine the routes:

• All private networks of the scale out column.

• IP address of the gateways in the base environment.

In the 1-GEO case you need following 2 routes:

In the 3-GEO case you need 18 routes, 6 routes per GEO. You need following routes for SiteX:

Using this information, run the capacity_scaleout_routes script. The script expects a number of --route <destination network> <gateway IP address> inputs (e.g. --route 192.168.54.0/24 192.168.56.5). See the table above to determine what routes you will have to specify.

For 3-GEO deployments, run the capacity_scaleout_routes script 3 times: once from each Site. The script will fail when you attempt to add routes with a gateway in one of the other sites.

For 1-GEO you have to specify 2 routes. For 3-GEO you have to specify 6 routes, as indicated in the table above

/opt/ampli/scripts/capacity_scaleout_routes.py --route route1 --route route2 --route routeN

An example of successful execution for 1-GEO is below:

The output will show the GUIDs for the network and created routes.

An example of failed execution is below:

The above scripts will wait until the NICs are configured. It’s also possible to specify a --nowait flag. The script then returns immediately, and you can use the task GUID (if there is one, there won’t be any if all routes are already in the model) to manually check the status of the NIC configure tasks (e.g using taskviewer).

You can run following on the base environment as marvin user, and check that the routes are as expected:

$ecmd -r all -- ip route

This should show the added routes on all nodes. In addition, a ping test towards the mgmt IP addresses on the ToR gateways for the scale out system can be performed to confirm connectivity:

ping 172.20.0.200

In case a wrong route was added, you can use the network and route GUIDs that are listed in the output of the capacity_scaleout_routes script to delete the routes from the model.

In sherpa execute the following:

cli.marvin.network.route.delete(<network guid>, <route guid>)

Then re-run the script with the correct destination network and gateway as input.

Note: Provisioning of the scale out rack can start only after connectivity has been confirmed.

The scale out rack has no column defined and the column_update_bucket task would throw a lot of critical events in the GUI. It’s best to disable this first, or these will show up in the event log. Similarly there are several flame jobs that don’t have to run on a converged scale out system.

Open sherpa on a management node of the scale out environment and execute following snippet:

disable_tasks = ['column_bucket_update', 'flame_scalerdb_foreground', 'flame_scalerdb_background', 'flame_scalerdb_verifier', 'flame_object_verifier_report_check', 'flame_job_status_check', 'flame_spark_health_checker', 'account_quota_checker', 'slow_blockstores', 'coldstorage_tape_media_verify']
taskguids = [t['guid'] for t in cli.marvin.taskschedule.list() if t['name'] in disable_tasks]
for tg in taskguids:
    cli.marvin.taskschedule.update(tg, enabled=False)

In this step, generate a column document for the scale out environment.

For the remainder of the procedure it is advised to open 2 terminals. One terminal in which you are logged in on one of the management nodes of the base environment, and the other terminal in which you are logged in on one of the management nodes of the scale out environment.

Open sherpa on a management node of the scale out environment and execute following snippet:

from marvin_lib.network import ha
import json
env_guid = cli.marvin.environment.list()[0]['guid']
column = json.loads(ha.post('/model/environment/%s/action/get_column_record/?insecure=True' % env_guid))['column_record']

This dict needs some cleanup. Change the X in ColumnX to the column number.

del column['column_id']
column['name'] = 'ColumnX' 
column['sdb_status'] = 'ACTIVE'

Then run following snippet:

for connection in column['connection']:
    if 'location' in connection:
        connection['location'] = cli.marvin.location.name(connection['location'])

To print the column dict, just type "column" (without quotes). This will print a datastructure like the following. Then copy the complete datastructure.

The column document from the previous step has to be manually copied to the base environment.

Open sherpa on a management node of the base environment and copy-paste the column document in a variable like in the following snippet:

column = <paste the column dict>

To add the column to the model and configure it, execute following snippet in that same sherpa session:

from marvin_lib.network import ha
import json
col_guid = json.loads(ha.post('/model/column/', json.dumps(column)))['column']['guid']
cli.marvin.column.configure(col_guid)

Verify that the column configure task succeeds by executing following taskviewer command on one of the nodes in the base environment:

taskviewer -a configure

The column configure task will start new tasks on the scale out environment. Before proceeding with the scale out procedure wait for all tasks on the scale out environment to be finished. Log in to one of the nodes on the scale out environment and execute following taskviewer command:

taskviewer -i no

Log in to one of the management nodes of the scale out environment and become marvin user. Then execute following command:

/opt/ampli/apps/sherpa/venv/bin/python /opt/ampli/scripts/clean_configrules.py

Log into one of the management nodes of the scale out rack and start sherpa. Execute following snippet.

%cpaste
from marvin_lib.network import ha
import json
private_nws = json.loads(ha.get('/model/network/?query="networktype=PRIVATE"'))['network']
[ha.put('/model/network/%s/' % private_nw['guid'], data=json.dumps({'shared': True})) for private_nw in private_nws]
--

Then log into one of the management nodes of the base environment. Start sherpa and execute the following snippet. Wait for it to finish.

%cpaste
import json
from marvin_lib.network import ha
from marvin_lib.task import wait_for_tasks
env_guid = json.loads(ha.get('/model/environment/'))['environment'][0]['guid']
task = json.loads(ha.post('/model/environment/%s/action/update_scaleout_routes/?configure=True' % env_guid))['task']
wait_for_tasks([task['guid']], stop_on_error=True)
--

The last snippet will reconfigure the private network interfaces of all machines in the base environment and on ALL scale out environments, so including previously installed scale out environments.

Log in to one of the management nodes of the scale out environment and become marvin user. Then execute following command:

/opt/ampli/apps/sherpa/venv/bin/python /opt/ampli/scripts/ensure_metadata_scaleout.py

This script will write what it is doing to terminal as well as to /opt/ampli/var/log/marvin/marvin.log on the node from which you are running the script.

In case the command fails, the script will write the failure to terminal and in the log. Please contact engineering for root cause analysis and to apply a fix. The script is rerunnable and will detect if metadata scale out already happened or not, or whether it happened partially due to an error during a previous attempt. Make sure to have run this script without errors before proceeding to the next step in the scale out procedure.

The rack serial for the scale out rack(s) needs to be known on the existing deployment. New Rack resource(s) should be added to the model of the existing deployment such that GUI and prometheus can also gather and display resources and metrics from the scale out rack(s).

Open sherpa on a management node of the base environment and execute following snippet:

%cpaste
from marvin_lib.network import ha
import json
col_guid = max(cli.marvin.column.list(), key=lambda c:c['column_id'])['guid']
scaleout_racks = json.loads(ha.get('/model/column/%s/action/proxy/location/?query="locationtype=RACK"' % col_guid))['location']
for scaleout_rack in scaleout_racks:
    scaleout_dc_name = json.loads(ha.get('/model/column/%s/action/proxy/location/%s/' % (col_guid, scaleout_rack['parent'])))['location']['name']
    new_location = {
        'address': scaleout_rack['address'],
        'locationtype': 'RACK',
        'name': scaleout_rack['name'],
        'serial': scaleout_rack['serial'],
        'parent': scaleout_dc_name
    }
    ha.post("/model/location/", data=json.dumps(new_location))
--

This step only applies when the customer does not want the additional performance of (some of) the scale out nodes, so the customer won’t send S3 requests to (some of) the scale out nodes. In that case it is suggested, but not necessary, to pull the cables on the public interfaces of these scale out nodes. If left uncabled, the customer will see NIC_DOWN events for the public interfaces in the GUI. If there are no NIC_DOWN events in the GUI, you can skip this step. Otherwise we can unconfigure the public interfaces of the nodes. This step can be applied for both bonded and non-bonded public interfaces.

To resolve the NIC_DOWN events, log into one of the management nodes of the scale out environment and become marvin user. Most likely all cables of all nodes have been pulled, in which case you will see NIC_DOWN events for all nodes. Determine the hardware platform of the scale out system (X200/P200/P100E3/Z200) and use it to run following command. Notice that it says “all”, so the script will fix all nodes in the scale out environment:

performance_scaleout_networks.py --unconfigure all --system <X200/P200/P100E3/Z200>

If the customer has a system that is different from X200/P200/P100E3/Z200, it is also possible to define the public interfaces. Determine which interface belongs to which public network. This is also required if nic bonding is applied. Then you run the script like this:

performance_scaleout_networks.py --unconfigure all --eth-devices <eth for public nw 1> <eth for public nw 2>

For example, the equivalent command for X200 would be the following (using --system is more convenient though)

performance_scaleout_networks.py --unconfigure all --eth-devices eth3 eth5

In some cases the customer has some nodes that are cabled and some nodes that are not. In those cases you can specify which nodes are NOT cabled, so the nodes for which you’ll see NIC_DOWN events in the GUI.

For example, the customer has a 3GEO system and only wants S3 traffic on all nodes of Site1. Then you can unconfigure the public interfaces of Site2 and Site3 as follows:

performance_scaleout_networks.py --unconfigure Site2 Site3 --system <X200/P200/P100E3/Z200>

If, for example, the customer has a 6-node 1GEO system and only wants S3 traffic on node CMB04-6, then you can unconfigure the public interfaces of CMB01-3 as follows:

performance_scaleout_networks.py --unconfigure Site1-R02-C02-CMB01 Site1-R01-C02-CMB02 Site1-R02-C02-CMB03 --system <X200/P200/P100E3/Z200>

If, for example the customer has a 3GEO system with 6 nodes per GEO and the customer only wants S3 traffic on nodes CMB01-3 of Site3, then you can unconfigure the public interfaces of the nodes on Site1, on Site2 and nodes CMB04-6 on Site3 as follows:

performance_scaleout_networks.py --unconfigure Site1 Site2 Site3-R02-C02-CMB04 Site3-R02-C02-CMB05 Site3-R02-C02-CMB06 --system <X200/P200/P100E3/Z200>

This way you can unconfigure the public interfaces of the nodes in every way that the customer wants.

The script will always tell you which interfaces of which nodes that it unconfigured. An example output is the following:

Mind the last line in the output! Make sure that cables are actually unplugged for all public interfaces that are unconfigured. After running the script, the NIC_DOWN events will gradually go away. You will also see that the public interfaces are not listed anymore in the Resources page of the GUI.

To configure the metrics, execute the following command in sherpa on the existing deployment.

cli.marvin.environment.configure_metrics(cli.marvin.environment.list()[0]["guid"])

The result of the configure_metrics task is accessible trough taskviewer. Run following as marvin user:

taskviewer -a configure_metrics

Note: Please note that metrics on the scaleout rack will only be available when the scaleout completes.

At this point in the configuration, you can SSH to any node in the scale out environment using the password, but without the support password prefix. For security reasons this has to be rectified.

This is currently a manual procedure. The system admin will need to log into the GUI and re-enter the support password prefix again. You will need to contact the customer’s system administrator and request that the support code prefix be re-entered .

Note: When clicking the Save button, the GUI might seem unresponsive, but the support password prefix will now be set for the scale out environment as well. Please verify that this is the case.

If the customer has an external Prometheus and/or external Grafana to monitor ActiveScale metrics, the new racks that have been added to the deployment via this scale out procedure also have to become visible to the external monitoring framework. This will not happen automatically as part of the previous steps. Instead the configuration of the external Prometheus and/or external Grafana has to be manually adjusted such that these new racks are also monitored.

Please consult the documentation about External Grafana and Prometheus in the Administration Guide (support edition) External Grafana and Prometheus to have an in depth explanation of the steps that are required to have an external Prometheus scrape a Global or Local Prometheus in ActiveScale and/or have an external Grafana display metrics for a Global or Local Prometheus in ActiveScale.

External Prometheus

• If the customer’s external Prometheus is scraping a Global Prometheus in ActiveScale, no configuration changes are needed. The metrics for the scale out racks will automatically be included.

• If the customer’s external Prometheus is scraping Local Prometheus instances in ActiveScale, we will have to add a scrape_config job for every scale out rack to the external Prometheus configuration file. Please consult the Administration Guide for the necessary changes to the configuration file. It is very important to set the systemid correctly for the scale out racks. Make sure that the systemid is the same in all scrape_configs for Local Prometheus instances in the ActiveScale deployment. Otherwise Grafana will incorrectly and incompletely show metrics for the ActiveScale deployment.

External Grafana

• If the customer’s external Grafana has a data source for an external Prometheus, no configuration changes are needed. Make sure that the configuration changes for the external Prometheus are correctly applied though.

• If the customer’s external Grafana has a data source for a Global Prometheus instance in ActiveScale, no configuration changes are needed. The metrics for the scale out racks will automatically be included.

• If the customer’s external Grafana has data sources for Local Prometheus instances in ActiveScale, first ask the customer if they want to add data sources for the scale out racks as well. If so, please consult the Administration guide for the necessary steps to add a new Local Prometheus data source for each scale out rack