Converged Scale Out Guide

See networking guide for scaling out with X200 hardware: Combined Scale Out Networking

Configuring the public networks for X200 is outlined in the networking guide. Should the customer choose not to cable public network interfaces of the X200 nodes, extra care has to be taken when configuring settings for the installer: This topic provides instructions and resources to install software using the VM installation procedure..

Determine the current scenario you are in and follow the guide to configure the network according to recommendations:

• X100 + X200:Scaling out X100 with X200 (Single GEO)

• X200 + X200. A cluster is the set of ActiveScale nodes that are connected to the same pair of ToR switches. A cluster can be up to 18 twin nodes, spread over 1 to 6 columns (in multiples of 3 per column):

  • 2 clusters: Scaling out X200 with X200 - 2 clusters

  • 3 clusters:Scaling out X200 with X200 - 3 clusters

  • > 3 clusters: Scaling out X200 with X200 - beyond 3 clusters

Switch configuration must allow bidirectional connectivity between the (private) networks for the scale out rack and (private or scaleout network) on the base rack prior to install.

• X100 + X200 3-Geo:Combined Scale Out Networking for 3-GEO deployments

Determine the selected policy on the existing deployment. We need:

• policy type

• policy category

• policy name

Using sherpa on a scaler management node on the existing deployment:

%cpaste
policy_type = !/opt/ampli/apps/marvinbase/venv/bin/python -c "from installer.common import find_policy_group; print find_policy_group()"
policy_type = policy_type[0]
policy_category = cli.marvin.environment.list()[0]['selectedstoragepolicy']['category']
supportedstoragepolicies = cli.marvin.environment.list()[0]['supportedstoragepolicy']
policy_name = next((policy['name'] for policy in supportedstoragepolicies if policy['category'] == policy_category), None)
if not policy_name:
    from marvin_lib.utils import load_yaml
    dssconfig_path = "/opt/ampli/apps/marvinbase/config/dssconfig.yml"
    dss_config = load_yaml(dssconfig_path)
    policy_name = dss_config.get('policies').get(policy_type).get(policy_category).get('name')
print "policy type: %s\npolicy category: %s\npolicy name: %s" % (policy_type, policy_category, policy_name)
--

Make sure to note these values. The customer can only choose a policy of the same category on the scaleout rack.

The converged scale out procedure depends on the location names that are defined in the model of the existing deployment. Make sure it is known in which datacenter each rack is situated. This information is visible in the GUI and can be provided by the customer. It is also possible to determine it from the model. Execute following snippet in sherpa to get a list of datacenters and corresponding racks:

for dc in cli.marvin.location.list(query="locationtype=DATACENTER"):
    for rack in [l for l in cli.marvin.location.list(query="locationtype=RACK") if l['parent']==dc["guid"]]:
        print "DATACENTER %s with RACK %s (serial: %s)" % (dc["name"], rack["name"], rack["serial"])

example output:

See installer guide: Step 4: Cable the Components

Follow the different steps in the Quantum VM installer guide to prepare and set up the installer. Make sure that the host machine has the necessary hardware requirements and that the VM installer can actually reach the ActiveScale nodes that are going to be installed. As this is an installation of a scale out rack, make sure that the VM is connected to one of the switches of the scale out rack and that the specific port on that switch is on the same VLAN as the nodes that are being installed.

Then follow the VM installer guide steps to build the inventory and configure the ActiveScale system options for the scale out rack. For the converged scale out it is important to make sure the settings in the following section are correctly specified in the inventory file.

Among the asnodes variables in the inventory file there will be a parameter called scale out_rack. By default its value is set to false, but when installing a capacity only scale out rack, the value has to be set to true:

all:
  children:
    ...
    asnodes:
      vars:
        ...
        scaleout_rack: true
        ...

The purpose of this setting is to skip column configure and scaler account generation on the scale out rack.

AS 6.1 and above leverages the federation concept for scaling out. The federation is stored in a separate database (a dedicated mongodb cluster). During the install of the scale out environment, it will contact the base rack for information. To get the credentials to access the base rack, execute the following sherpa script on the base rack:

%cpaste
from marvin_lib.network import ha
import json
cluster = [c for c in cli.marvin.mongocluster.list() if c['name'] == 'federated_model']
for c in cluster:
    for a in c['application']:
        machine = cli.marvin.machine.get(a['machine'])
        for n in machine['nic']:
            if 'network' not in n or n['network'] is None:
                continue
            nw = cli.marvin.network.get(n['network'])
            if nw['networktype'] == 'PRIVATE':
                print("%s [%s/%s]" % (n['address'], nw['network'], nw['netmask']))
        print a['ports']['general']
    print('')
print('')
print(json.loads(ha.get('/model/environment/?insecure=True'))['environment'][0]['ssl_key'])
print(json.loads(ha.get('/model/environment/?insecure=True'))['environment'][0]['ssl_certificate'])
--

It will list the IP addresses (with network and netmask) and port to connect to the (usually 3) mongodb instances that make up the federated cluster, e.g. when executed on an X100:

This information is then input into the federated_api section of the VM installer inventory. The keys and certificate need to be copied into separate files in the quantum_installer/federated_api_files folder, the filenames can then be input into the VM installer inventory, together with the information for the mongodb endpoints:

 # Below section is not relevant if this is not a scaleout rack
        federated_api:
          ssl_key: federated_ssl.key # File name of file containing the SSL key for the federated api, located in quantum_installer/federated_api_files folder
          ssl_cert: federated_cert.crt # File name of file containing the certificate for the federated api, located in quantum_installer/federated_api_files folder
          endpoints:
            - endpoint: # endpoint 1 for federated api
                ip1: 172.16.11.31
                ip2: 172.16.21.31
                port: 27018 # Port number for federated api on endpoint 1
            - endpoint: # endpoint 2 for federated api
                ip1: 172.16.11.32
                ip2: 172.16.21.32
                port: 27018 # Port number for federated api on endpoint 2
            - endpoint: # endpoint 3 for federated api
                ip1: 172.16.11.33
                ip2: 172.20.21.33
                port: 27018 # Port number for federated api on endpoint 3
        # end scaleout rack section

The inventory file of the VM installer has a datacenter section where the datacenter name can be specified. The converged scale out procedure requires that the datacenter name that is defined at installation of the scale out rack has to be same as the one specified in the model of the existing deployment. Datacenters and rack names of the existing deployment were determined in the prerequisite section (Determine datacenter and rack names of existing deployment). Determine which rack of the existing deployment is in the same physical datacenter, and get the corresponding datacenter name. For a 1 GEO deployment this will be trivial as all racks are in that same datacenter, and there is really only 1 datacenter name. In the 3 GEO case you’ll have to make sure to select the correct datacenter.

In either case, the datacenter should be defined in the model of the existing deployment. This can be verified in the GUI or by entering following command (change <dc name> to the datacenter name you’ve found) in sherpa on one of the scaler management nodes of the existing deployment:

cli.marvin.location.get(<dc name>)

Once you are certain that you’ve found the correct datacenter name, adjust the inventory file of the VM installer as outlined in the installer guide:

all:
  children:
    ...
    asnodes:
      vars:
        ...
        datacenter:
          - name: <dc name>
        ... 

The inventory file of the VM installer will also have a rack section where the rack name and serial can be specified. The converged scale out procedure requires that the rack name has to be unique. All rack names in the existing deployment were already determined in the prerequisite section (Determine datacenter and rack names of existing deployment), so make sure to not pick a name that is already defined. It is also crucial that the rack serial is correctly specified:

all:
  children:
    ...
    asnodes:
      vars:
        ...
        rack:
          - name: <rack name>
              serial: <rack serial>
        ... 

The inventory file of the VM installer will have a network section where we can specify the private networks that have to be modeled on the scale out rack. Using the specific guide on the networking scenario that was determined in the prerequisite section Networking, the network and netmask of the 2 private networks should be determined.

In case the customer wants to have the machines of the scale out rack on the same subnets as the machines of the existing deployment, the startip and endip should be carefully chosen such that there is no overlap with the ip range that is specified for the private networks in the model of the existing deployment. The ip range should be big enough to cover all machines. In case of scale out with X200 there should also be extra ip addresses for IPMI and JBODs.

The scale out rack needs to be able to route to the base rack during install. These are also added to the private network definitions in the installer. The information for the networks that need to be routed to was listed as output of the command to get the credentials to access the federation (federated_api section). Note that when following the recommended network numbering, systems consisting only of X200/P200 can use aggregated routes towards all “private 1” and “private 2” (destination 172.20.128.0/17) networks (See the X200 networking guide).

• Single GEO aggregated routes for a column X:

  • private 1: 172.20.0.0/17 via ToR 1: 172.20.X.200

  • private 2: 172.20.128.0/17 via ToR 2: 172.20.[128+X].200

• 3-GEO systems aggregated routes for column X

  • private 1: destination networks 172.21.0.0/17, 172.22.0.0/17, 172.23.0.0/17 via ToR 1

  • private 2: destination networks 172.21.128.0/17, 172.22.128.0/17, 172.23.128.0/17 via ToR 2

all:
  children:
    ...
    asnodes:
      vars:
        ...
        private1:
          network: <nw 1>
          netmask: <nm 1>
          startip: <startip 1>
          endip: <endip 1>
          routes:
            - name: <route name>
              destination_network: <private network 1 on base rack>
              destination_netmask: <network mask for destination network>
              destination_gateway: <gateway in private network 1> 
        ...
        private2:
          network: <nw 2> 
          netmask: <nm 2>
          startip: <startip 2>
          endip: <endip 2>
          routes:
            - name: <route name>
              destination_network: <private network 2 on base rack>
              destination_netmask: <network mask for destination network>
              destination_gateway: <gateway in private network 2> 

        ...

The inventory file of the VM installer will also contain the public network definitions. In case the customer decides to not cable public nics, these public network definitions can just be omitted from the file.

The inventory file of the VM installer will have a dssconfig section where the customer can specify the storage policy. The converged scale out procedure requires the policy type, category and name to be the same on the scale out rack as for the existing deployment. Please make sure the dss policy values that were determined in the prerequisite section Determine selected policy type, category and name are correctly specified in the inventory file.

all:
  children:
    ...
    asnodes:
      vars:
        ...
        dssconfig:
          policy-type: <policy_type>
          policy-category: <policy_category> 
          policy-name: '<policy_name>'
          spread_width: <spread width>
          blockstore_safety: <blockstore safety>
          max_blocks_per_node: <max blocks>
        ... 

Complete the connectivity from the base system towards the scale out column. To easily configure the network routes, use the support script: /opt/ampli/scripts/capacity_scaleout_routes.py

In this step you need following information to determine the routes:

• All private networks of the scale out column.

• IP address of the gateways in the base environment.

In the 1-GEO case you need following 2 routes:

In the 3-GEO case you need 18 routes, 6 routes per GEO. You need following routes for SiteX:

Using this information, run the capacity_scaleout_routes script. The script expects a number of --route <destination network> <gateway IP address> inputs (e.g. --route 192.168.54.0/24 192.168.56.5). See the table above to determine what routes you will have to specify.

For 3-GEO deployments, run the capacity_scaleout_routes script 3 times: once from each Site. The script will fail when you attempt to add routes with a gateway in one of the other sites.

For 1-GEO you have to specify 2 routes. For 3-GEO you have to specify 6 routes, as indicated in the table above

/opt/ampli/scripts/capacity_scaleout_routes.py --route route1 --route route2 --route routeN

An example of successful execution for 1-GEO is below:

The output will show the GUIDs for the network and created routes.

An example of failed execution is below:

The above scripts will wait until the NICs are configured. It’s also possible to specify a --nowait flag. The script then returns immediately, and you can use the task GUID (if there is one, there won’t be any if all routes are already in the model) to manually check the status of the NIC configure tasks (e.g using taskviewer).

You can run following on the base environment as marvin user, and check that the routes are as expected:

$ecmd -r all -- ip route

This should show the added routes on all nodes. In addition, a ping test towards the mgmt IP addresses on the ToR gateways for the scale out system can be performed to confirm connectivity:

ping 172.20.0.200

In case a wrong route was added, you can use the network and route GUIDs that are listed in the output of the capacity_scaleout_routes script to delete the routes from the model.

In sherpa execute the following:

cli.marvin.network.route.delete(<network guid>, <route guid>)

Then re-run the script with the correct destination network and gateway as input.

Note: Provisioning of the scale out rack can start only after connectivity has been confirmed.

The scale out rack has no column defined and the column_update_bucket task would throw a lot of critical events in the GUI. It’s best to disable this first, or these will show up in the event log. Similarly there are several flame jobs that don’t have to run on a converged scale out system.

Open sherpa on a management node of the scale out environment and execute following snippet:

disable_tasks = ['column_bucket_update', 'flame_scalerdb_foreground', 'flame_scalerdb_background', 'flame_scalerdb_verifier', 'flame_object_verifier_report_check', 'flame_job_status_check', 'flame_spark_health_checker', 'account_quota_checker', 'slow_blockstores']
taskguids = [t['guid'] for t in cli.marvin.taskschedule.list() if t['name'] in disable_tasks]
for tg in taskguids:
    cli.marvin.taskschedule.update(tg, enabled=False)

In this step, generate a column document for the scale out environment.

For the remainder of the procedure it is advised to open 2 terminals. One terminal in which you are logged in on one of the management nodes of the base environment, and the other terminal in which you are logged in on one of the management nodes of the scale out environment.

Open sherpa on a management node of the scale out environment and execute following snippet:

from marvin_lib.network import ha
import json
env_guid = cli.marvin.environment.list()[0]['guid']
column = json.loads(ha.post('/model/environment/%s/action/get_column_record/?insecure=True' % env_guid))['column_record']

This dict needs some cleanup. Change the X in ColumnX to the column number.

del column['column_id']
column['name'] = 'ColumnX' 
column['sdb_status'] = 'ACTIVE'

Then run following snippet:

for connection in column['connection']:
    if 'location' in connection:
        connection['location'] = cli.marvin.location.name(connection['location'])

To print the column dict, just type "column" (without quotes). This will print a datastructure like the following. Then copy the complete datastructure.

The column document from the previous step has to be manually copied to the base environment.

Open sherpa on a management node of the base environment and copy-paste the column document in a variable like in the following snippet:

column = <paste the column dict>

To add the column to the model and configure it, execute following snippet in that same sherpa session:

from marvin_lib.network import ha
import json
col_guid = json.loads(ha.post('/model/column/', json.dumps(column)))['column']['guid']
cli.marvin.column.configure(col_guid)

Verify that the column configure task succeeds by executing following taskviewer command on one of the nodes in the base environment:

taskviewer -a configure

The column configure task will start new tasks on the scale out environment. Before proceeding with the scale out procedure wait for all tasks on the scale out environment to be finished. Log in to one of the nodes on the scale out environment and execute following taskviewer command:

taskviewer -i no

Log in to one of the management nodes of the scale out environment and become marvin user. Then execute following command:

/opt/ampli/apps/sherpa/venv/bin/python /opt/ampli/scripts/ensure_metadata_scaleout.py

This script will write what it is doing to terminal as well as to /opt/ampli/var/log/marvin/marvin.log on the node from which you are running the script.

In case the command fails, the script will write the failure to terminal and in the log. Please contact engineering for root cause analysis and to apply a fix. The script is rerunnable and will detect if metadata scale out already happened or not, or whether it happened partially due to an error during a previous attempt. Make sure to have run this script without errors before proceeding to the next step in the scale out procedure.

The rack serial for the scale out rack(s) needs to be known on the existing deployment. New Rack resource(s) should be added to the model of the existing deployment such that GUI and prometheus can also gather and display resources and metrics from the scale out rack(s).

Open sherpa on a management node of the base environment and execute following snippet:

%cpaste
from marvin_lib.network import ha
import json
col_guid = max(cli.marvin.column.list(), key=lambda c:c['column_id'])['guid']
scaleout_racks = json.loads(ha.get('/model/column/%s/action/proxy/location/?query="locationtype=RACK"' % col_guid))['location']
for scaleout_rack in scaleout_racks:
    scaleout_dc_name = json.loads(ha.get('/model/column/%s/action/proxy/location/%s/' % (col_guid, scaleout_rack['parent'])))['location']['name']
    new_location = {
        'address': scaleout_rack['address'],
        'locationtype': 'RACK',
        'name': scaleout_rack['name'],
        'serial': scaleout_rack['serial'],
        'parent': scaleout_dc_name
    }
    ha.post("/model/location/", data=json.dumps(new_location))
--

To configure the metrics, execute the following command in sherpa on the existing deployment.

cli.marvin.environment.configure_metrics(cli.marvin.environment.list()[0]["guid"])

The result of the configure_metrics task is accessible trough taskviewer. Run following as marvin user:

taskviewer -a configure_metrics

Note: Please note that metrics on the scaleout rack will only be available when the scaleout completes.

At this point in the configuration, you can SSH to any node in the scale out environment using the password, but without the support password prefix. For security reasons this has to be rectified.

This is currently a manual procedure. The system admin will need to log into the GUI and re-enter the support password prefix again. You will need to contact the customer’s system administrator and request that the support code prefix be re-entered .

Note: When clicking the Save button, the GUI might seem unresponsive, but the support password prefix will now be set for the scale out environment as well. Please verify that this is the case.

If the customer has an external Prometheus and/or external Grafana to monitor ActiveScale metrics, the new racks that have been added to the deployment via this scale out procedure also have to become visible to the external monitoring framework. This will not happen automatically as part of the previous steps. Instead the configuration of the external Prometheus and/or external Grafana has to be manually adjusted such that these new racks are also monitored.

Please consult the documentation about External Grafana and Prometheus in the Administration Guide (support edition) External Grafana and Prometheus to have an in depth explanation of the steps that are required to have an external Prometheus scrape a Global or Local Prometheus in ActiveScale and/or have an external Grafana display metrics for a Global or Local Prometheus in ActiveScale.

External Prometheus

• If the customer’s external Prometheus is scraping a Global Prometheus in ActiveScale, no configuration changes are needed. The metrics for the scale out racks will automatically be included.

• If the customer’s external Prometheus is scraping Local Prometheus instances in ActiveScale, we will have to add a scrape_config job for every scale out rack to the external Prometheus configuration file. Please consult the Administration Guide for the necessary changes to the configuration file. It is very important to set the systemid correctly for the scale out racks. Make sure that the systemid is the same in all scrape_configs for Local Prometheus instances in the ActiveScale deployment. Otherwise Grafana will incorrectly and incompletely show metrics for the ActiveScale deployment.

External Grafana

• If the customer’s external Grafana has a data source for an external Prometheus, no configuration changes are needed. Make sure that the configuration changes for the external Prometheus are correctly applied though.

• If the customer’s external Grafana has a data source for a Global Prometheus instance in ActiveScale, no configuration changes are needed. The metrics for the scale out racks will automatically be included.

• If the customer’s external Grafana has data sources for Local Prometheus instances in ActiveScale, first ask the customer if they want to add data sources for the scale out racks as well. If so, please consult the Administration guide for the necessary steps to add a new Local Prometheus data source for each scale out rack.