Software Installation

The Quantum Installer VM consists of an OVA image which contains both the installation VM and the corresponding ActiveScale ISO. Download the appropriate OVA file to your laptop.

3.1.1 When You Can't Use the OVA

In case OVA can't be used (MacBook with Apple M1/M2, VM provided by 3rd party, ESXi host, ...) you can also use the TGZ archive.

Refer to 3.7 Setup Installer via TGZ on Your Own VM for instructions.

Note: If you use the TGZ archive, you will still need to run through the other sections listed below, as they remain applicable, regardless if you use OVA or the TGZ archive.

The Installer VM can be imported by double-clicking the downloaded file and importing it via the VirtualBox Wizard. All default values should suffice. Alternatively, the OVA can also be imported via the Command Line on Unix based host systems by executing:

To be able to communicate with the ActiveScale system, the host system needs to able to communicate in Private VLAN1 for X200, P200 and Z200 or Private VLAN2 for P100E3. To achieve this, the host system which will run the VM installer will need to be physically connected to the switch which also connects the nodes of the ActiveScale system.

The Installer VM will connect to this network by using the bridged networking mode of VirtualBox. We recommend attaching an extra VNIC to the just-imported VM so communication between the host system and the VM is easier. An extra benefit of this is the fact that by default the VNIC of the VM is in NAT-networking mode which means it can access all systems which the hosts system can reach (including the Internet if applicable).

To attach an extra VNIC to the Installer VM you can use either the UI of VirtualBox or run the following commands on the host system:

Note: eth666 in the above example is the physical NIC of the host systems which is connected to the appropriate Private VLAN of the ActiveScale system.

Note: This port should be located in PrivateVLAN1 for X200, P200 and Z200 and in PrivateVLAN2 for P100E3!

To do this via the VirtualBox UI, you need to edit the just-imported VM and add a new VNIC. Make sure to select the correct mode: BRIDGED and connect it to the physical NIC of the host system which is connected to the ActiveScale VLAN. In most, if not all cases, this will be the Ethernet interface of the laptop.

Leave the already present VNIC in NAT-mode. This makes life simpler to connect to the installer, and enables the installer to reach the Internet if the laptop itself has Internet access through e.g. WiFi.

Note: Internet access is not a requirement to perform the install, but it’s a nice to have to sync things like time over NTP.

Once the above network config is completed, it is time to start the VM. This can be done by using the UI of VirtualBox or run the following command on the host system:

This will start the Ubuntu VM. Please note that by default the newly added bridged networkadapter is not connected to the VM. Once the VM has booted, it is time to connect it to the VM by using the UI or running on the host system:

Note: If you opted to add an extra bridged VNIC, you can now reach the VM via SSH over the default VNIC in NAT mode.

Final step of the setup of the Installer VM itself is to assign an IP in the Private VLAN of the ActiveScale system. Important to note here is that this IP adresss should be unique in the VLAN. This means that it should not be in use by any device in the VLAN and it cannot be assigned to any of the to-be-installed nodes. It should be a valid IP in the subnet of the Private VLAN.

By default for X200, P200 and Z200, this is Private VLAN1, VLANid 1000, 172.20.0.0/24. We recommend using: 172.20.0.240

By default for P100E3, this is Private VLAN2, VLANid 210, 172.16.210.0/24. We recommend using: 172.16.210.240

Note: These subnets/VLANids can differ depending of the customer’s requirements. But it’s important that for X200, P200 and Z200 the installer can reach Private VLAN1 and for P100E3 the installer can reach Private VLAN2

This IP can be set via the Ubuntu UI of the VM by navigating to Settings > Network and selecting the correct NIC. If you opted for the 2 VNIC solution, by default 1 NIC in the Ubuntu VM will have an IP in 10.0.2.0/24. Use the other NIC to set the IP in the Private VLAN. In most situations, the bridged NIC has the name ‘enp0s8’ in the Ubuntu VM.

Via commandline, the IP can be set in the VM by running the following command on the VM itself:

Or alter the config file under /etc/netplan/network-config.yaml as root to:

And the run the following command on the VM:

Note: By using the Netplan approach, the IP is persistent across reboots of the VM.

Because an ActiveScale system does not like major time jumps, we sync the date and time of the to-be installed ActiveScale nodes with the time defined on the Installer VM during installation. If Internet access is available to the VM installer, it can sync over NTP with a timeserver and no explicit steps should be needed.

Note: Always double check the date and time defined on the Installer VM before continuing.

However if there’s no Internet available and/or you see the date/time in the Installer VM being off as oposed to the actual UTC time, it’s a good idea to set the clock correctly via the Installer VM UI. We don’t need split-second precision, but we do want to have the correct hour and of course date set, so the ActiveScale system has a nice starting point and does not need to jump time later on.

In case OVA can't be used (MacBook with Apple M1/M2, VM provided by 3rd party, ESXi host, ...) one can also use the TGZ archive.

This sections assumes the following:

• Sections 3.5 and 3.6 above were executed.

• You have downloaded the .tgz file and extracted it to the ~/quantum_installer on the VM.

To begin, you will want to install the following packages:

• ubuntu-desktop-minimal

• firefox

• virtualbox-guest-additions-iso

• virtualenv

• nmap

• vlan

• ipmitool

• isc-dhcp-server

• tftpd-hpa

• nfs-kernel-server

• nfs-common

• ntp

• curl

• expect

• socat

• ntpdate

• iperf

• yamllint

• sshpass

Next, create the virtualenv. There might already be one already there, but it was packaged on Ubuntu 20.04 (but not a constant upstream), so remove it. Your system python 2.7 might not be the one packaged in the VM installer OVA.

Upload the ISO you want to install ActiveScale from to the VM under `~/<your ISO>`, and run the following snippet:

Do the same as you would in case you started from the standard OVA.

The inventories used in the installer are in the yaml format. This means whitespace determines the structure and precedence of items in the file. Be aware of this! Also mind the difference of TAB and spaces on Unix-based systems of which the installer VM is an example.

Generally speaking, the installer inventory structure looks like this:

So we have 3 types of systems:

• installers: There will always only be one installer defined under installers. In essence this is the installer VM, but specified in the Private VLAN.

• basenodes: Only used if this is an installation of a scale out system. This section will be automatically ignored otherwise. It defines networking information to connect to a single node in the base system. There should only be one basenode defined.

• asnodes: The number of asnodes is variable. It ranges from 3 for P100E3 to a minimum of 6, all the way up to 36 for X200. Under asnodes we will enter the required info on a per-node basis, allowing for full flexibility.

First step in building the inventory is defining the installer. For this you will need the IP you set on the VM in 3.0 Setup the Quantum Installer VM.

Below is an example;

Note: This section can be ignored if you are not installing a scale out system (scaleout_rack: false). Just leave the fields empty, they will be ignored anyway.

In case of scale out the installer will contact one of the nodes of the base system to retrieve specific information that is used to install the scale out system. At the same time this validates network connectivity between the base networks and the scale out networks. Provide following information in the inventory:

• network: Private VLAN1 of the base system. For P100E3 this has to be Private VLAN2. In case of 3-GEO you can pick Private1 (or Private2 for P100E3) in any of the GEOs.

• netmask: The corresponding netmask (e.g. 255.255.255.0)

• ip_address: IP address of one of the base system management nodes in that network

• gateway: IP address of the gateway in the network where the installer is running. This is a Private VLAN1 network of the scale out system. For P100E3 it is the IP address of the gateway in Private VLAN2.

For 3-GEO systems you only have to specify this section in 1 of the GEO yaml files, e.g. in GEO1.yaml. It does not matter which of the 3-GEO Private1 networks (or Private2 for P100E3) you specify in the basenodes section since the installer should be able to connect to any of GEOs at install time.

Below is an example:

In this example the installer will try to connect to a node of the base system with IP address 172.16.110.1 in network 172.16.110.0/24 via the gateway with address 172.20.0.241.

Next section to tackle is the vars section under asnodes. This will be based upon the customer requirements and networks provided. Great care needs to be taken here, because this models the ActiveScale system fully. For simplicity we will split this up in different subsections.

• networks: The next section defines the private and public networks being used in this install.

  # Definition of PrivateVLAN1

  # Make sure the startip is the first available IP for ActiveScale if not using explicit IP assignment!

  private1:

  network: 172.20.0.0

  netmask: 255.255.255.0

  startip: 172.20.0.1

  endip: 172.20.0.254

  dhcp_range: 10

  # Definition of PrivateVLAN2

  # Make sure the startip is the first available IP for ActiveScale if not using explicit IP assignment!

  private2:

  network: 172.20.128.0

  netmask: 255.255.255.0

  startip: 172.20.128.1

  endip: 172.20.128.254

  # Definition of Customer's Public Network 1

  public1:

  network: 192.168.1.0

  netmask: 255.255.255.0

  startip: 192.168.1.1

  endip: 192.168.1.254

  routes:

  # Below is the default route for the public networks

  - name: PublicNetwork1_route

  destination_network: 0.0.0.0

  destination_netmask: 0.0.0.0

  destination_gateway: 192.168.1.1 # Default gateway

  # Definition of Customer's Public Network 2

  public2:

  network: 192.168.2.0

  netmask: 255.255.255.0

  startip: 192.168.2.1

  endip: 192.168.2.254

  Note: While the default for the private VLANs should suffice, it could be a customer has different requirements. Make sure to collect the correct values for the public networks being used. Setting these here correctly prevents a lot of work once the system is installed, because the public interfaces will be reachable immediately.

  The above definitions are quite self-explanatory, it’s important to note that if the different private IPs are not set explicitly on a per node basis, the installer will use the value of startip as the first IP to assign to a node in a logical order. Also, if NIC bonding is used (see below), you do not need to define public2 because it’s not used. We recommend assigning the entire subnet (startip and endip), although if you must limit it, note that there need to be enough IPs for each machine and the IPMI for each machine in private1, which means at least 2 times (number of machines)!

  Note: Please note when defining public1_ip_address and/or public2_ip_address on ASNode level (see below), public1 and/or public2 defined here need to match. So don’t leave the defaults for these networks if the customer is going to use anything else than these defaults.

  Next you can define if NIC bonding on the public NICs will be used or not. If this is not the case you can safely ignore this section and leave bonded:false in place. Otherwise put it to true and configure the other options as needed. Example:

  bonded: true

  bonding:

  # Bonding mode

  # Choices are roundrobin, lacp or alb

  mode: lacp

  # Link monitoring frequency in milliseconds (default 100)

  miimon: 100

  # The rate in which the driver requests its link partner to transmit LACPDU packets in lacp mode

  # Choices are fast or slow

  lacp_rate: fast

  # The transmit hash policy to use for slave selection, applicable to modes lacp and alb

  # Choices are layer2, layer2+3 or layer3+4

  xmit_hash_policy: layer3+4

The final section to define in the inventory determines the amount of nodes, and which physical node is which. For simplicity we have defined 3 nodes for P100E3 and 6 for X200, P200 and Z200. To expand the number of nodes for X200, P200, and Z200, you simply need to add the next CMB0X definition and define all parameters. If you’re installing a P200 or Z200 system with less than 6 nodes, remove the unsued defintions!

For each node there are 2 required parameters and 4 optional.

If the BMC/IPMI is fully configured when the rack is installed and a (default) username and password are known, we can opt to use the fully automated install flow. Via this flow it is no longer needed for an operator to manually power-on/cycle the ASnodes when PXE-booting. To achieve this we need to edit one general config file and add some extra data in the inventory.

4.7.1 Configure IPMI Address

Configure the IPMI IP address from the BIOS setup utility for each node by performing the following:

1. On boot-up when the Quantum screen is displayed, select the DEL key until the system enters the BIOS setup utility.

   

   The setup utility will open at the Main menu

   

2. Tab or arrow over to the IPMI tab and select Enter.

   

3. Select BMC Network Configuration and press Enter.

   The BMC network configuration will display.

   

4. Select Update IPMI Lan configuration [Yes].

5. Scroll down to Configuration Address Source and select Static.

   

6. Scroll down to Station IP Address and select Enter.

   

7. Enter the IPMI IP Address of the node and select Enter.

8. Scroll down to Subnet Mask and select Enter.

   

9. Enter the Subnet Mask for the node and select Enter.

10. Select Escape (ESC).

11. Tab over to Save & Exit and select Enter.

    

12. Select Save Changes and Reset and select Enter.

13. Select Yes. The Quantum screen will display.

    

14. Repeat Steps 1 - 13 for each node in the system.

4.7.2 Configure the IPMI Username and Password

1. Enter the IPMI IP Address into the browser from within the Quantum VM Installer. The login prompt will display.

   

2. Enter the username ADMIN and the password.

   X200

   Password is on the label on the twin server.

   

   Note: Server number one is the bottom server, server two is the top.

3. Select the Configuration tab. Then select Account Services and select the Users tab.

   

4. Select an empty User ID line, it will highlight in yellow.

   

5. Select line 10 and enter marvin as the user name.

   

6. Enter the marvin user name and enter the password Act1v3Sc4l3!#.

   

7. The new user added screen will display.

   

8. Select Logout.

9. Verify the added user by entering the Username marvin and the Password Act1v3Sc4l3!#.

   

4.7.3 Verify BIOS Settings (X200 and P200 Only)

Verify BIOS for IPV4 (only) PXE Boot by performing the following:

1. Upon boot-up, select del at the Quantum screen until the BIOS utility opens.

   

   The BIOS utility will open at the main menu

   

2. Select the Boot tab and select Enter.

   

3. Scroll down to UEFI NETWORK Drive BBS Priorities and select Enter.

   

4. Verify Options 1through 4 are PXE IPV4 only and that 5 through 8 are disabled. There should be no IPV6 options.

This step checks if the installer can reach itself over the Private VLAN (basic sanity check) and also extracts the ISO and configure TFTP. We opted to do this here, to shrink the size of the OVA as much as possible.

To run this step, either open a terminal on the VM or ssh into the VM from the host via: sshinstaller@127.0.0.1-p2222 and execute the following:

Next, execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4 Building the inventory, but without the .yaml extension.

Note: This step is only required if this is an installation of a scale out system. If this is not the case, please ignore this step and proceed with the PXE boot step.

Note: This step can also be run after PXE booting. It just has to be executed before deploying ActiveScale.

5.2.1 Usage

This step consists of the installer connecting to the base system and gathering information that is needed for installation of the scale out system. This includes federated API config and keys, routes… The installer uses the values that are specified in the basenodes section to connect to the base system, so make sure to define it: 4.4 Defining the Basenode (Scale Out Only). Note that this step also serves as a basic network check between the scale out network and the base system network!

To run this step, either open a terminal on the VM or ssh into the VM from the host via: sshinstaller@127.0.0.1-p2222 and execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4.0 Build the Inventory, but without the .yaml extension.

After a second or two the script will halt and prompt for the support username and password. Please provide the support username and password that you would use to log into the base system! This is the password that contains the support passcode (if the customer configured it). The prompt is the following for the username:

If you have entered the wrong username and/or password the script will fail later at this step:

And you will see following error

Just rerun the script and enter the correct username and password.

If there are any other errors, please have a look at the following section about basic debugging. In case of success the installer stores the information that it gathered locally. Note that if you were to rerun this step, that the script will not prompt for username and password anymore, but it will just read the values that are stored locally.

After running the 001-get-scaleout-info script we recommend to run additional inventory validation that is targeted specifically towards scale out. Please run following:

This script checks whether scale out related values in the inventory are valid. Note that running 001-get-scaleout-info.sh is a necessary requirement. Otherwise the script will fail with following error:

Please have a look at the following section about basic debugging if the validate script fails with an error.

5.2.2 Basic Debug

This section handles several errors that one can encounter when running the 001-get-scaleout-info and/or the 000-validate-inventory-scaleout scripts. When encountering an error, please investigate and rectify. Then you can rerun the script.

Not a scale out system

This system is not defined as a scale out system!

Depends on your intention. If this is not an installation of a scale out system you don’t have to run 001-get-scaleout-info. If this is a scale out system, please make sure to set scaleout_rack: true in the inventory.

Cached scale out info was edited

The cached scale out information on the installer had an unexpected content and has been removed. Please rerun the current script to regenerate it.

Someone or something edited the scale out info that was gathered in a previous run of the 001-get-scaleout-info script. It was unreliable so it got automatically removed. Please rerun the 001-get-scaleout-info script again

Basenodes section

The script validates values in the basenodes section. If this section is not defined, following error is generated. Please follow 4.4 Defining the Basenode (Scale Out Only).

A scale out installation requires a basenodes section.

Please follow documentation on how to define the basenodes section in the inventory.

The values in the basenode section are sanity checked. 4 values are expected: network, netmask, ip_address and gateway. The scripts will fail if any of these is missing. It also checks for incorrectly formatted IP addresses and whether the given ip_address is part of the given network/netmask. If the script fails for any of these checks, please resolve and rerun the script.

Furthermore the script will check if the given gateway IP is part of the same network as the installer. If this is not the case, the script shows following type of error. Please define the correct IP address of the gateway and rerun the script.

The given basenode gateway (<gateway>) is not in subnet <scale out network/ scale out netmask>.

This is the required VLAN for this platform.

Failed to ping basenode gateway

Unable to ping basenode gateway (<gateway>) from this installer.

Is this the correct gateway?

The gateway IP that is defined in the basenodes section of the inventory cannot be pinged from the installer. Note that this IP should be in the same network as the installer. Is the gateway online? Is the IP address correct? Please investigate and rectify.

Failed to ping basenode ip address

Note: Please take a look at Prepare gateways of base system and scale out system. Also note that the installer is only checking Private1 (or Private2 in case of P100E3) since that’s the network that it is part of. Please make sure to always configure both networks in all GEOs!

Unable to ping basenode ip_address (<ip_address>) from this installer. Please check the following:

This can have multiple causes. The output of the script provides several pointers:

This can have multiple causes. The output of the script provides several pointers:

• Is the IP address of the basenode correct?

• Is the machine to which the IP address belongs running?

• Is its interface down?

• Did you run the capacity scale out script on the base system? This has to be done for every GEO. See Adding routes from the base system to the scale out system

• Does the gateway of the base system network have a route to the scale out network. See Prepare gateways of base system and scale out system

• Does the gateway of the scale out network have a route to the base system network. See Prepare gateways of base system and scale out system

Failed to ping one or more management nodes of base system

Note: Please take a look at Prepare gateways of base system and scale out system. Also note that the installer is only checking Private1 (or Private2 in case of P100E3) since that’s the network that it is part of. Please make sure to always configure both networks in all GEOs!

The installer failed to ping following management nodes on the base system: <list of IPs>.

Please check the following:

This can have multiple causes. The output of the script provides several pointers:

• Is the machine running?

• Is its interface down?

• Did you run the capacity scale out script on the base system? This has to be done for every GEO. See Adding routes from the base system to the scale out system

• Does the gateway of the base system network have a route to the scale out network. See Prepare gateways of base system and scale out system

• Does the gateway of the scale out network have a route to the base system network. See Prepare gateways of base system and scale out system

The cached deployment_id is from a different installation

The cached deployment_id is from a different installation.

The scale out cache has been removed.

Please rerun this step such that it gets recreated for the correct system.

This error occurs when you are re-using this installer for a different customer. The script has found locally cached scale out information but it seems to belong to another system than the one that you are trying to install. The script has automatically removed that cached information. Please rerun the 001-get-scaleout-info script.

Unknown datacenter names

Datacenter names of the scale out system have to be the same as for the base system, which are: <list of names>.

Please change following datacenter name(s) in the inventory: <list of names>.

The name of the datacenter(s) that you defined in the inventory has/have to be the same ones as for the base system. Please make sure this is the case and rerun the script.

Rack names are not unique

Rack names of the scale out system have to be different from rack names in the existing deployment, which are: <list of names>.

Please change following rack name(s) in the inventory: <list of names>.

The name of the rack(s) that you defined in the inventory has/have to be different from the ones that are already known by the base system. Typically they are named R01, R02, R03… just pick the following in line. It is possible that the customer did not buy a new physical rack and that the nodes of the new scale out system are placed in a previously bought rack. In that case it is still needed to define a different rack name!

Missing route to base system

2 variants, for 1-GEO and 3-GEO

Network pivate1/private2 in the inventory file is missing a route to the base system!

Network pivate1/private2 in the inventory file of GEO1/GEO2/GEO3 is missing a route to the base system!

The output of the script tells you exactly what route you are missing. Pay attention that you have defined routes to the private networks of the base system. Add the missing route(s) to the inventory and rerun the script.

5.3.1 Usage

This step replaces the tedious USB boot part from the field installation procedure by PXE booting the ASnodes and installing our ActiveScale to disk automatically based upon the data entered in the inventory. This step also allows for easier re-do of the install if something goes wrong in the final step.

This step consists of the installer setting up al needed DHCP rules, the operator manually powering on/cycle all ASnodes and then the installer assigning an IP-address and performing the PXE-boot and install.

Note: The manual power-on/cycle by the operator can be circumvented by having BMC/IPMI configured at rack time. See: 4.7 Opting for the Fully Automated Install (Recommended)

1. Validate required BIOS settings (Z200 only).

2. Setup DHCP rules.

3. Installer prompt operator to power cycle/on the ASnodes.

4. ASnodes send DHCPDISCOVER and get an IP address from the Installer VM.

5. Installer VM serves a PXE boot image to the ASnodes.

6. ASnodes boot from this PXE image and request the same IP once more.

7. Installer waits until all ASnodes are up and reachable over SSH or it times out and notifies the operator.

8. Once all ASnodes are reachable over SSH, they are NTP time synced with the Installer VM..

9. The ActiveScale software is deployed to disk and the node info (IP,name, ..) are persisted to disk.

10. ASnodes are rebooted from disk.

11. Installer checks if all ASnodes are booted from disk.

Note: We recommend to power off all ASnodes before starting the installer so that all nodes are roughly in the same state at the same time. It will also ease potential debugging because if something is wrong, a certain node will run behind in the output. Leave the JBODs powered on!

To run this step, either open a terminal on the VM or ssh into the VM from the host via sshinstaller@127.0.0.1-p2222 and execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4 Building the inventory, but without the .yaml extension.

When not opting for the fully automatic approach, after a short while, the output will pause at:

This is the point where you need to manually power on the ASNodes.

From the moment where the nodes are powered on/cycled until the next output it should take around 10/15 minutes. You can always follow the PXE boot process live on ASnode by connecting a display to one of the nodes.

5.3.2 Basic Debug

The most common issues an operator can encounter here is either a mistake in the inventory or a hardware issue.

MAC Address

It’s critically important to have the MAC-address correct in the inventory before running this step. If you notice that a node is not PXE-booted or much slower then the others to PXE-boot, it is best to check which MAC-address is being seen on the switch on the corresponding port. If this matches with the input in the inventory you can check the syslog of the installer VM to see which MAC-addresses are sending a DHCPDISCOVER.

DHCP Server

Also make sure that no other DHCP servers are running on the Private VLAN. This is especially the case with customer’s using shared switches instead of dedicated switches. Having an other DHCP server present in this VLAN will cause collisions on the network.

P100E3

For P100E3 specific, please make sure the correct NIC is set as first boot NIC in the BIOS. Should be: NIC2 IPv4 Intel(R) Ethernet Connection X552 10 GbE Backplane.

Bandwidth

If the installation fails during PXE-boot with the message there’s not enough bandwidth available, please double check how much bandwidth there is available between the VM installer and the ActiveScale system. By default we require and check on a value around 1Gbit/sec (actually 0,9 Gbit/sec as a practical minimum). If this value is not present, and you want to overwrite this maximum, you can do this by adding the following to you inventory file:

all:

vars:

bandwidth_required: 900000000

children:

installers:

Where 700000000 is the amount of bits per second you want as a minimum. Please also note that how lower the available bandwidth, the longer it will take to PXE-boot and perform the install to disk, because we need to copy around 3GB of data per ActiveScale node during this install. It can very well be you run into timeouts. These are set to 1500 seconds by default. If you want to overwrite these, you can do so by editing: inventories/group_vars/all.yaml

himalaya_install: true

bmc_boot: false

initial_delay: 180

total_timeout: 1800

private1.name: PrivateNetwork1

private2.name: PrivateNetwork2

public1.name: PublicNetwork1

public2.name: PublicNetwork2

Add a definition for himalaya_install_timeout and set a new value, e.g. 2000. To extend the timeout when deploying ActiveScale to disk, the default value is 1500s. In the case the install fails during the post-install phase you can define himalaya_post_install_timeout to override this timeout. Default value is 600s (10 minutes).

Please take into account that the post-install consists of copying about 3GB of data from the installer to each node, so in low bandwidth situations this can cause havoc. In very rare cases you need to define a new value for pxe_timeout. This is the time the installer maximum waits before a node should be PXE-booted ready to receive further instructions. Most pxe_timeout related errors are caused by cabling errors or incorrect BIOS boot settings. Or: the operator not being quick enough to power cycle the nodes in the manual power cycle scenario.

This step configures the ASnodes fully as ActiveScale systems, by installing all the different components, configuring them, formatting disks, assigning and activating blockstores, apply policies, …

If this is the installation of a scale out system please make sure to first follow section 5.2 Get Scale Out Info.

All queue files which needed to be altered manually in the field installation procedure are generated automatically here based upon the data entered in the inventory. This step consists of the following subtasks:

1. Generate the different queue files and config files

2. Copy these to CMB01

3. Run the install_activescale.yaml playbook on CMB01

This step will take several hours to complete and total run time is dependent on the amount of disks and total config of the ASnodes. We estimate a minimal run time of 4 to 6 hours.

To run this step, either open a terminal on the VM or ssh into the VM from the host via: sshinstaller@127.0.0.1-p2222 and execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4 Building the inventory, but without the .yaml extension.

Situations can arise where one needs to connect to one of the nodes from the installer node without the use of a username/password. This can be done by using the factory installer ssh key. Please note that after CBW is installed, the default factory ssh key is replaced.

The key (with name id_rsa_factory) can be found in the quantum_installer/keys folder. One can also locate the factory installer ssh key by using:

The found key can be used to connect to a node:

where KEY denotes the key’s path and NODE_IP the IP address of the node one wants to connect to.

Define that this installation is for a scale out system by setting scaleout_rack: true in the inventory file. In case of 3GEO, make sure to set scaleout_rack: true in all 3 GEO yaml files.

The installer will connect to one of the nodes of the base system to gather some information that will be used for validation and configuration of the scale out system. You have to provide 4 values as indicated in below snippet. For P100E3 you are required to define network Private VLAN2 of the base system.

The inventory file of the VM installer has a datacenter section where you have to specify the datacenter name. For scale out systems it is required that the datacenter names (multiple in case of 3GEO) that are defined at installation of the scale out system have to be same as the those of the base system. Datacenters and rack names of the existing deployment were determined in the prerequisite section. See Determine datacenter and rack names of existing deployment.

Whether datacenter names are the same will be automatically validated by the installer.

The inventory file of the VM installer will also have a rack section where the rack name and serial can be specified. For scale out systems it is required that the rack names are unique. All rack names in the existing deployment were already determined in the prerequisite section (see See Determine datacenter and rack names of existing deployment), so make sure to not pick a name that is already defined. It is also crucial that the rack serial is correctly specified.

Note: It is possible that the customer did not buy a new physical rack and that the nodes of the new scale out system are placed in a previously bought rack. In that case it is still needed to define a different rack name! You can use the same rack serial though.

Whether the rack names are unique will be automatically validated by the installer.

This defines the column number of this rack, should be in format CXX, defaults to C01, but if this is a scale-out rack it should be set to 1 higher than the current highest column number. E.g for first scale-out rack: C02

Note: The installer will validate that routes to the scale out system are correctly defined and will throw an error if this is not the case.

The inventory has a network section where you have to specify the private networks of the scale out rack. Fill in the network and netmask of private1 and private2. The IP range should be big enough to cover all machines. In case of scale out with X200 make the range big enough such that there are enough IP addresses for IPMI and JBODs.

The scale out system needs to be able to route to the base system during install. These are also added to the private network definitions in the inventory. For 1-GEO scale out systems you need a route per private network, and for 3-GEO scale out systems you need 3 extra (on top of the 2 existing) routes per private network.

Note: Don’t set destination_network_guid for routes to the base system. These are only needed for the routes to other GEOs of the scale out system.

This is an example for 1-GEO:

and for 3-GEO, for SiteX (assuming SiteY and SiteZ are the other sites):

Note that when the recommended network numbering is used, systems consisting only of X200/P200 can use aggregated routes towards all “private 1” and “private 2” (destination 172.20.128.0/17) networks.

• Single GEO aggregated routes for a column X:

  • private 1: 172.20.0.0/17 via ToR 1: 172.20.X.200

  • private 2: 172.20.128.0/17 via ToR 2: 172.20.[128+X].200

• 3-GEO systems aggregated routes for column X:

  • private 1: destination networks 172.21.0.0/17, 172.22.0.0/17, 172.23.0.0/17 via ToR 1

  • private 2: destination networks 172.21.128.0/17, 172.22.128.0/17, 172.23.128.0/17 via ToR 2

An ActiveScale 3GEO system, is an ActiveScale system spread accross 3 different physical sites. This means there are 3 times the amount of network subnets involved. It’s up to the customer to decide how he sets up the public networking side of things, but for ActiveScale, each ActiveScale instance in each site has 2 seperate private networks which need to be able to access one of the private networks in the other sites to spread the data.

Most customers opt to use a WAN connection to transparently connect all private vlans and we simply have to add a route in ActiveScale to a specific gateway which is part of the local private network. In this guide we will not explain the 3GEO networking setup, but having a key understanding on how this all ties together is paramount in understanding how to correctly use the VM installer.

The VM installer PXE boots all the different nodes of an ActiveScale system to be able to install them. With 3GEO however, it’s possible that there is simply not enough bandwidth to PXE boot the nodes in Site 2 or 3 from a VM Installer which is located in Site 1. It could be that the PXE boot step needs to be performed seperately in 3 distinct physical locations, this guide will explain which commands to use to achieve this. It’s paramount however that the inventory used in all 3 sites matches and to achieve this, we recommend building the inventory before heading out to the customer’s sites to do the actual install. It’s much easier to spot potential issues in the inventory when not being stressed about performing the actual install.

Note: It’s important that the routing between the different private networks in the different sites is up and running when doing the install. Especially because the actual deployment of ActiveScale will happen from Site 1.

Note: You will notice that Site 1 in a 3GEO setup can be seen as the ‘parent’ site. Next to the actual deployment of ActiveScale being started here, it’s important to note that also the mini-CBW can only be run from Site 1! It will configure all sites, but needs to be started from Site 1.

The inventory for a 3GEO system is in essence a folder in the inventories subfolder of the VM Installer. It consists of some additional configuration files and 3 inventory files. Each of these 3 inventory files are very similar to the inventory file of a single GEO system, as they also model 1 specific GEO.

The 3 files are named: GEO1.yaml, GEO2.yaml and GEO3.yaml. There are 2 very important differences between the different files: - Only GEO1.yaml can have the stroage policy configuration present because this configuration needs to be consistent accross all GEOs - The definition of the private networks WILL differ in each GEO, the definition of the public networks CAN differ. Either way both private and public networks need to be defined and modeled correctly for each GEO even if they are the same.

In the case where we cannot PXE-boot accross different GEOs, also the definition in the installers section of each file will differ, even if you re-use the same VM with the same set of files, because of the fact the private networks differ, the IP of the installer in each GEO will differ. Don’t forget to change the installer IP in each GEO accordingly! We recommend to use a similar IP in each GEO, keep the last octet the same and simply alter the subnet portion of the IP according to the definition of each private network in the GEOs. Each inventory file needs to specify the ip address of the installer node. This address must be the same in all inventory files when running the 003 script.

To ease the configuration, we have provided a folder and example files (templates) for each supported hardware type. These can be found under:

• ~/quantum_installer/inventories/3GEO-X200

• ~/quantum_installer/inventories/3GEO-P200

• ~/quantum_installer/inventories/3GEO-Z200

The only files which should be altered without the support of engineering are GEO1.yaml, GEO2.yaml and GEO3.yaml.

Note: Don’t forget to edit all 3 files! Especially things like serial number, etc are key differences across GEOs

The templates have some explanation internally, but for a full overview you can have a look at: 4.0 Build the Inventory. Below we will explain some key differences:

As mentioned above, the main difference in the inventory is the fact that for 3GEO the inventory is actually a folder of files instead of 1 single file. Depending on the scenario defined below, you will have to point the script to either the folder or a specific file.