Combined Scale Out Networking for 3-GEO Deployments

This guide documents the recommended network configuration (Switch layout, cabling, VLANs, IP Subnets) for 3-GEO X200 and P200 systems and for scaling out by adding 3-GEO X200 or P200 converged columns to an existing site.
This guide uses the term cluster for the set of ActiveScale nodes that are connected to the same pair of ToR switches. A cluster can be up to 18 twin nodes, spread over 1 to 6 columns (in multiples of 3 per column).

As a refresher, the internal networks for X100/P100 and X200/P200 in the 172.16.0.0/12 block are show in the table below. ActiveScale X100 and P100 can scale up to 9 racks. For X100 this is 3 columns 3 GEO, for P100 this is 9 columns 3 GEO.
For the column number, all means that the same VLAN and subnet is used for all racks/columns in the GEO (X100 and P100), 1 is for non-scaleout deployment, n is for a scaled out deployment.

ActiveScale X100/P100

GEO

Column (P100: 1 <= n <= 9,

X100: 1 <= n <= 3)

VLAN IP subnet

Site 1

Scaleout Network 1_1

all

110

172.16.110.0/24

Storage Network 1_1

1

111

172.16.111.0/24

n

11n

172.16.11n.0/24

Scaleout Network 2_1

all

210

172.16.210.0/24

Storage Network 2_1

1

211

172.16.211.0/24

n

21n

172.16.21n.0/24

Site 2

Scaleout Network 1_2

all

120

172.16.120.0/24

Storage Network 1_2

1

121

172.16.121.0/24

n

12n

172.16.12n.0/24

Scaleout Network 2_2

all

220

172.16.220.0/24

Storage Network 2_2

1

221

172.16.221.0/24

n

22n

172.16.22n.0/24

Site 3

Scaleout Network 1_3

all

130

172.16.130.0/24

Storage Network 1_3

1

131

172.16.131.0/24

 

n

13n

172..16.13n.0/24

Scaleout Network 2_3

all

230

172.16.230.0/24

Storage Network 2_3

1

231

172.16.231.0/24

n

23n

172.16.23n.0/24

ActiveScale X200/P200

GEO

Converged Column (1 <= n <= 128)

VLAN (0 <= N <= 127)

IP subnet

Site 1

Private Network 1_1

1

1000

172.21.0.0/24

n

1000 + N

172.21.N.0/24

Private Network 2_1

1

1128

172.21.128.0/24

n

1128 + N

172.21.N+128.0/24

Site 2

Private Network 1_2

1

2000

172.22.1.0/24

n

2000 + N

172.22.N.0/24

Private Network 2_2

1

2128

172.22.128.0/24

n

2128 + N

172.22.N+128.0/24

Site 3

Private Network 1_3

1

3000

172.23.1.0/24

n

3000 + N

172.23.N.0/24

Private Network 2_3

1

3128

172.23.128.0/24

n

3128 + N

172.23.N+128.0/24

The reference customer network uses addresses in a 10.0.0.0/8 range. We consider each customer site to be a /16 network, which is further subdivided into smaller subnets of which the smallest are /24 networks. Each /16 network can use a (set of) private Autonomous System(s) for BGP between sites/ within sites, but the customer’s internal routing strategy (BGP, OSPF, IS-IS) is not very important for ActiveScale. This document presents the customer network as static routes.

Site

Autonomous System (AS) number

Network

Router/Gateway Towards Other Sites

Customer Site 1

65110

10.11.0.0/16

10.11.0.1

Customer Site 2

65120

10.12.0.0/16

10.12.0.1

Customer Site 3

65130

10.13.0.0/16

10.13.0.1

Interconnections between customer sites are considered to be routed over direct point-to-point WAN links organized in /31 networks (/31 networks have two host addresses and no network or broadcast address). This allows for 128 such links in the 10.10.32.0/24 range. Each site is directly connected to the other sites, WAN Link 1 is on 10.10.32.0/31, WAN Link 2 is on 10.10.32.2/31 and WAN Link 3 is 10.10.32.4/31, with redundant paths (with higher weight) in case one of the WAN links goes down. The terminology “router” is used in the document, but this could be a beefy switch with Layer 3 capability.

In 3-GEO configuration, a single ActiveScale column (erasure encoding domain) is spread over 3 distinct locations to improve reliability and availability in case one of the sites or WAN links goes down. The private networks need to be routed across GEOs. Each scaler node needs to be able to route to the scale-out networks (or Private Network in case of a combined system) at the other 2 customer sites. In addition, for X100 and P100, each column node needs to be able to route to the corresponding storage network in the other GEOs.

For the public networks, the customer can choose any VLANs and subnetworks that are available to him at each GEO, there is no additional requirement when compared to a single GEO deployment. We will focus on the private/internal networks in this guide.

The reference strategy for interconnecting sites for 3-GEO X100 and P100 systems was to integrate each site into the customer network by extending the internal VLANs all the way to the customer’s router. The router is assigned IP address .241 on each VLAN. From that router, the traffic is routed to the other locations (GEOs). Recall from the table above that each X100 / P100 site has 2 scaleout networks (VLANs 110 and 210) and 2 storage networks per column, so the number of VLANs extended into the customer network at each site for n columns is 2 + 2 * n. For our maximum P100 offering of 3 racks / GEO (9 columns), this is 20 VLANs. These VLANs should be chosen such that they do not conflict with VLANs in use at the customer site. In addition, extending the VLANs means that all traffic towards the router originates from either from 1 of the ToR switches or uses NIC bonded links if we stack the ToR switches and the customer’s aggreggation switch supports this.

The router(s) at each site will also have to add routing entries for all these (up to 20) networks, which is up to 60 entries in the customer router(s) at each site: 20 outgoing entries each for the two remote GEOs, and 20 local VLAN routes for incoming packets from the remote GEOs. Redundant routes of the WAN links may add up to 40 entries more (shown with a link weight of 20 in the figure). This guide draws the configuration for X100 (max 3 columns), but note that the IP addressing is shown for up to 9 columns, which is possible for P100. This baseline configuration is the starting point for 3-GEO scaleout of X100 with X200 (and P100 with P200). From here on, only 1 WAN link is drawn in the diagrams, but the interconnection should be interpreted as in the reference figure above.

The configuration (for site 1) is also summarized in the table below. The ActiveScale nodes are able to reach the private networks in the other GEOs via a route with the .241 gateway IPs as the next hop. Inside the GEO, traffic stays within the layer 2 broadcast domain. The customer’s router’s routing table then has entries for all the IP networks towards the other GEOs, and VLAN entries for incoming traffic towards the directly connected local VLANs (blue).

Site

Network

Column Node Routing Table ToR Routing Table

Customer Router Routing Table

Site 1

scaleout network 1

all

172.16.120.0/24 via 172.16.110.241

172.16.130.0/24 via 172.16.110.241

-

172.16.120.0/24 via 10.10.32.1

172.16.130.0/24 via 10.10.32.1

172.16.110.0/24 vlan 110

scaleout network 2

all

172.16.220.0/24 via 172.16.210.241

172.16.230.0/24 via 172.16.210.241

-

172.16.220.0/24 via 10.10.32.1

172.16.230.0/24 via 10.10.32.1

172.16.210.0/24 vlan 210

storage network 1

1

172.16.121.0/24 via 172.16.111.241

172.16.131.0/24 via 172.16.111.241

-

172.16.121.0/24 via 10.10.32.1

172.16.131.0/24 via 10.10.32.1

172.16.111.0/24 vlan 111

n

172.16.12n.0/24 via 172.16.11n.241

172.16.13n.0/24 via 172.16.11n.241

-

172.16.12[1-9].0/24 via 10.10.32.1

172.16.13[1-9].0/24 via 10.10.32.1

172.16.11[1-9].0/24 vlan 11[1-9]

storage network 2

1

172.16.221.0/24 via 172.16.121.241

172.16.231.0/24 via 172.16.121.241

-

172.16.221.0/24 via 10.10.32.1

172.16.231.0/24 via 10.10.32.1

172.16.211.0/24 vlan 211

n

172.16.22n.0/24 via 172.16.121.241

172.16.23n.0/24 via 172.16.121.241

-

172.16.22[1-9].0/24 via 10.10.32.1

172.16.23[1-9].0/24 via 10.10.32.1

172.16.21[1-9].0/24 vlan 21[1-9]

In 3 GEO configuration, there will be at most 6 twin servers (+ 6 4u JBODS) per column per GEO (so 18 twins in a column). The general playbook for integrating ActiveScale X200 is to let the deployments live in their own “natural” network in the default private address ranges in the 172.16.0.0/12 private address block, and route the internal traffic over the customer network between each GEO. The objectives for 3-GEO X200 scale out networking are to

  • Aggregate addresses to keep customer routing tables lean

    • Additionally allow assigning internal AS numbers to ActiveScale and use BGP to disseminate routes between ActiveScale sites.

  • Reduce the number of VLANs to be added in the customer network

  • Allow for multiple switches to connect towards the (border) router for increased bandwidth

  • Keep the traffic separate as much as possible for HA

The X100 approach for extending VLANs inside the customer network becomes intractible with X200 scaling out to (in theory) over 100 columns per site. For X200, the recommendation is to reserve 2 /24 subnets (and 2 VLANs) for connecting the ActiveScale ToR switches into the network to the main (border) router(s) for each site. These VLANs do not need to be extended (trunked) between sites. As such the traffic is routed independently to provide some resiliency to internal network outages at a customer site.

The VLANs should be chosen to be outside the ranges used by ActiveScale internally, the recommendation is to re-use VLANs 4011 and 4021 at each site. At the connection point within the customer router, we use the .254 address as shown in the table below.
With at most 6 twins per converged column per GEO, a ToR switch pair can support at least 3 columns in the GEO. Reserving a single /24 network in the customer site allows enough room to assign an IP address to each of the ToR 1 switches needed for 128 X200/P200 converged columns (and up to 3 ToR 1 switches for X100 or P100 racks). This document uses the 10.1X.110.0/24 and 10.1X.210.0/24 in a nod to ActiveScale X100 internal network convention, but these 2 /24 subnets in the customer network can be chosen freely by the customer without impact on the configuration complexity of the network. The routing convention allows for a uniform look on the network from a customer perspective at each site.

Site

ActiveScale Network Switches

VLAN towards customer IP address range in customer network

Customer Aggregation point IP address at customer router

   

 

 

  

Site 1

ToR 1’s

4011

10.11.110.0/24

10.11.110.254

ToR 2’s

4021

10.11.210.0/24

10.11.210.254

Site 2

ToR 1’s

4011

10.12.110.0/24

10.12.110.254

ToR 2’s

4021

10.12.210.0/24

10.12.210.254

Site 3

ToR 1’s

4011

10.13.110.0/24

10.13.110.254

ToR 2’s

4021

10.13.210.0/24

10.13.210.254

The routing convention above allows for a uniform network integration of Activescale at each site from a customer perspective.

A single column 3-GEO X200 consists of 3 or 6 twin servers per site in a 3-GEO configuration. Only the shortest routes are shown in the figure and tables to avoid too much clutter.

The X200 network is fully IP routed, no internal VLANs are extended into the customer network. Each router has at most 6 entries for ActiveScale X200 (regardless of scale out, see below). The complete routing tables for this deployment are shown below, additional routes in the nodes for future scale-out are not really required (but set by default at install) and shown in grey.

Site

Network

Node Routing Table

ToR Routing Table

Customer Router Routing Table

Site 1

private network 1

172.22.0.0/17 via 172.21.0.241

172.23.0.0/17 via 172.21.0.241

172.21.0.0/17 via 172.21.0.241

172.22.0.0/17 via 10.11.110.254

172.23.0.0/17 via 10.11.110.254

172.21.0.0/17 vlan 1000

172.22.0.0/17 via 10.10.32.1

172.23.0.0/17 via 10.10.32.5

172.21.0.0/17 via 10.11.110.1

+ redundant routes

private network 2

172.22.128.0/17 via 172.21.128.241

172.23.128.0/17 via 172.21.128.241

172.21.128.0/17 via 172.21.128.241

172.22.128.0/17 via 10.11.210.254

172.23.128.0/17 via 10.11.210.254

172.21.128.0/17 vlan 1128

172.22.128.0/17 via 10.10.32.1

172.23.128.0/17 via 10.10.32.5

172.21.128.0/17 via 10.11.210.1

+ redundant routes

Site 2

private network 1

172.21.0.0/17 via 172.22.0.241

172.23.0.0/17 via 172.22.0.241

172.22.0.0/17 via 172.22.0.241

172.21.0.0/17 via 10.12.110.254

172.23.0.0/17 via 10.12.110.254

172.22.0.0/17 vlan 2000

172.21.0.0/17 via 10.10.32.0

172.23.0.0/17 via 10.10.32.3

172.22.0.0/17 via 10.12.110.1

+ redundant routes

private network 2

172.21.128.0/17 via 172.22.128.241

172.23.128.0/17 via 172.22.128.241

172.22.128.0/17 via 172.22.128.241

172.22.128.0/17 via 10.12.210.254

172.23.128.0/17 via 10.12.210.254

172.21.128.0/17 vlan 2128

172.21.128.0/17 via 10.10.32.0

172.23.128.0/17 via 10.10.32.3

172.21.128.0/17 via 10.12.210.1

+ redundant routes

Site 3

private network 1

172.21.0.0/17 via 172.23.0.241

172.22.0.0/17 via 172.23.0.241

172.23.0.0/17 via 172.23.0.241

172.21.0.0/17 via 10.13.110.254

172.22.0.0/17 via 10.13.110.254

172.23.0.0/17 vlan 3000

172.21.0.0/17 via 10.10.32.4

172.22.0.0/17 via 10.10.32.2

172.23.0.0/17 via 10.13.110.1

+ redundant routes

private network 2

172.21.128.0/17 via 172.23.128.241

172.22.128.0/17 via 172.23.128.241

172.23.128.0/17 via 172.23.128.241

172.21.128.0/17 via 10.13.210.254

172.22.128.0/17 via 10.13.210.254

172.23.128.0/17 vlan 3128

172.21.128.0/17 via 10.10.32.4

172.22.128.0/17 via 10.10.32.2

172.23.128.0/17 via 10.13.210.1

+ redundant routes

The network layout for 3-GEO X200 at Site 1 with up to 3 ToR switches is shown below.

The customer can connect multiple ActiveScale ToR 1 switches to a VLAN (4011) which is extended to the (border) router, and similarly connects multiple ToR 2 switches to another VLAN (4012), also extended to the border router. The aggregation switches here are optional, the VLANs could be directly connected from ActiveScale ToR switches to the main router. The example has two of the switches connected, and the traffic is divided over the 2 ToRs, for instance using 2 /22 IP routes (more optimal routes can be added). Once arrived at a ToR switch, traffic is routed as in a single GEO deployment between the columns.

Site

Network

Column

Node Routing Table ToR Routing Table

Customer Router Routing Table

Site 1

private network 1

n

172.22.0.0/17 via 172.21.n.241

172.23.0.0/17 via 172.21.n.241

172.21.0.0/17 via 172.21.n.241

172.22.0.0/17 via 10.11.110.254

172.23.0.0/17 via 10.11.110.254

+ trunk routes
Scaling out X200 with X200 - 3 clusters

172.22.0.0/16 via 10.10.32.1

172.23.0.0/16 via 10.10.32.1

172.21.0.0/22 via 10.11.110.1

172.21.128.0/22 via 10.11.210.1


172.21.4.0/22 via 10.11.110.2
172.21.132.0/22 via 10.11.210.2

private network 2

n

172.22.128.0/17 via 172.21.n.241

172.23.128.0/17 via 172.21.n.241

172.21.0.0/17 via 172.21.n.241

172.22.128.0/17 via 10.11.210.254

172.23.128.0/17 via 10.11.210.254

+ trunk routes
Scaling out X200 with X200 - 3 clusters

When scaling out beyond the capability of 3 ToR switch pairs, a leaf-spine architecture is recommended. In 3-GEO, the spine switches can be integrated into the customer network, so traffic that was carried over dedicated direct links between our ToR switches (which may form a bottleneck) will now all be sent over the spine switches. The spine switches, as the aggregation switches in the setups before, do not need IP routing entries. With such large deployments, it is likely that the border routers will be redundant. Many options are possible, a routing example is shown below where we keep the VLANs traffic flows separated over the routers, but options where there is a full mesh between the aggregation switches can add additional robustness (at the cost of used switch ports). The ToR switches get assigned address .241 on the internal subnets for the private networks for inter-GEO routing in addition to the .200 address for management.

Site

Network

Column

Node routing table

ToR Routing table

Customer Router Routing Table

 

    

 

    

Site 1

private network 1

all

172.22.0.0/17 via 172.21.n.241

172.23.0.0/17 via 172.21.n.241

172.22.0.0/17 via 10.11.110.254

172.23.0.0/17 via 10.11.110.254

[+ internal GEO routes via 10.11.110.x]

172.22.0.0/16 via 10.10.32.1

172.23.0.0/16 via 10.10.32.1

172.21.0.0/22 via 10.11.110.1

172.21.128.0/22 via 10.11.210.1


172.21.4.0/22 via 10.11.110.2
172.21.132.0/22 via 10.11.210.2

private network 2

all

172.22.128.0/17 via 172.21.n.241

172.23.128.0/17 via 172.21.n.241

172.22.128.0/17 via 172.16.210.241

172.23.128.0/17 via 172.16.210.241

[+ internal GEO routes via 10.11.210.x]

There are a lot of options for scaling out 3-GEO X100, and it looks like the deployments in the field use customer-tailored IP addressing schemes, so this will usually require customized configuration.

The overall recommendations to connect additional X200 converged columns to an existing X100 in each GEO are:

  • Leave all X100 hardware and cabling untouched.

  • Connect the X200 via its 2 ToR switches to the customer router as per the normal operation for X200 3-GEO on VLANs 4011 and 4012.

  • Connect the X200 ToR switch pair to the X100 ToR switch pair as per normal scale-out

  • Add routing entries for the X100 deployment in the X200 ToR switches, and for incoming routes to VLANs 110-119.

  • Add routing entries for the X200 deployment in the X100 ToR switches

The most important routing entries are shown in the Table below. We assume the following trunk between the X100 ToR and X200 ToR switches:

Trunk VLAN IP Network IP Address X100 Switch

IP Address X200 Col 1 ToR

IP Address X200 Col 2 ToR

10

172.16.10.0/31

172.16.10.0

172.16.10.1

 

11

172.16.10.2/31

172.16.10.3

 

172.16.10.4

20

172.16.20.0/31

172.16.20.0

172.16.20.1

 

21

172.16.20.2/31

172.16.20.3

 

172.16.20.4

 

Site Network

Column

Node Routing Table ToR Routing Table

Customer Router Routing Table

Site 1

scaleout network 1

all X100 columns

172.16.120.0/24 via 172.16.110.241

172.16.130.0/24 via 172.16.110.241

172.21.0.0/17 via 172.16.110.200

172.22.0.0/17 via 172.16.110.241

172.23.0.0/17 via 172.16.110.241

172.16.110.0/24 vlan 110

172.21.0.0/17 via 172.16.10.2

172.16.120.0/24 via 10.10.32.1

172.16.130.0/24 via 10.10.32.1

172.16.110.0/24 vlan 110

scaleout network 2

all X100 columns

172.16.220.0/24 via 172.16.210.241

172.16.230.0/24 via 172.16.210.241

172.21.128.0/17 via 172.16.210.200

172.22.128.0/17 via 172.16.210.241

172.23.128.0/17 via 172.16.210.241

172.16.210.0/24 vlan 210

172.21.128.0/17 via 172.16.20.2

172.16.220.0/24 via 10.10.32.1

172.16.230.0/24 via 10.10.32.1

172.16.210.0/24 vlan 210

Storage network 1

X100 column (1 <= n <= 9)

172.16.12n.0/24 via 172.16.11n.241

172.16.13n.0/24 via 172.16.11n.241

-

172.16.12[1-9].0/24 via 10.10.32.1

172.16.13[1-9].0/24 via 10.10.32.1

172.16.11[1-9].0/24 vlan 11[1-9]

Storage network 2

X100 column (1 <= n <= 9)

172.16.22n.0/24 via 172.16.21n.241

172.16.23n.0/24 via 172.16.21n.241

-

172.16.22[1-9].0/24 via 10.10.32.1

172.16.23[1-9].0/24 via 10.10.32.1

172.16.21[1-9].0/24 vlan 21[1-9]

private network 1

all X200 columns (0 <= n < 128)

172.22.0.0/17 via 172.21.n.241
172.23.0.0/17 via 172.21.n.241

172.16.0.0/16 via 172.21.n.241

172.22.0.0/17 via 10.11.110.254

172.23.0.0/17 via 10.11.110.254

172.16.0.0/16 via 172.16.10.1 (first X200)

172.16.0.0/16 via 10.11.110.2 (other X200’s)

[+ internal GEO routes via 10.11.110.x]

172.22.0.0/16 via 10.10.32.1

172.23.0.0/16 via 10.10.32.1

172.21.0.0/22 via 10.11.110.1

172.21.128.0/22 via 10.11.210.1


172.21.4.0/22 via 10.11.110.2
172.21.132.0/22 via 10.11.210.2

private network 2

all X200 columns (0 <= n < 128)

172.22.128.0/17 via 172.21.128+n.241

172.23.128.0/17 via 172.21.128+n.241
172.16.0.0/16 via 172.21.128_n.241

172.22.128.0/17 via 10.11.210.254

172.23.128.0/17 via 10.11.210.254

172.16.0.0/16 via 172.16.20.1 (first X200)
172.16.0.0/16 via 10.11.210.2 (other X200s)

[+ internal GEO routes via 10.11.210.x]