Overview

The StorNext File System (SNFS) on the DXi has dedicated space for the metadata of the files on the SNFS. Like regular disk space, the metadata is capable of filling up, which can put the DXi into low space conditions.

 

---

Problem

Here is the warning from the tsunami.log when metadata hits 85% full:

 

WARN   - 11/04/13-06:51:34 - SpaceManagement spacemgr.cpp(811) [spaced] runStateMachine() - In NO SPACE (Stop IO)State: Available disk capacity: 59236 GB

WARN   - 11/04/13-06:51:34 - SpaceManagement spacemgr.cpp(813) [spaced] runStateMachine() - In NO SPACE (Stop IO)State: Available Metadata LUN percentage: 15

 

The Available disk capacity is not the problem (58TB or so free), but the Metadata LUN only has 15% space available, hitting the high water mark and triggering the above messages.

 

---

Cause

If you have root access to the command line on the system, you can check the metadata LUN with the /usr/cvfs/bin/cvadmin tool. It should open to a prompt similar to this (type “quit” to exit the tool):

 

[root@SES6702DXi55 ~]# /usr/cvfs/bin/cvadmin

StorNext Administrator

 

Enter command(s)

For command help, enter "help" or "?".

 

List FSS

 

File System Services (* indicates service is in control of FS):

1>*vol0[0]              located on 127.0.0.1:40910 (pid 26216)

 

Select FSM "vol0"

 

Created           :    Mon Oct 24 15:48:51 2011

Active Connections:    1

Fs Block Size     :    64K

Msg Buffer Size   :    4K

Disk Devices      :    9

Stripe Groups     :    9

Fs Blocks         :    369867260 (22.05 TB)

Fs Blocks Free    :    293117856 (17.47 TB) (79%)

 

snadmin (vol0) >

 

You should automatically be at the above prompt. Just type in "show long" (no quotes) and capture the output. What we are most interested in is how much space is available in the Stripe Groups marked as "MetaData", it should look something like this:

 

snadmin (vol0) > show long

Show stripe groups (File System "vol0")

 

Stripe Group 0 [A0_Metadata]  Status:Up,MetaData,Journal,Exclusive

  Total Blocks:1048431 (63.99 GB)  Reserved:0 (0.00 B) Free:1029673 (62.85 GB) (98%)

  MultiPath Method:Rotate

  Stripe Depth:1  Stripe Breadth:1 blocks (64.00 KB)

  Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0

    Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0

  Disk stripes:

    Primary  Stripe [A0_Metadata]  Read:Enabled  Write:Enabled

      Node 0 [Cvfs_default_1_8]

 

The output above indicates that this system has 98% free space in the Metadata LUN (highlighted in red).

 

What does filling the SNFS MetaData space mean to the customer? Once the metadata space fills, StorNext cannot create new files until files/directories are deleted from the system (SNFS re-uses inodes after the associated files/directories are deleted). So, even if the customer has data disk space available for ingest, the lack of metadata space will keep the system from creating the new files.

 

---

Solution

The only work around of fix is to have the customer remove some of the files. We generally see this problem on the DXi when the customer is not using the appliance as a backup device, but as a NAS for live data, or if the customer is using a backup program that creates extensive temporary files as part of the backup, like "rsync." In those cases, the customer will need to conform to our best business practices for the DXi or change to a Quantum-tested backup product.