Quantum
Recovering from Data Cluster Corruption (DRAFT)

Overview

An event such as an unclean shutdown or disk failure can be a possible cause of data corruption. If you suspect data corruption, run a data integrity check. If corruption is detected, follow the procedure in this article to recover the corrupted data.

 

This procedure helps you address a specific corruption condition, but does not cover root cause analysis.

 

The information in this article is Quantum confidential. 

Data Integrity Check

Use cvfsck -nv (cluster fast, table, blob data fast) and grep for "corrupt" as in the example below. The output of this example verification shows no blobs reporting corruption, but some clusters are reporting corruption.

 

$ grep Verify2  messages | grep corrupt

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3193 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3254 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3504 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3195 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3230 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3256 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3210 contains corrupt data.

Feb  5 12:13:02 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3196 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3309 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3188 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3272 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3567 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3371 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3271 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3209 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3170 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3187 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3189 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3497 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3496 contains corrupt data.

Feb  5 12:13:03 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3255 contains corrupt data.

Feb  5 13:07:41 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 1 contains corrupt data.

Feb  5 14:11:54 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3505 contains corrupt data.

Feb  5 14:17:28 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3506 contains corrupt data.

Feb  5 17:47:06 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 90 contains corrupt data.

Feb  5 17:47:27 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 3194 contains corrupt data.

Feb  5 19:35:26 EIPDSQDXI61 Blockpool[22910]: W: [216] (Verify2) Cluster 1120739 contains corrupt data.

 

Recovery Procedure

Step 1

 

Execute the blockpool cluster dump in order to collect the list of blocklets and the flags for each cluster:

 

(If you have several clusters on which you need to run the command, create a working directory for each.)

 

# mkdir –p /scratch/SRnnnn/dumpcluster

 

# cd /scratch/SRnnnn/dumpcluster

 

# /hurricane/blockpool/bin/blockpool dump cluster <cluster#> +Nlocal@localhost 2>&1 | cat > /scratch/SRnnnn/dumpcluster/cluster_<cluster#>.dump

 

Replace <cluster#> with the cluster number you find in the messages. You will execute the blockpool command for each corrupt cluster.

 

Step 2

 

Open each of the dumps and look at the corrupt blocklets listed in each of the cluster dumps. If the ref count of a blocklet marked ‘CO’ is zero, it means that compaction should take care of that specific blocklet.

 

If blocklet has a flag ‘CO’ but shows a value other than zero under the reference count, please upload the cluster dump file to your senior or backline engineer for help with the next steps.

  

An example of checking the blocklets inside a cluster dump:

 

               Serial'd Data      Data       Input

    Ref-Count  Offset Length  Offset Length  Length Flags                   Hash

------------------------------------------------------------------------------------------------------------

H:          0       0      0       0   8000    8000 CO,DL,SI                5EE18B5EE50EE206733E6C26769AC3B6

 

H:          0       0      0       0   4c60    4c60 CO,DL,SN,IC             22537744EBECF60680D082035997ACE4

 

 

Above is an example of a corrupt blocklet (see the ‘CO’ under the column Flags).

 

Notice two things:

 

In cases like the example above, run gc followed by compaction in order to remove the corrupted blocket.

 

For a healthy blocklet we would expect to find something like the following example. Note that the blocklet hash has a header (indicated by ‘H:’) and a body (indicated by ‘B:’)  and no ‘CO’ flag under the column flag.

 

               Serial'd Data      Data       Input

    Ref-Count  Offset Length  Offset Length  Length Flags                   Hash

------------------------------------------------------------------------------------------------------------

H:         21       0   6d2f       4   6d2b    6d2b IC                      D140C0135A9F4A23FCB788EC78769F81

B:                  0   6d2f       4   6d2b                                 D140C0135A9F4A23FCB788EC78769F81

 

 

In cases where you find a corrupted blocklet but with a non-zero ref-count (see the example below), you must proceed with Step 3 of this procedure.

 

               Serial'd Data      Data       Input

    Ref-Count  Offset Length  Offset Length  Length Flags                   Hash

------------------------------------------------------------------------------------------------------------

H:          1       0      0       0   8000    8000 CO,DL,SI                5EE18B5EE50EE206733E6C26769AC3B6

 

H:         12       0      0       0   4c60    4c60 CO,DL,SN,IC             22537744EBECF60680D082035997ACE4

 

 

Step 3

 

Execute this step when you find a corrupt blocklet with a non-zero ref count. This procedure may take a while to run, so it's advised that you run it in the background. For help with nohup and screen, please consult the man pages.

 

Generate the tag/blob list for all the clusters you found showing corrupt blocklets with a non-zero ref count:

 

/hurricane/blockpool/bin/blockpool dump blob-list +C<cluster#>,<cluster#>  +O/scratch/SRnnnn/dumpcluster/blob-list.out +Nlocal@localhost

 

Replace <cluster#> with the cluster number. The example below generates a list of two clusters:

 

/hurricane/blockpool/bin/blockpool dump blob-list +C3193,3154 +O/scratch/SRnnnn/dumpcluster/blob-list.out +Nlocal@localhost

 

After you get the list, use Unix commands to format the list of tags/blobs as follows:

 

FFD206415449A23978718BF5A5574D3E "FFD206415449A23978718BF5A5574D3E"

FFD786C0A192339C4808A4427F4D5983 "FFD786C0A192339C4808A4427F4D5983"

FFDBEE2204B9DB1E8E753F06367DDF62 "FFDBEE2204B9DB1E8E753F06367DDF62"

FFE19B6CF1FC959C62BC13C998B00D53 "FFE19B6CF1FC959C62BC13C998B00D53"

 

When the tag/blob list is formatted as shown above, run a blob verify for that list:

 

/hurricane/blockpool/bin/blockpool verify +B/scratch/SRnnnn/dumpcluster/blob-list.out +Nlocal@localhost > /scratch/SRnnnn/dumpcluster/verifybloblog.lout 2>&1

 

At the end of the blockpool verify, the resulting log will be in the file verifybloblog.out.  Confirm that all verifies executed successfully, then separate the list of blobs that issued errors during the verification process. Take this list into Step 4.

 

Important Note:

If all blob verifications are successful, please consult your senior or backline engineer and escalate if needed. Make sure to provide all the information you generated and the processes you worked so far.

 

Step 4

 

To perform this step, you must have a list of blobs that failed the verify process in Step 3.

 

For those corrupted blobs, you will either recover the blob from a target DXi, OR if there is no replication or good data to be recovered, identify the user file(s) associated to that blob and ask customer either to reingest the file/cartridge or expire/delete the file/cartridge.

 

To replicate the blob use the command:

 

/hurricane/blockpool/bin/blockpool replicate 000230108307F9D4E6BB7095E90639EB +Slocal@localhost +Nremote/SHA-AES:password@<ip-address> 2>&1

 

The command above must be executed on the DXi where you have the good blob stored (target DXi) and replace <ip-address> with the IP of the DXi which has the corrupt blob.

 

If the customer has not done replication and the blob can't be recovered, you will need to identify the files/cartridges associated to that blob using the command bpw_service:

 

/opt/DXi/bpw/bin/bpw_service --tagsearch <blob#> /snfs/ddup

 

Replace <blob#> with the tag/blob from the list.

 

For all commands you can use the Unix bash shell to automate the execution of the procedure if you have more than one item to process and verify.

 

It may be helpful to use nohup and screen to execute commands in the background. For info about nohup and screen, check the man pages.

 

 

 

 


This page was generated by the BrainKeeper Enterprise Wiki, © 2018