Debugging StorNext for Object Storage Systems and Cloud Providers

When an error is detected, StorNext generates as much error information as possible to describe the action and the error encountered. Errors may originate internally within StorNext or be generated by one of the StorNext components, including libcurl, SSL, or the HTTP server.

For example, an error in the log file may look like:

... rc=412, error: HTTP status 404: 404 Not Found ..

where, 412 is the unique error formed by StorNext and 404 is an HTTP error.

Or a specific libcurl error may look like:

.. CURLE_OUT_OF_MEMORY ..

For more information, click the links below:

In general, all providers provide the same error status code.

  1. What operation are you doing when the error occurred?
  2. Find the error message in:
    1. /usr/adic/TSM/tac/tac_00
    2. /usr/adic/TSM/trace/trace_04 or trace_05
    3. Look at the extended error message. Not all messages have extended verbose error messages from the server, but most do. These extended messages are very useful. An extended error message looks like:
      .. <Error><Code>SignatureDoesNotMatch</Code> ..
  3. Which component failed?
    1. StorNext? Storage Manager or CVFS, etc.
    2. Is the failure in libcurl?
    3. Is the failure an HTTP error - Server error?

Oct 03 10:37:06.486626 cngam-mdc-1 sntsm fsobjcfg[22238]: E1201(8)<05247>:mdtWAStorage1226: WAStorageRestIF::validateUrlConnectivity: can't access url https://google.com/cngam-c2s1, rc=412, error: HTTP status 404: 404 Not Found^M

for url: https://google.com/cngam-c2s1 Oct 03 10:37:06.487035 cngam-mdc-1 sntsm fsobjcfg[22238]: E1200(7)<05247>:mui5objcfg3606: mui5WAStorageValidateUrlConnectivity: No connectivity to url https://google.com/cngam-c2s1

Oct 03 10:29:30.245375 cngam-mdc-1 sntsm fsobjcfg[16600]: E1201(8)<05194>:mdtWAStorage1226: WAStorageRestIF::validateUrlConnectivity: can't access url https://s3.amazonaws.com/cngam-c2s1, rc=427, error: HTTP status 403: 403 Forbidden^M

for url: https://s3.amazonaws.com/cngam-c2s1

Additional response info: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIAJ4WZ7RFOD5Q7NEDQ</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256 20161003T152930Z 20161003/us-east-1/s3/aws4_request 5cc6438b6ddf7f91058ec6a9daab21062b49b9ff4f32ba7f675dc50fc41fcf8d</StringToSign><SignatureProvided>2d13c2c3a314a207b4907ca71d24e64c8cf92b9cd6fb51de2e113028d5f9c78a</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 36 31 30 30 33 54 31 35 32 39 33 30 5a 0a 32 30 31 36 31 30 30 33 2f 75 73 2d 65 61 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 35 63 63 36 34 33 38 62 36 64 64 66 37 66 39 31 30 Oct 03 10:29:30.245784 cngam-mdc-1 sntsm fsobjcfg[16600]: E1200(7)<05194>:mui5objcfg3606: mui5WAStorageValidateUrlConnectivity: No connectivity to url https://s3.amazonaws.com/cngam-c2s1

Nov 03 12:06:17.918816 cngam-mdc-1 sntsm fsobjcfg[29802]: E1201(8)<05323>:mdtWAStorage1226: WAStorageRestIF::validateUrlConnectivity: can't access url https://s3.amazonaws.com/cngam-c2s1, rc=427, error: HTTP status 403: 403 Forbidden^M

for url: https://s3.amazonaws.com/cngam-c2s1

Additional response info: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeTooSkewed</Code><Message>The difference between the request time and the current time is too large.</Message><RequestTime>20161103T170617Z</RequestTime><ServerTime>2016-10-03T16:21:22Z</ServerTime><MaxAllowedSkewMilliseconds>900000</MaxAllowedSkewMilliseconds><RequestId>4F66316E808CABB0</RequestId><HostId>mdeGOfGuAt30QSwk45eOwvrVVrKsDuZHEhUD4HRDbGKTO0XBz/7EEyYg7NpJ9bogbSDhgmyCZY4=</HostId></Error> Nov 03 12:06:17.919199 cngam-mdc-1 sntsm fsobjcfg[29802]: E1200(7)<05323>:mui5objcfg3606: mui5WAStorageValidateUrlConnectivity: No connectivity to url https://s3.amazonaws.com/cngam-c2s1

Oct 03 13:21:43.908897 cngam-mdc-1 sntsm fs_fmover[21386]: E1200(7)<05525>:mdtWAStorage2098: {1}: initiateWrite: REST_save_object_at_url returned error 427 - HTTP status 403: 403 Forbidden^M

for url: https://s3.amazonaws.com/cngam-c2s1/sm1147BF2AF83D05000000000000000041000000010000JWWN2RZTJG6ESMZCCW

Additional response info: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2594CCC315BB7922</RequestId><HostId>xrwfQYNI6PY5Ekxv9xZqth1xI2AVSFMsFQ8on5Pw04Pc7B2yNLfwrF1qU8Lj4oEPBK3O0Q3h59A=</HostId></Error>289caea90fbe Oct 03 13:21:43.909328 cngam-mdc-1 sntsm fs_fmover[21386]: E1201(8)<05525>:mdm3ioc6052: {1}: mdm3iocSetObjStorErr: Encountered forbidden error with Object Storage media Oct 03 13:21:43.909545 cngam-mdc-1 sntsm fs_fmover[21386]: E1202(9)<05525>:mdm3mov5242: {1}: Object Storage encountered write error media index 4. Status=16, marking media protect(NOT WRITEABLE). Oct 03 13:21:43.912328 cngam-mdc-1 sntsm fs_fmover[21386]: E1200(7)<05525>:mdm3mvr751: {1}: A fatal error: 16 has been encountered, failing all files in request