Scanning and Reading Files from StorNext Tapes

There are times when you may want to find out what files are on a StorNext tape and perhaps read them back independently of StorNext. Either on a StorNext system where the media has been cleaned (ie fsclean), on a different StorNext system or on a non-StorNext system altogether. As most StorNext Customer use the ANTF format it’s not possible to use generic Unix tools to do this.

The fsmedscan and fsmedread utilities detailed below can be used on non-StorNext systems provided binaries compatible with the OS can be found. Most (if not all) versions of the utilities should be able to read StorNext tapes regardless of the version they were written with as the tape format shouldn’t change.

The techniques detailed below can be adapted according to the circumstances you find yourself in to investigate a variety of StorNext media and database issues. Sometimes all you need to do is confirm that a piece of media is a StorNext tape and hasn’t been overwritten by something else.

For detailed information of the utilities used please refer to their man pages.

Note that these methods won’t work for LTFS, there’s a separate Qwiki article on how to independently mount and read LTFS tapes.

ANTF = ADIC Native Tape Format

LTFS = Linear Tape File System

Scanning a StorNext Tape

The StorNext utility “fsmedscan” is be used to scan a tape to determine its contents and provide the information necessary to read the files back. In the following example a StorNext system is used, for non-StorNext systems you’ll need to mount the tape using whatever tools are available. (EG mtx or manually).

Use fsmount to direct MSM to mount the tape into a drive. Using fsmount ensures that StorNext knows that both the tape and drive are in use and avoids any errors that would be caused if something else tries to use them. In the following example tape ID 000020 is used :

# fsmount 000020

FS0589 08 0001246169 fsmount interim: Tertiary Manager software request received.

FS0670 08 0001246169 fsmount interim: Media mounted on device /dev/sg31.

FS0000 08 0001246169 fsmount completed: Command Successful.

Using the device ID returned from the fsmount command, scan the tape to a file :

# fsmedscan /dev/sg31 > 000020.fsmedscan

To confirm this is a StorNext ANTF tape take a look at the first 2 lines of the file, it should show you, amongst other things, the media ID and generation number as stored in the MEDIADIR database table and in fsfileinfo of a file (for more details see “More on MediaIDs and Generation Numbers” below) :

# head -2 fsms_000020.fsmedscan

devtype:LTO(14) medtype:LTO(17) fd=0

volId:' 12', impId:'ADIC000000002', ioSize=524288, mediaGen=34

The information following the volume header on an ANTF tape will show a wealth of information regarding the files stored and their position on the tape. Use the file information to locate the file(s) you need to read back.

If the tape is not an ANTF tape you could try dumping the first block from the tape (using dd or fs_scsi) to see if it contains any clues as to why it’s not the expected format. (Use “strings” or ”od –c” to read the binary file). Sometimes in mixed usage libraries or in environments where the tape devices are accessed outside of StorNext you might find the tape was overwritten by other backup software or utilities, EG Networker, tar, fbackup. In such cases you’ll have to tell the Cutomer the StorNext data is lost and provide the header information to help them find the overwriting culprit.

Reading Files From The Tape

The scan of the tape will list all the files stored on the tape regardless of their current status in StorNext. Inactive files cleaned from the database won’t be accessible through StorNext but will still appear in the scan. Note however that the files are listed with the path and filename of the file when it was stored, if the file was moved or renamed after storing this won’t be reflected in the scan. All we have is the information from the tape.

The fsmedscan information includes the position of the file on the tape which can be provided to fsmedread to read the file back. In the example below we’re going to retrieve a file that was stored on 10-Mar-2014 that was called /stornext/PCG_managed/policy1/test. In the fsmedscan output it appears as :

DIR ENTRY 2, offset 480

. path len:34 name:'/stornext/PCG_managed/policy1/test'

. own:grp 'root':'root'

. mode(oct) 664

. migrTime 1394458008 - Mon Mar 10 13:26:48 2014

. modTime 1387277193 - Tue Dec 17 10:46:33 2013

. oneup 19

. file Size:8

. segment Num:1 Version:1 Offset:0 Size:8 Tot:1

. position fsn:33 cdbn:338305 ldbn:136171995

. copyId 1

. fcompRec key: 19 path: /stornext/PCG_managed/policy1/test insert into filecompXX values(136171995,338305,8,19,NDX,1,1387277193,1394458008,ETIME,33,0,0,33,1,1,1,hex('11'),' ');

We use the position information and the device where the tape is loaded to bring the file back and store it to a new filename :

fsmedread -f 33 -c 338305 -l 136171995 -d /dev/sg31 /tmp/test.restored

fsn = file sequence number

cdbn = cumulative data block number

ldbn = logical data block number

After recovering all the files needed from the tape be sure to unmount it so that Stornext can continue using both the media and drive in it’s normal operation:

# fsdismount -m 000020

More on MediaIDs and Generation Numbers

Each piece of StorNext media is identified by a unique media ID associated with it’s physical label (the barcode). This media ID is the key to tracking file locations for restoration. The barcode is shown in the fsfileinfo output while the media ID and it’s generation number are shown in the dm_info output and all recorded in the filecomp and mediadir database tables. In the following example we use a different file to the previous examples as the purpose of fsmedscan and fsmedread is to read/recover files that are no longer on the filesystem. This example is for a file called testfile.txt that is on the filesystem and is stored on the media with barcode 000020. The mediaid for this barcode is 12 and media generation number is 34. We use the “key” given in the dm_info output to find the file in the filecomp database.

fsfileinfo :

# fsfileinfo testfile.txt

-------------------------------------------------------------------------------

File Information Report Fri Feb 24 14:12:22 2017

Filename: /stornext/managed/onecopy/pcg/qwiki/testfile.txt

Stored Name: /stornext/managed/onecopy/pcg/qwiki/testfile.txt

-------------------------------------------------------------------------------

Last Modification: 25-jun-2015 08:54:28

Owner: root Location: DISK AND TAPE

Group: root Existing Copies: 1

Access: 644 Target Copies: 1

Target Stub: 0 (KB) Existing Stub: n/a

File size: 31,161,699 Store: MINTIME

Affinity: n/a Reloc: MINTIME

Class: one_tape_copy Trunc: MINTIME

Clean DB Info: NO

Media: 000020(1)

Checksum: Y

Object Ids: N

FS0000 24 0000125828 fsfileinfo completed: Command Successful.

dm_info :

# dm_info testfile.txt

Filename: testfile.txt

handle (hex): 00051856ebe280c9000e00000000004a000000000c09c58a

fsid: 0x00051856ebe280c9 dev: 31 rdev: 31 aff: n/a

size: 31161699 block size: 4096 number of blocks: 60864

ino: 201966986 gen: 74 type: S_IFREG mode: 0100644 lnks: 1

extended attributes: attrname: SNEA attr_ver: 6.82 key: 3492922

class: 2 oneup: 3492922 vsn: 001 totvers: 1 totseg: 1

media: ndx1,gen1: 12,34 ndx2,gen2: 0,0

ldbn: 0 slen: 31161699 fsn: 49

flags: 0x2 events: 0x160000 cpymap: 0x1 stub size,len: -1,0

flags: ALL_COPIES_MADE

objids: NONE

eventlist: 0x00160000 WRITE TRUNCATE DESTROY

region events: WRITE TRUNCATE

atime = 1461247060 -> Thu Apr 21 14:57:40 2016

ctime = 1487692070 -> Tue Feb 21 15:47:50 2017

mtime = 1435218868 -> Thu Jun 25 08:54:28 2015

filecomp :

# mysql -e "select File_key, Mediandx, Medgen from filecomp1 where file_key=3492922;" tmdb

+----------+----------+--------+

| File_key | Mediandx | Medgen |

+----------+----------+--------+

| 3492922 | 12 | 34 |

+----------+----------+--------+

With the media identified from the file perpective we can check that the current media generation number matches that in the mediadir table :

mediadir

# mysql -e "select MediaID, Mediandx, VolID, Mediagen from mediadir where mediandx=12;" tmdb

+---------+----------+--------+----------+

+---------+----------+--------+----------+

| 000020 | 12 | 12 | 34 |

+---------+----------+--------+----------+

And as we did before, check the media header with fsmedscan :

fsmedscan

# fsmount 000020

FS0589 24 0000125838 fsmount interim: Tertiary Manager software request received.

FS0670 24 0000125838 fsmount interim: Media mounted on device /dev/sg34.

FS0000 24 0000125838 fsmount completed: Command Successful.

# fsmedscan /dev/sg34 | head

devtype:LTO(14) medtype:LTO(17) fd=0

volId:' 12', impId:'ADIC000000002', ioSize=524288, mediaGen=34

If at any point the media is missing or the media generation number doesn’t match what the file is expecting then we’ll have a problem !

Tape mounts will also fail with “Invalid label data” if the header on the tape loaded doesn’t show the VolID and MediaGen of the tape that StorNext requested. (Corrupted or overwritten tape header, wrong drive, wrong media but same barcode etc.). Note that the Mediandx and VolID do not have to be the same.