Import Object Storage Media

On media imports, the media specified for import is retained in the Storage Manager System and the metadata required to map the contents of an object storage bucket into a StorNext directory tree are created for each file found on the media or defined in the manifest file. The metadata created allows the data to be retrieved through StorNext. A media import is requested by specifying the media import request type with option -t of the fsobjimport command

The file list to be processed for media import can either be obtained from the object storage system or alternatively listed in an object import file.

Options can also be specified to filter out files for processing.

• The –T time option processes only those files created after time time. The time must be specified in YYYY-MM-DD HH:MM:SS format.
• The p prefix option processes only those files beginning with prefix.
• You must specify a destination directory, and it must be an archival directory, managed by the Storage Manager. The path hierarchy which is specified by the key names on the object storage system are maintained in the destination directory unless overridden by directives in an import manifest file. By default, the forward slash (/) is interpreted as a directory separator. If the object storage system uses a distinctive character as a directory delimiter, you can specify this character with the –D option to allow the directory structure to be recreated in the destination directory. For example, if you use the colon (:) as a directory delimiter on the object storage system, object ID xx:yy:zz is created under the destination directory as xx/yy/zz.
• You can specify default file attributes with command options to set the group, file permissions, and user for all files being imported. In the case where default attributes are not specified or are not configurable, they are inherited from the destination directory, class configuration, or truncate exclusion list. If you ingest from a manifest file, the attributes defined for each file override the command options.
• You can tune the media import request by modifying the system parameters, which define the number of meta data threads and database update threads used on the request:
  • The parameter IMP_DBASE_NUM_THREADS defines the number of threads used to update the database.
  • The parameter IMP_META_NUM_THREADS defines the number of threads used to create the file metadata.

You can specify the following options to import object storage media into the Storage Manager:

Examples

• Issue a request to import media AWS0000, creating the truncated file hierarchy under directory /stornext/snfs1/import1, for each object found on the media:

  # fsobjimport -d /stornext/snfs1/import1 -m AWS0000 -t media

• Issue a request to import AWS0000 and to create the truncated files under directory /stornext/snfs1/import1. Import only those files which are prefixed with accounts and replace the object storage delimiter ":"with a forward slash "/" in the destination file path:

  # fsobjimport -d /stornext/snfs1/import1 -m AWS0000 -t media -p accounts -D:

• Issue a request to import only those files on AWS0000 which were created after 7:30 P.M., August 10, 2017. Create the truncated files under directory /stornext/snfs1/import1:

  # fsobjimport -d /stornext/snfs1/import1 -m AWS0000 -t media -T "2017-08-10 19:30:00"

On file imports, the contents of the media are copied into a destination directory and the media can be exported once the import completes. You can request a file import by specifying the files import request type with option -t of the fsobjimport command.

The file list to be processed for media import can either be obtained from the object storage system or alternatively listed in an object import file.

You can also specify options to filter out files for processing:

• The -T time option processes only those files created after time time. You must specify the time in YYYY-MM-DD HH:MM:SS format.
• The -p prefix option processes only those files beginning with prefix.
• You must specify a destination directory. The destination directory may be associated with a relation point, but it is not required. If the destination directory is not under a relation point, the files are ingested but are not managed by Storage Manager.
• The path hierarchy, which you specify by the key names on the object storage system, are maintained in the destination directory unless overridden by directives in an import manifest file. By default, the forward slash ("/") is interpreted as a directory separator. If the object storage system uses a distinctive character as a directory delimiter, you can specify this character with the –D option to allow the directory structure to be recreated in the destination directory. For instance, if you use the colon (":") as a directory delimiter on the object storage system, object ID xx:yy:zz is created under the destination directory as xx/yy/zz.
• You can specify the option -W to request that a file be overwritten if it already exists. If you do not specify the option -W, then a version number is appended to the file name if a unique name is needed.
• You can specify file attributes with command options to set the group, file permissions, and user for all files being imported. If you do not specify the group or user, then they are inherited from the destination directory. If you do not specify the file permissions, then they default to 0440. If you ingest from a manifest file, the attributes defined for each file override the command options.
• You can tune the file import request by modifying the system parameter, IMP_COPY_NUM_THREADS, which defines the number of threads used to copy data. By default, five copy threads are used.

You can specify the following options to import the contents of object storage media:

Example

• Issue a request to import files from media AWS0000, copying the files to directory /stornext/snfs1/import2. Create the files with an owner ID of 101, a group ID of 120, and with file permissions of 0660. Overwrite the file if it already exists in the destination directory:

  # fsobjimport -d /stornext/snfs1/import2 -m AWS0000 -t files -u 101 -g 120 -M 0660 -W

You can use a manifest file to identify a list of files to import and their associated metadata. The file must be in JSON format and you can create it using the -t list option of the fsobjimport command. You must format the file as follows:

{
    "media": "STS0001",
    "file_list": [
         {
               "objectId": "0000BBF3-3A01-46CD-9476-BFA0EAAFF676",
               "size": 100,
               "time": 1496387370,
               "mode": 511,
               "gid": 0,
               "uid": 0,
               "destPath": ""
         },
         {
               "objectId": "0001996D-BBDF-4F7E-A2D3-DA7A7FEE74A5",
               "size": 100,
               "time": 1496385290,
               "mode": 511,
               "gid": 0,
               "uid": 0,
               "destPath": ""
         }
    ]
}

• You can use destPath to define a path relative to the destination directory (-d) which differs from the object ID returned by the object storage system. If you specify a delimiter with option -D, each delimiter in objectId is replaced with a forward slash ("/"’) in the destination path for each file. By default, -t list does not set this value and, when you use this file for import, the object ID is used for the file name.
• By default, the mode, gid, and uid are set to the option values used with the fsobjimport -t list command or the destination directory attributes if the attribute options are not specified.
• Options can be specified to filter out files included in the manifest file.
  • The -T time option processes only those files created after time time. You must specify the time in YYYY-MM-DD HH:MM:SS format.
  • The -p prefix option processes only those files beginning with prefix.

You can specify the following options to create a manifest file for a specified media:

Example of How to Create a Manifest File

• Query media AWS0000 for a list of the files on the media. Write this information to file /manifest/aws-filelist. Use options -u, -g and -M to specify the default user ID, group ID, and mode for each file entry. Replace the object storage delimiter ":"with a forward slash ("/") in the destination path for each file:

  # fsobjimport-m AWS0000 -t list -f /manifest/aws-filelist -u 101 -g 120 -M 0660 -D:

Example of How to Import from a Manifest File

• Modify the contents of the manifest file created with option -t list, modifying, if necessary, the attributes and destination path to values specific for each file. Issue a media import request, generating a file list from the contents of file /manifest/aws-filelist and creating the truncated files under directory /stornext/snfs1/import3:

  # fsobjimport -d /stornext/snfs1/import3 -m AWS0000 -t media -f /manifest/aws-filelist

Checkpoint information is maintained for an fsobjimport request. If you suspend an import or it terminates abnormally, you can use this information to resume the import.

• To suspend an import, specify the -S option with either the -m option or the -i option to identify the import by media ID or request ID. Use the -s option of the fsobjimport command to display the media ID and request ID of each pending imports.
• To resume an import, specify the -R option with the -m option to identify the import. The import request restarts from the point of failure.
• If you do not resume or undo an import, you can clear the checkpoint information for the import by specifying the -C option along with the -m option to identify the import. If you specify -C without the -m option, then the import information is cleared for all imports.

Example of How to Suspend an Import

To suspend an import of media AWS0000:

Example of How to Resume an Import

Issue a request to resume a suspended import of media AWS0000:

You can undo a completed import request by specifying the -U option with the previously specified fsobjimport request, specifying the same media, destination, import type and filtering options. If the import request terminated before completion and the import request information has not been cleared from the database, you can request an undo of this import by specifying the -U option with the -m option to identify the import. Once you request an undo, then the import request is no longer recoverable.

An undo of an import removes each imported file from the destination directory. If you undo a media import, then the file information is removed from the filecomp, fileinfo and mediadir database tables. Any directories created as part of the import are also removed.

Note: If you manually remove the files after a media import, then Storage Manager removes these files from the media if a clean is requested using the fsclean command.

You can tune the file import request by modifying system parameter, IMP_REMOVE_NUM_THREADS, which defines the number of threads used to remove the imported files or metadata. By default, five threads are used.

Example of How to Undo a Completed Import

• To undo the following media import request:

  # fsobjimport -d /stornext/snfs1/import5 -m AWS0000 -t media -p accounts -D:

• Specify the same import request options with -U:

  # fsobjimport -d /stornext/snfs1/import5 -m AWS0000 -t media -p accounts -D: -U

Example of How to Undo a Suspended Import

• To undo a suspended import of media AWS0000:

  # fsobjimport -m AWS0000 -U

You can use the –s option of the fsobjimport command to display the status of pending import requests. You can display the status of all pending imports, or use the –m option to display the status of a specific import request. You can specify the -F option to set the output format to TEXT, XML or JSON.

The short text report displays as follows:

  Media    Type      Start        Time     Status    State    Complete   Total
--------- ------- ------------ ---------- --------- -------- ---------- --------
 STS0001   media   Mon Jul  3   16:53:40   suspend   meta        93160   856138
 STS0002   files   Mon Jul  3   18:20:04   active    ingest      24050   230157

The default short report produces the following information for each pending import:

If you specify -l, then a long report on pending or failed imports is displayed. The import information is output as follows for the text format:

The following information is displayed:

If an import fails for a file, the error is logged, and processing continues with the next queued file. If consecutive file failures occur, the fsobjimport command is terminated. Checkpoint entries for the failed files remain in the database and are used to resume an import if requested.

Error information for file failures are logged to the file:

Each log entry identifies the time of the error, the media, request ID of the failed fsobjimport, the object ID of the file, the error message and the errno terminating the file processing. Each message is formatted as follows: