Using the smartcp Command

Overview

The smartcp command can be used to directly replicate how we move data within a customer environment. Think of the Linux DD command as a "raw" approach to moving data. Smartcp employs a lot of the "bells and whistles" that vmPRO can use to move data.

 

This article includes the following sections:

 


Running smartcp

In the excerpt below from the SmartMotion log, we see Tomato.py running smartcp:


2013-10-14 14:10:21,223 smartmotion/INFO:: Tomato.py:461 Running SmartMotion copy: ['/usr/local/pancetera/bin/smartcp', '--preserve=timestamps', '--sparse=always', '--stats-output=json', '/export/LinuxLab/Fedora/Fedora_1.vmdk', '/storage/.0a929102-6933-307e-6927-83e88d1c4a00/2013-10/2013-10-14-141019/LinuxLab/Fedora']


We can also directly access the Smartcp executable from the /usr/local/Pancetera/bin/ directory, as shown here:

 

bash-4.1# pwd

/usr/local/pancetera/bin

bash-4.1# file smartcp

smartcp: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped

 


Viewing Options with smartcp --help

Check out the --help option to learn what can be done with smartcp:


Usage: smartcp [OPTION] source  destination
 

   or: smartcp [OPTION] source [more source] directory

 

Options:

 -v, --verbose

         Dump internal information for diagnostic purposes.

 -g, --debug

         Dump more detailed internal information for diagnostic purposes.

         This option can be on only when verbose option is also on.

 -p <attribute>, --preserve=<attribute>

         Specify the file attribute to preserve.

         Currently suppot 'timestamps' only. Default is on.

 -B <size>, --buffersize=<size>

         Specify copy buffer size (default = 128k).

         The parameter is a number or one followed by a size modifier

         (k, K, m, M, g, G).  Example: '24k', '4m', '3G'.

 -s <when>, --sparse=<when>

         Specify sparse output support.

         Valid values for <when> are 'always' and 'never'.  Default is 'always'.

 -c, --cal-md5sum

         Specify to calculate md5 checksum.

 -z <scan>, --scan-zeros=<scan>

         Scan and skip writing zeros when sparse output is on.

         Valid values for <scan> are 'always' and 'never'.  Default is 'always'.

 -S <use>, --use-smartread=<use>

         Utilize SmartRead if <use> is 'yes'.

         Valid values for <use> are 'yes' and 'no'.  Default is 'yes'.

 -P <bypass>, --bypass-vmproxy=<bypass>

         Bypass vm_proxy and access datastore directly if <bypass> is 'yes'.

         Valid values for <bypass> are 'yes' and 'no'.  Default is 'no'.

 -T <format>, --stats-output=<format>

         Statistics output format.

         Valid values for <format> are 'json' and 'none'.  Default is 'none'.

 -I <interval>, --stats-interval=<interval>

         Statistics output interval in second.  Default is 1.

 -R, --read-only

         Read the source file and write to /dev/null in read only mode.

         Note that only the source file name is allowed after the options.

 -W <size>, --write-only=<size>

         Read /dev/zero and write the size as specified in write only mode.

         Note that only the destination file name is allowed after the options.

 -h, --help

          Print usage information and quit.

Source and Destination Files (with the exception of read or write only mode):

    1) <source filename> <desitination filename>

       The destination file may or may not exist. If one exist, it will be

       overwritten.

    2) <source filename> [more source filenames] <desitination directory>

       One or multiple source filenames are specified along with a

       destination directory. In this case, the directory must exist.

 


Saving smartcp Output for Diagnostic Purposes

Saving your smartcp output can help diagnose problems.

 

The following command increases the debugging level:

smartcp --preserve=timestamps --sparse=always --stats-output=json --verbose --debug /export/LinuxLab/Fedora/Fedora_1-flat.vmdk /storage/.0a929102-6933-307e-6927-83e88d1c4a00/

 

Pushing the stdout of the command to an ASCII file can be helpful in gathering logs to review.

 

Command:

smartcp --preserve=timestamps --sparse=always --stats-output=json --verbose --debug /export/LinuxLab/Fedora/Fedora_1-flat.vmdk /storage/.0a929102-6933-307e-6927-83e88d1c4a00/ > /var/upgrade/sm_tape_test

 

Output to /var/upgrade. This will provide a log of a smartcp run, as shown in the example below:

 

Options: smart read   : on

         sparse output: no

         md5 checksum : no

         buffer size  : 131072

         bypass vmp   : no

         zero scanning: yes

Copying: /export/LinuxLab/Fedora/Fedora_1-flat.vmdk

     To: /storage/.0a929102-6933-307e-6927-83e88d1c4a00/Fedora_1-flat.vmdk

         ---- time on open files: 9358 ms

         ---- 131072 bytes processed, zeros: 0, elapsed time: 9, time on scan: 0 ms, time on read: 5 ms, time on write: 0 ms

{ "r": "data", "v": 1, "t": "9", "total": "21474836480", "copied": "131072", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "10", "total": "21474836480", "copied": "1966080", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "11", "total": "21474836480", "copied": "33685504", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "12", "total": "21474836480", "copied": "71958528", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "13", "total": "21474836480", "copied": "112066560", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "14", "total": "21474836480", "copied": "138018816", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "15", "total": "21474836480", "copied": "156106752", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "16", "total": "21474836480", "copied": "183894016", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "17", "total": "21474836480", "copied": "219938816", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "18", "total": "21474836480", "copied": "257032192", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "19", "total": "21474836480", "copied": "290324480", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "20", "total": "21474836480", "copied": "304480256", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "21", "total": "21474836480", "copied": "327811072", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "22", "total": "21474836480", "copied": "370016256", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "23", "total": "21474836480", "copied": "412483584", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "24", "total": "21474836480", "copied": "438435840", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "25", "total": "21474836480", "copied": "460718080", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "26", "total": "21474836480", "copied": "496631808", "write_skipped": "0", "read_skipped": "0" }

{ "r": "data", "v": 1, "t": "27", "total": "21474836480", "copied": "525729792", "write_skipped": "0", "read_skipped": "0" }

         ---- 537001984 bytes processed, zeros: 0, elapsed time: 27, time on scan: 0 ms, time on read: 17430 ms, time on write: 702 ms

 

……(I’ve Truncated the logging here)….

 

Stats:   file size    :  21474836480

         read skipped :            0

         write skipped:            0

         copied bytes :  21474836480

         total bytes  :  21474836480

         scanned zeros:            0

         elapsed time :        719 s

         time on scan :         0 ms

         time on read :    706993 ms

         time on write:     21590 ms

         enqueue r q  :    708315 ms

         dequeue r q  :      9736 ms

         enqueue w q  :       901 ms

         dequeue w q  :    695781 ms

 

Troubleshooting Tip:

When using smartcp to reproduce a problem while troubleshooting, keep in mind that you'll want to redirect stderr to stout, or the last entry will be the stats, not the error from the binary.

 

Example command:

smartcp --preserve=timestamps --sparse=always --stats-output=json --verbose --debug /export/LinuxLab/Fedora/Fedora_1-flat.vmdk /storage/.0a929102-6933-307e-6927-83e88d1c4a00/ 2>&1 /usr/upgrade/jontest.txt

 

Example of pan_pwrite error:

 

2013-12-20 20:58:38,666 smartmotion/ERROR:: Tomato.py:354 smartcp: pan_pwrite(4, 0x7fbe99840a40, 131072, 82072305664) failed: No such file or directory

4
                - The file descriptor
0x7fbe99840a40 - The memory offset in smartcp of the buffer
131072         - The size of the read/write
82072305664    - The offset in the file of the read/write

 

 

 

 

 

 

 



This page was generated by the BrainKeeper Enterprise Wiki, © 2018