Quantum| Using the smartcp Command |
The smartcp command can be used to directly replicate how we move data within a customer environment. Think of the Linux DD command as a "raw" approach to moving data. Smartcp employs a lot of the "bells and whistles" that vmPRO can use to move data.
This article includes the following sections:
In the excerpt below from the SmartMotion log, we see Tomato.py running smartcp:
2013-10-14 14:10:21,223 smartmotion/INFO:: Tomato.py:461 Running SmartMotion copy: ['/usr/local/pancetera/bin/smartcp', '--preserve=timestamps', '--sparse=always', '--stats-output=json', '/export/LinuxLab/Fedora/Fedora_1.vmdk', '/storage/.0a929102-6933-307e-6927-83e88d1c4a00/2013-10/2013-10-14-141019/LinuxLab/Fedora']
We can also directly access the Smartcp executable from the /usr/local/Pancetera/bin/ directory, as shown here:
bash-4.1# pwd
/usr/local/pancetera/bin
bash-4.1# file smartcp
smartcp: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
Check out the --help option to learn what can be done with smartcp:
Usage: smartcp [OPTION] source destination
or: smartcp [OPTION] source [more source] directory
Options:
-v, --verbose
Dump internal information for diagnostic purposes.
-g, --debug
Dump more detailed internal information for diagnostic purposes.
This option can be on only when verbose option is also on.
-p <attribute>, --preserve=<attribute>
Specify the file attribute to preserve.
Currently suppot 'timestamps' only. Default is on.
-B <size>, --buffersize=<size>
Specify copy buffer size (default = 128k).
The parameter is a number or one followed by a size modifier
(k, K, m, M, g, G). Example: '24k', '4m', '3G'.
-s <when>, --sparse=<when>
Specify sparse output support.
Valid values for <when> are 'always' and 'never'. Default is 'always'.
-c, --cal-md5sum
Specify to calculate md5 checksum.
-z <scan>, --scan-zeros=<scan>
Scan and skip writing zeros when sparse output is on.
Valid values for <scan> are 'always' and 'never'. Default is 'always'.
-S <use>, --use-smartread=<use>
Utilize SmartRead if <use> is 'yes'.
Valid values for <use> are 'yes' and 'no'. Default is 'yes'.
-P <bypass>, --bypass-vmproxy=<bypass>
Bypass vm_proxy and access datastore directly if <bypass> is 'yes'.
Valid values for <bypass> are 'yes' and 'no'. Default is 'no'.
-T <format>, --stats-output=<format>
Statistics output format.
Valid values for <format> are 'json' and 'none'. Default is 'none'.
-I <interval>, --stats-interval=<interval>
Statistics output interval in second. Default is 1.
-R, --read-only
Read the source file and write to /dev/null in read only mode.
Note that only the source file name is allowed after the options.
-W <size>, --write-only=<size>
Read /dev/zero and write the size as specified in write only mode.
Note that only the destination file name is allowed after the options.
-h, --help
Print usage information and quit.
Source and Destination Files (with the exception of read or write only mode):
1) <source filename> <desitination filename>
The destination file may or may not exist. If one exist, it will be
overwritten.
2) <source filename> [more source filenames] <desitination directory>
One or multiple source filenames are specified along with a
destination directory. In this case, the directory must exist.
Saving your smartcp output can help diagnose problems.
The following command increases the debugging level:
smartcp --preserve=timestamps --sparse=always --stats-output=json --verbose --debug /export/LinuxLab/Fedora/Fedora_1-flat.vmdk /storage/.0a929102-6933-307e-6927-83e88d1c4a00/
Pushing the stdout of the command to an ASCII file can be helpful in gathering logs to review.
Command:
smartcp --preserve=timestamps --sparse=always --stats-output=json --verbose --debug /export/LinuxLab/Fedora/Fedora_1-flat.vmdk /storage/.0a929102-6933-307e-6927-83e88d1c4a00/ > /var/upgrade/sm_tape_test
Output to /var/upgrade. This will provide a log of a smartcp run, as shown in the example below:
Options: smart read : on
sparse output: no
md5 checksum : no
buffer size : 131072
bypass vmp : no
zero scanning: yes
Copying: /export/LinuxLab/Fedora/Fedora_1-flat.vmdk
To: /storage/.0a929102-6933-307e-6927-83e88d1c4a00/Fedora_1-flat.vmdk
---- time on open files: 9358 ms
---- 131072 bytes processed, zeros: 0, elapsed time: 9, time on scan: 0 ms, time on read: 5 ms, time on write: 0 ms
{ "r": "data", "v": 1, "t": "9", "total": "21474836480", "copied": "131072", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "10", "total": "21474836480", "copied": "1966080", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "11", "total": "21474836480", "copied": "33685504", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "12", "total": "21474836480", "copied": "71958528", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "13", "total": "21474836480", "copied": "112066560", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "14", "total": "21474836480", "copied": "138018816", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "15", "total": "21474836480", "copied": "156106752", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "16", "total": "21474836480", "copied": "183894016", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "17", "total": "21474836480", "copied": "219938816", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "18", "total": "21474836480", "copied": "257032192", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "19", "total": "21474836480", "copied": "290324480", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "20", "total": "21474836480", "copied": "304480256", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "21", "total": "21474836480", "copied": "327811072", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "22", "total": "21474836480", "copied": "370016256", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "23", "total": "21474836480", "copied": "412483584", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "24", "total": "21474836480", "copied": "438435840", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "25", "total": "21474836480", "copied": "460718080", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "26", "total": "21474836480", "copied": "496631808", "write_skipped": "0", "read_skipped": "0" }
{ "r": "data", "v": 1, "t": "27", "total": "21474836480", "copied": "525729792", "write_skipped": "0", "read_skipped": "0" }
---- 537001984 bytes processed, zeros: 0, elapsed time: 27, time on scan: 0 ms, time on read: 17430 ms, time on write: 702 ms
……(I’ve Truncated the logging here)….
Stats: file size : 21474836480
read skipped : 0
write skipped: 0
copied bytes : 21474836480
total bytes : 21474836480
scanned zeros: 0
elapsed time : 719 s
time on scan : 0 ms
time on read : 706993 ms
time on write: 21590 ms
enqueue r q : 708315 ms
dequeue r q : 9736 ms
enqueue w q : 901 ms
dequeue w q : 695781 ms
Troubleshooting Tip:
When using smartcp to reproduce a problem while troubleshooting, keep in mind that you'll want to redirect stderr to stout, or the last entry will be the stats, not the error from the binary.
Example command:
smartcp --preserve=timestamps --sparse=always --stats-output=json --verbose --debug /export/LinuxLab/Fedora/Fedora_1-flat.vmdk /storage/.0a929102-6933-307e-6927-83e88d1c4a00/ 2>&1 /usr/upgrade/jontest.txt
Example of pan_pwrite error:
2013-12-20 20:58:38,666 smartmotion/ERROR:: Tomato.py:354 smartcp: pan_pwrite(4, 0x7fbe99840a40, 131072, 82072305664) failed: No such file or directory
4 - The file descriptor
0x7fbe99840a40 - The memory offset in smartcp of the buffer
131072 - The size of the read/write
82072305664 - The offset in the file of the read/write
This page was generated by the BrainKeeper Enterprise Wiki, © 2018 |