import_fs

Import_fs is a writable FUSE based file system used from importing virtual machine into hypervisor. It is a virtual file system mounted within the appliance at /import, used by the Recover Wizard UI, and also accessible as a CIFS share for third party backup product for single step restore. Import_fs is a proxy that redirects virtual disk data copied in to datastore_fs which will then be send to the appropriate vSphere datastores. When the VMX file is finally copied into import_fs, it is also responsible for registering the virtual machine with VMware.

Users of import_fs should create a directory for each VM to be imported. The user can copy (or create) the VM CFG file into the directory. Each directory should have at most one CFG file. The CFG file will define an "Import Session", specifying how the VM will be imported. If the configuration has any error, such as if another VM of the same name already exists (and when register-after-import is 1), import_fs will let the user know by generating an error file (with suffix *.cfg.err). Once all the error is corrected and there are no error files, the user can start copying the virtual machine data into directory, in the order of 1. VMDK extent files, 2. PANCBT files (if any), and finally 3. the VMX file. If there are multiple disks to restore per VM, one can copy multiple VMDK extent files in parallel. If the configuration is set to register the VM after import, then once the VMX file is copied, the VM will be registered with vSphere.

Virtual Machine Configuration File (CFG file):

The CFG file is generated by vm_proxy_fs in /export and backed up with the virtual machine. It contains comments explaining each attribute. Here is a sample file:

#
# Virtual Machine '1GBDirectAttachedEmpty(Production)' configuration file from '10.30.240.240'
#
version = "4"

#
# The virtual machine name
#
# Modify this parameter during recovery if necessary to rename
# the recovered VM
#
vmname = "1GBDirectAttachedEmpty(Production)"

#
# The virtual machine UUID
#
# This is a read-only parameter. After recovery, virtual machine may be
# assigned a different UUID
#
vmuuid = "50133967-ae5a-052c-0140-e47ce07219ee"

#
# The hypervisor server name
#
# Modify this parameter during recovery if necessary to recover to
# a different hypervisor
#
servername = "10.30.240.240"

#
# The datacenter name
#
# Modify this parameter during recovery if necessary to recover to
# a different datacenter
#
datacenter = "Production"

#
# The location of virtual machine folder within the data center
# without leading or trailing directory separator
#
# e.g. "", or "Discovered Virtual Machines"
# Modify this parameter during recovery if necessary to recover to
# a different folder
#
folder = "Test VMs"

#
# The host system the virtual machine is registered to
#
# Modify this parameter during recovery if necessary to register
# the virtual machine to a different host, or leave this parameter
# empty (i.e. "") to allow the hypervisor to determine the destination host
#
hostsystem = "10.30.240.197"

#
# The resource pool the virtual machine belongs to
#
# Modify this parameter during recovery if necessary to register
# the virtual machine to a different resource pool
#
resourcepool = "Low Priority resource"

#
# The datastore name or UUID where the virtual machine files are stored
#
# Modify this parameter during recovery if necessary to recover the
# virtual machine to a different datastore
#
dsname = "4e3193a0-698e20ea-f9ef-180373f1beff"

#
# The directory within the datastore where the virtual machine files are stored
#
# Modify this parameter during recovery if necessary to recover the virtual
# machine to a different directory within the datastore
#
directory = "1GBDirectAttachedEmpty"

#
# The file name of the virtual machine VMX file
#
# This is a read only parameter.
#
vmx-name = "1GBDirectAttachedEmpty.vmx"

#
# The new file name of the virtual machine VMX file
#
# Modify this parameter during recovery if necessary to rename the VMX file.
# If not specified, then the old VMX file name is used.
#
#vmx-rename = "1GBDirectAttachedEmpty.vmx"

#
# Virtual Disk 1
#
# The name of the virtual disk 1 descriptor file
#
# This is a read only parameter.
#
vmdk.0.name = "1GBDirectAttachedEmpty(Production).vmdk"

#
# The name of the destination virtual disk 1 descriptor file
#
# Modify this parameter during recovery if necessary to rename
# the virtual disk descriptor file.
# If not specified, then the old file name is used.
#
#vmdk.0.rename = "1GBDirectAttachedEmpty(Production).vmdk"

#
# The name of the virtual disk 1 extent file
#
# This is a read only parameter.
#
vmdk.0.extent-name = "1GBDirectAttachedEmpty(Production)-flat.vmdk"

#
# The datastore name or UUID where the virtual disk is stored
#
# Modify this parameter during recovery if necessary to recover
# the virtual disk to a different datastore
#
vmdk.0.dsname = "4e3193a0-698e20ea-f9ef-180373f1beff"

#
# The directory within the datastore where the virtual disk is stored
#
# Modify this parameter during recovery if necessary to recover
# the virtual disk to a different directory within the datastore
#
vmdk.0.directory = "1GBDirectAttachedEmpty"

#
# The size of the virtual disk in bytes
#
# This is a read only parameter.
#
vmdk.0.size = 1073741824

#
# The type of the virtual disk, "thick", or "thin"
#
# Modify this parameter during recovery if necessary to convert
# the virtual disk to a thick or thin disk
#
vmdk.0.type = "thick"

#
# On-conflict mode
#
# This parameter specify the behavior during recovery when there is a
# conflict, such as when the datastore directory already exists.
# Valid values are "stop", or "rename". "stop" will cause the
# recovery to return an error if the destination directory already exists.
# "rename" will cause the recovery to be automatic redirected to a
# different directory on the same datastore.
# Modify this parameter during recovery if necessary.
#
on-conflict = "stop"

#
# Register VM with hypervisor after import?
#
# This parameter specify whether the recovered VM will be registered
# with the hypervisor or not. 0 will not register the VM after it is recovered.
# Users are responsible for registering the VM using datastore browser.
# 1 will register the VM after it is recovered, using the vm name,
# datacenter and other parameter specified in this file.
# Modify this parameter during recovery if necessary.
#
register-after-import = 0

#
# Last access time: Wed Feb 22 14:54:18 2012
#
access-time = 1329951258

==============================================

Import Session Log File

In vmPRO 3.0, each import session directory will have an import.log file logging information about what has happened during the import session. For example:

2012-02-28 13:51:49.750429: Open /import.log
2012-02-28 17:38:32.149042: Open /1GBDirectAttachedEmpty.cfg
2012-02-29 15:31:42.228616: Create file /.1GBDirectAttachedEmpty.cfg.swp
2012-02-29 15:31:42.228661: ERROR [thread:3055209360, where:import_fuse.c:1241, error:1] Cannot create /.1GBDirectAttachedEmpty.cfg.swp of unsupported file type
2012-02-29 15:31:49.930786: Create file /4913
2012-02-29 15:31:49.930843: ERROR [thread:3065838480, where:import_fuse.c:1241, error:1] Cannot create /4913 of unsupported file type
2012-02-29 15:31:49.931340: Open /1GBDirectAttachedEmpty.cfg
2012-02-29 15:31:49.931795: Create file /1GBDirectAttachedEmpty.cfg~
2012-02-29 15:31:49.931825: ERROR [thread:3076606864, where:import_fuse.c:1241, error:1] Cannot create /1GBDirectAttachedEmpty.cfg~ of unsupported file type
2012-02-29 15:31:49.947062: Truncate /1GBDirectAttachedEmpty.cfg to offset 0
2012-02-29 15:31:49.947772: Open /1GBDirectAttachedEmpty.cfg
2012-02-29 15:31:49.949173: import_session: created /
2012-02-29 15:31:51.012744: import session redirecting recover from 1GBDirectAttachedEmpty to 1GBDirectAttachedEmpty(1)
2012-02-29 15:31:51.031206: import session redirecting recover from 1GBDirectAttachedEmpty to 1GBDirectAttachedEmpty(1)
2012-02-29 15:31:51.031235: Import Session /:
2012-02-29 15:31:51.031257: vm name      : 1GBDirectAttachedEmpty(Production)
2012-02-29 15:31:51.031270: server       : 10.30.240.197
2012-02-29 15:31:51.031283: host system : 10.30.240.197
2012-02-29 15:31:51.031296: datacenter   : Production
2012-02-29 15:31:51.031308: resource pool: Low Priority resource
2012-02-29 15:31:51.031321: mode         : 1
2012-02-29 15:31:51.031334: register vm : 0
2012-02-29 15:31:51.031347: vmx:         : 1GBDirectAttachedEmpty.vmx => [4e3193a0-698e20ea-f9ef-180373f1beff] 1GBDirectAttachedEmpty(1)/1GBDirectAttachedEmpty.vmx
2012-02-29 15:31:51.031361: disk 0       : 1GBDirectAttachedEmpty(Production).vmdk => [4e3193a0-698e20ea-f9ef-180373f1beff] 1GBDirectAttachedEmpty(1)/1GBDirectAttachedEmpty(Production).vmdk
2012-02-29 15:31:51.031376: disk 0       : 1GBDirectAttachedEmpty(Production)-flat.vmdk => [4e3193a0-698e20ea-f9ef-180373f1beff] 1GBDirectAttachedEmpty(1)/1GBDirectAttachedEmpty(Production)-flat.vmdk (1073741824 bytes thick)
2012-02-29 15:31:51.031838: stats: /1GBDirectAttachedEmpty.cfg total 4 kb, write thru 4 kb, skipped 0 kb (2 kb/s), keep alive 0, incoming avg write size 4938, outgoing avg write size 4938
2012-02-29 15:31:49.948194: Flush /1GBDirectAttachedEmpty.cfg
2012-02-29 15:32:45.648118: Create file /1GBDirectAttachedEmpty(Production)-flat.vmdk
2012-02-29 15:33:00.641405: Open /1GBDirectAttachedEmpty(Production)-flat.vmdk
2012-02-29 15:34:07.750009: stats: /1GBDirectAttachedEmpty(Production)-flat.vmdk total 1048576 kb, write thru 0 kb, skipped 1048576 kb (15887 kb/s), keep alive 1, incoming avg write size 512, outgoing avg write size 0
2012-02-29 15:34:07.749900: Flush /1GBDirectAttachedEmpty(Production)-flat.vmdk
2012-02-29 15:34:38.822017: Create file /1GBDirectAttachedEmpty.vmx
2012-02-29 15:34:38.843826: Open /1GBDirectAttachedEmpty.vmx
2012-02-29 15:34:38.870554: importing /vmfs/volumes/4e3193a0-698e20ea-f9ef-180373f1beff/1GBDirectAttachedEmpty(1)/1GBDirectAttachedEmpty.vmx
2012-02-29 15:34:38.880287: redirecting disk from 1GBDirectAttachedEmpty(Production).vmdk to 1GBDirectAttachedEmpty(Production).vmdk
2012-02-29 15:34:44.943081: done importing /vmfs/volumes/4e3193a0-698e20ea-f9ef-180373f1beff/1GBDirectAttachedEmpty(1)/1GBDirectAttachedEmpty.vmx
2012-02-29 15:34:44.943185: stats: /1GBDirectAttachedEmpty.vmx total 2 kb, write thru 2 kb, skipped 0 kb (0 kb/s), keep alive 0, incoming avg write size 2600, outgoing avg write size 2600
2012-02-29 15:34:38.870284: Flush /1GBDirectAttachedEmpty.vmx

VM Name Conflict During an import_fs Session

When importing a virtual machine, a conflict may occurs if the destination directory for the VMX file or any of the VMDK file already exist, because we do not want to mix files from possibly a different virtual machine in the same directory. The "on-conflict" property in the configuration file specifies how import_fs will handle the conflict. The default is "stop", which will generate an error in the error file. The other option is "rename", in which case import_fs will rename any conflicting directory to a different name (by appending "(#)" to the end of the path) to avoid the conflict.

In vmPRO 3.0, it will also generate a .vmcfg file (same format as the configuration file) describing the destination path after the renaming:

#
# Virtual Machine '1GBDirectAttachedEmpty(Production)' configuration file from '10.30.240.240'
#
version = "4"
vmname = "1GBDirectAttachedEmpty(Production)"
vmuuid = "50133967-ae5a-052c-0140-e47ce07219ee"
servername = "10.30.240.240"
datacenter = "Production"
folder = "Test VMs"
hostsystem = "10.30.240.197"
resourcepool = "Low Priority resource"
dsname = "4e3193a0-698e20ea-f9ef-180373f1beff"
directory = "1GBDirectAttachedEmpty(1)"
vmx-name = "1GBDirectAttachedEmpty.vmx"
vmx-rename = "1GBDirectAttachedEmpty.vmx"
vmdk.0.name = "1GBDirectAttachedEmpty(Production).vmdk"
vmdk.0.rename = "1GBDirectAttachedEmpty(Production).vmdk"
vmdk.0.extent-name = "1GBDirectAttachedEmpty(Production)-flat.vmdk"
vmdk.0.extent-rename = "1GBDirectAttachedEmpty(Production)-flat.vmdk"
vmdk.0.dsname = "4e3193a0-698e20ea-f9ef-180373f1beff"
vmdk.0.directory = "1GBDirectAttachedEmpty(1)"
vmdk.0.size = 1073741824
vmdk.0.type = "thick"
on-conflict = "rename"
register-after-import = 0
access-time = 1329951258

When the import session is configured to register the VM, then the import VM name may conflict with an existing VM name. The "on-conflict" will also determine how import_fs will resolve that conflict, either by generating an error ("stop"), or by renaming the destination VM name ("rename").

There are other on-conflict resolution mode that is not fully implemented or supported yet ("overwrite" to overwrite an existing VM, or "snapshot" to restore by creating and restoring to a new snapshot of an existing VM).

Cleaning Up import_fs and Partially Recovered Files

When the import is completed, the user responsible for deleting the directory in import_fs. Note that this will only remove the import session from import_fs and some cached data in /var/pancetera/import_cache. It will not remove any files on the datastore.

This is true even if the import has failed for any reason, the partially recovered file will not be removed by import_fs. The reasoning behind this design is because import_fs really has no idea whether a recover has completed successfully or not. Failures can happen outside of import_fs and only data mover has the full visibility to make that determination. Therefore, import_fs cannot clean up after failures that it does not know. As of 2.3.2, failed recover attempts may result in orphaned partially recovered files on the datastores. This needs to be addressed in the future, potentially by make the data mover (e.g. Recover Wizard) clean up after failures by deleting files thru datastore_fs.

PANCBT File Merging

Differential virtual disk data are stored in our formatted PANCBT file. This differential data has be merged with a full backup of its corresponding virtual disk extent file. The merging can be done in 2 ways:

import_fs, by copying the virtual disk extent file (*-flat.vmdk or *-rdm.vmdk) into import_fs first, and then copy the pancbt file into import_fs next.
recovery_fs, by mounting the network storage containing both the virtual disk extent file and the pancbt file, /recover/images will present the merged version of the virtual disk. Copy that merged virtual disk into import_fs

Approach #1 is the only viable option when dealing with 3rd party backup product and when the customer do not want to recover the files from tape to a staging area first. It was also the approach taken by the Recover Wizard UI before 3.0. However, if the extent file and the pancbt file is already on a network storage on disk, Approach #2 can be more efficient.

Consider the case where you have a 100GB virtual disk, a 20GB changes in the pancbt file, and the disk is 20GB full after the change. For approach #1, you will have move 120GB (100 + 20) of data over the network from its backup storage into import_fs. For approach #2, because recovery_fs supports SmartRead and merges the pancbt file on the fly, it will only be moving the used blocks from the network storage into appliance, which is 20GB. Therefore, starting in 3.0, the Recovery Wizard GUI is taking approach #2 and taking advantage of SmartRead to reduce the network traffic during recovery.

Thinning Disk and Skipping Zeros

In a newly created thin virtual disk, the disk blocks do not use any datastore disk space until it has been written to, and those unallocated disk blocks will return 0s when read. To keep a virtual disk thin, preserve disk space, and reduce I/O, import_fs will skip writing virtual disk sectors that are all 0s.

When restoring a virtual disk, users can control whether the disk will be thin or thick using the CFG file. This applies to virtual disks there were either thick or thin, you can convert a thick disk to a thin disk and vice versa as it is imported.

There are a few subtle details to be aware of about 0 skipping:

The buffer offsets after stripping the 0s should be sector aligned or else datastore_fs will try to read/modify/write the boundary sectors and will be slow.
Some third party backup product implicitly skip 0s (e.g. TSM sparse handling) and their write requests may not be sector aligned, and thus resulting in slower performance. It may be faster if that is turned off.
The code in import_fs to skip 0s is fast but it still takes time and CPU to move the data into import_fs thru FUSE (~700MB/s on my desktop), it will be more efficient if the data mover (e.g. smartcp) can skip the 0s by not writing to import_fs at all while maintaining sector alignment to avoid the read/modify/write problem described above.
A large virtual disk may have long runs of 0s, and because it takes time to skip the zeros (see above point), we may end up with an idling vddk connection with no data for a long period of time, which can be timed out by VMware or the network firewall. We have code to send keep alive periodically to avoid these timeouts.
The 0 skipping algorithm today only looks at the 0s at the beginning and end of each write request buffer. It will not trim the 0s in the middle of the buffer. So the thinness of the result virtual disk may vary as the write buffer size changes.
The 0 skipping algorithm trims a write request buffer at a time, and therefore may result in small writes that are 4k or less. These small writes are extremely slow in VDDK and datastore_fs works around it by doing a read-modify-write into a larger buffer. This may need some optimization, perhaps by gathering multiple continuous buffers from multiple write requests to trim and write more efficiently.

Restoring Virtual RDM Disk

Import_fs and datastore_fs do not have the ability to recreate a virtual RDM disk yet. Restoring virtual RDM extent files (*-rdm.vmdk) must be redirected to a virtual disk file on a regular datastore.

Performance

Import_fs depends on datastore_fs to write data to vSphere. See datastore_fs for more information about its performance.

Import_fs will log a lline of statistics when done with importing a virtual disk:

2012-02-29 15:34:07.750009: stats: /1GBDirectAttachedEmpty(Production)-flat.vmdk total 1048576 kb, write thru 0 kb, skipped 1048576 kb (15887 kb/s), keep alive 1,
incoming avg write size 512, outgoing avg write size 0

It contains to total amount of data written into the file (not including data skipped by the data mover such TSM or SmartCP sparse handling), the amount of data it write thru to datastore_fs, the amount of data it skipped, and throughput (based on the total amount of data coming in). It also logs the number of keep alive sent, the average incoming buffer size (bigger the better), and the average outgoing buffer size to datastore_fs (bigger the better).

Virtual Machine Registration

When an import session is specified to register the virtual machine, then import_fs will perform some extra steps:

When configuration file is written, it does additional "preflight" check to make sure that the parameters such as "datacenter", "host", "vm-folder" and "resource-pool" are valid.
At the end of the import, when the VMX file is completely written during the "flush()" call, it will register the VM with vSphere using the parameters specified in the configuration file.

Process Communication Flow

Recover Wizard =>

Third party backup client

Import_fs =>

datastore_fs =>

vSphere datastore

Recover Wizard =>

Third party backup client

/import =>

/vmfs/volumes =>

vSphere datastore

Troubleshooting

The import_fs process currently writes all log entries to “/var/log/import_fs” but starting with release 3.0 an extra log file “/var/log/import.log” will be logged per “import session”.

The following registry settings impact import_fs:

appliance.debug.level - sets the appliance debug level. Default value is 0. Will enable debug for all vmPRO processes / daemons.
appliance.cbt.debug - sets the CBT debug level. Set to non-zero to log CBT debug messages.
import.skip_zero - set to 0 to disable zero skipping. Default is 1.
import.keep_alive - sets the number of seconds of idling before a keep alive read operation is sent to avoid the VDDK connections from terminating. Default is 120 seconds.

What's Next?

Licensing >