A developer’s view on VSS for SMB File Shares

Note: this post originally appeared on https://aka.ms/clausjor by Claus Joergensen.

VSS for SMB File Shares is an extension to the existing VSS infrastructure which consists of four parts:

In this post I want to look at VSS for SMB File Shares from a developers point of view and how to support it with minimal changes to 3^rd party VSS requestors, writers and providers, as well as look at how to investigate backup issues with betest, vsstrace and Microsoft Netmon trace tools.

Hopefully you have already read the blog covering technical overview and deployment scenarios of VSS for SMB File Shares. For more technical details of this feature, please refer to Molly Brown’s talk at Storage Developers Conference 2012 and the MS-FSRVP protocol document.

VSS changes

The summary of backup application related VSS changes are listed below.

• VSS Requestor

• COM Security
• Support UNC path
• UNC Path normalization
• VSS Writer
• VSS Providers

• VSS requestor

  Requestor is one of the key VSS components that drives the application consistent shadow copy creation and backup/restore.
  

  COM Security

  As a COM client to VSS coordinator service, the VSS requestor compatible with this feature must satisfy the following security requirements:

• Run under a user account with local administrator or backup operator privilege on both application server and file servers.
• Enable Impersonation and Cloaking so that the user token running backup application can be authenticated on the file server.
• Below is a sample code to achieve this. More detail can be found in Security Considerations for Requestors.

  // Initialize COM security.
  CoInitializeSecurity(
  NULL,                           //  PSECURITY_DESCRIPTOR         pSecDesc,
  -1,                             //  LONG                         cAuthSvc,
  NULL,                           //  SOLE_AUTHENTICATION_SERVICE *asAuthSvc,
  NULL,                           //  void                        *pReserved1,
  RPC_C_AUTHN_LEVEL_PKT_PRIVACY,  //  DWORD                       dwAuthnLevel,
  RPC_C_IMP_LEVEL_IMPERSONATE,    //  DWORD                       dwImpLevel,
  NULL,                           //  void                        *pAuthList,
  EOAC_STATIC_CLOAKING,           //  DWORD                        dwCapabilities,
  NULL                            //  void                        *pReserved3
  );

  Support UNC path

  Prior to this feature, VSS only supported requestor adding shadow copies of data stored on local volume. With the new “File Share Shadow Copy Provider” available by default on all Windows Server 2012 editions, VSS now allows adding SMB UNC share path to a shadow copy set by calling the same IVssBackupComponents::AddToSnapshotSet method. Below is a simple code snippet to add a SMB UNC share path \\server1\guests to the shadow copy set.

  VSS_ID shadowCopySetId = GUID_NULL;

  VSS_ID shadowCopyId = GUID_NULL;

  CComPtr <IVssBackupComponents> spBackup;

  LPWSTR pwszPath == L“\\server1\guests”

  spBackup = CreateVssBackupComponents (VssBackupComponents);

  spBackup->StartSnapshotSet(&shadowCopySetId);

  spBackup->AddToSnapshotSet(pwszPath, GUID_NULL, &shadowCopyId);

  UNC Path Normalization

  Similar to local volume, uniquely identify a UNC share path in a shadow copy set is key to VSS-based shadow copy creation and backup/recovery. An UNC share path is composed of two parts: server name and share name. For the same share, the server name part of a UNC path can be configured in writer component in different formats below with many variations.

• Host name
• FQDN (Fully Qualified Domain Name)
• IPv4
• IPv6 or IPv6 literal format
• This feature adds a new method IVssBackupComponents4::GetRootAndLogicalPrefixPaths that normalizes a given UNC share path to its unique root path and logical prefix format. The unique root path and logical prefix path are designed to be used for shadow copy creation and backup/restore file path formation, respectively.

  Note that a VSS requestor need to:

• Call IVssBackupComponents::AddToSnapshotSet with unique root path of UNC share in hostname or FQDN format but not IPv4 or IPv6 address format.

  If a UNC share path is in IPv4 or IPv6 address format, its root path must be normalized into hostname or FQDN by calling IVssBackupComponents4::GetRootAndLogicalPrefixPaths.

   

• Consistently normalize the file share UNC path into either host name or FQDN format in the same shadow copy set before IVssBackupComponents::AddToSnapshotSet.
• Don’t call mix hostname or FQDN format in the same shadow copy set. Note the default root path format returned is hostname format. You can specify the optional 4^th parameter to require FQDN format in the returned unique root path.
• IPv4/IPv6 DNS reverse lookup must be configured appropriately on DNS infrastructure when normalizing UNC share path in IPv4/IPv6 format. Below are the examples to determine if your DNS server enables reverse IP address lookup for the specific address:
• Example of IPv4 reverse look up is enabled on DNS. In this case, you can find the highlighted hostname/FQDN is resolved from IP address.

  F:\>ping -a 10.10.10.110

  Pinging filesever.contoso.com [10.10.10.110] with 32 bytes of data:

  Reply from 10.10.10.110: bytes=32 time=1ms TTL=57

  Reply from 10.10.10.110: bytes=32 time=1ms TTL=57

  Reply from 10.10.10.110: bytes=32 time=1ms TTL=57

  Example of IPv6 reverse lookup is not configured for 2001:4898:dc3:1005:21c:c4ff:fe68:e88. In this case, you can find that the hostname/FQDN cannot be resolved as the highlighted part is still IPv6 address.

  F:\>ping -a 2001:4898:dc3:1005:21c:c4ff:fe68:e88

  Pinging 2001:4898:dc3:1005:21c:c4ff:fe68:e88 with 32 bytes of data:

  Reply from 2001:4898:dc3:1005:21c:c4ff:fe68:e88: time=3ms

  Reply from 2001:4898:dc3:1005:21c:c4ff:fe68:e88: time=1ms

  Reply from 2001:4898:dc3:1005:21c:c4ff:fe68:e88: time=1ms

  If your DNS server does not have IPv6 DNS reverse lookup configured, you can manually add the IP address to hostname/FQDN mapping to your local DNS cache directly. To achieve this, open %SystemRoot%\system32\drivers\etc\hosts with notepad.exe from an elevated command window and add one line for each IP address to hostname/FQDN mapping as shown below. Most of the deployments we tested have IPv4 reverse DNS lookup available. But not all of them have IPv6 DNS reverse lookup configured.

  2001:4898:2a:3:6c94:5149:2f9c:f083 fileserver.contoso.com

  2001:4898:2a:3:6c94:5149:2f9c:f083 fileserver

  To unify the normalization of all VSS supported paths, this API also supports

• DFS-N path pointing to another physical server, which returns the fully evaluated physical share path on the DFS target server.
• Local volume, which returns the results of GetVolumePathName and GetVolumeNameForVolumeMountPoint on the input path.
• Below is a sample requestor code snippet to illustrate how backup applications use IVssBackupComponentsEx4::GetRootAndLogicalPrefixPaths to normalize root path and logical prefix path for shadow copy creation and backup of files under DFS-N path.

  #define CHK_COM(X) \

  {\

      hr = X; \

      if (FAILED(hr)) \

  {\

  wprintf(L”COM Call %S failed: 0x%08lx”, #X, hr );\

  goto _Exit;\

  }\

  }

  …

  CComPtr <IVssBackupComponents> spBackup;

  CComPtr<IVssBackupComponentsEx4>spBackup4;

  CComPtr <IVssAsync> spAsync;

  VSS_ID shadowCopySetId = GUID_NULL;

  VSS_ID shadowCopyId = GUID_NULL;

  VSS_SNAPSHOT_PROP shadowCopyProp;

  // GatherWriterMetadata and metadata enumeration are not shown below.

  // Instead, we assumes one writer component path to be shadow copied and backed up.

  // \\dfsworld\logical\path\guests is a DFS-N link to \\server1\guests

  // vm1 is a directory under \\server1\guests which contains the vhd file vm1.vhd that needs to

  // be backed up.

  LPWSTR pwszWriterComponentPath = L“\\dfsworld\logical\path\guests\vm1”;

  LPWSTR pwszWriterComponentFileSpec = L“vm1.vhd”;

  LPWSTR pwszRootPath = NULL;

  LPWSTR pwszLogicalPrefix = NULL;

  HRESULT hr = S_OK;

  CHK_COM(CreateVssBackupComponents (&spBackup));

  CHK_COM(spBackup->SafeQI(IVssBackupComponentsEx4, &spBackup4));

  // The caller must free the pwszRootPath and pwszLogicalPrefix strings allocated by GetRootAndLogicalPrefix()

  CHK_COM(spBackup->GetRootAndLogicalPrefixPaths(writerComponentPath, &pwszRootPath, &pwszLogicalPrefix));

  // pwszRootPath == “\\server1\guests”

  // pwszLogicalPrefix == “\\dfsworld\logical\path\guests”

  CHK_COM(spBackup->StartSnapshotSet(&shadowCopySetId));

  CHK_COM(spBackup->AddToSnapshotSet(rootPath, GUID_NULL, &shadowCopyId));

  CHK_COM(spBackup->DoSnapshotSet(&spAsync));

  CHK_COM(spAsync->Wait());

  CHK_COM(spBackup->GetSnapshotProperties(shadowCopyId, &shadowCopyProp));

  // Build snapshot relative path for file copy

  // 1. Initialize it to shadow copy share path \\server1\guests@{guid}

  CComBSTR shadowCopyPath = shadowCopyProp->m_wszSnapshotDevicePath;

  // 2. Get the relative path below the logical prefix path, “\vm1\vm1.vhd”

  //This is based on the logical prefix path returned from GetRootAndLogicalPrefixPaths,

  CComBSTR relativePath = writerComponentPath + wcslen(logicalPrefix);

  // 3. Append the relativePath and the wrierComponentFileSpec \\server1\guest@{guid}\vm1\

  shadowCopyPath += relativePath;

  // For each file (backupItem) that matches the file spec under the shadowCopyPath, copy the file to backup store

  // \\server1\guest@{guid}\vm1\vm1.vhd

  CComBSTR backupItem = shadowCopyPath+writerComponentFileSpec;

  HANDLE h = CreateFile(

  backupItem,

  …

  );

  // BackupRead and Copy from backItem

  …

  _Exit:

  // The caller must free the string allocated by GetRootAndLogicalPrefix()

  if (pwszRootPath!=NULL)

  CoTaskMemFree(&pwszRootPath);

  If (pwszLogicalPrefix!=NULL)

  ContaskMemFree(&pwszLogicalPrefix);

  VSS writer

  There is no change needed from VSS writer to support this feature as long as the application itself allows storing application data on SMB share.

  VSS provider

  On application server, the new file share shadow copy provider handles the file share shadow copy life cycle management. There is no change needed from existing VSS software/hardware provider to support this feature.

  Developer tools

  Betest for backup/restore Hyper-V over SMB

  BETest is a VSS requester that tests advanced backup and restore operations. It is included in the Microsoft Windows Software Development Kit (SDK) since Windows Vista and later. However, only the BETest in Windows Server 2012 SDK has been extended to support this feature. Full detail about betest can be found in the BETest document.

  Below we give an example to online backup and restore Hyper-V VM running over SMB shares. Please refer to the Hyper-V over SMB guide to configure Hyper-V over SMB.

  1. Download Windows 8 SDK which includes BETest tool.

  2. Get the VMId of the Hyper-V VM to be backed up by running powershell cmdlet. Assuming the VM to be backed up is named test-vm10, you run:

  PS C:\test> get-vm -name test-vm10|select vmid

  VMId

  —-

  c8a460ae-8aa2-4219-8c4f-532479fb854a

  3. Create the backup component file vm10.xml for backup.

  The Hyper-V writer ID is a GUID constant {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}. Each VM is handled by Hyper-V writer individually as a separate component with a componentName equals to its VMId. To select more VMs for backup, you can repeat step 2 to get all the VMIDs and put each into <Component> </Component>.

  <BETest>

  <Writer writerid=”66841cd4-6ded-4f4b-8f17-fd23f8ddc3de”>

  <Component componentName=”c8a460ae-8aa2-4219-8c4f-532479fb854a”></Component>

  </Writer>

  </BETest>

  4. Create a shadow copy and Full backup based on the components selected by vm10.xml.

  Upon completion of execution, the application consistent Hyper-V VM VHD and VM configuration file for VM (test-vm10) will be snapshotted and copied to backup destination specified with “/D” parameter (C:\backupstore). A new backup document vm10backup.xml specified by /S parameter will be created as well. It will be used with the file in the backup destination for restore later.

  betest /b /V /T FULL /D C:\backupstore /X vm10.xml /S vm10backup.xml

  If you just want to create the snapshot for testing purpose without waiting for the long file copying, you can just run the following command without /D and /S parameters like below.

  betest /b /V /T FULL /X vm10.xml

  During step 4, the betest output includes lots of writer information during shadow copy creation and backup. The key to determine if the backup succeeds or not is to check writer status at the end of the output as highlighted below after backup complete. You need to rerun step 4 until you get an application consistent backup indicated by STABLE writer state after backup. If you keep getting errors during step 4, vsstrace and netmon traces discussed below are the most useful tools to help investigation.

  status After Backup Complete (14 writers)

  Status for writer Task Scheduler Writer: STABLE(0x00000000)

  Status for writer VSS Metadata Store Writer: STABLE(0x00000000)

  Status for writer Performance Counters Writer: STABLE(0x00000000)

  Status for writer Microsoft Hyper-V VSS Writer: STABLE(0x00000000)

  Status for writer System Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer ASR Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer BITS Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer Shadow Copy Optimization Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer Dedup Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer Registry Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer COM+ REGDB Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer WMI Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer Cluster Database: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  Status for writer Cluster Shared Volume VSS Writer: STABLE(0x00000000)

  Application error: (0; <unknown error>)

  5. Restore the VM

  You can restore the VM by specifying the backup document using /S and the backup store location using /D parameter.

  betest /R /D c:\backupstore /S vm10backup.xml

  VSStrace for file share shadow copy

  This feature adds two major components to Windows Server 2012:

• File Share Shadow Copy Provider running on the application server
• File Share Shadow Copy Agent running on the file server or file server clusters
• Both of them support detailed ETW traces that are compatible with existing VSSTrace tool included in Windows 8 SDK, which makes it easy to correlate File Share Provider VSS Provider/Agent activities with VSS infrastructure for trace analysis.

  To turn on logging on the application server, open an elevated command prompt and run

  Vsstrace.exe -f 0 +WRITER +COORD +FSSPROV +indent -o vssTrace_as.log

  To turn off tracing on the application server, go back to the command prompt on that machine and hitting “ctrl + c”. The log file vssTrace_as.log generated is a text file that contains detail information about activities of file share shadow copy provider, VSS and VSS writers.

  To turn on logging on the file server, open an elevated command prompt and run:

  Vsstrace.exe -f 0 +WRITER +COORD +FSSAGENT +indent -o vssTrace_fs.log

  To turn off tracing on the file server, go back to the command prompt on that machine and hitting “ctrl + c”. The log file vssTrace_fs.log generated is a text file that contains detail information about activities of file share shadow copy agent, VSS and VSS writers.

  If you hit Hyper-V host-based backup issue, it is useful to gather local VSS trace inside the guest OS. To turn on logging inside the guest OS, open an elevated command prompt in VM and run:

  Vsstrace.exe -f 0 +WRITER +COORD +SWPRV +indent -o vssTrace_guest.log

  To turn off tracing on the VM, go back to the command prompt on that machine and hitting “ctrl + c”. The log file vssTrace_guest.log generated is a text file that contains detail information about activities of VSS, VSS SYSTEM Provider and VSS writers.

  MS-FSRVP RPC protocol

  The Windows 2012 Server “File Share Shadow Copy Provider” and “File Share Shadow Copy Agent” communicate through a new RPC-based protocol called MS-FSRVP. Its open protocol architecture offers the flexibility to allow 3^rd party ISV/IHV to implement their file share shadow copy agent RPC server on non-Windows servers and interop with VSS-based backup application running on Windows Server 2012. There are 13 protocol messages for shadow copy life cycle management. In addition to the protocol document available http://msdn.microsoft.com/en-us/library/hh554852(v=prot.10).aspx, an FSRVP netmon parser is provided to understand and investigate the protocol sequence issues.

  To trace the FSRVP activities with Netmon:

  1. Download and install Microsoft Netmon and parser package on the application server where the shadow copy and backup are initiated.

• Netmon from http://www.microsoft.com/en-us/download/details.aspx?id=4865
• Netmon parser from http://nmparsers.codeplex.com/
• 2. Start the Netmon, click “Tools->Option” menu and active “Windows Profile” to enable FSRVP parser as shown below.

  3. Start a new capture, key in “FSRVP” and apply the protocol filter in the Filter window

  4. Create file share shadow copy using diskshadow/betest or 3^rd party backup software that is compatible with this feature.

  In the example below, I create a shadow copy for SMB share \\yxy-vm1\data on yxy-vm1 (file server) from yxy-vm2 (application server)

  C:\>diskshadow

  Microsoft DiskShadow version 1.0

  Copyright (C) 2012 Microsoft Corporation

  On computer: YXY-VM2, 7/2/2012 10:54:57 AM

  DISKSHADOW> add volume \\yxy-vm1\data

  DISKSHADOW> create

  Alias VSS_SHADOW_1 for shadow ID {3c36b6e5-ba12-4ba4-92c7-fa9cf1e35bcc} set as environment variable.

  Alias VSS_SHADOW_SET for shadow set ID {9c647280-b2db-40c7-b729-3b82fd71e851} set as environment variable.

  Querying all shadow copies with the shadow copy set ID {9c647280-b2db-40c7-b729-3b82fd71e851}

  * Shadow copy ID = {3c36b6e5-ba12-4ba4-92c7-fa9cf1e35bcc} %VSS_SHADOW_1%

  - Shadow copy set: {9c647280-b2db-40c7-b729-3b82fd71e851} %VSS_SHADOW_SET%

  - Original count of shadow copies = 1

  - Original volume name: \\YXY-VM1\DATA\ [volume not on this machine]

  - Creation time: 7/2/2012 10:56:02 AM

  - Shadow copy device name: \\YXY-VM1\DATA@{009F5DC8-856A-4102-9313-5BBB00024F29}

  - Originating machine: YXY-VM1

  - Service machine: yxy-vm2.dfsr-w8.nttest.microsoft.com

  - Not exposed

  - Provider ID: {89300202-3cec-4981-9171-19f59559e0f2}

  - Attributes: Auto_Release FileShare

  Number of shadow copies listed: 1

  5. As show in the Netmon trace below, the complete shadow copy creation sequence with FSRVP protocol includes IsPathSupported, GetSupportedVersion, SetCOntext, StartShadowCopySet, AddToShadowCopySet, PrepareShadowCopySet, CommitShadowCopySet, ExposeShadowCopySet, GetShareMapping and RecoveryCompleteShadowCopySet. If any error happens in the middle, the AbortShadowCopySet message will be sent to cancel the file server shadow copy processing.

  Conclusion

  I hope this introduction makes it easier for backup application developers to add support for this feature, which provides backup of application servers that store their data files on SMB file shares.

  Xiaoyu Yao

  Software Development Engineer

  
  Additional resources

  Application consistent VSS snapshot creation workflow:

  http://msdn.microsoft.com/en-us/library/aa384589(v=vs.85).