Safely Virtualizing DFSR

Hi folks, Ned here again. With the massive growth of virtualization, odds are you want to safely and reliably run Distributed File System Replication on Hyper-V, Xen, KVM, or VMware – heck, you may already be doing so.

With that in mind, I am here today to save your job! OK that might overstating things; let me try again: I am here today to save your file servers!

Multi-Master Background

You can virtualize many server workloads transparently without requiring any special considerations; the hypervisor just becomes a standard “hardware platform” within the company. When you deploy multi-master replication technologies, though - such as Active Directory Domain Services, the File Replication System, or DFSR – you must exercise greater care.

DFSR contains a database on every volume participating in file replication. This extensible storage engine (aka Jet) database records every replicated update with a set of metadata we call the “summary of knowledge”, or the “version vector”. This means that each server keeps track of:

The UID and GVSN amount to:

• A GUID that matches the DFSR database ID of a given server
• The version high watermark on that database’s volume when we notice the file creation, update, or deletion.

You can see these records for a given file with the DFSRDIAG IDRECORD command and if you want to know which GUIDs originate from which servers, use the DFSRDIAG GUID2NAME command.

Clearly a file of critical importance

This version vector number (aka “Logical Clock”) never decrements and each DFSR server keeps track of other servers’ latest known version number as part of replication. While the UID never changes for a file, the GVSN updates with every file or folder alteration.

Consider table below that describes the database of “ServerA”: it lists a replicated folder named “rf1” with two files named “canary” and “cat” created on ServerA:

Note: When DFSR creates the first replicated folder, the UID of the replicated folder is marked with the GUID of the content set from Active Directory and “v1”, and the GVSN of the first server in the Replicated Folder and “v10”.

At this point, the downstream ServerB database only knows about the RF:

The servers then exchange version vectors. Since ServerB knows that ServerA has a high watermark of 12 and that ServerB is missing 11 through 12 from ServerA by looking in its own database, it requests those updates, replicates them, and updates its own database to match.

When ServerA replicates outbound to its partner ServerB – a process called “joining” it sends along the version vector chain “ServerA –> 11..12” then replicates those files in that chain:

When complete, both server databases look identical and ServerB knows of the latest changes from ServerA:

If ServerB originates a modification to cat.docx and a subsequent new file dog.txt later, it uses the high watermark of ServerB at that point and the process flows in the other direction, with ServerA then knowing the latest VV from ServerB. Moreover, if I add a “Fried” subfolder with a file called “chicken.rtf” to ServerA, this replicates to ServerB with the deeper parent relationship.

Now the databases on both servers look like this:

And the file system looks like this:

Mmmm… fried chicken…

The Big Cannoli – do not restore snapshots or whole machine backups

By now, you’re asking yourself “interesting, but… what does this have to do with virtualization?” Multi-master replication relies on this incrementing logical clock and – as any sci-fi buff knows– time travel carries a whole host of problems when going backwards and forwards.

So if you read nothing else in this blog post, read this:

• Virtual Machine Saved States/Snapshots. When virtualizing DFSR, start the virtual machine, run DFSR and if you need to stop the virtual machine, fully shut down the guest OS. Do not use saved states or snapshots.
  
• Backing Up Virtualized DFSR. When backing up virtualized DFSR, perform a guest side backup using a backup product that is VSS aware. Do not perform or restore from host side backups of virtualized DFSR servers.

Custom DFSR

If you don’t follow this guidance and attempt to use saved states/snapshots or attempt to restore a virtual machine using a custom host side restore solution, that VM will stop replicating forever. The server logs DFSR events 2212, 2104, 2004, and 2106. DFSR overwrites locally originating changes on a restored server as conflicts from other servers. The restored server and its partners will not be able to figure out what happened, as they do not ever expect the clock to go backwards. A regular backup restore of the database inside the VM using the internal VSS writer is the only way DFSR handles this correctly– and even that is rather overkill; easier to simply backup the files and restore them without worrying about the database at all. If you have to follow KB2517913, you are in a world of hurt and must work with MS Support to delete your DFSR Database structure on any restored servers, perhaps requiring some setting of Primary replication to ensure files flow in or out in the manner you need.

Don’t create snapshots of custom DFSR servers. If you need backups, back up the replicated files and the system state, by running your backup agents within the VM, not outside of it - just like a physical machine. If anyone tells you different, point to this article and wag your finger. If you absolutely must backup entire machines, restore the virtual disks, mount them offline and extract the files from for the restore – do not instantiate a running instance of the virtual machine (i.e. start them and run them). If you really must restore the entire machine to get it back in working order, understand that DFSR is now broken until its database is correctly recreated and the server will replicate inboundnon-authoritatively when you delete that database.

DCs and SYSVOL

The only time you can “safely” restore DFSR (and FRS) snapshots or whole machine backups is when using the Virtualized Domain Controller feature introduced in Windows Server 2012, and only for the built-in SYSVOL replica created automatically by domain controllers. Domain controllers understand a feature called VM-Generation ID and when they detect a snapshot restore or a whole machine restore (e.g. from DPM 2012), they delete the DFSR database that references SYSVOL, which forces replication to completely re-run inbound non-authoritatively. This also requires a hypervisor such as Windows Server 2012 Hyper-V or Microsoft Hyper-V Server 2012 that supports VM Gen-ID and a snapshot/backup solution that does as well, naturally.

Why is safely in quotes above? Because there is still risks when restoring DC snapshots or full machine backups:

1. If you restore all DCs in a domain, SYSVOL still stops replicating. There is no authoritative DC and all will be attempting non-auth sync simultaneously.

2. Any custom DFSR replicas on other volumes than the one containing SYSVOL stop replicating and require repair. Moreover, even though custom RFs on the same volume as SYSVOL “accidentally” recover non-authoritatively (they share the same database), they are likely to see perceived data loss from any changes not replicated out before the snapshot restored and eliminated them.

3. Even though SYSVOL replicating inbound non-authoritatively is generally ok - it does not usually have many changes and the changes mostly originate from one server, the PDC Emulator - group policies will certainly be disrupted on that node until sync completes. In addition, if it was the PDCE, recent GP changes that never replicated out will be lost forever just like in #2.

4. This requires homogenous virtualization of Windows Server 2012 or later. Windows Server 2008 R2 and older domain controllers do not support the VM Generation ID, so they just end up in USN rollback for AD with ruined SYSVOL databases.

Do I sound like a broken record? This is old news for AD administrators – they have been reading this for years in KB articles, TechNet articles, and blog posts, going all the way back to Virtual Server 2005 – but we want to clear on this critical guidance.

The Rest

What are the other recommendations around safely virtualizing DFSR servers?

Note: I can only speak to Microsoft Hyper-V in this article; please contact your third party hypervisor vendor details, capabilities, and limitations.

1. Microsoft recommends Hyper-V as the preferred hypervisor for virtualized DFSR. As stated previously, Windows Server 2012 Hyper-V/Microsoft Hyper-V Server 2012 both include a critical feature, VM Generation ID, designed to help alleviate issues with workloads such as Active Directory, File Replication Services and DFSR that don’t like to be time shifted. Furthermore, Microsoft develops, tests, optimizes and validates its products with Hyper-V as part of its Common Engineering Criteria.

Finally, Microsoft does not recommend using hypervisors that aren’t supported via the Microsoft Server Virtualization Validation Program (SVVP).

2. Avoid Snapshots and host side virtual machine backups – enough said on the matter. Restoring an entire VM from a saved state or a snapshot breaks custom DFSR. Use VSS backups inside the VM.

3. Reliable and high-performing hypervisor host disk subsystem that write through to disk– The host physical disk system must satisfy at least one of the following criteria to ensure the virtualized workload data integrity through power faults:

• The system uses server-class disks (SCSI, Fiber Channel, etc.)
• The system uses disks connected to a battery-backed caching host bus adapter (HBA)
• The system uses a storage controller (for example, a RAID system) as the storage device
• The system uses is protected by an uninterruptible power supply (UPS)
• The system’s disk write-caching feature is disabled

Without these safeguards in place, database corruption during unexpected power loss becomes much more likely. When DFSR detects database corruption in Windows Server 2012 and older operating systems, it deletes the database and forces inbound non-authoritative sync.

When virtualizing DFSR with Hyper-V, use virtual disks attached to virtual SCSI controllers for the DFSR data (via a VM’s Settings page). Doing so means significantly improved I/O speeds, as virtual SCSI outperforms the virtual IDE stack (even without write caching). Keep in mind that Windows Server 2012 Hyper-V and earlier versions can only boot from the virtual disks attached to the virtual IDE controller. Thus, your virtual machines should be configured to:

       A. Boot the OS from a virtual disk attached to the virtual IDE controller.

       B. Attach one or more “data” disks to the virtual SCSI controller and only replicate data on those disks.

Why yes, that is my host name. I don’t have a big hardware budget, I am a PM…

When using Hyper-V, place your DFSR data on virtual disks attached via virtual SCSI!

4. Do not clone, export, or copy VMs– As you have seen in the first section, DFSR relies heavily on the certain unique aspects of a computer such as the volume and the database signatures. If you create multiple instances of DFSR servers then all of them will believe themselves to be the same server from a DFSR perspective – even if you were to rename the servers and rejoin them to the domain. Clones create black holes in the replication topology if you were to reuse these machines with the same local DFSR metadata (especially if you accidentally left them with the same name and IP – this is a possibility even with VDC). Always create new servers from Sysprepped images or fresh media and never create copies of existing servers.

5. P2V with care - While physical to virtual conversion of DFSR servers is supported using SCVMM or Disk2Vhd, you must not allow both computers to run simultaneously. With SCVMM, this means choosing “offline mode conversion” and with Disk2VHD it simply means being careful. In both cases, once you have a working VM, you must never allow the physical machine back on your network again (See #3).

6. Do not pause or save state VMs – while not intrinsically bad, a paused or saved VM is like one that is turned off: the longer you wait, the more divergent its replication gets. Wait too long and it will never replicate again, especially since Windows Server 2012 enables content freshness by default. Getting into the habit of pausing and saving VMs is like the habit of turning off physical machines – eventually, you are going to forget you did it on one of them.

7. Use multiple virtualization hosts – This is the first “all eggs in one basket” scenario. If you only use one host, there may come a day when that host croaks. Use Failover Clustering, Hyper-V Live Migration, or Hyper-V Replica to ensure that even if a hypervisor host is down, its workloads are not. If not deploying true high availability, make sure that you at least run VMs on more than one host machine. This is a (very) basic assurance that a single hardware failure will not stop all your file replication.

8. Secure your guests and hosts– Unlike the other best practices, this is not about reliability but true data safety:

• Ensure that only your most trusted administrators have local administrator privileges on the hypervisor (i.e. with unfettered access to the virtual disks). It’s impossible to stop an administrator from simply copying a virtual disk onto an external disk for later attack, so the MS Support recommendation is legally bonding your staff against malfeasance and keeping good audit and physical security practices.
• Your Hyper-V admins use the Hyper-V Administrators group and only have file system access to their own VMs.
• Physical disks containing your virtual disk should use encryption, such as BitLocker (with a TPM and multiple protectors). Even better, ensure the hosts also use BitLocker Network Unlock so that they cannot easily walk out of the building. (It’s not supported to run BitLocker within a virtual machine, as you would then need to use Password Protectors, and rebooting thousands of servers on patch Tuesday would turn ugly.)

Last Words

Remember: the operating system, the domain controller, the virtual machines, DFSR, etc. – none of that matters to the business folks. All that matters is the data. That is where most companies make their money. The rest is just necessary cost, like desks and pens and a microwave in the break room. Everything must bend to the need for protecting that data from harm.

This may be a TechNet article someday, but in the meantime, use this blog post as your go-to guidance for virtualizing DFSR. If you find other issues, please raise them in the comments – I’m happy to add updates to the main article in the future.

Until next time,

- Ned “Don’t save state!” Pyle