Hi folks, Ned here again. Yesterday I introduced my new video series on Storage Replica in Windows Server 2016 TP2 with an example of Stretch Cluster disaster recovery. Today I show off a brand new feature, the Test-SRTopology cmdlet.

The Test-SRTopology cmdlet arrives when you install the “Storage Replica Module for Windows PowerShell” feature using Server Manager or Install-WindowsFeature.

Here’s the beauty of it - you don’t need to install SR to find out how well SR will work. You just need this cmdlet. When run, it verifies that servers meet requirements, tells you the write IO load the servers handle currently, estimates how long initial sync will take, and gives you advice on log sizing – all in a pretty graphical report.

But this is supposed to be a video blog post – to the Tubes!

There’s that crummy voice again. Argh, stop sounding like you are from Chicago, Ned!

There you have it, a cmdlet so simple even I can use it, with results so plain even I can understand them. There are tons of improvements coming to this tool, I can’t wait to share them.

Tune in tomorrow for Storage Replica running over a 25km fiber network!

Until next time,

  - Ned “Maybe I could direct one of those craptastic lost footage horror movies” Pyle

Hi folks, Ned here again. This week I introduced my new video series on Storage Replica in Windows Server 2016 TP2. First it was Stretch Cluster disaster recovery, then Test-SRTopology. Today you get to see replication over a real 25km fiber network using Chelsio iWARP RDMA.

Wait, you don’t believe that I have a 25km network? OK smarty-pants, here’s Greg holding it (hopefully you can see this, it seems our blog software is out to lunch today).

Our little Greglet, dwarfed by unshielded fiber spools… He wrote SMB Direct, BTW. Not a dope.

The demo video today shows a pair of white box servers running a single NVME drive apiece, with a fiber optic network that takes a 50km round trip for every Storage Replica packet, all plugged into RDMA. I demonstrate the “before and after” of configuring replication, while hitting disks with a reasonable simulated workload using diskspd. Storage Replica uses SMB 3.1.1 and that means you get RDMA support – along with encryption, signing, and multi-channel – all for free.

Let’s rot our brains with TV!

So nasal…

We also have a 40km (!!!) network plugged into our Mellanox InfiniBand MetroX test setup. With this amazingly low latency RDMA rig - usually a few hundred microseconds at 80km roundtrip - you have to have incredible storage to even tax the network – and we just took delivery of some last week. I’ll share the crazy results at a later date.

Tune in tomorrow for server–to-server replication provisioning.

Until next time,

  - Ned “Greg may be a stand-in for Paul Rudd that new Ant Man movie” Pyle

Hi folks, Ned here again. We are a few days into my new video series on Storage Replica in Windows Server 2016 TP2. Look back a few posts to see what I’ve covered already. Today it’s provisioning and observing a server-to-server replication partnership using Windows PowerShell.

The thing that rubs me raw about a lot of replication technology – even ones that I own, like DFSR– is that setting them up takes too many steps. A single command should be enough. Moreover, we should be able to easily tell what replication is doing, with one command. And the event logs and performance counters should be clear, concise, and intuitive. I don’t want to “learn to ignore” certain messages.

The demo video today shows me configuring replication for the first time between a pair of servers; there’s no cluster here, just a couple of machines in a pair of sites that I want to protect with synchronous replication. Then I poke around a bit for state, health, and history.

Engage the Stereopticon, Smithers!

I wonder if a nose job would help? Too many punches to the face in my youth…

There you go, some Windows PowerShell and status messaging so simple that you could teach your grandmother. She’d bake you cookies and tell her bridge club what a nice young engineer you turned out to be. Then it’s off to the slots, Granny gots to get paid!

Tune in tomorrow for more server-to-server replication management.

Until next time,

  - Ned “Now that I think about, I should have added Jake animations to these flicks” Pyle

Hi folks, Ned here again. I’m nearing the end of my new video series on Storage Replica in Windows Server 2016 TP2. Check the last few posts to see what I’ve already covered. Today I alter replication direction – i.e. a planned failover - of a server-to-server replication partnership using Windows PowerShell. Oh, and I show you how to remove replication. Not that you will ever do this.

Never. Never ever.

Altering replication in TP2 requires a little forethought. One, you have to do it on a particular server in TP2, because our remoting of server-to-server SR wasn’t done when we shipped this release. Two, block replication means a single writeable source and an unmounted destination – so if someone is doing something important on the source volume and you change replication direction without warning them, their writes will fail. They may come after you with pitchforks and torches. Don’t be The Monster, be Dr. Frankenstein.

Wait, I think they came after him with torches too. Uuuhhh, don’t be Mr. Hyde, be Dr. Jekyll! Yeah, that’s better. What were we talking about? Oh yeah, Storage Replica.

To the video graveyard, Igor!

Perhaps it’s a deviated septum? Or tuberculosis?

Simple as that. A single command to change replication, three commands to tear down replication without a trace. We’ve been thinking about making those single commands as well; as always, let us know what you think via email or UserVoice.

Tune in Monday for the final* installment in this series: managing a Stretch Cluster using the Failover Cluster Manager GUI. It’s a howl.

*For now.

Until next time,

  - Ned “O. Selznick?” Pyle

Hi folks, Ned here again. You have reached the end of my new video series on Storage Replica in Windows Server 2016 TP2. We’ve covered Stretch Cluster automatic failover, Test-SRTopology, RDMA Fiber performance, creating and managing server-to-server replication, switching replication direction, and removal. Today I show building and managing your stretch cluster, including both CSV and non-CSV storage.

Tomorrow I show nothing. Hey pal, this was a lot of work, I need a break from Camtasia!

I saved the best for last; Stretch Clusters are the coolest. A nice familiar graphical interface, a simple and intuitive wizard, status management tabs, and a polished experience. Once you’re running a stretch cluster, you get all the benefits of a normal single-site cluster, as well as peace of mind from synchronous replication protecting your data. Aaaaaaah.

Let’s warm ourselves around the 21^st century hearth.

Finally, our long national nightmare is over. No more hearing Ned.

I’d say this was so easy even a monkey could do it, but monkeys can be pretty smart. Therefore, it’s so easy even an Elden can do it.

Zing!

Any comments, suggestions, anger, love? Find us through email or UserVoice. I hope you’ve enjoyed this brief series. Rest assured that plenty of deeper content is coming. You know me, I’m a constant self-promoter.

Until next time,

  - Ned “Now to reap the rewards of points on the backend” Pyle

Hi folks, Ned here again. Rudolf Vesley has provided everyone with great community tool for DFSR. Quoting time:

I decided to write Windows PowerShell Cmdlet (Advanced Function) to get report of configuration, status and health of Distributed File System Replication (DFS-R) in large enterprise environment (of course it is possible to use it for small deployment). I created this script because there are no native tools for such report and I did not find a single PowerShell script that would satisfy my needs.

Key concepts

• The script is in PowerShell but the script does not use native DFSR module so it is possible to use it on system with any version of the DFSR module or even without this module.
• The script was designed to never stop. That means that when there is an error on a single server or on a single replication group, folder or connection then script will log this error and continue with gathering data from other servers, connections, groups or folders.
• It is possible to target script to a single main (hub) DFS-R server and count backlogs in both directions (from source to destination and from destination to source). Of course it is possible to target all servers and do not count these reverse backlogs.

Slick stuff to make your life easier and fill in the gaps. Get it here!

 - Ned "sorry this took me so long, Rudolf!" Pyle

Hi folks, Ned here again. Below are all the videos I created for Storage Replica on Windows Server 2016 Technical Preview 2 in my recent series. Yes, this is one of those boring "all my links" blog posts. Not my fault, people asked me for it!

• Stretch Cluster Automatic Site Failover in Windows Server 2016 Technical Preview 2
• Plan Your Replication using Test-SRTopology in Windows Server 2016 Technical Preview 2
• Storage Replica with RDMA and 25km Fiber in Windows Server 2016 Technical Preview 2
• Provision and Manage Server-to-Server Replication in Windows Server 2016 Technical Preview 2
• Alter and Remove Server-to-Server Replication in Windows Server 2016 Technical Preview 2
• Configure and Manage Stretch Clusters in Windows Server 2016 Technical Preview 2

Enjoy.

- Ned "mmm, these reheated leftovers are tasty!" Pyle

Hi folks, Ned here again. We created DFSR Cloning in Windows Server 2012 R2 to make initial synchronization faster. Today I talk about file attributes, how they sneak inefficiency into your cloning process, and what can you do about them.

Let’s get to it.

Attributes, DFSR, and Cloning

Attributes are simply metadata on files and folders that describe special states, like hidden or read-only. They often don’t change the file in a meaningful way. DFSR handles attribute changes with a few methods.

If you change these attributes, DFSR does replicate the attributes to the other server:

• FILE_ATTRIBUTE_HIDDEN
• FILE_ATTRIBUTE_READONLY
• FILE_ATTRIBUTE_SYSTEM
• FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
• FILE_ATTRIBUTE_OFFLINE
• FILE_ATTRIBUTE_REPARSE_POINT (note: it’s not that simple – see this treatise on what actually does and doesn’t work. Files with the IO_REPARSE_TAG_DEDUP, IO_REPARSE_TAG_SIS, or IO_REPARSE_TAG_HSM reparse tags are replicated as normal files)
• FILE_ATTRIBUTE_SPARSE_FILE
• FILE_ATTRIBUTE_DIRECTORY
• FILE_ATTRIBUTE_COMPRESSED

If you change these attributes, DFSR doesnot trigger replication of the attributes to the other server (but if the file is altered in another way that does trigger replication, these attributes come along for the ride):

• FILE_ATTRIBUTE_ARCHIVE
• FILE_ATTRIBUTE_NORMAL

There is the FILE_ATTRIBUTE_TEMPORARY attribute (good call, Dragos!), which makes DFSR ignore the file.

This is normal, day-to-day replication – Bobby Joe in Accounting sets the quarterly report to hidden and read-only, then DFSR updates all the other replicated copies of that file with these attributes. This gets more interesting in DFSR Cloning. The cloning process bypasses the step of always exchanging file information between servers during initial sync, by simply providing the new server with a copy of the old server’s database. This means that when you perform Import-DfsrClone, we need to check the local copy of the preseeded data with whatever is in the imported database. If the file dates, file sizes, or file ACL differ, we know that someone messed with the file between the export and the import, at least on this destination server.

However, we also check the attributes – if they differ between the preseeded files and the database records, we consider the file mismatched. Any mismatched files replicate to the destination using the normal initial sync mechanism, after cloning completes. Unlike usual, this is a full file replication, not just the metadata.

In other words:

1. If someone decides to change attributes on the files after they cloned the database, those files replicate again. Even if they are files with the Archive or Normal bit being changed.
2. If someone changes attributes on the source copy of the files, and they are in the list above that trigger replication, those files are going to queue into the source server’s backlog, and replicate once you finish cloning. I.e. a little later, and metadata replication only, but you still pay a price.

#2 is out of your control, and frankly, who cares? You created a replication topology, there’s no worry about it actually replicating. #1 is avoidable. What could be changing these files? Here are some possible culprits:

• Archive bit – Usually disappearing, giving the file the Normal bit. The Archive bit is an idiotic legacy that supposedly tells you a file has not been backed up. Windows Server stopped using it many years ago. If this attribute is changing, you are likely running third party backup software on the destination server. Update your software or yell at kindly ask your vendor why they are still using this junk bit instead of the correct USN journal methodology.
  
• Hidden, read-only, compressed, and/or system bit – This is all you, buddy! Some application, some script, some automation, or - hopefully not – some individual user is changing things on the destination prior to cloning completion. Get out the ProcMon, I have no way to tell you who, when, or how.
  
• Reparse point and sparse file bit – This is likely Windows Server Deduplication. Not that dedup is at fault; you or your colleagues did the dirty deed. When you run an optimization job in dedup to dehydrate files, they have to be marked for the chunk store. That mark means setting the sparse file attribute and a reparse point, in this case with the IO_REPARSE_TAG_DEDUP tag. When you decided to turn on dedup and alter all the files in the middle of your cloning operations, you altered their attributes – probably a lot of files too, dedup is good at its job. This is one of those “Doctor, it hurts when I do this” scenarios. Don’t do that.

Detecting It

That’s all fine, but how to tell if you are getting attribute-based cloning inefficiency? After all, when you look at your event logs after cloning, you only see a count of mismatches. Those could be anything:

Four? That’s not so bad.

To the debug logs!

First, look for lines that start with “[WARN] DBClone::IDTableImportUpdate Mismatch record was found”. For instance, here is our filewith that warning:

Look closely. The ACL hashes match, the write times match, and the files sizes match. But the attributes?

20150616 15:00:01.037 2980 DBCL  4054 [WARN] DBClone::IDTableImportUpdate Mismatch record was found. Local ACL hash:3D1F6474-928B8530-1E0A6559-5F02A2C7LastWriteTime:20150605 16:52:48.545FileSizeLow:402432 FileSizeHigh:0Attributes:128Clone ACL hash:3D1F6474-928B8530-1E0A6559-5F02A2C7LastWriteTime:20150605 16:52:48.545FileSizeLow:402432 FileSizeHigh:0Attributes:32

Aha! To the Internet!

A 128 is FILE_ATTRIBUTE_NORMAL. Someone removed the archive bit. Ok, that’s not too bad. How about:

To the calculator! We know that 32 is a file with the archive bit. 34-32=2. A value of 2 is FILE_ATTRIBUTE_HIDDEN. Starting to make sense?

I wonder why it’s hidden?

       ==============================================================

Ok, one more:

Got the hang of it? Good. What a strange set of files names…

Call me Hank

To the wrap-up! At Microsoft these are often called “learnings”, which makes me want to hurtle across the conference table and shake the person until they admit that learnings isn’t a $%#^#^%& word.

Ahem.

The Lesson

Don’t monkey with file attributes on the destination server. For that matter, don’t changes any files in any fashion on the destination while cloning; this adds inefficiency. Letting users party on the downstream file server isn’t a good idea – you are about to enable replication and DFSR will reconcile the differences. If users on the destination alter files first, their changes will go buh-bye.

 
What part didn’t you understand?

Not to mention the confusion if you start seeding data onto your destination while they accessed it. “Hey Martha, what’s with this empty share? Oh, now it has some files. And some more. And some more. Ok, I’m going to lunch.”

You don’t have to do anything if you run into attribute changes causing less efficient replication – DFSR will fix everything up by performing initial sync on just those files. Seeing a couple of mismatches is no reason to get in a twist. If a sizable number mismatch, however, you need to evaluate what’s going on and decide if fixing the issue and re-importing is going to save you time in the end.

I want to thank Jeroen de Bonte, Dutch Microsoft Support Engineer extraordinaire, for working with us these behaviors, which pointed out that we had no documentation on it. Good man.

Until next time,

  - Ned “Destitute and in Disrepute” Pyle

Hi folks, Ned here again. While we designed Storage Replica in Windows Server 2016 for synchronous, zero data-loss protection, it also offers a tantalizing option: extreme data mover. Today I compare Storage Replica’s performance with Robocopy and demonstrate using SR for more than just disaster planning peace of mind.

Oh, and you may accidentally learn about Perfmon data collector sets.

In this corner

Robocopy has been around for decades and is certainly the most advanced file copy utility shipped in Windows. Unlike the graphical File Explorer, it has many options for multithreading, security, large files, retry, logging, and more. You’ve seen me use it with DFSR preseeding. It’s a great tool for IT professionals who want real control over moving files.

That said, it is still a user-mode file-level utility running at the very top of the stack. Windows Server 2016 Storage Replica is a kernel-mode, block-level filter system operating near the bottom of the stack. It only does one thing – copy blocks of data under a Windows volume - but it does it very well.

How well, though?

Heavyweight bout

I started by stealing borrowing a couple of 1.45TB Intel P3700 SSDs. I wanted that model for its solid-state consistency. It’s very fast and low latency, thanks to its NVME PCIe architecture. Moreover, it has excellent drivers and firmware.

To ensure that the network did not throttle my NVME disks, I grabbed Chelsio’s latest Terminator 5 40Gb RDMA cards. With two interfaces per server, SMB Direct and SMB Multichannel had 80Gb of ultra-low latency throughput – there was no chance that my NVME would run out of pipe, which is a real possibility on these drives.

I then snagged two servers that I knew wouldn’t give me unreasonable bottlenecks in memory or CPU. That meant a couple of whitebox 2Us, each with 16 cores of Intel Xeon E5-2660 at 2.2Ghz, and 256GB of memory. As Bill Gates once famously said to Scarlett O’Hara, “Frankly my dear, 256GB of RAM should be enough for anyone.”

“Don’t make me throw a sherry glass at you, Ned.”

Ahem.

Finally, I needed to come up with some tests that included various file and size characteristics. I generated my files using a combination of FSUTIL and an internal tool:

Now I was ready for Robocopy and Storage Replica to meet at the pinnacle of the crucible of the octagon of the ultimate mêlée on the field of peril, where second is the first loser and… you know, Richard Hammond metaphors.

There are a few ways to approach this kind of testing: you can measure time taken to complete and declare victory. You can also see system impact and resource utilization.

Let’s start simply.

Round 1 – A single huge file

I began with the single file test. To create this hulk, I ran:

Then I copied it several times using differing syntax. First:  

These command-line parameters ensured that I copied the file metadata (security, attributes, etc.) as well as used the restartable option, ensuring that if the file copy failed mid-stream due to an interruption, I could pick up where I left off. Moreover, I didn’t need permissions to the file, thanks to the backup API call.

Then again, with:  

The /j means unbuffered IO, which makes it much faster to copy very large files – only! Don’t use that with a bunch of tiny ones.

Then I split the difference with:  

Finally, I enabled Storage Replica and replicated that whole 1.45TiB volume, with its single 1.45TiB file. Then I removed replication so that the destination volume remounted.

The results:

Round 1 went to Storage Replica– not even close, except for robocopy /j, and then you must hope nothing interrupts my copy, which would mean starting all over again. Any interruption to SR just means it picks up where it left off, automagically.

Round 2

For my next test, I had 1.1 million files ranging from 1KiB to 10GiB, weighing in at 1.45TiB – a typical mixed bag of shared user data on a file server. This is where Robocopy should be in its element – not too many files and the files aren’t too small or too big.  

You can see I added an extra set of cmdline options. One was to copy all the files in the single subfolder I created on my D drive, and the other was to enable multithreading and copy 128 files simultaneously – the maximum robocopy supports. I sent everything to a log file and stopped the screen output, which makes robocopy faster (really!)

The results:

Round 2 went to SR, beating robocopy by nearly half the time. Spoiler alert: SR is very consistent in its performance on a given set of hardware. On these servers, it will always be around ~14.5 minutes unless some outside force acts upon it.

What was the performance like during all this, though?

I fired up Performance Monitor on the source server and created a User Defined Data Collector Set. This gets me Process, Processor, Memory, Disk, Network, and System performance data.

I ensured its stop condition was set long enough to capture the entire operation and let robocopy rip again. Then I ate a burrito. It’s a well-known constant that a proper burrito takes exactly 25 minutes to eat. It’s called Señor Nedo’s Law. You can look it up.

I started another run of the same data collector, but shortened it to 15 minutes, and examined the reports.

Uh oh

Robocopy looked angry. I cracked open the BLG from the report and look at the chart.

You’re getting your money’s worth on these processors, at least

To use a technical term, that is “dang near pegged”. How was Storage Replica looking during its run?

Oh yeah!

Not bad at all, it’s quite efficient on CPU thanks to operating in kernel.

Hello, Intel? I’d like to return some unused cores. Yes, I’ll hold.

That was the source machine. What did Storage Replica resource consumption look like on the destination?

The CPU was higher, but not much – in this case it averages 14%. The memory usage was much more aggressive than the source, as the destination is checksuming blocks for its bitmap during asynchronous initial sync. To perform that on an incredibly fast set of NVME storage, we need RAM. Storage Replica throttles at 50% of the total physical memory to ensure we don’t use too much. Robocopy doesn’t do this – depending on the copy characteristics in my three rounds of testing the source and destination server would use anywhere from 2% to 80%. Quite a bit of fluctuation on both nodes.

For comparison’s sake, when using some plain old hard disk drives, my memory usage dropped to around 20% used by Storage Replica. That’s spinning rust for ya – too slow to pressure the system. The future is solid state, dear reader.

Round 3

For my final test, I used one hundred million very small files, only 1KiB apiece. Remember, this is still the same 1.45TB drive. Meaning that Robocopy only had to copy 95GiB of data while Storage Replica had to copy more than 15 times as much data – every single block of that volume.  

I feel like this picture should have Dr. Evil ‘shopped in

The results: 

Round 3 went to SR - understatement of the year! As you can see, SR was its usual 14+ minutes, but Robocopy was an all-day affair.

By the way, just like with round 2, CPU usage of robocopy with tiny files was gnarly and it was using a 25% of the RAM on the source and 64% on the destination at about an hour into the copy.

Robocopy source

Robocopy destination

In the end, depending on the files you were copying, SR should be gentler on CPU and perhaps even on memory than robocopy in many circumstances.

The winner by knockout

Storage Replica won. Sheesh. Didn’t you read the last three sections?

Down goes robocopy, down goes robocopy, down goes robocopy!

Here’s a roundup:

The kicker? I made it much easier for Robocopy than reality will dictate. Think about it:

• When I copied a file with robocopy /J, I could not use /ZB to gain resiliency; any interruption would force me to recopy the entire file all over again. The one and only time robocopy was almost as fast as SR, it was in the danger zone.
• I wasn’t making robocopy write through to disk in my tests; it was hitting a file server share with no clustering, meaning no Continuous Availability and always using cache. This gave robocopy an advantage in performance, at the expense of data resiliency. As you know from my previous blog post, robocopy would not write this fast to a share marked with CA. Storage Replica always hardens data to the disk.
• I didn’t introduce any users or apps into the tests. Robocopy might hit exclusively locked files if anyone was actually using the source machine, which would require retry attempts (note: consider DiskShadow as a potential workaround for this). Storage Replica operates beneath the file layer, where this is not an issue.
• I had no anti-virus software installed, where file scanning would have come into play. I won’t be mean to any anti-virus vendors here, but let’s all agree that you don’t use them for their performance enhancement.
• I wasn’t adding any more files! With Storage Replica, users can continue to add data that SR automatically replicates. You would need to run robocopy repeatedly for any new files.

The only realistic chance robocopy has to win? By copying a relatively small set of data from a very large volume. Even then, I could just shrink the volume temporarily, sync my data with SR, remove SR, then expand the volume. SR rides again!

Robocopy can be the right tool for the job. When moving a metric poop ton of data, SR may be better.

I’m no fool. I realize this requires you run our latest Windows Server 2016 preview build on both nodes, and you aren’t going to get such amazing numbers with terrible old HDDs and 1Gb Ethernet and 4GB of RAM. With that, now that you know the advantages of going all-in on our latest OS and its hardware capabilities when it releases, you have something to think about when you need to move some serious data.

Until next time,

- Ned “Sugar Ray” Pyle

Feast your eyeballs on our new Software-Defined Storage Design Calculator and Design Considerations Guide! Be amazed at how designing a storage solution with Windows Server, Storage Spaces, and SOFS (Scale-Out File Server) can be simultaneously simplified with a colorful spreadsheet and concurrently complexified with a detailed guide to everything you wanted to know about Storage Spaces design, but weren’t afraid to ask (we’ve been getting a lot of questions…).

Your hosts for today’s blog are Josh Adams (Senior PM) and Jason Gerend (Senior Content Developer), and we’ll give you a little peek at our spreadsheet & document package and then send you out to go design some storage. So let’s begin.

First – go get the Software-Defined Storage Design Calculator spreadsheet. Once you’ve opened it in Excel, you should see theQuickDesign sheet, which lets you see several typical storage designs. Here’s what it looks like – pretty simple (as far as storage designs go anyway).

If you want to get into more detail or make changes to the design, click the AdvancedDesign sheet near the bottom of Excel. This sheet lets you start from one of the same templates and then customize the design to fit your particular requirements. Here’s what this sheet looks like – a bit more complex, but still something you can glance over.

Now’s when you’ll probably want to take a peek at theSoftware-Defined Storage Design Considerations Guide, which is definitely more complex, as you might imagine from a detailed design guide. It’s chock full of design help for each step in the spreadsheet (and then some), but still organized in a streamlined way that matches the workflow of the spreadsheet so that you can easily switch between them and not get bogged down in details.

That’s it for now - we hope you find them helpful, and thanks for joining us!

Josh Adams – Senior PM Manager

Jason Gerend - Senior Content Developer

Hello everyone,

From time to time we receive support inquiries about Offline File related problems. The symptoms range from clients which do not come online anymore, which cannot sync, which throw plenty of sync-errors or sync-conflicts up to paths that cannot be reached in SlowLink mode although there should be no offline files active on these paths.

These problems have one thing in common: the root cause can be found in the employed Offline File concepts and can only be solved through a new and sustainable concept for the deployment of Offline Files in that environment.

The migration from an old concept to a new concept often implies the usage of the FormatDatabase Registry Keys (http://support.microsoft.com/kb/942974/en-us) to clean up the faulty client configuration. Obviously this brings up questions regarding backup of the offline available data on the clients (that might not have synced for months), which usually lead to months long migration projects.

To avoid the lengthy migration, I want to help the Administrators to build sound Offline Files concepts, to enable them develop flexible plans.

The following are some key concepts in Offline Files.

The 5 ways of offline availability

There are 5 ways files/folders can get into the Offline Cache of the Client and a related scope and partnership will be created within Offline Files. We need to get administrative control over all of them and maintain them.

• Files made offline available automatically by Folder Redirection.

When configuring a Folder Redirection policy, an Offline Partnership will be automatically created for each redirected folder.

Through configuration of “Do not automatically make Redirected Folders available Offline“, this automatism can be disabled. See http://gpsearch.azurewebsites.net/#293

• Administrative assigned Offline Files

The admin can configure paths in a group policy that will be made available offline by the client. See http://gpsearch.azurewebsites.net/#2056

• User making content ‘always available’ by selecting this option in the context menu of a file/ folder

By default the user can decide what paths he/she wants to make offline available.

This can and should be disabled by the Policy “Remove Make Available Offline Command” in order to not get unforeseen partnerships e.g. on group drives. See: http://gpsearch.azurewebsites.net/#7857

• Caching settings of a share on a file server

For each CIFS share these settings are available:
“Only the files and programs that users specify are available offline” (basically: The Client can cache if it wants to) which is the default in Windows,
“No files and programs from the shared folder are available offline” (basically: No caching allowed here), and “All files and programs that users open from the shared folder are automatically available offline” (basically: The Client has to cache).

When the Client access a share that is configured to “The Client has to cache” the Client will create a sync partnership and will start to sync the files that have been opened to the client.

In that Regards the settings on the respective shares have to be checked / corrected.

•  LinkTargetCaching behavior when creating an .lnk file on an offline available path

Per default the target of .Ink Shortcuts will be made offline available if the .Ink file itself is made offline available.

A user could therefore unintentional create an Offline partnership against a group share if this target is on a share that isn’t yet offline available. This option is by default enabled to make sure that users have the expected files available when working with shortcuts. The drawback is that slow link policies will also apply to this new share and data that isn’t cached will not be available any more when the share transitions to slow connection.

This can be controlled by the LinkTargetCaching Registry Key:
http://support.microsoft.com/kb/817275/en-us

Permission on the file server structure

Because Offline Files also need to sync data from other (not currently logged on) users on the computer, the Offline File sync does not exclusively take place in the rights context of the logged on user. Thus, Offline Files require a set of minimum permissions to be able to sync (and in fact work) correctly. Not setting the permissions correctly on the file server will cause sync problems, can prevent the client from switching to online mode and a host of other problems (files created offline won’t sync back to the server).

The required minimum permissions for the share, the NTFS permissions of the shared folder and the permissions for each folder in the path are documented in: http://support.microsoft.com/kb/2512089/en-us.

Scopes andcachedfiles

Offline Files considers each share (\\Server\Share) as a scope. Each access to a network path will be checked against the list of scopes which Offline Files has partly or fully cached.

Note here that \\Server\Share is treated completely independently from \\Server.contoso.com\Share. As a result administrators must ensure that only the FQDN paths are used for drive mappings, redirections and DFSN paths.

If a match is found Offline files will be handling the request and among other things decide if the request should be satisfied from the Server (Online Mode) or from the local Cache (Offline mode).
Thus the Scope is the instance that decides if a paths is online or offline and it is always the complete share that is taken offline. That said, there are situations where only specific files are treated as offline for example when they are in a synchronization conflict. You can also configure a subset of a scope to be always offline and only synced during logon and logoff (this is called suspending and can be configured through http://gpsearch.azurewebsites.net/#2584).

If Offline Files treats a scope as offline, you can only use what is locally cached.
In Case you only made a subset of \\Server\Share available offline, you can only reach this subset of data in offline mode and you lose access to other branches that are not in the cache until you switch back to online mode. While offline greyed out icons identify data that is NOT cached but available as name in the Offline Files database only for consistency.

What part of a scope is offline available is very important to understand especially in case of DFS Namespaces in order to make meaningful decisions about how to design the directory structures on the server site.

It makes sense to host data that should be offline available in a different scope than the data that should not be offline available. Here it is common practice to host users offline available home shares in a separate DFS Namespace than not offline available Group shares.

Slowlink Mode and Background Sync.

With the policy “Configure slow-link mode” threshold values for latency and/or bandwidth can be defined. Those values can be applied to all (indicated by an ‘*’) or individual paths.

If for example the latency on the connection raises above the threshold value offline files will switch into offline state. See http://gpsearch.azurewebsites.net/#2093

When deploying Offline Files on DFS paths, it might make sense to defined very high latency values for the DFS-root so the root itself cannot switch to offline mode due to a slow link, but the deeper paths can. This is because the DFSN root share is treated by offline files in the same way as any other share that can go offline (slow connection).
For all the details see the blog of my colleague Ned Pyle:

http://blogs.technet.com/b/askds/archive/2011/12/14/slow-link-with-windows-7-and-dfs-namespaces.aspx

When the Client switches to offline (slow link) mode, the policy “configure background sync” governs if and how often the client will try to sync with the server which is important to still sync the data back to the server for backup and consistency:
See http://gpsearch.azurewebsites.net/#2095

It is not recommended to configure the background sync frequency to a very low value (minimum should be 30 minutes).

Migration of offline available data to other paths

Avoid this Scenario at all cost!

In http://support.microsoft.com/kb/977229/en-us two supported ways to migrate offline available data without reset of the client cache (and therefore resync) are described.
One way applies only to Offline Availability via Folder Redirection (automatic move done by the folder redirection service), the other to all other ways of Offline availability (manual by script).

Both ways have in common that you have to follow the KB Article to the letter if you want to have success and you must be up to date with all binaries on the client (see http://support.microsoft.com/kb/2820927).

As you can see from the first KB you have to follow a sequence of steps including Changes in AD/Fileserver/Infrastructure and on the Client.
From my experience this is seldom possible to do for a larger number of clients and it is very hard to automate.

Therefore the use of DFS for abstraction of logical paths and physical paths is strongly recommended.
In case you want to migrate offline available data on a DFS paths you just need to logoff the client, and change the paths the DFS Link is pointing to.

As the path used by offline files does not change, offline files won’t realize the migration as long as you preserved the filestate for example by using robocopy.

However when using Offline Files on DFS (or Roaming Profiles or DFSR) make sure you stay within the supported limits:

http://blogs.technet.com/b/askds/archive/2010/09/01/microsoft-s-support-statement-around-replicated-user-profile-data.aspx

Also note here that the path change from NetBios name to FQDN name is also a full blown move as Offline Files treats these 2 although ending up at the same data as different paths. Here the ‘Verify Old and New folder redirection targets‘ policy mentioned in 2820927 has to be set.

Configurations & scenarios that should be avoided

• Firstly the same set of data should not be made available offline in different ways!
  For example: Folder Redirection with offline availability on the path \\Server\Share\Username\MyDocuments, in addition administrative assigned Offline Files on the path \\Server\Share\Username Here the subfolder MyDocuments has been made available offline in two different ways. This nested offline availability is something offline files doesn’t cope well with in my experience.
   

• Another scenario is the use of several machines simultaneously by the same user.

  For example, a laptop with offline availability and a desktop machine without offline availability.
  In this scenario you have to ensure that the client using offline availability does not fall into offline mode for example due to SlowLink Mode. If it does, the user can inadvertently change a file on the server and in the local cache of the offline client, thus resulting in a sync conflict, the next time the laptop switches to online mode. The same is valid for data that is used by several users. These data are unfit to be used with Offline Files.

• Also, avoid redirecting the appdata directory. This can cause severe delays (depending on network performance) for applications on the client that expect that their data to be stored on a local disk!

I think it should be self-explanatory that before Offline Files are deployed to the clients, they should get the latest hotfixes for Offline Files (2820927), Fileservices (client and server 2899011) as well as the latest Security Updates installed. Furthermore a patch management is required to keep those components up to date.

On a related note, if you are planning for new client deployment, and exploring new solutions for user data offline access, I highly recommend you to look into the new feature “Work Folders” introduced in Windows Server 2012 R2 and Windows 8.1. To find more about the topic, see here https://technet.microsoft.com/en-us/library/dn265974.aspx.

I hope this helped you understanding offline files a bit better and enables you to build sound and solid concepts.

With best Regards
Hubert

Heya folks, Ned here again. The inimitable Claus Jorgenson just posted about the Microsoft and Intel showcase Storage Spaces Direct with NVM Express at IDF ’15. He demonstrates a 16-node hyper-converged system running Windows Server 2016 TP3, with 4.2M read IOPS - all while using only 4 data drives per node. Pretty amazing bang for the buck. It goes on to show Storage QoS as well as mixed read-write performance, and has great info from Dan Lovinger, one of our oldest wisest developers here in the High Availability and Storage group.

Get on over there to read more about the configuration (Chelsio RNICs! Intel E5-2699 v3 CPUs! ReFS file system! A crazy lighted rack door!) and be sure to catch the videos, they are a treat. Here's a taste:

There are three more videos and plenty more info, head over there now.

   - Ned "this private cloud goes to 11" Pyle

Heya folks, Ned here again with a public service announcement from the Microsoft Container and Filter Team and registering for their upcoming free plugfest. Hurry, the registration period ends on August 31st! Here’s the skinny:

We are pleased to announce Windows Server Container Plugfest. As you know, we recently announced Container Technologies on Windows Server. In today’s cloud-first world, businesses increasingly rely on applications to fuel innovation and productivity. As the cloud evolves, containers are emerging as an attractive way to efficiently build and deploy these applications at the speed of business. Containers offer developers and IT professionals the ability to deploy applications from a workstation to a server in mere seconds, taking development to a whole new level.

It is super important that your filters work well with Windows Container technology. Plugfest provides you the opportunity to test your applications (Anti-Virus, Replication, Redirector, Activity Monitor, Security Enhancer, HSM, and other types) on Windows Container technology. Come test the interoperability of your file system filters and network filters with Microsoft Container Technology on Windows Server. Here are some preliminary event details:

When

Monday, November 9^th to Thursday, November 12^th, 2015.

The event begins at 9 am and ends at 6 pm each day, except for Thursday, when it ends at 3 pm

Where

Building 37, room 1717, Microsoft Campus, Redmond, Washington. Map.

Registration

To register, submit a completed Registration Form no later than August 31^st, 2015. We will follow up through email to confirm the registration. Due to constraints in space and resources at this Plugfest, ISVs are required to limit their participation to a maximum of two persons representing a product to be tested for interoperability issues. There will be no exceptions to this rule, so please plan for the event accordingly. Please look for messages from fsfcomm@microsoft.com for registration confirmation.

Cost

Free – There is no cost for this event.

Audience

ISVs (Independent Software Vendor) and developers writing file system filter drivers and network filter driver for Windows Server.

Goal

Compatibility testing with Windows Server Container and Hyper-V Container

Benefits

• Test products extensively for interoperability with Windows Server Container and Hyper-V Container. This has traditionally been a great way to understand interoperability scenarios and flush out any interoperability related bugs.
• Attend talks and informative sessions on Windows Server Container and Hyper-V Container.
• Meet and interact with engineers from the Microsoft Container and the Microsoft Filter teams and get answers to technical questions.

Please familiarize yourself with the information in Windows Server Containers and Filter Drivers. Prior to Plugfest, we will send instructions for trying out our Container technology with your filter and running some basic tests. This will help you get your filter running with Containers before you come to the event. At Plugfest, you can focus on more complex interop issues.

Regards,

Microsoft Container and Filter Team

With the release of Windows Server Technical Preview 3, it’s time to provide updates on the feature content for Data Deduplication. Rather than only provide the delta from Technical Preview 2 to 3, and make you go back and forth between a couple of blog posts, I’m including the full article intact in this post and just adding the changes for TP3.

For those familiar with the Data Deduplication in Windows Server Technical Preview 2 article, you can jump down to the new entry for “Dedup Improvement #4: Support for the Nano Server”, since that is primarily what is new in this release.

Everything else still applies, so if you have been experimenting with Data Deduplication in the Technical Preview releases, please continue. If you have been waiting to jump in, Technical Preview 3 is a great release to get started with. And of course send email to dedupfeedback@microsoft.com and let us know how your evaluation goes and any questions you may have.

What’s New in Data Deduplication?

If I had to pick two words to sum up the major changes for Data Deduplication coming in the next version of Windows Server, they would be “scale” and “performance”. In this posting, I’ll explain what these changes are and provide some recommendations of what to evaluate in Windows Server Technical Preview 3.

In Windows Server 2016, we are making major investments to enable Data Deduplication (or “dedup” for short) to more effectively scale to handle larger amounts of data. For example, customers have been telling us that they are using dedup for such scenarios as backing up all the tenant VMs for hosting businesses, using from hundreds of terabytes to petabytes of data. For these cases, they want to use larger volumes and files while still getting the great space savings results they are currently getting from Windows Server.

Dedup Improvement #1: Use the volume size you need, up to 64TB

Dedup in Windows Server 2012 R2 optimizes data using a single-threaded job and I/O queue for each volume. It works great, but you do have to be careful not to make the volumes so big that the dedup processing can’t keep up with the rate of data changes, or “churn”. In a previous blog posting (Sizing Volumes for Data Deduplication in Windows Server), we explained in detail how to determine the right volume size for your workload and typically we have recommended to keep volume size <10TB.

That all changes in Windows Server 2016 with a full redesign of dedup optimization processing. We now run multiple threads in parallel using multiple I/O queues on a single volume, resulting in performance that was only possible before by dividing up your data into multiple, smaller volumes:

The result is that our volume guidance changes to a very simple statement: Use the volume size you need, up to 64TB.

Dedup Improvement #2: File sizes up to 1TB are good for dedup

While the current version of Windows Server supports the use of file sizes up to 1TB, files “approaching” this size are noted as “not good candidates” for dedup. The reasons have to do with how the current algorithms scale, where, for example, things like scanning for and inserting changes can slow down as the total data set increases. This has all been redesigned for Windows Server 2016 with the use of new stream map structures and improved partial file optimization, with the results being that you can go ahead and dedup files up to 1TB without worrying about them not being good candidates. These changes also improve overall optimization performance by the way, adding to the “performance” part of the story for Windows Server 2016.

Dedup Improvement #3: Virtualized backup is a new usage type

We announced support for the use of dedup with virtualized backup applications using Windows Server 2012 R2 at TechEd last November, and there has been a lot of customer interest in this scenario since then. We also published a TechNet article with the DPM Team (see Deduplicating DPM Storage) with a reference configuration that lists the specific dedup configuration settings to make the scenario optimal.

With a new release we can do more interesting things to simplify these kinds of deployments and in Windows Server 2016 we have combined all the dedup configuration settings into a new usage type called, as you might expect, “Backup”. This both simplifies the deployment as well as helps to “future proof” your configuration since any future setting changes can be included to be automatically changed by setting this usage type.

Dedup Improvement #4 (new for TP3): Nano Server support

Nano Server is a new installation option in Windows Server Technical Preview that provides a cloud-optimized Windows Server environment. Data deduplication is fully supported in Nano Server. What works differently? Nano Server is a headless deployment option for Windows Server providing a deeply refactored and reduced environment optimized for cloud deployments. Data deduplication has been tuned and validated to operate in the Nano Server environment.

Note that deduplication support in Nano Server is in “preview” status and currently has the following restrictions:

• Support has only been validated in non-clustered configurations

• Deduplication job cancellation must be done manually (using the Stop-DedupJob PowerShell command)

Suggestions for What to Check Out in Windows Server TP3

What should you try out in Windows Server TP3? Of course, we encourage you to evaluate overall the new version of dedup on your own workloads and datasets (and this applies to any deployment you may be using or interested in evaluating for dedup, including volumes for general file shares or for supporting a VDI deployment, as described in our previous blog article on Large Scale VDI Deployment).

Also, if you are evaluating Nano Server for TP3, it would be great for you to try out dedup in your environment.

But specifically for the new features, here are a couple of areas we think it would be great for you to try.

Volume Sizes

Try larger volume sizes, up to 64TB. This is especially interesting if you have wanted to use larger volumes in the past but were limited by the requirements for smaller volume sizes to keep up with optimization processing.

Basically the guidance for this evaluation is to only follow the first section of our previous blog article Sizing Volumes for Data Deduplication in Windows Server, “Checking Your Current Configuration”, which describes how to verify that dedup optimization is completing successfully on your volume. Use the volume size that works best for your overall storage configuration and verify that dedup is scaling as expected.

Virtualized Backup

In the TechNet article I mentioned above, Deduplicating DPM Storage, there are two changes you can make to the configuration guidance.

Change #1: Use the new “Backup” usage type to configure dedup

In the section “Plan and set up deduplicated volumes” and in the following section “Plan and set up the Windows File Server cluster”, replace all the dedup configuration commands with the single command to set the new “Backup” usage type.

Specifically, replace all these commands in the article:

…with this one new command:

Change #2: Use the volume size you need for the DPM backup data

In the article section “Plan and set up deduplicated volumes”, a volume size of 7.2TB is specified for the volumes containing the deduplicated VHDX files containing the DPM backup data. For evaluating Windows Server TP2, the guidance is to use the volume size you need, up to 64TB. Note that you still need to follow the other configuration guidance, e.g., for configuring Storage Spaces and NTFS. But go ahead and use larger volumes as needed, up to 64TB.

Conclusion

We think that these improvements to Data Deduplication coming in Windows Server 2016 and available for you to try out in Windows Server Technical Preview 3 will give you great results as you scale up your data sizes and deploy dedup with virtualized backup solutions.

And we would love to hear your feedback and results. Please send email to dedupfeedback@microsoft.com and let us know how your evaluation goes and, of course, any questions you may have.

Thanks!

At Microsoft, we have Work Folders deployed for our internal users. I have worked closely with people from the IT group to investigate issues and learn about management operations which relate to Work Folders. One such operation is to recover data after an encryption key is revoked. Erez Benari from Microsoft IT is the expert on the data recovery topic, and we collaborated on this blog post to give an in-depth look into the Work Folders data encryption on Windows.

Client encryption

When we were designing Work Folders, one of the goals was to ensure data security, as the devices which have Work Folders may not be under IT management. The policy we exposed to the IT admins is “Require encryption” on the Sync Share, which indicates whether the sync server requests that the contents of Work Folders be encrypted on each PC and device that syncs with the sync share. I.e. if the admin configures “encryption required” on the server, the Work Folders client will enforce it.

The Work Folders client gets the policy from the sync server, and enforce it by setting the encryption on the Work Folders path and all the files “touched” (e.g. created/moved/synced) under the path will be protected with the encryption. Existing files will be encrypted during the next run of the Work Folders Maintenance Work configured in the Task Scheduler.

If the admin disables the encryption policy on the server at any time, the client will get the change on the next sync, and remove the encryption at the root path of the Work Folders, so new files generated under the Work Folders path will not be encrypted, but existing files will remain encrypted. User can remove the encryption by opening the folder, right clicking on the encrypted folder or file and clicking Remove enterprise control to decrypt the files on the client.

If the user chooses to remove enterprise control on files to decrypt the data, and where policy on the server indicate “require encryption”, the Work Folders client will encrypt the files during the next maintenance task run.

Encryption under the cover

The encryption technology that Work Folders leverages on Windows to protect the data is called “Selective Wipe”[1]. At a high level, the encryption uses a key which is associated an Enterprise ID (EID), which is the company’s top-level domain (for example contoso.com). The selective-wipe encryption mechanism is very similar to EFS, but a major difference is that it is non-PKI based. The encryption key is stored in the Windows Vault (a.k.a. Credentials Manager), which is not stored in Active Directory as part of the users’ profile, and windows does not offer a backup mechanism for it (the built-in credentials manager backup mechanism doesn’t backup that key as part of the backup process). One potential mitigation comes from the fact that when a device is domain-joined (note that this is different than “Workplace joined”), the EID encryption will use both the EID key on the user device and the domain’s Data Recovery Agent (DRA) key. This will allow an enterprise administrator to decrypt the files in case of the key revocation or loss on the domain joined devices.

Key revocation

In case of a device compromise, the encryption key can be revoked by an admin using a Mobile Device Management (MDM) solution such as InTune, which can issue a command to set the device’s status to “revoke access”. When the Work Folders client attempts to sync, it first checks the Work Folders encryption status. If the status shows “revoke access”, the Work Folders client immediately revokes the sync partnership, and no further sync can take place. The user will see the following message on the control panel:

The Enterprise ID for this PC was remotely revoked by the issuing authority. Access to files in WorkFolders is blocked on this PC. To resume syncing, in the Work Folders Control Panel, click Stop using Work Folders and then recreate the Work Folders synchronization partnership.

If the user doesn’t wish to use Work Folders anymore, he can click “Stop using Work Folders” from the Work Folders control panel:

At any time, when user lost access to the encryption key. This could happen due to a corruption or damage to the hard-drive, or if the user’s profile is deleted or damaged in some way (for example, because of a virus or hack). If the key is lost, the user will not be able to open any of the files, and require a recovery.

Data recovery

Key revocation can be triggered remotely by the administrator at any time. After key revocation, the user will no longer be able to open any of the files. Any attempt to access a file will return an “Access denied” error.

If the user should no longer access the data, the admin should remove the user access to the sync share. If the user wants to get the data back, there are 2 options listed below.

Re-enable the sync

Work Folders syncs the data in the background between the client and the file server. The user can get the data back on the client by re-configuring the Work Folders client. After the setup, data will sync down from server to client.

During the Work Folders setup, the Work Folders default path is %user profile%\Work Folders. There is a specific check for the encryption status of the Work Folders root folder. If the folder is encrypted with a revoked key, the following message will be displayed to the user:  

This folder can't be used because it was wiped by your organization. Delete the wiped folder or choose another location

The user can either rename the existing folder path, if there is a need to recover un-synced files; or delete the existing Work Folders path if there is no need for recovery.

Recovery of un-synced data

In some cases, the client and server data are out of sync. The main reason which can cause the sync to fail on the client is requiring user to enter credential. If some files are not synced to the server after key revocation, user will need to request an administrator’s assistance to recover the un-synced files.

To decrypt the data on Windows 8.1 and RTM release on Windows 10, the admin will require console access to the computer holding the files. This could be either physical access to the computer, or access through remote desktop (RDP). With November release and later of Windows 10, this can also be done directly over the network (where the user can share the files over a regular SMB share). The primary challenge for data recovery is time consumption. In our experience, it requires about 20-60 minute per GB.

Risks of using DRA

When performing data recovery using the DRA, a specific risk is for the DRA to be used as a means for a user to access files encrypted by another (for example, an employee gaining access to his boss’s or the company’s confidential info). To reduce the risk, the person performing the recovery should take the necessary precautions to confirm that the user who is making the recovery request is in fact the owner of the files, or that he has the authority or permission for it. One way to do this is to run the DIR command with the /q option, which shows the file’s owner:

Decryption with Cipher

Decrypting a few files is simple, but when a large number of files with many sub-folder need to be decrypted, things can get tedious. Even though you can uncheck the encrypt option at the top-level folder, the engine might not succeed in decrypting all sub folders and files, resulting in a partially decrypted data set. To perform decryption more efficiently, you can use a command line tool Cipher, which is built into Windows. Cipher can be run at a top level of the drive or folder, or go through the entire drive.

If the DRA certificate is stored on a smartcard and requires a PIN to unlock, the administrator should provide the PIN quickly before Cipher times out and moves to the next file. Be cautious that Windows may show the PIN dialog as a system-tray notification icon or a pop-up that’s hiding behind the active Window (this might mislead one to think Cipher is stuck).

Another case which requires the PIN to be entered again is when the decryption engine’s key cache expired. This means that the administrator needs to pay attention to the recovery process and inspect the progress on a regular basis. If the pin is not provided quickly, Cipher will skip to the next file and might fail to decrypt many files. In such a case, you can re-run Cipher and it will skip over the already-decrypted files quickly. It’s generally a good idea to re-scan with Cipher at least once again after the decryption has completed, as it might be able to decrypt files that it skipped or missed in the initial scan. Also, during the 2^nd scan, the administrator should look for any files that haven’t been successfully decrypted. Such files can usually be decrypted manually using the Windows file GUI.

Conclusion

Encryption adds extra data security on the device, but you should be aware and plan for data recovery.

As always, hope you find the information helpful, and share your feedback on Work Folders with us.

Heya folks, Ned here again. Our enterprising tech evangelists Matt McSpirit and Rick Claus joined us back in July for a conversation about the end of Windows Server 2003 and what that means for SMB1. I’d hoped for an Irish wake, but unfortunately we stayed lucid. Probably for the best.

Here are 30 minutes of semi-amusing conversation about why SMB1 is better off dead and how SMB 2.1 and 3.x can help you work faster and safer. It’s delivered by the SMB team’s dev manager Dave Kruse, senior dev Greg Kramer, and yours truly; the cause of - and solution to - all of life’s problems.

For more information on SMB 3, point your eyeballs to:

• SMB 3.X information
• SMB 3.1.1 in Windows Server 2016 and Windows 10
• MUP hardening
• SMB 3 security information
• JoseLand, the abandoned SMB amusement park!

Thanks again Sharp Dressed Matt and The Rickster!

Ned “Got it in one take” Pyle

Offline Files is a feature in Windows that we have been improving upon since Windows XP. Many of our customers, especially large organizations, depend on Offline Files to provide access to the user data on Windows clients. Work Folders is a new solution we introduced in Windows 8.1 and is now also available for Windows 7, iPad and iPhone. After discussing both solutions with some customers, I see the need to have more in depth comparison between them, and this blog post will be focused on that.  

Why Offline Files?

Home folders and My Documents make it easier for an administrator to back up user files and manage user accounts by collecting many or all of a user's files in one location, but users must be connected to the corporate network in order to access those data. For mobile users, online connections are not always possible, the need to access files while the device is not connected to the network became a priority. Offline files was introduced to serve that need. It intercepts the API call to access the files on the file server, and get the data in the local device cache.

Why not Offline Files?

Most issues I’ve heard from the customers are result of the data stored in a hidden cache. The user view of the content is hosted on a file server, the cache is only used when the device is not connected to the corporate network. There is no easy way of figuring what’s in the cache, where is the cache pointing to, extracting data in the cache, permission changes etc. In addition, user may see different content depending on the device is online/offline. The list grows. Took these lessons, Work Folders stores the files in the local folder path, this made the user experience much simpler, as the file can be viewed/accessed just like any other local files.

Why Work Folders?

In addition to the “no more hidden cache” difference, we observed the trend with proliferation of devices in the consumer world, and anticipate an increasing demand to access the user data on file servers on those unmanaged devices which are not connected to the corporate network. With the goal to provide consistent user experience across different devices to access user data from anywhere, here are 3 main advantages Work Folders provides:

1. Access data over the internet. No more VPN or direct access to the corpnet for data access.

2. Access from non-Windows devices. In addition to Windows releases, Work Folders is also enabled for iPAD and iPhone.

3. Simple, local file access experience for users. No more hidden caches. Files under the Work Folders path are truly local files. The sync engine will keep the files in sync with the file server.

Data security with Work Folders

When we presented the Work Folders at the TechEd a couple of years ago, we demoed a number of scenarios to show Work Folders security features, as well as how Work Folders can integrate with other file server solutions such as FCI/RMS encryption to protect the data. You can find the video here. In this blog post, I’ll high light the security features we designed in Work Folders.

Data protection (encryption and wipe)

Data leakage has always been a concern for the Enterprise customers, to better protect the data on unmanaged devices, we provided the option for the admins to encrypt the data on the user device through a policy configuration on the server. The encryption key is tied with an Enterprise ID, and it is different from the encryption key if user wants to use EFS for data encryption on the same device. This separation allows admin to issue wipe to a device, which will only revoke the enterprise key, and leaving personal encrypted data intact.

Device password enforcement

Usually, password policy is enforced through group policy (GP), and GP can’t reach to those unmanaged devices, we provided a basic password policy to admins who want to ensure the devices are protected with password if they want to use the device for data access.

Encryption on the wire with https

With Work Folders, data transfer can happen on the corpnet or over the internet. To ensure data security, data will be encrypted on the wire using SSL. This adds additional file server managements, such as publishing the sync server URL for client to connect; adding web server certificate to have secure transfer.

Folder Redirection (FR) or not?

Folder Redirection is a feature allowing the content under the special folders (e.g. Documents, Desktop) to be stored on a file server. Offline Files is mostly used in combination with Folder Redirection to allow user or application offline access to data under the special folders.

When we were designing Work Folders, we deliberately moved away from the special folders, and introduced a standalone folder “Work Folders” which can be access on Windows as well as non-Windows devices. No matter what type of device the user is using, he/she can simply put all work related stuff under this folder, and it will simply get synced across to any other device which has Work Folders configured.

Early customer feedback has been a mixed. I have seen customers

• who are introducing Work Folders in parallel with FR and Offline Files, and Work Folders will sync a different set of files, and no need with integration with FR;

• who are thinking of replacing Offline Files with Work Folders, and strongly requesting FR the content of the special folders into Work Folders.

What do you vote for? http://windowsserver.uservoice.com/forums/295056-storage/suggestions/8342490-support-folder-redirection-of-well-known-folders-t

Conclusion

The world is changing, people want to get access to their data anytime anywhere. Has this change come to you yet? Share with us on your thoughts about Work Folders. Add your request here: http://windowsserver.uservoice.com/forums/295056-storage/category/121120-work-folders

Hi folks, Ned here again. Do you want your server OS deployment and servicing to move faster? We're a team of Microsoft engineers who want your experiences and ideas around solving real problems of deploying and servicing your server OS infrastructure. We prefer that you don't love server OS deployment already, and we’re interested even if you don’t use Windows Server. We need to learn it and earn it.

Click the link below if you wish to fill out a brief survey and perhaps participate in a short phone call.

https://aka.ms/deployland

Thanks, and talk to you soon,

Ned “virtual teams are fun” Pyle

Before I start

If you are not familiar with or not currently using Folder Redirection, you can simply ignore this blog post. It’s not my intention to evangelize Folder Redirection with Work Folders. This blog post is only for enterprises that have specific dependencies (such as work flows) on these Windows special folders and want to continue redirecting them as they migrate to Work folders.

Why and why not?

I often get questions on why we didn’t integrate Folder Redirection with Work Folders. When we were designing Work Folders, we debated this topic at length. There is no simple right or wrong, and it really depends on the individual customer IT culture. I’d like to share our thought on this, and open to hear yours.

We started by looking at the other file sync technologies offered at the time, which had a simple user experience by designate a specific folder which enables user to access the same content no matter which device they are using. We wanted to provide a similar user experience: a consistent view no matter on what devices. Because Work Folders is designed to work with non-domain joined devices, and non-Windows device (iPAD and iPhone for now), user can access data on their home PC, iOS devices, if using Folder Redirection on work PCs, imagine how confusing it could be to remember the different paths for file access on different devices, why not avoid it?

As we hear from customers requesting Folder Redirection integration with Work Folders, one common pattern emerges among these customers. They operate a very tightly controlled client environment: where the clients are all domain joined Windows PCs; IT doesn’t allow BYOD in the environment; or not allow those devices to access corporate files. For these customers, they like Work Folders’ file sync experience, but their users are accustomed to use Windows special folders or they may even have applications that default to save to those locations. And since they aren’t embracing personal devices, there is no confusion about access of those data.

How?

Folder Redirection setting is mostly managed through Group policy. User can also manually configure Folder Redirection on the client, but that solution is not scalable nor manageable. If you want to simply test out the user experience without the hassle of creating GPOs, you can open File Explorer, right click on one of the special folders, such as Documents, open Properties, click on the Location tab and to define the new location.

Group policy

I’ve tried the following 2 examples in my test environment; you need to change to your environment variables.

1. Redirecting the Document folder to a subfolder under Work Folders path. Use this configuration if you want to redirect more than one special folders under Work Folders.

2. Redirecting the Document folder to Work Folders. Use this configuration if you only want to redirect the document folder.

For the following GPOs, the assumption is, Work Folders has been already configured on the client, and your client setup for Work Folders is using the default path: C:\users\%username%\Work Folders.

Redirecting multiple special folders to Work Folders

In this example, I’m only showing redirecting Documents folder, but you can configure the other special folders by changing the GPO for other special folders. Make sure you redirect the special folders to a subfolder under the Work Folders path.

Create a GPO, and navigate to User Configuration -> Windows Settings -> Folder Redirection -> Documents, change

• the settings to: Basic – Redirect everyone’s folder to the same location;

• Target folder location: Redirect to the following location

• Root Path: %systemdrive%\users\%username%\Work Folders\Documents

Once the user gets the GPO, the documents folder will be redirected to a subfolder to the Work Folders.

Redirecting document to Work Folders

In this example, you only want to redirect the Documents folder, and nothing else. You can map point the root path in the GPO to the Work Folders itself. User can get a consistent view whether to access through Documents folder or Work Folders path.

Create a GPO, and navigate to User Configuration -> Windows Settings -> Folder Redirection -> Documents, change

• the settings to: Basic – Redirect everyone’s folder to the same location;

• Target folder location: Redirect to the following location

• Root Path: %systemdrive%\users\%username%\Work Folders

Once the user gets the GPO, the documents folder will be redirected to the Work Folders. Please note that, in Windows File explorer, user will not see the Work Folders folder. The folder is present in cmd window. User will access the files through Documents folder.

Which folders to be redirected?

There are 13 folders which can be configured using the group policy, but not all folders are suitable to be redirected to Work Folders. For example, AppData/roaming folder is designed to store application setting data, files are likely to be opened by the applications, and Work Folders client will not be able to sync open files. I have tested redirecting the following folders as subfolders under Work Folders:

1. Documents

2. Pictures

3. Videos

4. Music

5. Downloads

6. Desktop

Conclusion

Although this blog post covers how you can redirect some Windows special folders to Work Folders, I’d like to raise the awareness again that doing so can introduce user confusion once IT starts to embrace BYOD, as user will need to remember different file access path on different devices. However, if your environment does require this, let’s explore. What I have covered here is the very basic configuration. I’d like to see your comments to learn your needs, what works, and what’s not.

Are you currently using a file replication service (DFS-R or anything else) to keep files in sync across multiple locations? We’re exploring options to bring you a solution that radically streamlines the file replication process for you and we’d like the opportunity to make sure we’re building a solution that meets your needs.

If you’re open to a short discussion with our product team, please complete this 2 minute form.

http://aka.ms/hfsurvey

Thank you.