Hi folks, Ned here again. If you were smart/cool/lucky enough to land some Microsoft Ignite tickets for next week, here’s the nicely organized list of all the Windows Server 2016 sessions. Color-coding, filters, it’s very sharp.
Naturally, the killer session you should register for is Drill into Storage Replica in Windows Server 2016. I hear the presenter kicks ass and has swag for attendees.
– Ned “not so humble brag” Pyle
Hi all,
I’m Jeff Patterson, Program Manager for Work Folders.
I wanted to let you know that Digest authentication does not work on iOS 10. Please review the issue details below if you’re currently using the Work Folders iOS client in your environment.
After upgrading to iOS 10, Work Folders fails with the following error after user credentials are provided:
Check your user name and password
There’s a bug in iOS 10 which causes Digest authentication to fail.
We’re working with Apple to resolve the issue.
I will provide an update once this issue is resolved.
Thanks,
Jeff
This tiny two-server cluster packs powerful compute and spacious storage into one cubic foot.
The Challenge
In the Windows Server team, we tend to focus on going big. Our enterprise customers and service providers are increasingly relying on Windows as the foundation of their software-defined datacenters, and needless to say, our hyperscale public cloud Azure does too. Recent big announcements like support for 24 TB of memory per server with Hyper-V, or 6+ million IOPS per cluster with Storage Spaces Direct, or delivering 50 Gb/s of throughput per virtual machine with Software-Defined Networking are the proof.
But what can these same features in Windows Server do for smaller deployments? Those known in the IT industry as Remote-Office / Branch-Office (“ROBO”) – think retail stores, bank branches, private practices, remote industrial or constructions sites, and more. After all, their basic requirement isn’t so different – they need high availability for mission-critical apps, with rock-solid storage for those apps. And generally, they need it to be local, so they can operate – process transactions, or look up a patient’s records – even when their Internet connection is flaky or non-existent.
For these deployments, cost is paramount. Major retail chains operate thousands, or tens of thousands, of locations. This multiplier makes IT budgets extremely sensitive to the per-unit cost of each system. The simplicity and savings of hyper-convergence – using the same servers to provide compute and storage – present an attractive solution.
With this in mind, under the auspices of Project Kepler-47, we set about going small…
Meet Kepler-47
The resulting prototype – and it’s just that, a prototype – was revealed at Microsoft Ignite 2016.
Kepler-47 on expo floor at Microsoft Ignite 2016 in Atlanta.
In our configuration, this tiny two-server cluster provides over 20 TB of available storage capacity, and over 50 GB of available memory for a handful of mid-sized virtual machines. The storage is flash-accelerated, the chips are Intel Xeon, and the memory is error-correcting DDR4 – no compromises. The storage is mirrored to tolerate hardware failures – drive or server – with continuous availability. And if one server goes down or needs maintenance, virtual machines live migrate to the other server with no appreciable downtime.
(Did we mention it also has not one, but two 3.5mm headphone jacks? Hah!)
Kepler-47 is 45% smaller than standard 2U rack servers.
In terms of size, Kepler-47 is barely one cubic foot – 45% smaller than standard 2U rack servers. For perspective, this means both servers fit readily in one carry-on bag in the overhead bin!
We bought (almost) every part online at retail prices. The total cost for each server was just $1,101. This excludes the drives, which we salvaged from around the office, and which could vary wildly in price depending on your needs.
Each Kepler-47 server cost just $1,101 retail, excluding drives.
Technology
Kepler-47 is comprised of two servers, each running Windows Server 2016 Datacenter. The servers form one hyper-converged Failover Cluster, with the new Cloud Witness as the low-cost, low-footprint quorum technology. The cluster provides high availability to Hyper-V virtual machines (which may also run Windows, at no additional licensing cost), and Storage Spaces Direct provides fast and fault tolerant storage using just the local drives.
Additional fault tolerance can be achieved using new features such as Storage Replica with Azure Site Recovery.
Notably, Kepler-47 does not use traditional Ethernet networking between the servers, eliminating the need for costly high-speed network adapters and switches. Instead, it uses Intel Thunderbolt™ 3 over a USB Type-C connector, which provides up to 20 Gb/s (or up to 40 Gb/s when utilizing display and data together!) – plenty for replicating storage and live migrating virtual machines.
To pull this off, we partnered with our friends at Intel, who furnished us with pre-release PCIe add-in-cards for Thunderbolt™ 3 and a proof-of-concept driver.
Kepler-47 does not use traditional Ethernet between the servers; instead, it uses Intel Thunderbolt™ 3.
To our delight, it worked like a charm – here’s the Networks view in Failover Cluster Manager. Thanks, Intel!
The Networks view in Failover Cluster Manager, showing Thunderbolt™ Networking.
While Thunderbolt™ 3 is already in widespread use in laptops and other devices, this kind of server application is new, and it’s one of the main reasons Kepler-47 is strictly a prototype. It also boots from USB 3 DOM, which isn’t yet supported, and has no host-bus adapter (HBA) nor SAS expander, both of which are currently required for Storage Spaces Direct to leverage SCSI Enclosure Services (SES) for slot identification. However, it otherwise passes all our validation and testing and, as far as we can tell, works flawlessly.
(In case you missed it, support for Storage Spaces Direct clusters with just two servers was announced at Ignite!)
Parts List
Ok, now for the juicy details. Since Ignite, we have been asked repeatedly what parts we used. Here you go:
The key parts of Kepler-47.
| Function | Product | View Online | Cost |
| Motherboard | ASRock C236 WSI | Link | $199.99 |
| CPU | Intel Xeon E3-1235L v5 25w 4C4T 2.0Ghz | Link | $283.00 |
| Memory | 32 GB (2 x 16 GB) Black Diamond ECC DDR4-2133 | Link | $208.99 |
| Boot Device | Innodisk 32 GB USB 3 DOM | Link | $29.33 |
| Storage (Cache) | 2 x 200 GB Intel S3700 2.5” SATA SSD | Link | – |
| Storage (Capacity) | 6 x 4 TB Toshiba MG03ACA400 3.5” SATA HDD | Link | – |
| Networking (Adapter) | Intel Thunderbolt™ 3 JHL6540 PCIe Gen 3 x4 Controller Chip | Link | – |
| Networking (Cable) | Cable Matters 0.5m 20 Gb/s USB Type-C Thunderbolt™ 3 | Link | $17.99* |
| SATA Cables | 8 x SuperMicro CBL-0481L | Link | $13.20 |
| Chassis | U-NAS NSC-800 | Link | $199.99 |
| Power Supply | ASPower 400W Super Quiet 1U | Link | $119.99 |
| Heatsink | Dynatron K2 75mm 2 Ball CPU Fan | Link | $34.99 |
| Thermal Pads | StarTech Heatsink Thermal Transfer Pads (Set of 5) | Link | $6.28* |
* Just one needed for both servers.
Practical Notes
The ASRock C236 WSI motherboard is the only one we could locate that is mini-ITX form factor, has eight SATA ports, and supports server-class processors and error-correcting memory with SATA hot-plug. The E3-1235L v5 is just 25 watts, which helps keep Kepler-47 very quiet. (Dan has been running it literally on his desk since last month, and he hasn’t complained yet.)
Having spent all our SATA ports on the storage, we needed to boot from something else. We were delighted to spot the USB 3 header on the motherboard.
The U-NAS NSC-800 chassis is not the cheapest option. You could go cheaper. However, it features an aluminum outer casing, steel frame, and rubberized drive trays – the quality appealed to us.
We actually had to order two sets of SATA cables – the first were not malleable enough to weave their way around the tight corners from the board to the drive bays in our chassis. The second set we got are flat and 30 AWG, and they work great.
Likewise, we had to confront physical limitations on the heatsink – the fan we use is barely 2.7 cm tall, to fit in the chassis.
We salvaged the drives we used, for cache and capacity, from other systems in our test lab. In the case of the SSDs, they’re several years old and discontinued, so it’s not clear how to accurately price them. In the future, we imagine ROBO deployments of Storage Spaces Direct will vary tremendously in the drives they use – we chose 4 TB HDDs, but some folks may only need 1 TB, or may want 10 TB. This is why we aren’t focusing on the price of the drives themselves – it’s really up to you.
Finally, the Thunderbolt™ 3 controller chip in PCIe add-in-card form factor was pre-release, for development purposes only. It was graciously provided to us by our friends at Intel. They have cited a price-tag of $8.55 for the chip, but not made us pay yet. 
Takeaway
With Project Kepler-47, we used Storage Spaces Direct and Windows Server 2016 to build an unprecedentedly low-cost high availability solution to meet remote-office, branch-office needs. It delivers the simplicity and savings of hyper-convergence, with compute and storage in a single two-server cluster, with next to no networking gear, that is very budget friendly.
Are you or is your organization interested in this type of solution? Let us know in the comments!
// Cosmos Darwin (@CosmosDarwin), Dan Lovinger, and Claus Joergensen (@ClausJor)
In a previous blog post, I discussed setting up the Windows Standards-Based Storage Management Service (referred to below as Storage Service) on Windows Server 2012 R2. For Windows Server 2016 and System Center 2016 Virtual Machine Manager, configuration is much simpler since installation of the service includes setting up the necessary self-signed certificate. We also allow using CA signed certificates now provided the Common Name (CN) is “MSSTRGSVC”.
Before I get into those changes, I want to talk about the Transport Layer Security 1.2 (TLS 1.2) protocol, which is now a required part of the Storage Management Initiative Specification (SMI-S).
Secure communication through the Hyper Text Transport Protocol (HTTPS) is accomplished using the encryption capabilities of Transport Layer Security, which is itself an update to the much older Security Sockets Layer protocol (SSL) – although still commonly called Secure Sockets. Over the years, several vulnerabilities in SSL and TLS have been exposed, making earlier versions of the protocol insecure. TLS 1.2 is the latest version of the protocol and is defined by RFC 5246.
The Storage Networking Industry Association (SNIA) made TLS 1.2 a mandatory part of SMI-S (even retroactively). In 2015, the International Standards Organization (ISO) published ISO 27040:2015 “Information Technology – Security Techniques – Storage Security”, and this is incorporated by reference into the SMI-S protocol and pretty much all things SNIA.
Even though TLS 1.2 was introduced in 2008, it’s uptake was impeded by interoperability concerns. Adoption was accelerated after several exploits (e.g., BEAST) ushered out the older SSL 3.0 and TLS 1.0 protocols (TLS 1.1 did not see broad adoption). Microsoft Windows offered support for TLS 1.2 beginning in Windows 7 and Windows Server 2008 R2. That being said, there were still a lot of interop issues at the time, and TLS 1.1 and 1.2 support was hidden behind various registry keys.
Now it’s 2016, and there are no more excuses for using older, proven-insecure protocols, so it’s time to update your SMI-S providers. But unfortunately, you still need to take action to fully enable TLS 1.2. There are three primary Microsoft components that are used by the Storage Service which affect HTTPS communications between providers and the service: SCHANNEL, which implements the SSL/TLS protocols; HTTP.SYS, an HTTP server used by the Storage Service to support indications; and .NET 4.x, used by Virtual Machine Manager (VMM) (not by the Storage Service itself).
I’m going to skip some of the details of how clients and servers negotiate TLS versions (this may or may not allow older versions) and cipher suites (the most secure suite mutually agreed upon is always selected, but refer to this site for a recent exploit involving certain cipher suites).
How certificates are validated varies depending on whether the certificate is self-signed or created by a trusted Certificate Authority (CA). For the most part, SMI-S will use self-signed certificates – and providers should never, ever, be exposed to the internet or another untrusted network. A quick overview:
A CA signed certificate contains a signature that indicates what authority signed it. The user of that certificate will be able to establish a chain of trust to a well-known CA.
A self-signed certificate needs to establish this trust in some other way. Typically, the self-signed certificate will need to be loaded into a local certificate store on the system that will need to validate it. See below for more on this.
In either case, the following conditions must be true: the certificate has not expired; the certificate has not been revoked (look up Revocation List for more about this); and the purpose of the certificate makes sense for its use. Additional checks include “Common Name” matching (disabled by default for the Storage Service; must not be used by providers) and key length. Note that we have seen issues with certificates being valid “from” a time and there is a time mismatch between the provider and the storage service. These tend to cure themselves once the start time has been passed on both ends of the negotiation. When using the Windows PowerShell cmdlet Register-SmisProvider you will see this information.
In some instances, your provider may ignore one or more of the validation rules and just accept any certificate that we present. A useful debugging approach but not very secure!
One more detail: when provisioning certificates for the SMI-S providers, make sure they use key lengths of 1024 or 2048 bits only. 512 bit keys are no longer supported due to recent exploits. And odd length keys won’t work either. At least I have never seen them work, even though technically allowed.
This article will discuss Windows Server and System Center Releases, and the .NET Framework. It should not be necessary to mess with registry settings that control cipher suites or SSL versions except as noted below for the .NET framework.
Since the initial releases of these products, there have been many security fixes released as patches, and more than a few of them changed SCHANNEL and HTTP.SYS behavior. Rather than attempt to enumerate all of the changes, let’s just say it is essential to apply ALL security hotfixes.
If you are using Windows Server 2016 RTM, you also need to apply all available.
There is no .NET dependency.
SC 2012 R2 VMM uses the .NET runtime library but the Storage Service does not. If you are using VMM 2012 R2, to fully support TLS 1.2, the most recent version of .NET 4.x should be installed; this is currently .NET 4.6.2. Also, update VMM to the latest Update Release.
If, for some reason, you must stay on .NET 4.5.2, then a registry change will be required to turn on TLS 1.2 on the VMM Server(s) since by default, .NET 4.5.2 only enables SSL 3.0 and TLS 1.0.
The registry value (which changes to allow TLS 1.0, TLS 1.1 and TLS 1.2 and not SSL 3.0 which you should never use anyway) is:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319 “SchUseStrongCrypto”=dword:00000001
You can use this PowerShell command to change the behavior:
Set-ItemProperty -Path “HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319” -Name “SchUseStrongCrypto” -Value “1” -Force
(Note that the version number highlighted applies regardless of a particular release of .NET 4.5; do not change it!)
This change will apply to every application using the .NET 4.x runtime on the same system. Note that Exchange 2013 does not support 4.6.x, but you shouldn’t be running VMM and Exchange on the same server anyway! Again, apply this to the VMM Server system or VM, which may not be the same place you are running the VMM UI.
VMM 2016 uses .NET 4.6.2; no changes required.
Repeating the information from a previous blog, follow these steps on the VMM Server machine:
Note: if you deployed VMM in a HA configuration, you will need to repeat these steps on each VMM Server instance. Your vendor’s SMI-S provider must support a certificate store that allows multiple certificates.
Microsoft is actively involved in SNIA plugfests and directly with storage vendors to ensure interoperability. Some providers may require settings to ensure the proper security protocols are enabled and used, and many require updates.
Many SMI-S providers and client applications rely on the open source project OpenSSL.
Storage vendors who use OpenSSL must absolutely keep up with the latest version(s) of this library and it is up to them to provide you with updates. We have seen a lot of old providers that rely on the long obsolete OpenSSL 0.9.8 releases or unpatched later versions. Microsoft will not provide any support if your provider is out-of-date, so if you have been lazy and not keeping up-to-date, time to get with the program. At the time of this writing there are three current branches of OpenSSL, each with patches to mend security flaws that crop up frequently. Consult the link above. How a provider is updated is a vendor-specific activity. (Some providers – such as EMC’s – do not use OpenSSL; check with the vendor anyway.)
This step will vary greatly among providers. You will need to consult the vendor documentation for how to import the certificate into their appropriate Certificate Store. If they do not provide a mechanism to import certificates, you will not be able to use fully secure indications or mutual authentication with certificate validation.
To ensure you are using TLS 1.2 (and enabling indications), you must do the following:
Howdy, Claus here again, this time with Dan Lovinger.
At our recent Ignite conference we had some very exciting results and experiences to share around Storage Spaces Direct and Windows Server 2016. One of the more exciting ones that you may have missed was an experiment we did on a set of systems built with the help of Mellanox and Hewlett-Packard Enterprise’s NVDIMM-N technology.
What’s exciting about NVDIMM-N is that it is part of the first wave of new memory technologies referred to as Persistent Memory (PM), sometimes also referred to as Storage Class Memory (SCM ). A PM device offers persistent storage – stays around after the server resets or the power drops – but can be on the super high speed memory bus, and accessible at the granularity (bytes not blocks!) and latencies we’re more familiar with for memory. In the case of NVDIMM-N it is literally memory (DRAM) with the addition of natively persistent storage, usually NAND flash, and some power capacity and to allow capture of the DRAM to that persistent storage regardless of conditions.
These 8 HPE ProLiant DL380 Gen9 nodes had Mellanox CX-4 100Gb adapters connected through a Mellanox Spectrum switch and 16 8GiB NVDIMM-N modules along with 4 NVMe flash drives – each – for an eye-watering 1TiB of NVDIMM-N around the cluster.
Of course, being storage nerds, what did we do: we created three-way mirrored Storage Spaces Direct virtual disks over each type of storage – NVMe and, in their block personality, the NVDIMM-N – and benched them off. Our partners in SQL Server showed it like this:
What we’re seeing here are simple, low intensity DISKSPD loads – equal in composition – which lets us highlight the relative latencies of each type of storage. In the first pair of 64K IO tests we see the dramatic difference which gets PM up to the line rate of the 100Gb network before NVME is even 1/3rd of the way there. In the second we can see how PM neutralizes the natural latency of going all the way into a flash device – even as efficient and high speed as our NVMe devices were – and provides reads at less than 180us to the 99th percentile – 99% of the read IO was over three times faster for three-way mirrored, two node fault tolerant storage!
We think this is pretty exciting! Windows Server is on a journey to integrate Persistent Memory and this is one of the steps along the way. While we may do different things with it in the future, this was an interesting experiment to point to where we may be able to go (and more!).
Let us know what you think.
Claus and Dan.
p.s. if you’d like to see the entire SQL Server 2016 & HPE Persistent Memory presentation at Ignite (video available!), follow this link: https://myignite.microsoft.com/sessions/2767
Hi folks, Ned here again with another quickie engineering survey. As always, it’s anonymous, requires no registration, and should take no more than 30 seconds. We’d like to learn about your server firewalling, plus a couple drill-down questions.
Survey: What percentage of your Windows Servers have Internet access?
This may require some uncomfortable admissions, but it’s for a good cause, I promise. Honesty is always the best policy in helping us make better software for you.
Note: for a handful of you early survey respondents, the # of servers question had the wrong limit. It’s fixed now and you can adjust up.
– Ned “census taker” Pyle
Hi all,
I’m Jeff Patterson, Program Manager for Work Folders.
We’re excited to announce that we’ve released an updated version of Work Folders for Android to the Google Play Store which enables users to sync files that were created or edited on their Android device.
Work Folders is a Windows Server feature since 20012 R2 that enables individual employees to access their files securely from inside and outside the corporate environment. The Work Folders app connects to the server and enables file access on your Android phone and tablet. Work Folders enables this while allowing the organization’s IT department to fully secure that data.
Using the latest version of Work Folders for Android, users can now:
For the complete list of Work Folders for Android features, please reference the feature list section below.
If you’re interested in learning more about Work Folders, here are some great resources:
Introduction and Getting Started
Advanced Work Folders Deployment and Management
Videos
// This post was written by Dan Lovinger, Principal Software Engineer.
Howdy,
In the weeks since the release of Windows Server 2016, the amount of interest we’ve seen in Storage Spaces Direct has been nothing short of spectacular. This interest has translated to many potential customers looking to evaluate Storage Spaces Direct.
Windows Server has a strong heritage with do-it-yourself design. We’ve even done it ourselves with the Project Kepler-47 proof of concept! While over the coming months there will be many OEM-validated solutions coming to market, many more experimenters are once again piecing together their own configurations.
This is great, and it has led to a lot of questions, particularly about Solid-State Drives (SSDs). One dominates: “Is [some drive] a good choice for a cache device?” Another comes in close behind: “We’re using [some drive] as a cache device and performance is horrible, what gives?”
The flash translation layer masks a variety of tricks an SSD can use to accelerate performance and extend its lifetime, such as buffering and spare capacity.
Some background on SSDs
As I write this in late 2016, an SSD is universally a device built from a set of NAND flash dies connected to an internal controller, called the flash translation layer (“FTL”).
NAND flash is inherently unstable. At the physical level, a flash cell is a charge trap device – a bucket for storing electrons. The high voltages needed to trigger the quantum tunneling process that moves electrons in and out of the cell – your data – slowly accumulates damage at the atomic level. Failure does not happen all at once. Charge degrades in-place over time and even reads aren’t without cost, a phenomenon known as read disturb.
The number of electrons in the cell’s charge trap translate to a measurable voltage. At its most basic, a flash cell stores one on/off bit – a single level cell (SLC) – and the difference between 0 and 1 is “easy”. There is only one threshold voltage to consider. On one side the cell represents 0, on the other it is 1.
However, conventional SSDs have moved on from SLC designs. Common SSDs now store two (MLC) or even three (TLC) bits per cell, requiring four (00, 01, 10, 11) or eight (001, 010, … 110, 111) different charge levels. On the horizon is 4 bit QLC NAND, which will require sixteen! As the damage accumulates it becomes difficult to reliably set charge levels; eventually, they cannot store new data. This happens faster and faster as bit densities increase.
The FTL has two basic defenses.
Both defenses work like a bank account.
Over the short term, some amount of the ECC is needed to recover the data on each read. Lightly-damaged cells or recently-written data won’t draw heavily on ECC, but as time passes, more of the ECC is necessary to recover the data. When it passes a safety margin, the data must be re-written to “refresh” the data and ECC, and the cycle continues.
Across a longer term, the over-provisioning in the device replaces failed cells and preserves the apparent capacity of the SSD. Once this account is drawn down, the device is at the end of its life.
To complete the physical picture, NAND is not freely writable. A die is divided into what we refer to as program/erase “P/E” pages. These are the actual writable elements. A page must first be erased to prepare writing it, then the entire page can be written at once. A page may be as small as 16K, or potentially much larger. Any one single write that arrives in the SSD probably won’t line up with the page size!
And finally, NAND never re-writes in place. The FTL is continuously keeping track of wear, preparing fresh erased pages, and consolidating valid data sitting in pages alongside stale data corresponding to logical blocks which have already been re-written. These are additional reasons for over-provisioning.
In consumer devices, and especially in mobile, an SSD can safely leverage an unprotected, volatile cache because the device’s battery ensures it will not unexpectedly lose power. In servers, however, an SSD must provide its own power protection, typically in the form of a capacitor.
Buffers and caches
The bottom line is that a NAND flash SSD is a complex, dynamic environment and there is a lot going on to keep your data safe. As device densities increase, it is getting ever harder. We must maximize the value of each write, as it takes the device one step closer to failure. Fortunately, we have a trick: a buffer.
A buffer in an SSD is just like the cache in the system that surrounds it: some memory which can accumulate writes, allowing the user/application request to complete while it gathers more and more data to write efficiently to the NAND flash. Many small operations turn into a small number of larger operations. Just like the memory in a conventional computer, though, on its own that buffer is volatile – if a power loss occurs, any pending write operations are lost.
Losing data is, of course, not acceptable. Storage Spaces Direct is at the far end of a series of actions which have led to it getting a write. A virtual machine on another computer may have had an application issue a flush which, in a physical system, would put the data on stable storage. After Storage Spaces Direct acknowledges any write, it must be stable.
How can any SSD have a volatile cache!? Simple, and it is a crucial detail of how the SSD market has differentiated itself: you are very likely reading this on a device with a battery! Consumer flash is volatile in the device but not volatile when considering the entire system – your phone, tablet or laptop. Making a cache non-volatile requires some form of power storage (or new technology …), which adds unneeded expense in the consumer space.
What about servers? In the enterprise space, the cost and complexity of providing complete power safety to a collection of servers can be prohibitive. This is the design point enterprise SSDs sit in: the added cost of internal power capacity to allow saving the buffer content is small.
An (older) enterprise-grade SSD, with its removable and replaceable built-in battery!
This newer enterprise-grade SSD, foreground, uses a capacitor (the three little yellow things, bottom right) to provide power-loss protection.
Along with volatile caches, consumer flash is also universally of lower endurance. A consumer device targets environments with light activity. Extremely dense, inexpensive, fragile NAND flash – which may wear out after only a thousand writes – could still provide many years of service. However, expressed in total writes over time or capacity written per day, a consumer device could wear out more than 10x faster than available enterprise-class SSD.
So, where does that leave us? Two requirements for SSDs for Storage Spaces Direct. One hard, one soft, but they normally go together:
But … could I get away with it? And more crucially – for us – what happens if I just put a consumer-grade SSD with a volatile write cache in a Storage Spaces Direct system?
An experiment with consumer-grade SSDs
For this experiment, we’ll be using a new-out-of-box 1 TB consumer class SATA SSD. While we won’t name it, it is a first tier, high quality, widely available device. It just happens to not be appropriate for an enterprise workload like Storage Spaces Direct, as we’ll see shortly.
In round numbers, its data sheet says the following:
Note: QD (“queue depth”) is geek-speak for the targeted number of IOs outstanding during a storage test. Why do you always see 32? That’s the SATA Native Command Queueing (NCQ) limit to which commands can be pipelined to a SATA device. SAS and especially NVME can go much deeper.
Translating the endurance to the widely-used device-writes-per-day (DWPD) metric, over the device’s 5-year warranty period that is
185 TB / (365 days x 5 years = 1825 days) = ~ 100 GB writable per day
100 GB / 1 TB total capacity = 0.10 DWPD
The device can handle just over 100 GB each day for 5 years before its endurance is exhausted. That’s a lot of Netflix and web browsing for a single user! Not so much for a large set of virtualized workloads.
To gather the data below, I prepared the device with a 100 GiB load file, written through sequentially a little over 2 times. I used DISKSPD 2.0.18 to do a QD8 70:30 4 KiB mixed read/write workload using 8 threads, each issuing a single IO at a time to the SSD. First with the write buffer enabled:
diskspd.exe -t8 -b4k -r4k -o1 -w30 -Su -D -L -d1800 -Rxml Z:\load.binNormal unbuffered IO sails along, with a small write cliff.
The first important note here is the length of the test: 30 minutes. This shows an abrupt drop of about 10,000 IOPS two minutes in – this is normal, certainly for consumer devices. It likely represents the FTL running out of pre-erased NAND ready for new writes. Once its reserve runs out, the device runs slower until a break in the action lets it catch back up. With web browsing and other consumer scenarios. the chances of noticing this are small.
An aside: this is a good, stable device in each mode of operation – behavior before and after the “write cliff” is very clean.
Second, note that the IOPS are … a bit different than the data sheet might have suggested, even before it reaches steady operation. We’re intentionally using a light, QD8 70:30 4K to drive it more like a generalized workload. It still rolls over the write cliff. Under sustained, mixed IO pressure the FTL has much more work to take care of and it shows.
That’s with the buffer on, though. Now just adding write-through (with -Suw):
diskspd.exe -t8 -b4k -r4k -o1 -w30 -Suw -D -L -d1800 -Rxml Z:\load.binWrite-through IO exposes the true latency of NAND, normally masked by the FTL/buffer.
Wow!
First: it’s great that the device honors write-through requests. In the consumer space, this gives an application a useful tool for making data durable when it must be durable. This is a good device!
Second, oh my does the performance drop off. This is no longer an “SSD”: especially as it goes over the write cliff – which is still there – it’s merely a fast HDD, at about 220 IOPS. Writing NAND is slow! This is the FTL forced to push all the way into the NAND flash dies, immediately, without being able to buffer, de-conflict the read and write IO streams and manage all the other background activity it needs to do.
Third, those immediate writes take what is already a device with modest endurance and deliver a truly crushing blow to its total lifetime.
Crucially, this is how Storage Spaces Direct would see this SSD. Not much of a “cache” anymore.
So, why does a non-volatile buffer help?
It lets the SSD claim that a write is stable once it is in the buffer. A write-through operation – or a flush, or a request to disable the cache – can be honored without forcing all data directly into the NAND. We’ll get the good behavior, the stated endurance, and the data stability we require for reliable, software-defined storage to a complex workload.
In short, your device will behave much as we saw in the first chart: a nice, flat, fast performance profile. A good cache device. If it’s NVMe it may be even more impressive, but that’s a thought for another time.
Finally, how do you identify a device with a non-volatile buffer cache?
Datasheet, datasheet, datasheet. Look for language like:
… along with many others across the industry. You should be able to find a device from your favored provider. These will be more expensive than consumer grade devices, but hopefully we’ve convinced you why they are worth it.
Be safe out there!
/ Dan Lovinger
Hi! I’m Cosmos. Follow me on Twitter @cosmosdarwin.
The storage pool is the collection of physical drives which form the basis of your software-defined storage. Those familiar with Storage Spaces in Windows Server 2012 or 2012R2 will remember that pools took some managing – you had to create and configure them, and then manage membership by adding or removing drives. Because of scale limitations, most deployments had multiple pools, and because data placement was essentially static (more on this later), you couldn’t really expand them once created.
We’re introducing some exciting improvements in Windows Server 2016.
With Storage Spaces Direct, we now support up to 416 drives per pool, the same as our per-cluster maximum, and we strongly recommend you use exactly one pool per cluster. When you enable Storage Spaces Direct (as with the Enable-ClusterS2D cmdlet), this pool is automatically created and configured with the best possible settings for your deployment. Eligible drives are automatically discovered and added to the pool and, if you scale out, any new drives are added to the pool too, and data is moved around to make use of them. When drives fail they are automatically retired and removed from the pool. In fact, you really don’t need to manage the pool at all anymore except to keep an eye on its available capacity.
Nonetheless, understanding how the pool works can help you reason about fault tolerance, scale-out, and more. So if you’re curious, read on!
To help illustrate certain key points, I’ve written a script (open-source, available at the end) which produces this view of the pool’s drives, organized by type, by server (‘node’), and by how much data they’re storing. The fastest drives in each server, listed at the top, are claimed for caching.
The storage pool forms the physical basis of your software-defined storage.
Let’s start with three servers forming one Storage Spaces Direct cluster.
Each server has 2 x 800 GB NVMe drives for caching and 4 x 2 TB SATA SSDs for capacity.
We can create our first volume (‘Storage Space’) and choose 1 TiB in size, two-way mirrored. This implies we will maintain two identical copies of everything in that volume, always on different drives in different servers, so that if hardware fails or is taken down for maintenance, we’re sure to still have access to all our data. Consequently, this 1 TiB volume will actually occupy 2 TiB of physical capacity on disk, its so-called ‘footprint’ on the pool.
Our 1 TiB two-way mirror volume occupies 2 TiB of physical capacity, its ‘footprint’ on the pool.
(Storage Spaces offers many resiliency types with differing storage efficiency. For simplicity, this blog will show two-way mirroring. The concepts we’ll cover apply regardless which resiliency type you choose, but two-way mirroring is by far the most straightforward to draw and explain. Likewise, although Storage Spaces offers chassis and/or rack awareness, this blog will assume the default server-level awareness for simplicity.)
Okay, so we have 2 TiB of data to write to physical media. But where will these two tebibytes of data actually land?
You might imagine that Spaces just picks any two drives, in different servers, and places the copies in whole on those drives. Alas, no. What if the volume were larger than the drive size? Okay, perhaps it spans several drives in both servers? Closer, but still no.
What actually happens can be surprising if you’ve never seen it before.
Storage Spaces starts by dividing the volume into many ‘slabs’, each 256 MB in size.
Storage Spaces starts by dividing the volume into many ‘slabs’, each 256 MB in size. This means our 1 TiB volume has some 4,000 such slabs!
For each slab, two copies are made and placed on different drives in different servers. This decision is made independently for each slab, successively, with an eye toward equilibrating utilization – you can think of it like dealing playing cards into equal piles. This means every single drive in the storage pool will store some copies of some slabs!
The placement decision is made independently for each slab, like dealing playing cards into equal piles.
This can be non-obvious, but it has some real consequences you can observe. For one, it means all drives in all servers will gradually “fill up” in lockstep, in 256 MB increments. This is why we rarely pay attention to how full specific drives or servers are – because they’re (almost) always (almost) the same!
Slabs of our two-way mirrored volume have landed on every drive in all three servers.
(For the curious reader: the pool keeps a sprawling mapping of which drive has each copy of each slab called the ‘pool metadata’ which can reach up to several gigabytes in size. It is replicated to at least five of the fastest drives in the cluster, and synchronized and repaired with the utmost aggressiveness. To my knowledge, pool metadata loss has never taken down an actual production deployment of Storage Spaces.)
This may seem complicated, and it is. So why do it? Two reasons.
First, striping every volume across every drive unlocks truly awesome potential for reads and writes – especially larger sequential ones – to activate many drives in parallel, vastly increasing IOPS and IO throughput. The unrivaled performance of Storage Spaces Direct compared to competing technologies is largely attributable to this fundamental design. (There is more complexity here, with the infamous column count and interleave you may remember from 2012 or 2012R2, but that’s beyond the scope of this blog. Spaces automatically sets appropriate values for these in 2016 anyway.)
(This is also why members of the core Spaces engineering team take some offense if you compare mirroring directly to RAID-1.)
The second is data safety – it’s related, but worth explaining in detail.
In Storage Spaces, when drives fail, their contents are reconstructed elsewhere based on the surviving copy or copies. We call this ‘repairing’, and it happens automatically and immediately in Storage Spaces Direct. If you think about it, repairing must involve two steps – first, reading from the surviving copy; second, writing out a new copy to replace the lost one.
Bear with me for a paragraph, and imagine if we kept whole copies of volumes. (Again, we don’t.) Imagine one drive has every slab of our 1 TiB volume, and another drive has the copy of every slab. What happens if the first drive fails? The other drive has the only surviving copy. Of every slab. To repair, we need to read from it. Every. Last. Byte. We are obviously limited by the read speed of that drive. Worse yet, we then need to write all that out again to the replacement drive or hot spare, where we are limited by its write speed. Yikes! Inevitably, this leads to contention with ongoing user or application IO activity. Not good.
Storage Spaces, unlike some of our friends in the industry, does not do this.
Consider again the scenario where some drive fails. We do lose all the slabs stored on that drive. And we do need to read from each slab’s surviving copy in order to repair. But, where are these surviving copies? They are evenly distributed across almost every other drive in the pool! One lost slab might have its other copy on Drive 15; another lost slab might have its other copy on Drive 03; another lost slab might have its other copy on Drive 07; and so on. So, almost every other drive in the pool has something to contribute to the repair!
Next, we do need to write out the new copy of each – where can these new copies be written? Provided there is available capacity, each lost slab can be re-constructed on almost any other drive in the pool!
(For the curious reader: I say almost because the requirement that slab copies land in different servers precludes any drives in the same server as the failure from having anything to contribute, read-wise. They were never eligible to get the other copy. Similarly, those drives in the same server as the surviving copy are ineligible to receive the new copy, and so have nothing to contribute write-wise. This detail turns out not to be terribly consequential.)
While this can be non-obvious, it has some significant implications. Most importantly, repairing data faster minimizes the risk that multiple hardware failures will overlap in time, improving overall data safety. It is also more convenient, as it reduces the ‘resync’ wait time during rolling cluster-wide updates or maintenance. And because the read/write burden is spread thinly among all surviving drives, the load on each drive individually is light, which minimizes contention with user or application activity.
For this to work, you need to set aside some extra capacity in the storage pool. You can think of this as giving the contents of a failed drive “somewhere to go” to be repaired. For example, to repair from one drive failure (without immediately replacing it), you should set aside at least one drive’s worth of reserve capacity. (If you are using 2 TB drives, that means leaving 2 TB of your pool unallocated.) This serves the same function as a hot spare, but unlike an actual hot spare, the reserve capacity is taken evenly from every drive in the pool.
Reserve capacity gives the contents of a failed drive “somewhere to go” to be repaired.
Reserving capacity is not enforced by Storage Spaces, but we highly recommend it. The more you have, the less urgently you will need to scramble to replace drives when they fail, because your volumes can (and will automatically) repair into the reserve capacity, completely independent of the physical replacement process.
When you do eventually replace the drive, it will automatically take its predecessor’s place in the pool.
Check out our capacity calculator for help with determining appropriate reserve capacity.
New in Windows 10 and Windows Server 2016, slabs and their copies can be moved around between drives in the storage pool to equilibrate utilization. We call this ‘optimizing’ or ‘re-balancing’ the storage pool, and it’s essential for scalability in Storage Spaces Direct.
For instance, what if we need to add a fourth server to our cluster?
Add-ClusterNode -Name <Name>
The new drives in this new server will be added automatically to the storage pool. At first, they’re empty.
The capacity drives in our fourth server are empty, for now.
After 30 minutes, Storage Spaces Direct will automatically begin re-balancing the storage pool – moving slabs around to even out drive utilization. This can take some time (many hours) for larger deployments. You can watch its progress using the following cmdlet.
Get-StorageJobIf you’re impatient, or if your deployment uses Shared SAS Storage Spaces with Windows Server 2016, you can kick off the re-balance yourself using the following cmdlet.
Optimize-StoragePool -FriendlyName "S2D*"The storage pool is ‘re-balanced’ whenever new drives are added to even out utilization.
Once completed, we see that our 1 TiB volume is (almost) evenly distributed across all the drives in all four servers.
The slabs of our 1 TiB two-way mirrored volume are now spread evenly across all four servers.
And going forward, when we create new volumes, they too will be distributed evenly across all drives in all servers.
This can explain one final phenomena you might observe – that when a drive fails, every volume is marked ‘Incomplete’ for the duration of the repair. Can you figure out why?
Okay, that’s it for now. If you’re still reading, wow, thank you!
Let’s review some key takeaways.
In PowerShell, you can see the storage pool by running the following cmdlet.
Get-StoragePool S2D*And you can see the drives in the pool with this simple pipeline.
Get-StoragePool S2D* | Get-PhysicalDiskThroughout this blog, I showed the output of a script which essentially runs the above, cherry-picks interesting properties, and formats the output all fancy-like. That script is included below, and is also available at http://cosmosdarwin.com/Show-PrettyPool.ps1 to spare you the 200-line copy/paste. There is also a simplified version at here which forgoes my extravagant helper functions to reduce running time by about 20x and lines of code by about 2x. 
Let me know what you think!
# Written by Cosmos Darwin, PM
# Copyright (C) 2016 Microsoft Corporation
# MIT License
# 11/2016
Function ConvertTo-PrettyCapacity {
<#
.SYNOPSIS Convert raw bytes into prettier capacity strings.
.DESCRIPTION Takes an integer of bytes, converts to the largest unit (kilo-, mega-, giga-, tera-) that will result in at least 1.0, rounds to given precision, and appends standard unit symbol.
.PARAMETER Bytes The capacity in bytes.
.PARAMETER UseBaseTwo Switch to toggle use of binary units and prefixes (mebi, gibi) rather than standard (mega, giga).
.PARAMETER RoundTo The number of decimal places for rounding, after conversion.
#>
Param (
[Parameter(
Mandatory = $True,
ValueFromPipeline = $True
)
]
[Int64]$Bytes,
[Int64]$RoundTo = 0,
[Switch]$UseBaseTwo # Base-10 by Default
)
If ($Bytes -Gt 0) {
$BaseTenLabels = ("bytes", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
$BaseTwoLabels = ("bytes", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB", "ZiB", "YiB")
If ($UseBaseTwo) {
$Base = 1024
$Labels = $BaseTwoLabels
}
Else {
$Base = 1000
$Labels = $BaseTenLabels
}
$Order = [Math]::Floor( [Math]::Log($Bytes, $Base) )
$Rounded = [Math]::Round($Bytes/( [Math]::Pow($Base, $Order) ), $RoundTo)
[String]($Rounded) + $Labels[$Order]
}
Else {
0
}
Return
}
Function ConvertTo-PrettyPercentage {
<#
.SYNOPSIS Convert (numerator, denominator) into prettier percentage strings.
.DESCRIPTION Takes two integers, divides the former by the latter, multiplies by 100, rounds to given precision, and appends "%".
.PARAMETER Numerator Really?
.PARAMETER Denominator C'mon.
.PARAMETER RoundTo The number of decimal places for rounding.
#>
Param (
[Parameter(Mandatory = $True)]
[Int64]$Numerator,
[Parameter(Mandatory = $True)]
[Int64]$Denominator,
[Int64]$RoundTo = 1
)
If ($Denominator -Ne 0) { # Cannot Divide by Zero
$Fraction = $Numerator/$Denominator
$Percentage = $Fraction * 100
$Rounded = [Math]::Round($Percentage, $RoundTo)
[String]($Rounded) + "%"
}
Else {
0
}
Return
}
Function Find-LongestCommonPrefix {
<#
.SYNOPSIS Find the longest prefix common to all strings in an array.
.DESCRIPTION Given an array of strings (e.g. "Seattle", "Seahawks", and "Season"), returns the longest starting substring ("Sea") which is common to all the strings in the array. Not case sensitive.
.PARAMETER Strings The input array of strings.
#>
Param (
[Parameter(
Mandatory = $True
)
]
[Array]$Array
)
If ($Array.Length -Gt 0) {
$Exemplar = $Array[0]
$PrefixEndsAt = $Exemplar.Length # Initialize
0..$Exemplar.Length | ForEach {
$Character = $Exemplar[$_]
ForEach ($String in $Array) {
If ($String[$_] -Eq $Character) {
# Match
}
Else {
$PrefixEndsAt = [Math]::Min($_, $PrefixEndsAt)
}
}
}
# Prefix
$Exemplar.SubString(0, $PrefixEndsAt)
}
Else {
# None
}
Return
}
Function Reverse-String {
<#
.SYNOPSIS Takes an input string ("Gates") and returns the character-by-character reversal ("setaG").
#>
Param (
[Parameter(
Mandatory = $True,
ValueFromPipeline = $True
)
]
$String
)
$Array = $String.ToCharArray()
[Array]::Reverse($Array)
-Join($Array)
Return
}
Function New-UniqueRootLookup {
<#
.SYNOPSIS Creates hash table that maps strings, particularly server names of the form [CommonPrefix][Root][CommonSuffix], to their unique Root.
.DESCRIPTION For example, given ("Server-A2.Contoso.Local", "Server-B4.Contoso.Local", "Server-C6.Contoso.Local"), returns key-value pairs:
{
"Server-A2.Contoso.Local" -> "A2"
"Server-B4.Contoso.Local" -> "B4"
"Server-C6.Contoso.Local" -> "C6"
}
.PARAMETER Strings The keys of the hash table.
#>
Param (
[Parameter(
Mandatory = $True
)
]
[Array]$Strings
)
# Find Prefix
$CommonPrefix = Find-LongestCommonPrefix $Strings
# Find Suffix
$ReversedArray = @()
ForEach($String in $Strings) {
$ReversedString = $String | Reverse-String
$ReversedArray += $ReversedString
}
$CommonSuffix = $(Find-LongestCommonPrefix $ReversedArray) | Reverse-String
# String -> Root Lookup
$Lookup = @{}
ForEach($String in $Strings) {
$Lookup[$String] = $String.Substring($CommonPrefix.Length, $String.Length - $CommonPrefix.Length - $CommonSuffix.Length)
}
$Lookup
Return
}
### SCRIPT... ###
$Nodes = Get-StorageSubSystem Cluster* | Get-StorageNode
$Drives = Get-StoragePool S2D* | Get-PhysicalDisk
$Names = @()
ForEach ($Node in $Nodes) {
$Names += $Node.Name
}
$UniqueRootLookup = New-UniqueRootLookup $Names
$Output = @()
ForEach ($Drive in $Drives) {
If ($Drive.BusType -Eq "NVMe") {
$SerialNumber = $Drive.AdapterSerialNumber
$Type = $Drive.BusType
}
Else { # SATA, SAS
$SerialNumber = $Drive.SerialNumber
$Type = $Drive.MediaType
}
If ($Drive.Usage -Eq "Journal") {
$Size = $Drive.Size | ConvertTo-PrettyCapacity
$Used = "-"
$Percent = "-"
}
Else {
$Size = $Drive.Size | ConvertTo-PrettyCapacity
$Used = $Drive.VirtualDiskFootprint | ConvertTo-PrettyCapacity
$Percent = ConvertTo-PrettyPercentage $Drive.VirtualDiskFootprint $Drive.Size
}
$Node = $UniqueRootLookup[($Drive | Get-StorageNode -PhysicallyConnected).Name]
# Pack
$Output += [PSCustomObject]@{
"SerialNumber" = $SerialNumber
"Type" = $Type
"Node" = $Node
"Size" = $Size
"Used" = $Used
"Percent" = $Percent
}
}
$Output | Sort Used, Node | FT
Microsoft’s file systems organize storage devices based on cluster size. Also known as the allocation unit size, cluster size represents the smallest amount of disk space that can be used to hold a file. Both ReFS and NTFS support multiple cluster sizes, as different sized clusters can offer different performance benefits, depending on the deployment.
In the past couple weeks, we’ve seen some confusion regarding the recommended cluster sizes for ReFS and NTFS, so this blog will hopefully disambiguate previous recommendations while helping to provide the reasoning behind why some cluster sizes are recommended for certain scenarios.
IO amplification
Before jumping into cluster size recommendations, it’ll be important to understand what IO amplification is and why minimizing IO amplification is important when choosing cluster sizes:
ReFS cluster sizes
ReFS offers both 4K and 64K clusters. 4K is the default cluster size for ReFS, and we recommend using 4K cluster sizes for most ReFS deployments because it helps reduce costly IO amplification:
Additionally, 4K cluster sizes offer greater compatibility with Hyper-V IO granularity, so we strongly recommend using 4K cluster sizes with Hyper-V on ReFS. 64K clusters are applicable when working with large, sequential IO, but otherwise, 4K should be the default cluster size.
NTFS cluster sizes
NTFS offers cluster sizes from 512 to 64K, but in general, we recommend a 4K cluster size on NTFS, as 4K clusters help minimize wasted space when storing small files. We also strongly discourage the usage of cluster sizes smaller than 4K. There are two cases, however, where 64K clusters could be appropriate:
While a 4K cluster size is the default setting for NTFS, there are many scenarios where 64K cluster sizes make sense, such as: Hyper-V, SQL, deduplication, or when most of the files on a volume are large.
Hi folks!
Based on several customer bug reports, we have issued a critical fix for Data Deduplication in Windows Server 2016 in the most recent Windows update package, KB3216755. This patch fixes an issue where corruptions may appear in files larger than 2.2 TB. While we always recommend keeping your system up-to-date, based on the severity of any data corruption, we strongly recommend that everyone who is using Data Deduplication on Windows Server 2016 take this update!
Long-time users of Dedup on will note that we only officially support files with size up to 1 TB. While this is true, this is a “soft” support statement – we take your data integrity extremely seriously, and therefore will always address reported data corruptions. Our current defined support statement of 1 TB was chosen for two reasons: 1) for files larger than 1 TB, performance isn’t quite ‘up to snuff’ with our expectations, and 2) dynamic workloads with lots of writes may reach NTFS’ file fragmentation limits, causing the file to become read-only until the next optimization. In short, our 1 TB support statement is about preserving a high quality experience for you. Your mileage may vary… in particular, many users have reported to us that backup workloads that use VHDs or VHD-like container files sized over 1 TB work extremely well with Dedup. This is because backup workloads are typically append-only workloads. We do however recommend that you make use of the new Backup usage type in Windows Server 2016 to ensure the best performance with backup workloads.
Finally, I would just like to thank the three users who reached out to us with this issue and helped us validate the pre-release patch: thank you! We always love to hear from you, our customers, so please feel free to reach out to us with your questions, comments, or concerns anytime: dedupfeedback@microsoft.com!
We are happy to announce, that we’ve just released an update to the Work Folders app on iOS that now allows anyone to upload pictures and documents from other apps, take pictures or even write a simple note – right from within in the Work Folders App.
Work Folders is a Windows Server feature since 20012 R2 that enables individual employees to access their files securely from inside and outside the corporate environment. The Work Folders app connects to the server and enables file access on your Android phone and tablet. Work Folders enables this while allowing the organization’s IT department to fully secure that data.
Using the latest version of Work Folders for iOS, users can now:
For the complete list of Work Folders for iOS features, please reference the feature list section below.
Microsoft Office files are read-only when opening the files from the Work Folders app.
If you’re interested in learning more about Work Folders, here are some great resources:
Hello, Claus here again. It has been a while since I last posted here and a few things have changed since last time. Windows Server has been moved into the Windows and Devices Group, we have moved to a new building with a better café, but a worse view 
. On a personal note, I can be seen waddling the hallways as I have had foot surgery.
At Microsoft Ignite 2016 I did a demo at the 28-minute mark as part of the Meet Windows Server 2016 and System Center 2016 session. I showed how Storage Spaces Direct can deliver massive amounts of IOPS to many virtual machines with various storage QoS settings. I encourage you to watch it, if you haven’t already, or go watch it again . In the demo, we used a 16-node cluster connected over iWARP using the 40GbE Chelsio iWARP T580CR adapters, showing 6M+ read IOPS. Since then, Chelsio has released their 100GbE T6 NIC adapter, and we wanted to take a peek at what kind of network throughput would be possible with this new adapter.
We used the following hardware configuration:
We did not configure DCB (PFC) in our deployment, since it is not required in iWARP configurations.
Below is a screenshot from the VMFleet Watch-Cluster window, which reports IOPS, bandwidth and latency.
As you can see the aggregated bandwidth exceeded 83GB/s, which is very impressive. Each VM realized more than 1GB/s of throughput, and notice the average read latency is <1.5ms.
Let me know what you think.
Until next time
Hi folks,
To prioritize and plan for investments in vNext experiences for Windows Server, we could use input from you! We would like to understand more about how you are utilizing Windows File Server, especially as it relates to the size of your datasets. This survey should take approximately 2-5 minutes to complete. We appreciate your feedback!
Click here to take our survey!
Thanks,
The Windows Server Storage Team
Hello, Claus here again. Today Intel is announcing the Intel Optane SSD DC P4800X device family, and of course Windows Server 2016 Storage Spaces Direct will support this device. The P4800X device promises better latency and better endurance, so we took these devices for a spin in a Storage Spaces Direct configuration, comparing them to the P3700 devices:
Notice that we have half the devices and a quarter the capacity for the Intel™ Optane™ P4800X device. The software and workload configuration were:
The test results are captured in the diagram below:
The first observation is that we see a 90µs latency improvement across the board from low IOPS to high IOPS. This pretty much aligns with the latency improvements in the device itself, and notice how the improvement is realized at the top of the storage stack in a fully resilient storage configuration. This means that customers can realize the latency improvements provided by the P4800X device in their Storage Spaces Direct deployments.
The second observation is that we see the same CPU utilization at 880K IOPS (throttled by Storage QoS), with 258µs latency on the P4800X devices vs 344µs latency on the P3700 devices, meaning that Storage Spaces Direct customers can realize the latency improvements without any additional CPU consumption.
Let me know what you think.
Until next time
Claus
Hello, Claus here again. By now, you have probably seen some of my blogs and demos on Storage Spaces Direct performance. One of Storage Spaces Direct’s advantages is RDMA networking support that lowers latency and reduces CPU consumption. I often get the question “Is RDMA required for Storage Spaces Direct”. The answer to this question is: no. We support plain-old Ethernet as long as it’s 10GbE or better. But let’s look a bit deeper.
Recently we did a performance investigation on new hardware, comparing it with an in-market offering (more about that in another post). We ran the tests with RDMA enabled and RDMA disabled (Ethernet mode), which provided the data for this post. For this investigation, we used DISKSPD with the following configuration:
We used the following hardware configuration:
We are by no means driving this system hard, which is on purpose since we want to show the delta between RDMA and non-RDMA under a reasonable workload and not at the edge of what the system can do.
| Metric |
RDMA |
TCP/IP |
RDMA advantage |
| IOPS |
185,500 |
145,500 |
40,000 additional IOPS with the same workload. |
| IOPS/%kernel CPU |
16,300 |
12,800 |
3,500 additional IOPS per percent CPU consumed. |
| 90th percentile write latency |
250µs |
390µs |
140µs (~36%) |
| 90th percentile read latency |
260µs |
360µs |
100µs (28%) |
I think there are two key take-away’s from this data:
Let me know what you think.
Until next time
Claus
We’re excited to announce Work Folders now supports using Azure Active Directory Application Proxy to enable remote users to securely access their files on the Work Folders server.
Work Folders supports using VPN, Web Application Proxy (WAP) or a third-party reverse proxy solution to enable remote users access to their files on the Work Folders server. These remote access solutions require expensive hardware or additional on-premises servers that need to be managed.
Benefits of using Azure AD Application Proxy
To learn more about Azure Active Directory Application Proxy, please see the following article: How to provide secure remote access to on-premises applications
To enable Work Folders access using Azure AD Application proxy, please follow the steps below.
Before you can enable Work Folders access using Azure AD Application Proxy, you need to have:
High-level overview of the steps required:
Example
Note: If you have multiple Work Folders servers, you need to create a proxy application for each Work Folders server (repeat steps 1-12).
Example
Example
Note: If you have multiple Work Folders servers and you created multiple Work Folders proxy applications, please repeat the steps above to give the Work Folders native application access to all Work Folders proxy applications.
To learn more about the Application Proxy Connector and the outbound network ports that are required, please see the following article: Get started with Application Proxy and install the connector
The Work Folders server is configured by default to use Integrated Windows Authentication.
To verify the server is configured properly, perform the following steps:
Note: If the Work Folders environment is currently configured to use ADFS authentication, changing the authentication method from ADFS to Windows Authentication will cause existing users to fail to authenticate. To resolve this issue, the Work Folders clients will need to be re-configured to use the Work Folders proxy application URL or create another Work Folders server that will be used for Azure AD Application Proxy.
setspn -S http/workfolders.domain.com servername
Example: setspn -S http/workfolders.contoso.com 2016-wf
In the example above, the FQDN for the work folders server is workfolders.contoso.com and Work Folders server name is 2016-wf.
Note: The SPN value entered using the setspn command must match the SPN value entered in the Work Folders proxy application in the Azure portal.
Example
You can skip this section if you’re not using a self-signed certificate on the Work Folders server.
If the Work Folders server is using a self-signed certificate, you need to export the certificate on the Work Folders server and import the certificate on the App Proxy Connector server. This step is necessary for the App Proxy Connector server to communicate with the Work Folders server.
To export the certificate on the Work Folders server, follow these steps:
To import the certificate on the App Proxy Connector server, follow these steps:
Token Broker is an authentication broker that supports device registration. When using Token Broker with Azure AD Application Proxy for remote access, the client device can be registered in Azure AD when configuring the Work Folders client. Once the device is registered, device authentication will be used to access the Work Folders server.
Device registration provides the following benefits:
How to enable Token Broker
To enable Token Broker on a Windows 10 version 1703 system, enable the “Enables the user of Token Broker for AD FS authentication” group policy setting which is located under User Configuration\Administrative Templates\Windows Components\Work Folders.
For Android and iOS devices, Token Broker will be used automatically when using Azure AD Application Proxy.
How to register devices using the Work Folders client
When Token Broker is enabled on a Windows client, the user will be prompted to register their device in Azure AD when configuring the Work Folders client. If the Work Folders client is managed via group policy, the device is automatically registered in Azure AD.
For devices (Android and iOS), the device is automatically registered when configuring the Work Folders client.
How to configure a Windows 10 version 1703 client to use the Azure AD App Proxy URL:
How to configure an Android or iOS client to use the Azure App Proxy URL:
We’re excited to announce several improvements to the Work Folders clients for Windows 10 version 1703, Android and iOS:
For more details, please review the sections below.
Applies to: Windows 10 version 1703, Android and iOS
Work Folders supports using VPN, Web Application Proxy (WAP) or a third-party reverse proxy solution to enable remote users access to their files on the Work Folders server. These remote access solutions require expensive hardware or additional on-premises servers that need to be managed.
Work Folders now supports using Azure AD Application Proxy to enable remote users to securely access their files on the Work Folders server.
Benefits of using Azure AD Application Proxy
To learn more about Azure AD Application Proxy, please see the following article: How to provide secure remote access to on-premises applications
How to enable remote access to Work Folders using Azure Active Directory Application Proxy
For more details on how to configure Work Folders access using Azure AD Application Proxy, please see the following blog: Enable remote access to Work Folders using Azure Active Directory Application Proxy
Applies to: Windows 10 version 1703, Android and iOS
A common complaint when using AD FS authentication is the remote user is prompted for credentials every 8 hours if the device is not registered with the AD FS server. To reduce the frequency of credential prompts, you can enable the Keep Me Signed In (KMSI) feature but the maximum single sign on period for a non-registered device is 7 days. To register the device, the user needs to use the Workplace Join feature.
To improve the user experience when using Azure AD Application Proxy, Work Folders now supports Token Broker which is an authentication broker that supports device registration. When using Token Broker with Azure AD Application Proxy for remote access, the client device can be registered in Azure AD when configuring the Work Folders client. Once the device is registered, device authentication will be used to access the Work Folders server.
Device registration provides the following benefits:
For more details on Azure Active Directory device registration, please see the following article on TechNet: Get started with Azure Active Directory device registration
How to enable Token Broker
To enable Token Broker on a Windows 10 version 1703 system, enable the “Enables the user of Token Broker for AD FS authentication” group policy setting which is located under User Configuration\Administrative Templates\Windows Components\Work Folders
For Android and iOS devices, Token Broker will be used automatically when using Azure AD Application Proxy.
Note: Token Broker is currently supported when using Azure AD Application Proxy for remote access. Using Token Broker with AD FS authentication may be supported in a future update.
How to register devices using the Work Folders client
When Token Broker is enabled on a Windows client, the user will be prompted to register their device in Azure AD when configuring the Work Folders client. If the Work Folders client is managed via group policy, the device is automatically registered in Azure AD.
For devices (Android and iOS), the device is automatically registered when configuring the Work Folders client.
Applies to: Windows 10 version 1703
A common request when managing Work Folders clients via group policy is to configure the Work Folders client directory location.
How to configure the Work Folder client directory location using group policy
On Windows 10 version 1703, a group policy setting “Work Folders Local Path” has been added to configure the Work Folders client directory location. This group setting is located under User Configuration\Administrative Templates\Windows Components\Work Folders\Specify Work Folders settings.
Note: The Work Folders Local Path group policy setting applies to Windows 10 version 1607 and Windows 10 version 1703 systems. If the value is not defined, the client directory will be located under %userprofile%\Work Folders.
Hi folks, Ned here again. This blog post contains all products requiring SMB1, where the vendor explicitly states this in their own documentation or communications. This list is not complete and you should never treat it as complete; check back often.
All products arranged in alphabetical order, by vendor, by product, with a URL to their documentation stating SMB1 requirements.
Vendor – Product – Documentation
To update this list, please email StillNeedsSMB1@microsoft.com or tweet @nerdpyle with hashtag #StillNeedsSMB1.
Adding a product to this list requires direct quote or documentation from the vendor of that product, including their website, knowledgebase, support forums, or other vendor channels; third party forums are not enough to qualify. Consult your vendor for updates and newer product versions that support at least SMB 2.02. If you are a vendor and wish to report requirements for SMB1 or if information above has changed, email StillNeedsSMB1@microsoft.com.
There are vendors who are not publishing their SMB1 requirements. It is up to you, their customer, to have them publish this information – Microsoft cannot make them do so. If a vendor does not state if they require SMB1 but you believe they do, please contact that vendor directly. If you need assistance getting a vendor response, email StillNeedsSMB1@microsoft.com and we will try our best to assist. Politeness works best; the person you are speaking to at a vendor is extremely unlikely to have put SMB1 into the product & probably isn’t any happier about it than you are!
For more information on why using SMB1 is unsafe, see StopUsingSMB1. SMB1 has been deprecated for years and will be removed by default from many editions and SKUs of Windows 10 and Windows Server 2016 in the RS3 release.
Be safe out there,
Ned Pyle, Principal Program Manager of the SMB protocol family at Microsoft
Hello, Claus here again. As you have probably noticed by now, we have a great ongoing collaboration with Intel. In this blog post we are going to look at Windows Server 2016 Storage Spaces Direct on Intel’s latest and greatest hardware, which includes a new processor family, Intel® Xeon® Scalable Processors, an iWARP RDMA network adapter with the integrated Intel® Ethernet Connection X722, and “Intel® Optane™ Solid State Drives.
We use a 4 node cluster, each node with configured with the following hardware:
We deployed Windows Server 2016 Storage Spaces Direct and stood up VMFleet with:
Each VM runs DISKSPD, with 4K IO size at 90% read and 10% write rate limited at 7,500IOPS. This produces a total IOPS of ~720K (4 * 24 * 7,500).
Read IO is served at about 80 microseconds! This is significantly less than anything else we have seen before. Write IO is served at about 300 microseconds. The write latency is higher than read latency mostly due to network latency, as writes are mirrored in 3 copies to peer nodes in the cluster.
In addition, CPU consumption is less than 25%, which means there is plenty of headroom for applications to consume this storage performance.
We are very excited to see these numbers and the value that the new generation of Intel Scalable Processors, Intel Optane DC SSD devices combined with the Intel Ethernet Connection X722 with 4x 10GbE iWARP RDMA adapter can deliver to our joint customers. Let me know what you think.
Until next time
Claus
