Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and ...

WHITE PAPER

VIRTUALIZED DESKTOP INFRASTRUCTURE WITH DELL EMC XTREMIO X2 AND VMWARE HORIZON ENTERPRISE 7.2 Abstract This reference architecture evaluates the best-in-class performance and scalability delivered by Dell EMC XtremIO X2 for VDI. We present dataquantifying performance at scale for thousands of desktops in each stage of the VDI lifecycle. Datacenter design elements, both hardware and software, that synergize into achieving the optimum results, are also discussed in detail. August, 2017

Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and VMware Horizon Enterprise 7.2 © 2017 Dell Inc. or its subsidiaries.

Contents Abstract ................................................................................................................................................................................... 1 Executive Summary ................................................................................................................................................................ 4 Business Case ........................................................................................................................................................................ 4 Overview ................................................................................................................................................................................. 5 Test Results ............................................................................................................................................................................ 6 Summary ............................................................................................................................................................................. 6 Management Operations Performance Results .................................................................................................................. 7 Full Clone Provisioning ................................................................................................................................................... 7 Linked Clone Provisioning ............................................................................................................................................. 10 Linked Clone Pool Refresh ........................................................................................................................................... 13 Linked Clone Pool Recompose ..................................................................................................................................... 14 Production Use Performance Results ............................................................................................................................... 15 Boot Storms................................................................................................................................................................... 16 LoginVSI Results ........................................................................................................................................................... 19 Solution's Hardware Layer .................................................................................................................................................... 26 Storage Array: Dell EMC XtremIO X2 All-Flash Array ...................................................................................................... 26 XtremIO X2 Overview ................................................................................................................................................... 27 Architecture and Scalability ........................................................................................................................................... 27 XIOS and the I/O Flow .................................................................................................................................................. 30 System Features ........................................................................................................................................................... 34 XtremIO Management Server ....................................................................................................................................... 42 Test Setup ......................................................................................................................................................................... 48 Compute Hosts: Dell PowerEdge Servers ........................................................................................................................ 48 Storage Configuration ....................................................................................................................................................... 49 Zoning ........................................................................................................................................................................... 50 Storage Volumes ........................................................................................................................................................... 50 Initiator Groups and LUN Mapping................................................................................................................................ 50 Storage Networks .............................................................................................................................................................. 51 Solution's Software Layer...................................................................................................................................................... 51 Hypervisor Management Layer ......................................................................................................................................... 51 vCenter Server Appliance ............................................................................................................................................. 52 Hypervisor ..................................................................................................................................................................... 52 ESX Clusters ................................................................................................................................................................. 52 Network Configuration ................................................................................................................................................... 53 Storage Configuration, EMC SIS and VSI .................................................................................................................... 54 Compute Integration – Dell OpenManage .................................................................................................................... 55 Virtual Desktop Management Layer .................................................................................................................................. 56 VMware Horizon View Server Configuration ................................................................................................................. 56 2 | Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and VMware Horizon Enterprise 7.2 © 2017 Dell Inc. or its subsidiaries.

VMware Horizon View Desktop Pool Configuration ...................................................................................................... 59 Virtual Machine System Settings and Applications ....................................................................................................... 59 Conclusion............................................................................................................................................................................. 60 References ............................................................................................................................................................................ 61 Appendix A – Test Methodology ........................................................................................................................................... 62 How to Learn More ................................................................................................................................................................ 63

3 | Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and VMware Horizon Enterprise 7.2 © 2017 Dell Inc. or its subsidiaries.

Executive Summary This reference architecture serves as a reference point for deploying a Virtual Desktop Infrastructure (VDI) environment using Dell EMC XtremIO X2 as the storage array for the infrastructure. It also discusses design considerations for deploying such an environment. Based on the evidence presented herein, we firmly establish the value of XtremIO X2 as a best-in-class all-flash array for VMware Horizon Enterprise deployments. This reference architecture presents a complete VDI solution for VMware Horizon 7.2 delivering virtualized Windows 10 desktops with applications such as Microsoft Office 2016, Adobe Reader 11, Java and other common desktop user applications. It discusses design considerations that will give you a reference point for successfully deploying a VDI project using XtremIO X2, and as such, will focus on the storage aspects of the project. We are focusing this reference architecture on Full Clone desktops and Linked Clone desktops. A different reference architecture will be published to discuss Instant Clone desktops and the use of VMware App Volumes for policy-driven application delivery.

Business Case A well-known rationale behind virtualizing desktops is lowering the Total Cost of Ownership (TCO). TCO generally includes capital expenditures from purchasing hardware such as storage, servers, networking switches and routers, in addition to the software licensing and maintenance costs. The goals in virtualizing desktops is to improve economics and efficiency in desktop delivery, to ease maintenance and management and to improve desktop security. In addition to these goals, a key objective of a successful VDI deployment, and one that probably matters the most, is the end user experience. It is imperative for VDI deployments to demonstrate parity with that of physical workstations when it comes to the end user experience. The overwhelming value of virtualizing desktops in a software-defined datacenter and the need to deliver a rich end user experience compels us to select the best-of-breed infrastructure components for our VDI deployment. Selecting the best-in-class and performant storage system that is also easy to manage helps us in achieving our long-term goals in lowering the TCO, and hence is a critical piece of the infrastructure. While the "end user experience" metric is rather subjective and largely dependent on a user's choice of applications and its perception of "responsiveness", we are still able to assess a VDI solution by its ability to deliver consistent highperformance and low-latency while scaling up to thousands of desktops on a single storage array. A VDI solution should also be forward-looking, by factoring in future growth in desktop delivery and the management and complexities that come with the expansion. The shared storage infrastructure in a VDI solution should be robust enough to be geared for delivering consistent performance and scalability for thousands of desktops – irrespective of the desktop delivery mechanism (linked clones, full clones, etc.). XtremIO brings tremendous value by providing consistent performance at scale by means of always-on inline deduplication, compression, thin provisioning and unique data protection capabilities. Seamless interoperability with VMware vSphere by means of VMware APIs for Array Integration (VAAI), Dell EMC Solutions Integration Service (SIS) and Virtual Storage Integrator's (VSI) ease of management make choosing this best-of-breed all-flash array for desktop virtualization purposes even more attractive. XtremIO is a scale-out storage system that can grow in storage capacity, compute resources and bandwidth capacity whenever storage requirements for the environment are enhanced. With the advent of multi-core server systems and the number of CPU cores per processor (following Moore's law), we are able to consolidate an increasing number of desktops on a single enterprise-class server. When combined with XtremIO X2 All-Flash Array, we can consolidate vast numbers of virtualized desktops on a single storage array, thereby achieving high consolidation at great performance from a storage and a compute perspective.


Overview It is well known that implementing a complete VDI solution is a multi-faceted effort with nuances encompassing compute, memory, network – and most importantly – storage. Our focus in this reference architecture is on XtremIO X2 capabilities and benefits in such a solution; however, we intend to give a complete picture of a VDI solution. An XtremIO X2 cluster provides us with sufficient storage capacity and adequate performance for servicing the I/O requests and storage bandwidth required for a scale of thousands and tens of thousands of virtual desktops. This includes desktop delivery, management operations, login and boot storms, and production use at scale. The XtremIO X2 Storage Array demonstrates top class performance when deploying virtual desktops and running management operations on them, as well as when subjected to live user emulation tests using LoginVSI (Login Virtual Session Indexer – a software simulating user workloads for Windows-based virtualized desktops). The XtremIO All Flash Storage array is based upon a scale-out architecture. It is comprised of building blocks called X-Bricks which can be clustered together to grow performance and capacity as required. An X-Brick is the basic building block of an XtremIO cluster. Each X-Brick is a highly-available, high-performance unit that consists of dual Active-Active Storage Controllers, with CPU and RAM resources, Ethernet, FC and iSCSI connections and a Disk Array Enclosure (DAE) containing the SSDs that hold the data. With XtremIO X2, a single X-Brick can service the storage capacity and bandwidth requirements for a scale of 4000 desktops, with room to spare. XtremIO X2 All-Flash Array is designed to provide high responsiveness to an increasing amount of data for thousands of users, and is extremely beneficial for VDI projects. In subsequent sections of this reference architecture, we will observe XtremIO's compounding returns for its data reduction capabilities and the high performance it provides to VDI environments with thousands of desktops. We will see the benefits in deploying a Full Clone Desktop Pool (full copies of an OS template) as well as in deploying a Linked Clone Desktop Pool (VMware snapshots of an OS template) in terms of data reduction and storage performance. XtremIO's scale-out architecture allows scaling any environment, in our case VDI environments, in a linear way that satisfies both the capacity and performance needs of the growing infrastructure. An XtremIO X2 cluster can start with any number of required X-Bricks to service the current or initial loads, and can grow linearly (up to 4 X-Bricks in a cluster) to keep servicing the increasing environment appropriately (to be increased to 8 X-Bricks in the future, depending on the cluster's type). With X2, in addition to its scale-out capabilities, an XtremIO storage array has scale-up capabilities by adding extra SSDs to an X-Brick. An X-Brick can contain between 18 and 72 SSDs (in increments of 6) of fixed sizes (400GB or 1.92TB, depending on the cluster's type, with future versions allowing 3.84TB sized SSDs). In developing this VDI solution, we have selected VMware vSphere 6.5 as the virtualization platform, and VMware Horizon 7.2 for virtual desktop delivery and management. Windows 10 (32-bit) is the virtual desktops' operating system. EMC VSI (Virtual Storage Integrator) 7.2 and the vSphere Web Client are used to apply best practices pertaining to XtremIO storage Volumes and the general environment. To some degree, data in subsequent sections of this reference architecture helps us quantify the end user experience for a desktop user and also demonstrates the efficiency in management operations that a datacenter administrator may perceive when deploying their VDI environment on XtremIO X2 all-flash array. We begin the reference architecture by discussing test results, which are classified into the following categories: •

Management Operations – resource consumption and time to complete VMware Horizon management operations, such as desktop pool deployment, recomposing and refreshing a pool

•

Production Use – resource consumption patterns and time to complete a boot storm and resource consumption patterns and responsiveness when desktops in the pool are subjected to "LoginVSI Knowledge Worker" workloads, emulating real users' workloads

After presenting and analyzing the test results of our VDI environment, we will discuss the different elements of our infrastructure, beginning with the hardware layer and moving up to the software layer, including the features and best practices we recommend for the environment. This includes extensive details of the XtremIO X2 storage array, storage network equipment and host details at the hardware level, and the VMware vSphere Hypervisor (ESXi), vCenter Server and VMware Horizon Suite at the software level. The details of the virtual machine settings and LoginVSI workload profile provide us with the complete picture of how all building blocks of a VDI environment function together. 5 | Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and VMware Horizon Enterprise 7.2 © 2017 Dell Inc. or its subsidiaries.

Test Results In this section, we elaborate on the tests performed on our VDI environment and their results. We start with a summary of the results and their conclusions and dive deeper into each test's detailed results and analyzed data and statistics (including various storage and compute metrics such as bandwidth, latency, IOPS, and CPU and RAM utilization).

Summary A VMware Horizon View desktop pool undergoes the following basic lifecycle stages: •

Provisioning

•

Production work by active users

•

Maintenance operations (recomposing and refreshing, which are maintenance operations exclusive to "Linked Clone" desktops)

We will show brief and detailed results of tests made for these stages, divided into two types of lifecycle phases: Management Operations (Provisioning and Maintenance operations) and Production Use From the perspective of datacenter administrators, operational efficiency is translated to time to complete management operations. The less time it takes to provision desktops and perform maintenance operations, the faster the availability is of VDI desktop pools for production. It is for this reason that the storage array's performance capabilities in context of throughput deserve special attention – the more throughput the system can withstand, the faster those management operations will complete. The storage array throughput is perceived in terms of IOPS or bandwidth that manifest in terms of data transfer rate. During production, desktops are in actual use by end users via remote sessions. Two events are tested to examine the infrastructure's performance and ability to serve VDI users: Virtual desktops boot storm, and heavy workloads produced by high percentage of users using their desktops. Boot storms are measured by time to complete, and heavy workloads by the "user experience". The criteria dictating "user experience" is the applications' responsiveness and overall desktop experience. We use the proven LoginVSI tests (explained further in this paper) to evaluate user experience, and track storage latency during those LoginVSI tests. Table 1 shows a summary of the test results for all stages of a VDI desktop pool lifecycle with 4000 desktops for Full Clone and Linked Clone desktops, when deployed on an XtremIO X2 cluster as its storage array. Note that the Recompose and Refresh maintenance operations are not applicable for Full Clone desktops. Table 1.

VDI Performance Tests with XtremIO X2 – Results Summary

4000 DESKTOPS

FULL CLONES

LINKED CLONES

Elapsed Time – Deployment

110 Minutes

75 Minutes

Elapsed Time – Recompose

---

106 Minutes

Elapsed Time – Refresh

---

20 Minutes

LoginVSI Boot Storm

10 Minutes

10 Minutes

LoginVSI – VSI Baseline

756

776

LoginVSI – VSI Average

979

1240

LoginVSI – VSI Max

Not Reached

Not Reached

We can notice the excellent results for time to deploy, boot storms, and running maintenance operations, as well as the accomplished LoginVSI results (detailed in LoginVSI Results on page 19) that emulate production work by active users.


We are suggesting a scale-out approach for VDI environments, in which we add compute and memory resources (more ESX hosts) as we scale up in the quantity of desktops. In our tests, we are deploying Virtual Desktops with two vCPUs and 4GB of RAM (not all utilized since we are using a 32-bit operating system) per desktop. Following a number of tests to understand the appropriate scaling, we determined a scale of 125 desktops per single ESX host (with the given host configuration listed in Table 2). Using this scale, we deployed 4000 virtual desktops on 32 ESX hosts. We also determined a scale of 125 virtual desktops per XtremIO Volume of 10TB for Full Clones and 3TB for Linked Clones (the maximum number of desktops per a single LUN when provisioned with VAAI is 500). As we will see next, the total of 32 volumes of 320TB total size for Full Clones and 96TB for Linked Clones were easily handled by our single X-Brick X2 cluster, both in terms of capacity and performance (IOPS, bandwidth and latency). In the rest of this section, we take a deeper look into the data that was collected in our storage array and other environment components during each of the management operation tests, as well as during boot storms and LoginVSI's "Knowledge worker" workload tests. A data-driven understanding of our XtremIO X2 storage array's behavior provides us with evidence that assure a rich user experience and efficiency in management operations when using this effective all-flash array. This is manifested due to performance-at-scale, for thousands of desktops. The data collected below includes statistics of storage bandwidth, IOPS, I/O latency, CPU utilization and more. Using XtremIO RESTful API (Representational State Transfer Application Program Interface), a powerful feature that enables performance monitoring while executing management operation tests and running LoginVSI workloads, we collect performance statistics from the XtremIO Management Server (XMS) and provide a front-row seat to the exceptional capabilities of XtremIO X2 for VDI environments.

Management Operations Performance Results In this section, we take a deeper look at performance statistics from our XtremIO X2 array while performing management operations in the VDI environment that resides on it, such as Full Clone and Linked Clone desktop provisioning and Linked Clone refresh and recompose operations. Full Clone Provisioning The operational efficiency for datacenter administrators is determined mainly by the completion rate of desktop delivery (provisioning) and management operations. It is critical for datacenter administrators that the provisioning and maintenance operations on VDI desktops finish in a timely manner to be ready for production users. The time it takes to provision the desktops is directly related to storage performance capabilities. As shown in Figure 1 and Figure 2, XtremIO X2 is handling storage bandwidths as high as ~30GB/s with over 200k IOPS (read+write) during a 4000 Full Clone desktops provisioning phase, resulting in a quick and efficient desktop delivery (110 minutes for all 4000 Full Clone desktops).

Figure 1.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Full Clone Desktops Provisioning


Figure 2.

XtremIO X2 Read vs. Write IOPS and I/O Bandwidth – 4000 Full Clone Desktops Provisioning

It took 110 minutes for the system to finish the provisioning and OS customization of all 4000 desktops against our X2 array. We can deduce that desktops were provisioned in our test at an excellent rate of about 36.3 desktops per minute, or one desktop provisioned every 1.65 seconds. Figure 3 shows the block sizes distribution during the Full Clone provisioning process. We can see that most of the bandwidth used is 256kB blocks, as this is the block size that was configured at the software level (VMware) to use against our storage array.

Figure 3.

XtremIO X2 Bandwidth by Block Size – 4000 Full Clone Desktops Provisioning

In Figure 4, we can see the IOPS and latency stats in our Full Clone provisioning process of 4000 desktops. The graph shows again that IOPS are well over 200k but that the latency for all I/O operations remains less than 0.4msec, yielding the excellent performance and fast-paced provisioning of our virtual desktop environment.


Figure 4.

XtremIO X2 Latency vs. IOPS – 4000 Full Clone Desktops Provisioning

Figure 5 shows the CPU utilization of our Storage Controllers during the Full Clone provisioning stage. We can see that our CPUs are well utilized during this process and get close to 80% utilization. We can also see here the excellent synergy across our X2 cluster, when all our Active-Active Storage Controllers' CPUs share the load and effort, as the CPU utilization between all are virtually equal for the entire process.

Figure 5.

XtremIO X2 CPU Utilization – 4000 Full Clone Desktops Provisioning

Figure 6 shows XtremIO's incredible storage savings for the scenario of 4000 Full Clone desktops provisioned (each with about 13.5GB used space in their 40GB-sized C: drive volume). Notice that the physical capacity footprint of the 4000 desktops after XtremIO deduplication and compression is at 1.38TB, when the logical capacity is 51.42TB. This is a direct result of a data reduction factor of a whopping 40.4:1 (15.3:1 for deduplication and 2.6:1 for compression). Thin provisioning further-adds to storage efficiency, aggregating it to 141.3:1.


Figure 6.

XtremIO X2 Data Savings – 4000 Full Clone Desktops Provisioning

Linked Clone Provisioning As with Full Clones, we are also examining storage statistics for 4000 Linked Clone desktops provisioning. As Figure 7 and Figure 8 show, our X2 array is peaking to a ~30GB/s bandwidth only for a few moments at the beginning of the provisioning process, when the rest of the work is handled by an uneven amount of small I/O operations ranging mostly under 100k IOPS with recurring peaks of 150k IOPS (read+write). This I/O pattern is caused by the way Linked Clones work – using VMware snapshots, meaning no outstanding data is written to the array but pointers and VMware metadata. The entire provisioning process for the 4000 desktops took 75 minutes.

Figure 7.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Provisioning


Figure 8.

XtremIO X2 Read vs. Write IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Provisioning

The 75 minutes it took to provision our 4000 Linked Clone desktops are over 30% faster than what it took provisioning Full Clone desktops (110 minutes). It calculates to a rate of 53.3 desktops per minute, or an amazing one desktop provisioned every 1.125 seconds. Figure 9 shows the block sizes distribution during the Linked Clone provisioning process. We can see the 30GB/s peak at the beginning, with the rest of the I/O operations of small blocks which do not use much bandwidth.

Figure 9.

XtremIO X2 Bandwidth by Block Size – 4000 Linked Clone Desktops Provisioning

Examining the IOPS and latency stats during the Linked Clone provisioning process of our 4000 desktops in Figure 10, we can see the same mix of IOPS statistics that we saw in Figure 7, and a latency mostly below 0.5msec with some peaks of a higher latency, almost entirely under 1msec. These high-performance numbers are the reason for the excellent provision rate we achieved in our test.


Figure 10.

XtremIO X2 Latency vs. IOPS – 4000 Linked Clone Desktops Provisioning

Figure 11 shows the CPU utilization of our Storage Controllers during the Linked Clone provisioning process. This process is less demanding than the Full Clone provisioning process, due to significantly less data written, and we can see that the CPU utilization of the Storage Controllers stays normally at around 20%, with some peaks going over 40%.

Figure 11.

XtremIO X2 CPU Utilization – 4000 Linked Clone Desktops Provisioning

Figure 12 shows us the incredible efficiency that we get in storage capacity when using Linked Clones on XtremIO X2. The 4000 Linked Clone desktops we provisioned takes up a logical footprint of only 1.05TB, when an impressive data reduction factor of 14.3:1 (4.8:1 for deduplication and 3:1 for compression) lowers the physical footprint to an incredible 104.6GB for the entirety of the 4000 desktops! Thin provisioning is also a great saving factor especially with Linked Clones (here with almost 99% savings), as the desktops are merely VMware snapshots of an original parent machine, and consume no space until changes are being made on them.


Figure 12.

XtremIO X2 Data Savings – 4000 Linked Clone Desktops Provisioning

Linked Clone Pool Refresh A Linked Clone pool refresh operation restores the operating system disk of each of the Linked Clones in the pool to its original state and size as it was at the initial provisioning. Since Linked Clones are VMware snapshots of a template, they gain size and complexity as time goes on and the Linked Clone desktop is being used. Changes are made on the desktops during the users' work on them and the snapshots consume storage space and their load on the infrastructure increases. Refreshing a Linked Clone desktop pool frees up the space consumed by the snapshots over time and makes them lighter on the system again. This is of course an operation to be done on desktops where no changes are important to keep on the desktops themselves, which is the main objective of Linked Clone desktops. Scheduling a refresh operation periodically during off-peak hours is a common approach. Figure 13 and Figure 14 showcase bandwidth and IOPS stats on our XtremIO X2 single X-Brick cluster during a desktop pool refresh operation. The amount of IOPS and bandwidth required by the system is nothing our storage array cannot handle, as heavier loads were already viewed in the provisioning phases.

Figure 13.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Refresh


Figure 14.

XtremIO X2 Read vs. Write IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Refresh

The time it took for the system to refresh all 4000 Linked Clone desktops – only 20 minutes – translates to a fantastic rate of 200 desktop refreshes per minute, or one desktop refresh every 0.3 seconds. We can see from the graph in Figure 15 that the refresh operation does not strain the storage array from a compute perspective, as CPU utilization does not go above 10% for all Storage Controllers.

Figure 15.

XtremIO X2 CPU Utilization – 4000 Linked Clone Desktops Refresh

When using Linked Clone desktops, it is highly recommended to enable the "Reclaim VM disk space" option in the Horizon interface, which complements the refresh operation from the storage side: space that is freed up in the system during the refresh operation is later reclaimed by the storage array, completing the process of refreshing the infrastructure to its initial state. Space reclamation is further demonstrated in VMware Horizon View Server Configuration on page 56. Linked Clone Pool Recompose A Linked Clone pool recompose operation reassembles the operating system disk of each Linked Clone in the pool to an updated operating system version, after upgrading the parent virtual machine of the pool to a new required version or state. Any changes that are required to an entire VDI pool are performed on the parent virtual machine (template) and the recompose operation updates those changes to the entire pool. The recompose operation is another feature that exists in Linked Clone pools and not in Full Clone pools, and creates an advantage in the process of upgrading or updating virtual desktops' operating systems. Just as with the refresh operation, recomposing a Linked Clone pool is recommended during off-peak hours, and can be done in a phased manner. A recompose operation is relatively more expensive than a refresh operation. Figure 16 and Figure 17 showcase bandwidth and IOPS stats on our X2 cluster during a desktop pool recompose operation. We see IOPS peaks 10 times higher and bandwidth peaks 2 times higher than those in the refresh operation, but still more than manageable by our storage array.


Figure 16.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Recompose

Figure 17.

XtremIO X2 Read vs. Write IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Recompose

It took the system about 106 minutes to recompose all 4000 Linked Clone desktops, meaning a rate of 37.7 desktops recomposed per minute, or one desktop recomposed every 1.59 seconds. Figure 18 shows the CPU Utilization of X2 Storage Controllers during the recompose operation, which are heavier on the array than the refresh operation, but still does not peak above 40%.

Figure 18.

XtremIO X2 CPU Utilization – 4000 Linked Clone Desktops Recompose

Production Use Performance Results This section examines how an XtremIO X2 single X-Brick cluster delivers the best-in-class user experience with high performance during a boot storm and during the actual work of virtual desktop users, as emulated by LoginVSI "Knowledge Worker" – LoginVSI's workload that emulates more advanced users (details below). 15 | Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and VMware Horizon Enterprise 7.2 © 2017 Dell Inc. or its subsidiaries.

Boot Storms The rebooting of VDI desktops at a large scale is a process often orchestrated by administrators by invoking the vSphere task of rebooting virtual machines asynchronously (albeit sequentially), but it can also be performed by the end user. It is necessary, for instance, in scenarios where new applications or operating system updates are installed and need to be deployed to the virtual desktops. Desktops are issued with a reboot without waiting for previous ones to finish booting up. As a result, multiple desktops boot up at a given instant. The number of concurrent reboots is also affected by the limit configured in the vCenter server configurations. This configuration can be altered after some experimentation to determine how many concurrent operations a given vCenter server is capable of handling. Figure 19 and Figure 20 show storage bandwidth consumption and IOPS for the processes of rebooting 4000 Full Clone and Linked Clone virtual desktops respectively. In each case, the entire process took about 10 minutes when processed by our single X-Brick X2 cluster.

Figure 19.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Full Clone Desktops Boot Storm

Figure 20.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Linked Clone Desktops Boot Storm

The 10 minutes it took to reboot the 4000 desktops in both occasions translates to an amazing rate of 6.67 desktops every second, or one desktop boot per 150 milliseconds. Looking closely at the figures above, we can see that even though the process with Linked Clones required more IOPS with a lower bandwidth, it was still able to complete in 10 minutes, just as it took to the process with Full Clones. We will explain this next using the block distribution graphs and XtremIO X2 advanced Write Boost feature.


Figure 21 and Figure 22 show us the block distribution of I/Os to our X2 array during the 4000 Full Clone desktops boot storm. We can observe a majority of I/Os with block sizes that range between 4kB-32kB (Figure 22) while a sizable chunk of the bandwidth was actually used by bigger blocks, mainly 32kB, 512kB and >1mB sized blocks.

Figure 21.

XtremIO X2 Bandwidth by Block Size – 4000 Full Clone Desktops Boot Storm

Figure 22.

XtremIO X2 I/Os per Block Size – 4000 Full Clone Desktops Boot Storm

Figure 23 and Figure 24 show the block distribution during the 4000 Linked Clone desktops boot storm. We can see that the I/Os per block size remain the same for most sizes as they were in the same process for Full Clone desktops, except a big rise in I/O operations of sizes 512B and 4kB. On the bandwidth side, we can see significantly less big blocks going through the system, with 512kB blocks largely reduced in comparison to the Full Clone desktop boot storm process and >1mB blocks non-apparent. This results in a much lower bandwidth capacity (about half) with Linked Clone boot storm compared to Full Clone boot storm.


Figure 23.

XtremIO X2 Bandwidth by Block Size – 4000 Linked Clone Desktops Boot Storm

Figure 24.

XtremIO X2 I/Os per Block Size – 4000 Linked Clone Desktops Boot Storm

The reason for the change in block distribution between the two processes is due to the difference between Full Clones and Linked Clones. Since Full Clones are full copies of an OS image, they are more "data-driven" (big blocks), as each desktop is a full set of blocks in the system (though deduped inside the XtremIO storage array). Linked Clones, on the other hand, are not copies of an OS image, rather than VMware snapshots of one, making them more "metadata-driven" (higher number of small blocks). Though a boot storm of Linked Clone desktops requires much more I/Os that are smaller, the time it takes for the boot storm to complete isn't downgraded when compared to the Full Clone boot storm. This is due to XtremIO X2's advanced Write Boost feature, which groups together multiple small I/O operations before sending them to the array. This proactively solves the exact issue that could arise during the Linked Clone boot storm process – a big amount of small I/Os compromising the system's performance. With X2 and Write Boost, this becomes a non-issue for our VDI environment (though named "Write Boost", the feature also helps read operations the same way as with writes). For more information, see Write Boost on page 39. Figure 25 and Figure 26 show the CPU utilization for the 4000 Full Clone and Linked Clone boot storms, respectively.


Figure 25.

XtremIO X2 CPU Utilization – 4000 Full Clone Desktops Boot Storm

Figure 26.

XtremIO X2 CPU Utilization – 4000 Linked Clone Desktops Boot Storm

We can see that for the 4000 Full Clone boot storm, the CPU is well utilized in a range between 60% and 80%, while with Linked Clones the range is slightly higher and around 80% during the entire process, mainly due to the increase in I/O operation and the use of Write Boost when booting up Linked Clones. [8]

For a demo video of a boot storm with XtremIO, see XtremIO X2 4000 VM Boot Demo . LoginVSI Results In this section, we present the LoginVSI "Knowledge Worker" workload results for the 4000 Full Clone and Linked Clone desktops. The "Knowledge Worker" profile of LoginVSI emulates user actions such as opening a Word document, modifying an Excel spreadsheet, browsing a PDF document, web browsing or streaming a webinar. This emulates typical "advanced" user behavior and helps us characterize XtremIO's performance in such scenarios. While characterizing the user experience in those scenarios, any I/O latency that is detected in the storage array is of the utmost importance. This is a parameter that directly influences the end user experience in addition to CPU and memory stats on the ESX hosts, and the storage network's bandwidth utilization.


Figure 27.

LoginVSI's "Knowledge Worker" Workload Profile

We chose Microsoft Windows 10 (32-bit) as the desktop operating system. Additionally, Office 2016 suite, Adobe Reader 11 and the latest Oracle JRE are installed and used by LoginVSI's "Knowledge Worker" workloads. Figure 28 and Figure 29 show the LoginVSI results of our 4000 Full Clone and Linked Clone desktops respectively. LoginVSI scores are determined by observing the average application latencies, highlighting the speed at which user operations are completed. This directly lets us quantify user experience, since the measurements considered are at the application level. As a case in point, the blue line in each of the LoginVSI charts follows the progression of the "VSI average" against the number of active sessions. This is an aggregated metric, using average application latencies as more desktop sessions are added over time. The factor we should watch in these graphs is the VSImax threshold (the red line shown in Figure 29 and specified in both), which represents the threshold beyond which LoginVSI's methodology notifies us that the user experience has deteriorated and that the point where the maximum number of desktops that can be consolidated in a given VDI infrastructure has been reached.


Figure 28.

LoginVSI's "Knowledge Worker" Results – 4000 Full Clone Desktops

Figure 29.

LoginVSI's "Knowledge Worker" Results – 4000 Linked Clone Desktops

From the average shown in both graphs (the blue line), the application latency quantified is much lower than the VSImax threshold watermark for the 4000 active users (979 average vs. a 1756 threshold for Full Clones and 1240 average vs. a 1776 threshold for Linked Clones). This shows how XtremIO X2 all-flash single X-Brick cluster demonstrates a best-inclass delivery of user experience for up to 4000 VDI users, with room to scale even further. More details about LoginVSI test methodology can be found in Appendix A – Test Methodology and in the LoginVSI [25] documentation .


These LoginVSI results help us understand the user experience and are a testimony of the scalability and performance that manifests into an optimal end user experience with XtremIO X2. The obvious reason, as highlighted by Figure 30 and Figure 31, is none other than the outstanding storage latency demonstrated by XtremIO X2.

Figure 30.

XtremIO X2 Latency vs. IOPS – 4000 Full Clone Desktops In-Use

Figure 31.

XtremIO X2 Latency vs. IOPS – 4000 Linked Clone Desktops In-Use

For the Full Clone desktops, we can see a steady and remarkable ~0.1msec latency for the entire LoginVSI workload test. For the Linked Clone desktops, we see a small rise in latency numbers as IOPS accumulate, but never past 0.4msec. These numbers yield the great LoginVSI results we saw above, and provide a superb user experience for our VDI users. Figure 32 and Figure 33 present us with total IOPS and bandwidth seen during the LoginVSI "Knowledge Worker" profile workload on our 4000 Full Clone desktops and 4000 Linked Clone desktops respectively. In both occasions, the IOPS at the peak of the workload reach about 80k, while the peak bandwidth is different between the two setups: for Full Clones – ~7.5GB/s and for Linked Clones ~1.5GB/s. This fits our earlier remarks about Full Clones being more "data-driven" and consuming more bandwidth in the storage network than Linked Clones.


Figure 32.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Full Clone Desktops In-Use

Figure 33.

XtremIO X2 IOPS and I/O Bandwidth – 4000 Linked Clone Desktops In-Use

Figure 34 and Figure 35 show CPU Utilization of our 4000 Full Clone and Linked Clone desktops respectively during the LoginVSI "Knowledge Worker" profile workload. We can see that the CPU utilization at the peak of the workload get to about 20% for the Full Clone desktops, and about 40% for the Linked Clone desktops. This fits what we observed in the latency and LoginVSI results figures that with Linked Clones, although they save much space and provide various advantages, the workloads are slightly heavier due to them being based on VMware snapshots.

Figure 34.

XtremIO X2 CPU Utilization – 4000 Full Clone Desktops In-Use


Figure 35.

XtremIO X2 CPU Utilization – 4000 Linked Clone Desktops In-Use

Examining all the graphs collected during the LoginVSI "Knowledge Worker" profile workload test, we see that our X2 single-brick is more than capable of managing and servicing 4000 VDI working stations (Full Clones or Linked Clones), with room for additional volumes and workloads to serve. We are also taking a deeper look at our ESX hosts to see if our scaling fits from a compute-resources perspective as well. We are looking at both the CPU utilization of our ESX hosts (Figure 36) and their Memory utilization (Figure 37) during the LoginVSI "Knowledge Worker" profile workload test on the 4000 Linked Clone desktops.

Figure 36.

ESX Hosts CPU Utilization – 4000 Linked Clone Desktops In-Use


Figure 37.

ESX Hosts Memory Utilization – 4000 Linked Clone Desktops In-Use

We can see an approximate 70% utilization of both CPU and memory resources of our ESX hosts, indicating a wellutilized environment and good resource consumption of our ESX hosts, leaving room for some extra VMs in the environment, as well as spare resources for any need of a VMotion of VMs (due to hosts failures, planned upgrades, etc.). In Figure 38 below, we see the change in CPU utilization of a single ESX host in the environment as the LoginVSI "Knowledge Worker" profile workload test progresses. The test creates logins and workloads to the virtual desktops in an accumulative way, emulating a typical working environment in which users log in during a span of a few dozen minutes and not all at the same time. This behavior is seen clearly in the figure below, as the CPU utilization of this ESX host increases as time passes, until all virtual desktops in the host are in use and gets its CPU utilization to about 70%.


Figure 38.

A Single ESX Host CPU Utilization – 4000 Linked Clone Desktops In-Use

Solution's Hardware Layer Based on the data presented above, it is evident that storage/virtualization administrators must strive to achieve an optimal user experience for their VDI desktop end users. The following sections discuss how the hardware and software synergize in order to achieve these goals. We begin at the hardware layer, taking a wide look at our XtremIO X2 array and the features and benefits it provides to VDI environments, continue by discussing the details of our ESX hosts, based on Dell PowerEdge servers, on which our entire environment runs, and then scan our storage configuration and networks that connect the servers to the storage array, encompassing a complete look of the hardware components of the solution. We follow this up with details of the software layer by providing configuration details for VMware vSphere, VMware Horizon View Suite, Dell and EMC plugins for VMware and configuration settings on the "parent" virtual machine from which VDI desktops are deployed.

Storage Array: Dell EMC XtremIO X2 All-Flash Array Dell EMC's XtremIO is an enterprise-class scalable all-flash storage array that provides rich data services with high performance. It is designed from the ground up to unlock flash technology's full performance potential by uniquely leveraging the characteristics of SSDs and uses advanced inline data reduction methods to reduce the physical data that must be stored on the disks. XtremIO's storage system uses industry-standard components and proprietary intelligent software to deliver unparalleled levels of performance, achieving consistent low latency for up to millions of IOPS. It comes with a simple, easy-to-use interface for storage administrators and fits a wide variety of use cases for customers in need of a fast and efficient storage system for their datacenters, requiring very little planning to set-up before provisioning.


XtremIO leverages flash to deliver value across multiple dimensions: •

Performance (consistent low-latency and up to millions of IOPS)

•

Scalability (using a scale-out and scale-up architecture)

•

Storage efficiency (using data reduction techniques such as deduplication, compression and thin-provisioning)

•

Data Protection (with a proprietary flash-optimized algorithm named XDP)

•

Environment Consolidation (using XtremIO Virtual Copies or VMware's XCOPY).

We will further review XtremIO X2 features and capabilities. XtremIO X2 Overview XtremIO X2 is the new generation of the Dell EMC's All-Flash Array storage system. It adds enhancements and flexibility in several aspects to the already proficient and high-performant storage array's former generation. Features such as scale-up for a more flexible system, write boost for a more sensible and high-performing storage array, NVRAM for improved data availability and a new web-based UI for managing the storage array and monitoring its alerts and performance stats, add the extra value and advancements required in the evolving world of computer infrastructure. The XtremIO X2 Storage Array uses building blocks called X-Bricks. Each X-Brick has its own compute, bandwidth and storage resources, and can be clustered together with additional X-Bricks to grow in both performance and capacity (scale-out). Each X-Brick can also grow individually in terms of capacity, with an option to add to up to 72 SSDs in each brick. XtremIO architecture is based on a metadata-centric content-aware system, which helps streamlining data operations efficiently without requiring any movement of data post-write for any maintenance reason (data protection, data reduction, etc. – all done inline). The system lays out the data uniformly across all SSDs in all X-Bricks in the system using unique fingerprints of the incoming data and controls access using metadata tables. This contributes to an extremely balanced system across all X-Bricks in terms of compute power, storage bandwidth and capacity. Using the same unique fingerprints, XtremIO is equipped with exceptional always-on in-line data deduplication abilities, which highly benefits virtualized environments. Together with its data compression and thin provisioning capabilities (both also in-line and always-on), it achieves incomparable data reduction rates. System operation is controlled by storage administrators via a stand-alone dedicated Linux-based server called the XtremIO Management Server (XMS). An intuitive user interface is used to manage and monitor the storage cluster and its performance. The XMS can be either a physical or a virtual server and can manage multiple XtremIO clusters. With its intelligent architecture, XtremIO provides a storage system that is easy to set-up, needs zero tuning by the client and does not require complex capacity or data protection planning, as the system handles it on its own. Architecture and Scalability An XtremIO X2 Storage System is comprised of a set of X-Bricks that form together a cluster. This is the basic building block of an XtremIO array. There are two types of X2 X-Bricks available: X2-S and X2-R. X2-S is for environments whose storage needs are more I/O intensive than capacity intensive, as they use smaller SSDs and less RAM. An effective use of the X2-S is for environments that have high data reduction ratios (high compression ratio or a lot of duplicated data) which lower the capacity footprint of the data significantly. X2-R X-Bricks clusters are made for the capacity intensive environments, with bigger disks, more RAM and a bigger expansion potential in future releases. The two X-Brick types cannot be mixed together in a single system, so the decision which type is suitable for your environment must be made in advance.


Each X-Brick is comprised of: •

•

Two 1U Storage Controllers (SCs) with: o

Two dual socket Haswell CPUs

o

346GB RAM (for X2-S) or 1TB RAM (for X2-R)

o

Two 1/10GbE iSCSI ports

o

Two user interface interchangeable ports (either 4/8/16Gb FC or 1/10GbE iSCSI)

o

Two 56Gb/s InfiniBand ports

o

One 100/1000/10000 Mb/s management port

o

One 1Gb/s IPMI port

o

Two redundant power supply units (PSUs)

One 2U Disk Array Enclosure (DAE) containing: o

Up to 72 SSDs of sizes 400GB (for X2-S) or 1.92TB (for X2-R)

o

Two redundant SAS interconnect modules

o

Two redundant power supply units (PSUs)

1U

Second Storage Controller

1U

First Storage Controller

2U

DAE

4U

Figure 39.

An XtremIO X2 X-Brick

The Storage Controllers on each X-Brick are connected to their DAE via redundant SAS interconnects. An XtremIO storage array can have one or multiple X-Bricks. Multiple X-Bricks are clustered together into an XtremIO array, using an InfiniBand switch and the Storage Controllers' InfiniBand ports for back-end connectivity between Storage Controllers and DAEs across all X-Bricks in the cluster. The system uses the Remote Direct Memory Access (RDMA) protocol for this back-end connectivity, ensuring a highly-available ultra-low latency network for communication between all components of the cluster. The InfiniBand switches are the same size (1U) for both X2-S and X2-R cluster types, but include 12 ports for X2-S and 36 ports for X2-R. By leveraging RDMA, an XtremIO system is essentially a single sharedmemory space spanning all of its Storage Controllers. The 1GB port for management is configured with an IPv4 address. The XMS, which is the cluster's management software, communicates with the Storage Controllers via the management interface. Through this interface, the XMS communicates with the Storage Controllers and sends storage management requests such as creating an XtremIO Volume, mapping a Volume to an Initiator Group, etc. The second 1GB/s port for IPMI interconnects the X-Brick's two Storage Controllers. IPMI connectivity is strictly within the bounds of an X-Brick, and will never be connected to an IPMI port of a Storage Controller in another X-Brick in the cluster.


With X2, an XtremIO cluster has both scale-out and scale-up capabilities. Scale-out is implemented by adding X-Bricks to an existing cluster. The addition of an X-Brick to an existing cluster increases its compute power, bandwidth and capacity linearly. Each X-Brick that is added to the cluster brings with it two Storage Controllers, each with its CPU power, RAM and FC/iSCSI ports to service the clients of the environment, together with a DAE with SSDs to increase the capacity provided by the cluster. Adding an X-Brick to scale-out an XtremIO cluster is intended for environments that grow both in capacity and performance needs, such as in the case of an increase in the number of active users and the data that they hold, or a database which grows in data and complexity. An XtremIO cluster can start with any number of X-Bricks that fits the environment's initial needs and can currently grow to up to 4 X-Bricks (for both X2-S and X2-R). Future code upgrades of XtremIO X2 will allow up to 8 supported X-Bricks for X2-R arrays.

Figure 40.

Scale Out Capabilities – Single to Multiple X2 X-Brick Clusters

Scale-up of an XtremIO cluster is implemented by adding SSDs to existing DAEs in the cluster. Adding SSDs to existing DAEs to scale-up an XtremIO cluster is intended for environments that currently grow in capacity needs and has no need for extra performance. This occurs, for example, when the same number of users have an increasing amount of data to save, or when an environment grows in both capacity and performance needs but has only reached its capacity limits with still room to grow in performance with its current infrastructure. Each DAE can hold up to 72 SSDs, and is divided to up to 2 groups of SSDs called Data Protection Groups (DPGs). Each DPG can hold a minimum of 18 SSDs and can grow by increments of 6 SSDs up to the maximum of 36 SSDs. In other words, 18, 24, 30 or 36 is the possible number of SSDs per DPG, when up to 2 DPGs can occupy a DAE. SSDs are 400GB per drive for X2-S clusters and 1.92TB per drive for X2-R clusters. Future releases will allow customers to populate their X2-R clusters with 3.84TB sized drives, doubling the physical capacity available in their clusters.


Figure 41.

Scale Up Capabilities – Up to 2 DPGs and 72 SSDs per DAE

For more details on XtremIO X2 see the XtremIO X2 Specifications

[3]

[4]

and XtremIO X2 Datasheet .

XIOS and the I/O Flow Each Storage Controller within the XtremIO cluster runs a specially-customized lightweight Linux-based operating system as the base platform of the array. The XtremIO Operating System (XIOS) handles all activities within a Storage Controller and runs on top of the Linux-based operating system. XIOS is optimized for handling high I/O rates and manages the system's functional modules, RDMA communication, monitoring etc.

Figure 42.

X-Brick Components


XIOS has a proprietary process scheduling-and-handling algorithm designed to meet the specific requirements of a content-aware, low-latency, and high-performing storage system. It provides efficient scheduling and data access, full exploitation of CPU resources, optimized inter-sub-process communication, and minimized dependency between subprocesses that run on different sockets. The XtremIO Operating System gathers a variety of metadata tables on incoming data which includes data fingerprint, its location in the system, mappings and reference counts. The metadata is used as the fundamental insight for performing system operations such as laying out incoming data uniformly, implementing inline data reduction services and accessing the data on read requests. The metadata is also involved in communication with external applications (such as VMware XCOPY and Microsoft ODX) to optimize integration with the storage system. Regardless of which Storage Controller receives an I/O request from a host, multiple Storage Controllers on multiple XBricks cooperate to process the request. The data layout in the XtremIO system ensures that all components share the load and participate evenly in processing I/O operations. An important functionality of XIOS is its data reduction capabilities. This is achieved by using inline data deduplication and compression. Data deduplication and data compression complement each other. Data deduplication removes redundancies, whereas data compression compresses the already deduplicated data before it is written to the flash media. XtremIO is an always-on thin-provisioned storage system, further realizing storage savings by the storage system, which never writes a block of zeros to the disks. XtremIO integrates with existing SANs through 16Gb/s Fibre Channel or 10Gb/s Ethernet iSCSI connectivity to service hosts' I/O requests. Details of the XIOS architecture and its data reduction capabilities are available in the Introduction to DELL EMC XtremIO [2] X2 Storage Array document .

XtremIO Write I/O Flow In a write operation to the storage array, the incoming data stream reaches any one of the Active-Active Storage Controllers and is broken into data blocks. For every data block, the array fingerprints the data with a unique identifier and stores it in the cluster's mapping table. The mapping table maps the host's Logical Block Addresses (LBA) to the blocks' fingerprints and the blocks' fingerprints to its physical location in the array (the DAE, SSD and offset the block is located at). The fingerprint of a block has two objectives: to determine if the block is a duplicate of a block that already exists in the array; to distribute blocks uniformly across the cluster (the array divides the list of potential fingerprints among Storage Controllers in the array and gives each Storage Controller a range of fingerprints to be in charge of. The mathematical process that calculates the fingerprints results in a uniform distribution of fingerprint values and thus fingerprints and blocks are evenly spread across all Storage Controllers in the cluster). A write operation works as such: 1.

A new write request reaches the cluster

2.

The new write is broken into data blocks

3.

For each data block: 1.

A fingerprint is calculated for the block.

2.

An LBA-to-fingerprint mapping is created for this write request.

3.

The fingerprint is checked to see if it already exists in the array. • If it exists: o The reference count for this fingerprint is incremented by one.


• If it does not exist: 1.

A location is chosen on the array where the block will be written (distributed uniformly across the array according to fingerprint value).

2.

A fingerprint-to-physical location mapping is created.

3.

The data is compressed.

4.

The data is written.

5.

The reference count for the fingerprint is set to one.

Deduplicated writes are of course much faster than original writes. Once the array identifies a write as a duplicate, it updates the LBA-to-fingerprint mapping for the write and updates the reference count for this fingerprint. No data is further written to the array and the operation is completed quickly, adding an extra benefit to in-line deduplication. Figure 43 shows an example of an incoming data stream which contains duplicate blocks with identical fingerprints.

Figure 43.

Incoming Data Stream Example with Duplicate blocks

As mentioned, fingerprints also help to decide where to write the block in the array. Figure 44 shows the incoming stream demonstrated in Figure 43, after duplicates were removed, as it is being written to the array. The blocks are divided to their appointed Storage Controller according to their fingerprint value, which ensures a uniform distribution of the data across the cluster. The blocks are transferred to their destinations in the array using Remote Direct Memory Access (RDMA) via the low-latency InfiniBand network.


F, …

Storage Controller

Data

DAE Storage Controller

Data Data

InfiniBand

2, A, …

X-Brick 2 X-Brick 1

1, 9, …

Storage Controller

Data Data

Figure 44.

Data

Data

Data

Data

Data

Data

134F871

0325F7A

F3AFBA3

AB45CB7

20147A8

0, C, …

Data

CA38C90

Storage Controller

Data Data

963FE7B

DAE

Incoming Deduplicated Data Stream Written to the Storage Controllers

The actual write of the data blocks to the SSDs is carried out asynchronously. At the time of the application write, the system places the data blocks in the in-memory write buffer, and protects it using journaling to local and remote NVRAMs. Once it is written to the local NVRAM and replicated to a remote one, the Storage Controller returns an acknowledgment to the host. This guarantees a quick response to the host, ensures low-latency of I/O traffic and preserves the data in case of system failure (power-related or any other). When enough blocks are collected in the buffer (to fill up a full stripe), the system writes them to the SSDs on the DAE. Figure 45 demonstrates the phase of writing the data to the DAEs after a full stripe of data blocks is collected in each Storage Controller. Storage Controller Data Data Data

Data Data Data Data Data Data Data Data

P1 P2

DAE Data Data Data Data Data

Data Data Data Data Data Data Data

P1 P2

Storage Controller

X-Brick 2 X-Brick 1 Storage Controller Data Data Data Data Data


P1 P2

Data Data Data Data Data


P1 P2

DAE Storage Controller

Figure 45.

Full Stripe of Blocks Written to the DAEs


XtremIO Read I/O Flow In a read operation, the system first performs a look-up of the logical address in the LBA-to-fingerprint mapping. The found fingerprint is then looked up in the fingerprint-to-physical mapping and the data is retrieved from the right physical location. Just as with writes, the read load is also evenly shared across the cluster, as blocks are evenly distributed, and all volumes are accessible across all X-Bricks. If the requested block size is larger than the data block size, the system performs parallel data block reads across the cluster and assembles them into bigger blocks before returning them to the application. A compressed data block is decompressed before it is delivered to the host. XtremIO has a memory-based read cache in each Storage Controller. The read cache is organized by content fingerprint. Blocks whose contents are more likely to be read are placed in the read cache for a fast retrieve. A read operation works as such: 1.

A new read request reaches the cluster.

2.

The read request is analyzed to determine the LBAs for all data blocks and a buffer is created to hold the data.

3.

For each LBA:

4.

1.

The LBA-to-fingerprint mapping is checked to find the fingerprint of each data block to be read.

2.

The fingerprint-to-physical location mapping is checked to find the physical location of each of the data blocks.

3.

The requested data block is read from its physical location (read cache or a place in the disk) and transmitted to the buffer created in step 2 in the Storage Controller that processes the request via RDMA over InfiniBand.

The system assembles the requested read from all data blocks transmitted to the buffer and sends it back to the host.

System Features The XtremIO X2 Storage Array provides and offers a wide range of built-in features that require no special license. The architecture and implementation of these features is unique to XtremIO and is designed around the capabilities and limitations of flash media. We will list some key features included in the system.

Inline Data Reduction XtremIO's unique Inline Data Reduction is achieved by these two mechanisms: Inline Data Deduplication and Inline Data Compression Data Deduplication Inline Data Deduplication is the removal of duplicate I/O blocks from a stream of data prior to it being written to the flash media. XtremIO inline deduplication is always on, meaning no configuration is needed for this important feature. The deduplication is at a global level, meaning no duplicate blocks are written over the entire array. Being an inline and global process, no resource-consuming background processes or additional reads and writes (which are mainly associated with post-processing deduplication) are necessary for the feature's activity, which increases SSD endurance and eliminates performance degradation. As mentioned earlier, deduplication on XtremIO is performed using the content's fingerprints (see XtremIO Write I/O Flow on page 31). The fingerprints are also used for uniform distribution of data blocks across the array, which provides inherent load balancing for performance and enhances flash wear-level efficiency, since the data never needs to be rewritten or rebalanced.


XtremIO uses a content-aware, globally deduplicated Unified Data Cache for highly efficient data deduplication. The system's unique content-aware storage architecture enables achieving a substantially larger cache size with a small DRAM allocation. Therefore, XtremIO is the ideal solution for difficult data access patterns, such as "boot storms" that are common in VDI environments. XtremIO has excellent data deduplication ratios, especially for virtualized environments. With it, SSD usage is smarter, flash longevity is maximized, the logical storage capacity is multiplied (see Figure 6 and Figure 12 for examples) and total cost of ownership is reduced. Data Compression Inline data compression is the compression of data prior to it being written to the flash media. XtremIO automatically compresses data after all duplications are removed, ensuring that the compression is performed only for unique data blocks. The compression is performed in real-time and not as a post-processing operation. This way, it does not overuse the SSDs or impact performance. Compressibility rates depend on the type of data written. Data Compression complements data deduplication in many cases, and saves storage capacity by storing only unique data block in the most efficient manner. We can see the benefits and capacity savings for the deduplication-compression combination demonstrated in Figure 46 and some real ratios in the Test Results section in Figure 6 and Figure 12. This is the only data written to the flash media.

Data Written by Host

3:1 Data Deduplication

2:1 Data Compression

6:1 Total Data Reduction Figure 46.

Data Deduplication and Data Compression Demonstrated

Thin Provisioning XtremIO storage is natively thin provisioned, using a small internal block size. All volumes in the system are thin provisioned, meaning that the system consumes capacity only when it is needed. No storage space is ever pre-allocated before writing. Because of XtremIO's content-aware architecture, blocks can be stored at any location in the system (when the metadata is used to refer to their location) and the data is written only when unique blocks are received. Therefore, as opposed to disk-oriented architecture, no space creeping or garbage collection is necessary on XtremIO, volume fragmentation does not occur in the array and no defragmentation utilities are needed. This feature on XtremIO enables consistent performance and data management across the entire life cycle of a volume, regardless of the system capacity utilization or the write patterns of clients.


Integrated Copy Data Management XtremIO pioneered the concept of integrated Copy Data Management (iCDM) – the ability to consolidate both primary data and its associated copies on the same scale-out all-flash array for unprecedented agility and efficiency. XtremIO is one of a kind in its capabilities to consolidate multiple workloads and entire business processes safely and efficiently, providing organizations with a new level of agility and self-service for on-demand procedures. XtremIO provides consolidation, supporting on-demand copy operations at scale, and still maintains delivery of all performance SLAs in a consistent and predictable way. Consolidation of primary data and its copies in the same array has numerous benefits: •

It can make development and testing activities up to 50% faster, creating copies of production code quickly for development and testing purposes, and then refreshing the output back into production for the full cycle of code upgrades in the same array. This dramatically reduces complexity and infrastructure needs, as well as development risks, and increases the quality of the product.

•

Production data can be extracted and pushed to all downstream analytics applications on-demand as a simple inmemory operation. Copies of the data are high performance and can get the same SLA as production copies without compromising production SLAs. XtremIO offers this on-demand as both self-service and automated workflows for both application and infrastructure teams.

•

Operations such as patches, upgrades and tuning tests can be made quickly using copies of production data. Diagnosing problems of applications and databases can be done using these copies, and applying the changes back to production can be done by refreshing copies back. The same goes for testing new technologies and combining them in production environments.

•

iCDM can also be used for data protection purposes, as it enables creating many copies at low intervals point-intime for recovery. Application integration and orchestration policies can be set to auto-manage data protection, using different SLAs.

XtremIO Virtual Copies XtremIO uses its own implementation of snapshots for all iCDM purposes, called XtremIO Virtual Copies (XVCs). XVCs are created by capturing the state of data in volumes at a particular point in time and allowing users to access that data when needed, no matter the state of the source volume (even deletion). They allow any access type and can be taken either from a source volume or another Virtual Copy. XtremIO's Virtual Copy technology is implemented by leveraging the content-aware capabilities of the system, optimized for SSDs, with a unique metadata tree structure that directs I/O to the right timestamp of the data. This allows efficient copy creation that can sustain high performance, while maximizing the media endurance.

Figure 47.

A Metadata Tree Structure Example of XVCs


When creating a Virtual Copy, the system only generates a pointer to the ancestor metadata of the actual data in the system, making the operation very quick. This operation does not have any impact on the system and does not consume any capacity at the point of creation, unlike traditional snapshots, which may need to reserve space or copy the metadata for each snapshot. Virtual Copies capacity consumption occurs only when changes are made to any copy of the data. Then, the system updates the metadata of the changed volume to reflect the new write, and stores its blocks in the system using the standard write flow process. The system supports the creation of Virtual Copies on a single, as well as on a set, of volumes. All Virtual Copies of the volumes in the set are cross-consistent and contain the exact same point-in-time for them all. This can be done manually by selecting a set of volumes for copying, or by placing volumes in a Consistency Group and making copies of that Consistency Group. Virtual Copy deletions are lightweight and proportional only to the amount of changed blocks between the entities. The system uses its content-aware capabilities to handle copy deletions. Each data block has a counter that indicates the number of instances of that block in the system. If a block is referenced from some copy of the data, it will not be deleted. Any block whose counter value reaches zero is marked as deleted and will be overwritten when new unique data enters the system. With XVCs, XtremIO's iCDM offers the following tools and workflows to provide the consolidation capabilities: •

Consistency Groups (CG) – Grouping of volumes to allow Virtual Copies to be taken on a group of volumes as a single entity.

•

Snapshot Sets – A group of Virtual Copies volumes taken together using CGs or a group of manually-chosen volumes.

•

Protection Copies – Immutable read-only copies created for data protection and recovery purposes.

•

Protection Scheduler – Used for local protection of a volume or a CG. It can be defined using intervals of seconds/minutes/hours or can be set using a specific time of day or week. It has a retention policy based on the number of copies wanted or the permitted age of the oldest snapshot.

•

Restore from Protection – Restore a production volume or CG from one of its descendant snapshot sets.

•

Repurposing Copies – Virtual Copies configured with changing access types (read-write / read-only / no-access) for alternating purposes.

•

Refresh a Repurposing Copy – Refresh a Virtual Copy of a volume or a CG from the parent object or other related copies with relevant updated data. It does not require volume provisioning changes for the refresh to take effect, but only host-side logical volume management operations to discover the changes.

XtremIO Data Protection XtremIO Data Protection (XDP) provides a "self-healing" double-parity data protection with very high efficiency to the storage system. It requires very little capacity overhead and metadata space, and does not require dedicated spare drives for rebuilds. Instead, XDP leverages the "hot space" concept, where any free space available in the array can be utilized for failed drive reconstructions. The system always reserves sufficient distributed capacity for performing at least a single drive rebuild. In the rare case of a double SSD failure, the second drive will be rebuilt only if there is enough space to rebuild the second drive as well, or when one of the failed SSDs is replaced. The XDP algorithm provides: •

N+2 drives protection

•

Capacity overhead of only 5.5%-11% (depends on the number of disks in the protection group)

•

60% more write-efficient than RAID1

•

Superior flash endurance to any RAID algorithm, due to the smaller number of writes and even distribution of data

•

Automatic rebuilds that are faster than traditional RAID algorithms


As shown in Figure 48, XDP uses a variation of N+2 row and diagonal parity which provides protection from two simultaneous SSD errors. An X-Brick DAE may contain up to 72 SSDs organized in two Data Protection Groups (DPGs). XDP is managed independently on the DPG level. A DPG of 36 SSDs will result in capacity overhead of only 5.5% for its data protection needs.

k-1

D0

D1

D2

D3

D4

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

5

1

2

3

4

P

Q

5

k = 5 (prime) Figure 48.

N+2 Row and Diagonal Parity

Data at Rest Encryption Data at Rest Encryption (DARE) provides a solution to securing critical data even when the media is removed from the array, for customers in need of such security. XtremIO arrays utilize a high-performance inline encryption technique to ensure that all data stored on the array is unusable if the SSD media is removed. This prevents unauthorized access in the event of theft or loss during transport, and makes it possible to return/replace failed components containing sensitive data. DARE has been established as a mandatory requirement in several industries, such as health care, banking, and government institutions. At the heart of XtremIO's DARE solution lays the use of the Self-Encrypting Drive (SED) technology. An SED has a dedicated hardware which is used to encrypt and decrypt data as it is written to or read from the drive. Offloading the encryption task to the SSDs enables XtremIO to maintain the same software architecture whenever encryption is enabled or disabled on the array. All XtremIO's features and services (including Inline Data Reduction, XtremIO Data Protection, Thin Provisioning, XtremIO Virtual Copies, etc.) are available on an encrypted cluster as well as on a non-encrypted cluster, and performance is not impacted when using encryption. A unique Data Encryption Key (DEK) is created during the drive manufacturing process, and does not leave the drive at any time. The DEK can be erased or changed, rendering its current data unreadable forever. To ensure that only authorized hosts can access the data on the SED, the DEK is protected by an Authentication Key (AK) that resides on the Storage Controller. Without the AK, the DEK is encrypted and cannot be used to encrypt or decrypt data.


Figure 49.

Data at Rest Encryption in XtremIO

Write Boost In the new X2 storage array, the write flow algorithm was improved significantly to improve array performance, countering the rise in compute power and disk speeds and taking into account common applications' I/O patterns and block sizes. As mentioned when discussing the write I/O flow, the commit to the host is now asynchronous to the actual writing of the blocks to disk. The commit is sent after the changes are written to a local and remote NVRAMs for protection, and are written to the disk only later, at a time that best optimizes the system's activity. In addition to the shortened procedure from write to commit, the new algorithm addresses an issue relevant to many applications and clients: a high percentage of small I/Os creating load on the storage system and influencing latency, especially on bigger I/O blocks. Examining customers' applications and I/O patterns, it was found that many I/Os from common applications come in small blocks, under than 16K pages, creating high loads on the storage array. Figure 50 shows the block size histogram from the entire XtremIO install base. The percentage of blocks smaller than 16KB is highly evident. The new algorithm takes care of this issue by aggregating small writes to bigger blocks in the array before writing them to disk, making them less demanding on the system, which is now more capable taking care of bigger I/Os faster. The test results for the improved algorithm were amazing: the improvement in latency for several cases is around 400% and allows XtremIO X2 to address application requirements with 0.5msec or lower latency.


Figure 50.

XtremIO Install Base Block Size Histogram

VMware APIs for Array Integration (VAAI) VAAI was first introduced as VMware's improvements to host-based VM cloning. It offloads the workload of cloning a VM to the storage array, making cloning much more efficient: instead of copying all blocks of a VM from the array and back to it for the creation of a new cloned VM, the application lets the array do it internally, utilizing the array's features and saving host and network resources that are no longer involved in the actual cloning of data. This procedure of offloading the operation to the storage array is backed by the X-copy (extended copy) command to the array, which is used when cloning large amounts of complex data. XtremIO is fully VAAI compliant, allowing the array to communicate directly with vSphere and provide accelerated storage vMotion, VM provisioning, and thin provisioning functionality. In addition, XtremIO's VAAI integration improves X-copy efficiency even further by making the whole operation metadata driven. Due to its inline data reduction features and inmemory metadata, no actual data blocks are copied during an X-copy command, and the system only creates new pointers to the existing data, all carried out inside the Storage Controllers' memory. Therefore, the operation saves host and network resources and does not consume storage resources, leaving no impact on the system's performance, as opposed to other implementations of VAAI and the X-copy command. Performance tests of XtremIO during X-copy operations and comparison between X1 and X2 with different block sizes [10] can be found in a dedicated post written at XtremIO's CTO blog . Figure 51 illustrates the X-copy operation when performed against an XtremIO storage array and shows the efficiency in metadata-based cloning.


A X-Copy command (full clone)

VM1 Addr 1 Addr 2 Addr 3 Addr 4 Addr 5 Addr 6

Ptr

Ptr

Ptr

Ptr

Ptr

Ptr

Metadata in RAM

A B C D

XtremIO

Data on SSD

B VM1 Addr 1 Addr 2 Addr 3 Addr 4 Addr 5 Addr 6

Copy metadata pointers Ptr

Ptr

Ptr

Ptr

Ptr

Ptr

A B C D

XtremIO

Data on SSD

C • No data blocks are copied. • New pointers are created to the existing data.

VM1

VM2 New New New New New New Addr 1 Addr 2 Addr 3 Addr 4 Addr 5 Addr 6

Addr 1 Addr 2 Addr 3 Addr 4 Addr 5 Addr 6

Metadata in RAM Ptr

Ptr

Ptr

XtremIO

Figure 51.

Ptr

Ptr

Ptr

A B C D

Ptr

Ptr

Ptr

Ptr

Ptr

Ptr

Data on SSD

VAAI X-Copy with XtremIO

The XtremIO features for VAAI support include: •

Zero Blocks / Write Same – used for zeroing-out disk regions and provides accelerated volume formatting

•

Clone Blocks / Full Copy / X-Copy – used for copying or migrating data within the same physical array, an almost instantaneous operation on XtremIO due to its metadata-driven operations

•

Record Based Locking / Atomic Test & Set (ATS) – used during creation and the locking of files on a VMFS volumes, during power-down and powering-up of VMS, for example

•

Block Delete / Unmap / Trim – used for reclamation of unused space using the SCSI UNMAP feature

Other features of XtremIO X2 (some of them were already described in previous sections): •

Scalability (scale-up and scale-out)

•

Even Data Distribution (uniformity)

•

High Availability (no single points of failures)

•

Non-disruptive Upgrade and Expansion

•

RecoverPoint Integration (for replications to local or remote arrays)


XtremIO Management Server The XtremIO Management Server (XMS) is the component that manages XtremIO clusters (up to 8 clusters). It is preinstalled with the CLI, GUI and RESTful API interfaces, and can be installed on a dedicated physical server or a VMware virtual machine. The XMS manages the cluster through the management ports on both Storage Controllers of the first X-Brick in the cluster, and uses a standard TCPI/IP connection to communicate with them. It is not part of the XtremIO data path, thus can be disconnected from an XtremIO cluster without jeopardizing usual I/O tasks. A failure on the XMS only affects monitoring and configuration activities, such as creating and attaching volumes. A virtual XMS is naturally less vulnerable to such failures. The GUI is based on a new Web User Interface (WebUI), which is accessible via any browser, and provides easy-to-use tools for performing most system operations (certain management operations must be performed using the CLI). Some of the useful features of the new WebUI are described following.

Dashboard The Dashboard window presents a main overview of the cluster. It has three panels: •

Health - the main overview of the system's health status, alerts, etc.

•

Performance (shown in Figure 52) – the main overview of the system's overall performance and top used Volumes and Initiator Groups)

•

Capacity (shown in Figure 53) – the main overview of the system's physical capacity and data savings (the screenshots in this section should not be confused with the graphs in the Test Results section)

Figure 52.

XtremIO WebUI – Dashboard – Performance Panel


Figure 53.

XtremIO WebUI – Dashboard – Capacity Panel

The main Navigation menu bar is located on the left side of the UI. Users can select one of the navigation menu options pertaining to XtremIO's management actions. The main menus contain the Dashboard, Notifications, Configuration, Reports, Hardware and Inventory.

Notifications In the Notifications menu, we can navigate to the Events window (shown in Figure 54) and the Alerts window, showing major and minor issues related to the cluster's health and operations.

Figure 54.

XtremIO WebUI – Notifications – Events Window


Configuration The Configuration window displays the cluster's logical components: Volumes (shown in Figure 55), Consistency Groups, Snapshot Sets, Initiator Groups, Initiators, and Protection Schedulers. Through this window we can create and modify these entities, using the action panel on the top right side.

Figure 55.

XtremIO WebUI – Configuration

Reports In the Reports menu, we can navigate to different windows to show graphs and data of different aspects of the system's activities, mainly related to the system's performance and resource utilization. Menu options we can choose to view contain: Overview, Performance, Blocks, Latency, CPU Utilization, Capacity, Savings, Endurance, SSD Balance, Usage or User Defined reports. We can view reports using different resolutions of time and components: selecting specific entities we want to view reports on in the "Select Entity" option (shown in Figure 56) that appears above when in the Reports menus, or selecting predefined and custom days and times to review reports for (shown in Figure 57). The graphs in the Test Results section were taken from these menus.


Figure 56.

XtremIO WebUI – Reports – Selecting Specific Entities to View

Figure 57.

XtremIO WebUI – Reports – Selecting Specific Times to View


The Overview window shows basic reports on the system, including performance, weekly I/O patterns and storage capacity information. The Performance window shows extensive performance reports which mainly include Bandwidth, IOPS and Latency information. The Blocks window shows block distribution and statistics of I/Os going through the system. The Latency window (shown in Figure 58) shows Latency reports, including as a function of block sizes and IOPS metrics. The CPU Utilization window shows CPU utilization of all Storage Controllers in the system.

Figure 58.

XtremIO WebUI – Reports – Latency Window

The Capacity window (shown in Figure 59) shows capacity statistics and the change in storage capacity over time. The Savings window shows Data Reduction statistics and change over time. The Endurance window shows SSD's endurance status and statistics. The SSD Balance window shows how much the SSDs are balanced with data and the variance between them all. The Usage window shows Bandwidth and IOPS usage, both overall and divided to reads and writes. The User Defined window allows users to define their own reports to view.

Figure 59.

XtremIO WebUI – Reports – Capacity Window


Hardware In the Hardware menu, we can overview our cluster and X-Bricks with visual illustrations. When viewing the FRONT panel, we can choose and highlight any component of the X-Brick and view information about it in the Information panel on the right. In Figure 60 we can see extended information on Storage Controller 1 in X-Brick 1, but we can view information on more granular specifics such as local disks and Status LEDs. We can further click on the "OPEN DAE" button to see visual illustration of the X-Brick's DAE and its SSDs, and view additional information on each SSD and Row Controller.

Figure 60.

XtremIO WebUI – Hardware – Front Panel

In the BACK panel, we can view an illustration of the back of the X-Brick and see every physical connection to the X-Brick and inside of it, including FC connections, Power, iSCSI, SAS, Management, IPMI and InfiniBand, filtered by the "Show Connections" list at the top right. An example of this view is seen in Figure 61.

Figure 61.

XtremIO WebUI – Hardware – Back Panel – Show Connections

Inventory In the Inventory menu, we can see all components of our environment with information about them, including: XMS, Clusters, X-Bricks, Storage Controllers, Local Disks, Storage Controller PSUs, XEnvs, Data Protection Groups, SSDs, DAEs, DAE Controllers, DAE PSUs, DAE Row Controllers, Infiniband Switches and NVRAMs.


XMS Menus The XMS Menus are global menus for the XMS that can be accessed in the top right tools of the interface. We can use them to Search components in the system, view Health status of its managed components, view major Alerts, view and configure System Settings (shown in Figure 62) and use the User Menu to view login information (and logout), and support options.

Figure 62.

XtremIO WebUI – XMS Menus – System Settings

As mentioned, there are other interfaces available to monitor and manage an XtremIO cluster through the XMS server. The system's Command Line Interface (CLI) can be used for everything the GUI can be used for and more, a RESTful API is another pre-installed interface in the system which allows HTTP-based commands to manage clusters, and a PowerShell API Module is also an option to use Windows' PowerShell console to administer XtremIO clusters.

Test Setup We used an XtremIO cluster with a single X2-S X-Brick as the storage array for our environment. The X-Brick had 36 drives of 400GB size each which, after leaving capacity for parity calculations and other needs, amounts to about 11.2TB of physical capacity. As we saw in the Test Results section, this is more than enough capacity for our 4000 virtual desktops. 36 drives are half the amount that can fit in a single X-Brick. This means that in terms of capacity, we can grow to a maximum of x8 the capacity we have here in the same cluster with our scale-up (up to 72 drives per X-Brick) and scale-out (up to 4 X-Bricks per cluster) capabilities for X2-S (for X2-R we currently provide drives which are about 5 times bigger, yielding a much higher capacity. X2-R drives will soon be 10 times bigger, and X2-R clusters could grow to up to 8 X-Bricks). Performance-wise, we can also see from the Test Results section that our single X2-S X-Brick was enough to service our VDI environment of 4000 desktops, when storage traffic metrics (latency, bandwidth, IOPS) and resource consumption metrics (CPU, RAM) were excellent throughout all of the VDI environment's processes (X2-R clusters would have even higher abilities in terms of compute, as they have x3 the RAM of X2-S).

Compute Hosts: Dell PowerEdge Servers We have a homogenous cluster of 32 ESX servers for hosting VDI desktops and 2 ESX servers for virtual appliances, which are used to manage the VDI infrastructure. We chose Dell's PowerEdge FC630 as our ESX hosts, as they have the compute power to deal with an environment at such a scale (125 virtual desktops per ESX host), and are a good fit for virtualization environments. Dell PowerEdge servers work with the Dell OpenManage systems management portfolio that simplifies and automates server lifecycle management, and can be integrated with VMware vSphere with a dedicated plugin (see Compute Integration – Dell OpenManage on page 55).


Table 2 lists ESX hosts details at our environment. Table 2.

ESX Hosts Details Used for VDI Desktops and Infrastructure

PROPERTIES

2+32 ESX HOSTS

System make

Dell

Model

PowerEdge FC630

CPU cores

36 CPUs x 2.10GHz

Processor type

Intel Xeon CPU E5-2695 v4 @ 2.10GHz

Processor Sockets

2

Cores per socket

18

Logical processors

72

Memory

524 GB

Ethernet NICs

4

Ethernet NICs type

QLogic 57840 10Gb

iSCSI NICs

4

iSCSI NICs type

QLogic 57840 10Gb

FC adapters

4

FC adapters type

QLE2742 Dual Port 32Gb

On-board SAS controller

1

In our test, we used FC connectivity to attach XtremIO LUNs to the ESX hosts, but iSCSI connectivity could be used just as well and in the same manner. It is highly recommended to select and purchase servers after verifying the vendor, make and model from VMware's hardware compatibility list (HCL). It is also recommend installing the latest server chipset firmware code for the server and its adapters, and installing the latest GA release of VMware vSphere ESXi, including any of the latest update releases or express patch. For more information on Dell EMC PowerEdge FC630 see its specification sheet

[13]

.

Storage Configuration This section outlines the storage configuration in our test environment, highlighting zoning considerations, XtremIO Volumes, Initiator Groups, and mapping between Volumes and Initiator Groups.


Zoning In a single X-Brick cluster configuration, a host equipped with a dual port storage adapter may have up to four paths per device. Figure 63 shows the logical connection topology for four paths. Each XtremIO Storage Controller has two Fibre Channel paths that connect to the physical host, via redundant SAN switches.

Figure 63.

Dual Port HBA on an ESX Host to a Single X2 X-Brick Cluster Zoning

[6]

As recommended in EMC Host Connectivity Guide for VMware ESX Server : •

Use multiple HBAs on the servers.

•

Use at least two SAN switches to provide redundant paths between the servers and the XtremIO cluster.

•

Restrict zoning to four paths to the storage ports from a single host.

•

Use a single-Target-per-single-Initiator (1:1) zoning scheme.

Storage Volumes We provisioned two sets of XtremIO Volumes: •

1 Volume of 6TB for hosting all virtual machines for management functions of the VDI environment

•

32 Volumes for hosting VDI desktops (3TB sized for Linked Clones and 10TB sized for Full Clones)

We highly recommend leveraging the capabilities of EMC VSI plugin for vSphere Web client, to provision multiple XtremIO Volumes. Initiator Groups and LUN Mapping We configured a 1:1 mapping between Initiator Groups and ESX hosts in our test environment. Each of our ESX hosts has a dual port FC HBA, thus, each Initiator Group contains two Initiators mapped to the two WWNs of the FC HBA. Altogether 34 Initiator Groups were created, as follows: •

2 Initiator Groups for mapping volumes to 2 management servers

•

32 Initiator Groups for mapping volumes to all 32 ESX hosts hosting VDI desktops

The Initiator Groups and Volumes mapping was as follows: •

1 Volume (sized 6TB) mapped to the 2 management infrastructure's Initiator Groups

•

32 Volumes (3TB for Linked Clones, 10TB for Full Clones) mapped to the 32 ESX hosts' Initiator Groups hosting virtual desktops


Storage Networks We used FC connectivity between our X2 storage array and the ESX hosts of the environment to provision LUNs, but our environment is also iSCSI-ready. For SAN fabric, we used Brocade G620 switches connecting the HBAs on the host to the Storage Controllers on the X-Brick. Some important Brocade G620 details are summarized in Table 3. For more [21] details on the FC switch, refer to Brocade G620 Switch Datasheet . Table 3.

Brocade G620 FC Switch Details

Make/Model

Brocade 6510

Form factor

1U

FC Ports

64

Port Speed

32Gb

Maximum Aggregate Bandwidth

2048Gbps Full Duplex

Supported Media

128Gbps, 32Gbps, 16Gbps, 10Gbps

For iSCSI connectivity, we have Mellanox MSX1016 switches connecting host ports to the Storage Controllers on the XBrick. Some important Mellanox MSX1016 details are summarized in Table 4. For more details on the iSCSI switch, refer [22] to Mellanox MSX1016 Switch Product Brief . Table 4.

Mellanox MSX1016 10GbE Switch Details

Make/Model

Mellanox MSX1016 10GbE

Form factor

1U

Ports

64

Port Speed

10G

Jumbo Frames

Supported (9216 Byte size)

Supported Media

1GbE, 10GbE

We highly recommend installing the most recent firmware version for the FC and iSCSI switches when installing them in the datacenter.

Solution's Software Layer This section describes the software layers of our VDI environment, including configurations pertaining to VMware vSphere Hypervisor (ESXi), vCenter server and VMware Horizon View Suite. We also detail the virtual machines that enact specific roles in the software-defined datacenter for VDI.

Hypervisor Management Layer The following section describes configuration settings that were adopted for the Hypervisor Management Layer of the environment – namely VMware ESXi 6.5 components and relevant plugins for it.


vCenter Server Appliance We installed a single instance of VMware vCenter Server (VCS) 6.5, deployed with SQL server 2017, Platform Services Controller (PSC) and Single Sign-On service (SSO) embedded with it. For production, we recommend deploying 2 VCSs, one for the management infrastructure and one for the VDI infrastructure, and deploying a lightweight PSC and SSO appliance, federating the two VCSs to bind to the same SSO domain. The configuration of the VCS used in our environment is outlined in Table 5. Table 5.

vCenter Server Virtual Appliance Settings

PROPERTIES

VCSA

No. of vCPU

32

Memory

64GB

Database

SQL Server 2017

No. of virtual disks

1

Total Storage

60GB

Hypervisor All VMware Horizon View servers hosting virtual desktops and the two servers hosting the virtual machines for management functions are running VMware vSphere Hypervisor (ESXi) 6.5. We highly recommend installing the most recent ESX 6.5 update release. A single virtual datacenter was created to hold both the management cluster and the VDI cluster. ESX Clusters

Management Cluster A management ESX cluster for the VDI environment was created with two ESX servers for hosting server virtual machines and virtual appliances for the following functions: •

DHCP server

•

Active Directory co-located with a DNS server

•

VMware vCenter Server

•

SQL Server database for VMware View Composer and vCenter database (2016)

•

VMware Horizon View Composer service

•

VMware Horizon View Connection Server

•

VMware Horizon View Security Server

•

EMC Solutions Integration Service

•

LoginVSI file shares

•

LoginVSI management

•

LoginVSI launchers

Virtual machines for LoginVSI file shares, LoginVSI management and LoginVSI launchers are a part of the test platform, and should not be factored in while planning a production VDI infrastructure. Details on the LoginVSI infrastructure are [2425] discussed in Appendix A – Test Methodology and in LoginVSI's documentations .


VDI Cluster A VDI ESX cluster was created for the 32 ESX servers to host the virtual desktops of the environment. It is highly recommended to enable DRS (Dynamic Resource Scheduling) for the cluster. Collectively, the cluster has resources equivalent to approximately: •

Compute resources totaling 2.42THz

•

Memory resources totaling 16TB

Network Configuration For ease of management and to have a unified view into the network configuration, we recommend vSphere Distributed Switch (VDS) for the production environment. For best results, we highly recommended segregating network traffic by using multiple vSwitch Port groups, backed-up by its individual physical NICs. For production environments, we recommend configuring a VMkernel port group, using a 1GbE (or faster) interface for the management traffic between the ESX hosts and the vCenter Server Appliance. A second VMkernel port group located on a separate vSwitch, backed-up by a separate 1GbE (or faster) interface, should be configured for vMotion traffic. Lastly, we recommend having the virtual machine port group on another vSwitch, backed-up by a 10GbE interface for all virtual machine traffic. If iSCSI is used as the storage protocol, we need to have another VMkernel port group that is backed-up by a separate 10GbE NIC interface.

Figure 64.

In-Band Virtual Machine Networking Used in Test Environment

Figure 64 shows our non-production environment's in-band network configuration, where management traffic and virtual machine traffic flows through the same 10GbE NIC. The switch connected to the NICs was isolated at the physical layer from the outside world to avoid interference. We strongly advise following VMware's best practices for networking in vSphere environments.


Storage Configuration, EMC SIS and VSI EMC Solutions Integration Service 7.2 (EMC SIS) provides us with unique storage integration capabilities between VMware vSphere 6.5 and EMC XtremIO X2 (XMS 6.0.0 and above). The EMC VSI (Virtual Storage Integrator) 7.2 plugin for VMware vSphere web client can be registered via EMC SIS. New VMFS (Virtual Machine File System) datastores can be created using the VSI plugin, and backed-up by XtremIO Volumes at the click of a button. The VSI plugin interacts with EMC XtremIO to create Volumes of the required size, map them to the appropriate Initiator Groups, and create a VMFS datastore on vSphere, ready for use. Figure 65 shows an example of using the EMC plugin for datastore creation.

Figure 65.

Create a Datastore using the EMC VSI Plugin

The VSI plugin can also be used for modifying ESXi host/cluster storage-related settings, setting multipath management and policies, and for invoking space re-claim operations from an ESX server or from a cluster. The VSI plugin is the best way to enforce the following EMC-recommended best practices for ESX servers (see Figure 66): •

Enable VAAI.

•

Set Queue depth on FC HBA to 256.

•

Set multi-pathing policy to "round robin" on each of the XtremIO SCSI Disks.

•

Set I/O path switching parameter to 1.

•

Set outstanding number of I/O request limit to 256.

•

Set the "SchedQuantum" parameter to 64.

•

Set the maximum limit on disk I/O size to 4096.


Figure 66.

Configuring EMC Recommended Settings using the VSI Plugin

In our environment, we provisioned 32 XtremIO Volumes for storing all relevant virtual machine data. The VSI plugin can be used to enforce the above-mentioned configuration settings on all these datastores, across all ESX servers in the cluster, at the click of a button. As discussed in Storage Configuration on page 49, we have a dual port 16G FC HBA. An XtremIO single X-Brick cluster configuration has two Storage Controllers resulting into four Targets. In accordance with EMC XtremIO best practices for zoning, each LUN will have four paths. The same logic should be used when provisioning storage using iSCSI connectivity. Compute Integration – Dell OpenManage Dell OpenManage is a program providing simplicity and automation of hardware management tasks and monitoring for both Dell and multi-vendor hardware systems. Among its capabilities are: •

Rapid deployment of PowerEdge servers, operating systems, and agent-free updates

•

Maintenance of policy-based configuration profiles

•

Streamlined template-driven network setup and management for Dell Modular Infrastructure

•

Providing a "geographic view" of Dell-related hardware.

Dell OpenManage can integrate with VMware vCenter using the OpenManage Integration for VMware vCenter (OMIVV), which provides VMware vCenter with the ability to manage a datacenter's entire server infrastructure, both physical and virtual. It can assist with monitoring the physical environment, send system alerts to the user, roll out firmware updates to an ESXi cluster, etc. The integration is more profitable when using Dell PowerEdge servers as the ESX hosts of the VMware environment. Figure 67 shows an example of a cluster's hardware information provided by the OpenManage Integration for VMware vCenter. 55 | Virtualized Desktop Infrastructure with DELL EMC XtremIO X2 and VMware Horizon Enterprise 7.2 © 2017 Dell Inc. or its subsidiaries.

Figure 67.

Dell Cluster Information Menu provided by the Dell OpenManage Plugin for VMware

For more details see the Dell OpenManage Systems Management Tools link [15] OpenManage Integration for VMware vCenter .

[14]

Dell OpenManage and the Dell

Virtual Desktop Management Layer The following section describes configuration settings that are adopted for VMware Horizon View Suite and Horizon View Pool. VMware Horizon View Server Configuration As per the configuration maximum guidelines for VMware Horizon View 7, one connection server instance supports up to 4000 simultaneous connections. Horizon 7 also supports 4000 desktops (Full or Linked Clones) per one vCenter Server. These limitations fit our test environment precisely. For more than 4000 desktops, we would need additional connection server instances (replica servers) and additional vCenter Servers. For instance, in a fully-built View Pod of 12000 desktops, it is recommended having one instance of a standard View connection server, two instances of replica servers, and one additional replica server instance, for failover. We also need the same number of security servers. One security server is paired with a standard View connection server while the remaining servers are paired with replica View connection servers. It is also recommended having load balancers to evenly-distribute the incoming session initiation requests among the security servers. For achieving the best possible security, it is important that clients connect by using a URL pointing to a load balancer that distributes the requests to the security servers, and not directly to the standard View connection server or replica servers. Through Horizon's vCenter Server settings, we enabled the "Reclaim VM disk space" option to allow the storage array to reclaim space that was freed in the environment (see Figure 68). We did not enable the "View Storage Accelerator", an option that configures a host-based cache for every virtual machine disk, since our Flash-based array more than satisfies our storage needs in terms of performance and user experience.


Figure 68.

Horizon's vCenter Server Settings – Storage

Space reclamation can be performed on a single desktop or an entire pool, and can be scheduled or manually initiated. Figure 69 shows a summary report of space reclamation of a Desktop Pool.

Figure 69.

Space Reclamation Report within VMware Horizon


With VMware Horizon, we can set a limit to the number of concurrent tasks sent to the vCenter server during deployment and maintenance operations, which will limit the IOPS and bandwidth pushed by VMware Horizon. The following Horizon View configuration settings for a vCenter server and View Composer impact the execution times of management operations relevant to our environment: •

Max. concurrent vCenter provisioning operations

•

Max. concurrent power operations

•

Max. concurrent View Composer maintenance operations

•

Max. concurrent View Composer provisioning operations

With a high-performance array such as XtremIO X2 All-Flash Array, the default values of the above-mentioned configuration settings can be increased to achieve faster execution times during provisioning and maintenance operations. We recommend running a "pilot" phase in your project to narrow down the best possible values for your environment. The optimization of these configuration settings varies, based on the underlying hardware, the storage array and other clients using it and the method in which the Fibre Channel networking is deployed. In our environment, we used higher values for all maximum concurrent operations (see Figure 70). Keep in mind that either way, an ESX host can perform only up to eight concurrent operations. For more on Horizon limits and best [20] practices, see VMware Horizon 7 Sizing Limits and Recommendations .

Figure 70.

Horizon's Settings – Max. Concurrent Operations


VMware Horizon View Desktop Pool Configuration We scaled our 4000 desktops with a single Horizon View Desktop Pool. The entire pool was deployed on our 32 ESX hosts and 32 XtremIO LUNs datastores. By mapping all the datastores to all the hosts and scaling the entire desktop pool on them, VMware vSphere DRS can "vMotion" or migrate the virtual desktops to any host in the cluster. The View Pool default settings were kept, with the following exceptions: •

Selecting the Floating User assignment instead of Dedicated User assignment

•

Selecting to not re-direct disposable files to a non-persistent disk

Virtual Machine System Settings and Applications At this stage, we have covered most of the areas encompassing the VDI environment that is used for our 4000 desktops, including both the hardware and software layers. It is also important to examine the settings on the parent virtual machine, which is also referred to as the "golden master". It is from this virtual machine that we deploy both our Linked Clone pool and our Full Clone pool. No changes to CPU or memory reservations were made on the "golden master" virtual machine. Settings pertaining to hot-adding of CPU, memory, CD drive, floppy drive, swap file location, and Fibre Channel NPIV were all kept at their respective defaults. We recommend configuring the parent virtual machine so that it synchronizes its time with the ESX host. For time synchronization, it is very important to configure ESX hosts with accurate NTP servers. Table 6 summarizes the significant virtual machine settings. Table 6.

Parent Virtual Machine Settings

GUEST OPERATING SYSTEMS

32-BIT WINDOWS 10

VM Version

13

No. of vCPUs

2

Memory

4GB

No. of vNICs

1

No. of Virtual Disks

1

Total Storage Provisioned

40GB

Video Memory

35MB

As discussed in LoginVSI Results on page 19, the following applications were included in the virtual desktops: •

Microsoft Office 2016 Suite

•

Adobe Reader 11

•

Java 7 Update 13

We made minor tweaks and optimized the parent virtual machine in preparation for it to work with the LoginVSI test platform. We ran VMware OS's optimization tool, which is released as a VMware fling used for disabling most of the unwanted services and for removing unwanted applications that are irrelevant to LoginVSI's workload and the test platform.


Figure 71.

Horizon's Settings – Max. Concurrent Operations

We recommend engaging with LoginVSI's Support for any further tweaks and modifications that are necessary to successfully run and use the LoginVSI's workloads.

Conclusion From the results presented herein, we conclude that Dell EMC's XtremIO X2 All-Flash Storage Array offers a best-in-class performance and fulfils all our requirements for storage capacity and storage I/O processing for our VDI environments. This reference architecture provides the details of hardware and software components of our VDI infrastructure and their configurations, which gives datacenter architects a great starting point for designing their VDI environments for performance at scale.


References

1. Dell EMC XtremIO Main Page – http://www.dellemc.com/en-us/storage/xtremio-all-flash.htm 2. Introduction to EMC XtremIO 6.0 3. Dell EMC XtremIO X2 Specifications – http://www.emc.com/collateral/specification-sheet/h16094-xtremio-x2specification-sheet-ss.pdf 4. Dell EMC XtremIO X2 Datasheet – http://www.emc.com/collateral/data-sheet/h16095-xtremio-x2-next-generation-allflash-array-ds.pdf 5. XtremIO X2 vSphere Demo – http://www.emc.com/video-collateral/demos/microsites/mediaplayer-video/xtremio-x2vsphere-demo.htm 6. EMC Host Connectivity Guide for VMware ESX Server – https://www.emc.com/collateral/TechnicalDocument/docu5265.pdf 7. XtremIO X2 with VDI benefits Video – http://www.emc.com/video-collateral/demos/microsites/mediaplayervideo/transforming-desktop-virtualization-xtremio-x2.htm 8. XtremIO X2 4000 VM Boot Demo – http://www.emc.com/video-collateral/demos/microsites/mediaplayervideo/xtremio-x2-4000-vm-boot-demo.htm 9. XtremIO CTO Blog (with product announcements and technology deep dives) – https://xtremio.me/ 10. XtremIO XCOPY Chunk Sizes – https://xtremio.me/2017/07/31/xcopy-chunk-sizes-revisited-and-data-reduction-aswell/ 11. 2016 XtremIO with VDI Reference Architecture – https://xtremio.me/2016/07/25/a-new-vdi-reference-architecture/ 12. Dell EMC Virtual Storage Integrator (VSI) Product Page – https://www.emc.com/cloud-virtualization/virtual-storageintegrator.htm 13. Dell EMC PowerEdge FC630 Specification Sheet – https://i.dell.com/sites/doccontent/shared-content/datasheets/en/Documents/Dell-PowerEdge-FC630-Spec-Sheet.pdf 14. Dell OpenManage Systems Management Tools – http://en.community.dell.com/techcenter/systemsmanagement/w/wiki/1757.dell-openmanage-systems-management-tools 15. Dell OpenManage Integration for VMware vCenter – http://en.community.dell.com/techcenter/systemsmanagement/w/wiki/1961.openmanage-integration-for-vmware-vcenter 16. VMware vSphere 6.5 Configuration Maximum Guide – https://www.vmware.com/pdf/vsphere6/r65/vsphere-65configuration-maximums.pdf 17. Performance Best Practices for VMware vSphere 6.5 – https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/Perf_Best_Practices_v Sphere65.pdf 18. VMware Horizon 7 Documentations (including 7.2) – https://docs.vmware.com/en/VMware-Horizon-7/index.html 19. VMware Horizon 7.2 View Administration Guide – https://docs.vmware.com/en/VMware-Horizon-7/7.2/view-72administration.pdf 20. VMware Horizon 7 Sizing Limits and Recommendations – https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2150348 21. Brocade G620 Switch Datasheet – http://www.brocade.com/en/backend-content/pdfpage.html?/content/dam/common/documents/content-types/datasheet/brocade-g620-switch-ds.pdf 22. Mellanox MSX1016 Switch Product Brief – http://www.mellanox.com/related-docs/prod_eth_switches/PB_SX1016.pdf 23. LoginVSI Main Product Page – https://www.loginvsi.com/products/login-vsi 24. LoginVSI Main Documentation Page – https://www.loginvsi.com/documentation/index.php?title=Main_Page 25. LoginVSI VSImax and Test Methodology explained – https://www.loginvsi.com/documentation/index.php?title=VSImax


Appendix A – Test Methodology We used the LoginVSI 4.1.25 "Knowledge Worker" profile to emulate workloads on our VDI desktops. LoginVSI effectively emulates a typical office worker profile that logs into a VDI desktop, and performs activities representative of a desktop user, such as opening a Word document, modifying an Excel spreadsheet and browsing a PDF file or a web page. A LoginVSI setup has the following components driving the LoginVSI workload: •

A LoginVSI management server

•

A LoginVSI file share server

•

LoginVSI launcher machines

A LoginVSI management server has applications that launch the tests and monitor user session status, and has an analyzer application that calculates and presents the VSI baseline, VSI average and VSI threshold that we saw in Figure 28 and Figure 29. The file share server consists of folders and files that are accessed as part of the workload. The launcher machines launch the user sessions that initiate a connection to the desktop, which then start the workload. LoginVSI is a vendor-independent tool that helps characterize the desktop "user experience" of an environment, irrespective of the VDI vendor or the protocol used for remote sessions. The "Desktop Direct Connect" mode (known as DDC), is specifically geared towards storage vendors. DDC enables a user to directly log on to the desktop, bypassing the remote protocol but still driving I/O operations that are produced by a user logon and by application launches thereafter. For scaling 4000 desktop connections, we needed three "Launcher" virtual machine instances. To ensure that we are not limited by file share limitations, we used ten file server instances, across which the file access is distributed by LoginVSI. Table 7 shows the Launchers and file servers virtual machine settings. Table 7.

Launcher and File Server Virtual Machine Settings

PROPERTIES

3 LAUNCHER SERVERS + 10 FILE SERVERS

Operating System

64-bit Windows Server 2016

VM Version

13

No. of vCPUs

2

Memory

4GB

No. of vNICs

1

No. of Virtual Disks

1

Total Storage Provisioned

40GB

Video Memory

35MB

The management cluster, consisting of two ESX servers, hosted the "launcher" virtual machines. These launcher virtual machines are part of the LoginVSI test platform that initiates connections to the VDI desktops. For more details on [24][ 25] . LoginVSI components functionality see the LoginVSI documentation


How to Learn More For a detailed presentation explaining XtremIO Storage Array's capabilities and how XtremIO substantially improves performance, operational efficiency, ease-of-use, and total cost of ownership, please contact XtremIO at [email protected]. We will schedule a private briefing in person or via a web meeting. XtremIO provides benefits in many environments and mixed workload consolidations, including virtual server, cloud, virtual desktop, database, analytics, and business applications.

Learn more about Dell EMC XtremIO

Contact a Dell EMC Expert

High Availability, Data Protection, and Data Integrity in the

View more resources

© 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC and other trademarks are trademarks of Dell Inc. or Dell EMC XtremIO X2 Architecture its subsidiaries. Other trademarks may be trademarks of their respective owners. Reference Number H16607

© 2017 Dell Inc. or its subsidiaries.

Join the conversation @DellEMCStorage and #XtremIO