Building a Lab Server to Run ESXi and Hyper-V

One of the great things I enjoyed about working at Kelway was the access to a pretty solid lab environment. While I do have access to a lab environment at Atlantis (3 in fact), now that I work primarily from home, I really prefer a lab environment that can provide me more flexibility. Only a local environment can do that.

I originally started out with the aim of building a system that achieved 3 goals – low power consumption, enough grunt to deliver the workloads I’d like to run, plus be (relatively) quiet. As you can probably guess, I’ve achieved only two of those objectives (plus blowing my budget in the process).

Having recently moved back to Australia from the UK, I’ve gone from spending approximately £500 a year on electricity and gas, to what I estimate will be $1500 AU on electricity (no gas appliances in our current house). If you weren’t already aware, Australian summers can be hot (or bloody hot, as I prefer to say). Living in Melbourne means summers are shorter than Sydney or Brisbane, but I’ll still have two issues – heat and the cost of running a lab at home.

I first started looking at a Lenovo ThinkCentre M83 SFF Pro, as it’s the only small-form factor PC that I can find that can take 32 GB of RAM. While it looks like a good PC, it was coming out a bit expensive for what it is and an unknown quantity when it comes to noise.

It’s been a long time since I’ve built a PC from scratch, so to get exactly what I wanted, I thought this would be a good opportunity to do so. First though, a word of caution, I’m no expert, what I’ve ended up with worked for me and this spec could be definitely be improved upon.

Picking a CPU

My first step was to settle on a CPU – that way I can build everything else around it as the choice of CPU has an impact on the socket and motherboard plus the maximum amount of RAM. While I did do some research on various processors, I did find that I ended up being limited by availability.

Intel CPUs have been my personal preference, so I ruled out AMD immediately. If I’d used an AMD CPU, I probably could have saved a bit on the final build.

Starting with the Core i5 made some sense, both from a power consumption and cost perspective, but the i5 has two things going against it – no Hyper-Threading and a maximum of 32 GB of RAM. I didn’t want to limit my workloads, so I’ve gone with a Core i7.

To build a 32 GB system, you could go with the Core i7-4770K that uses a socket LGA1150. One problem with the 4th generation Core i7 (i.e. Haswell) is that they don’t yet support 64GB of RAM (as far as I can tell).

Core-i7-CPUFor a 64 GB system I picked the Core i7-4820K - this CPU is the older Ivy Bridge-E architecture. While I priced up a system based on the 4770K, I ended up going with the 4820K, so that I could build a machine with 64 GB of RAM.

As I’m doing testing and modelling with a solution that utilise a lot of RAM, I really needed to go with as much RAM as I could afford. There ended up being about $500-$600 AU difference between a 32 GB system and a 64 GB system, so budget be damned, 64 GB PC is what I got.

Selecting a Motherboard

I had originally wanted to build a system around a min-ITX or mini/microATX board to build a smaller PC but the choice of CPU using the socket LGA2011 and 64 GB of RAM has forced me on to a full ATX board.

As I’m not interested in over-clocking, my requirements for a motherboard were simple – I went for the lowest price board that could fit 64 GB of RAM (8 DIMM slots) and supports a minimum of 4 x SATA3 (6Gbps) ports. Unless you’re looking at server boards, you’ll find most desktop motherboards come with a lot of extra features that are just not required for a lab server (sound, SPDIF, heaps of USB ports, FireWire etc.).

ASRock-Extreme6Ultimately I went for the ASRock X79 Extreme 6 with 5 x SATA 3 ports. One thing I’ve found out in building this PC, is that the Intel chipsets typically provide 2 x SATA3 ports only. The remaining SATA ports are provided by a second chipset (or sometimes even a third chipset). Not a problem for Windows, but not the best choice for ESXi (more on that later).

Filling it With RAM

G.Skill-RipjawsX

With 8 DIMM slots to fill and 64 GB of RAM to get to, I needed to find 8 x 8 GB DIMMs. The ASRock X79 Extreme 6 supports 2400 / 2133 / 1866 / 1600 / 1333 / 1066 MHz clock speeds.

I originally looked at Corsair Vengeance RAM but settled on G.Skill RipjawsX DDR3-1600 F3-1600C9Q-32GXM, at $100 AU less than the Corsair. With a CAS latency of 9-9-9-24, I think this ended up being good value for money.

Housing the Rig in a Case

Corsair-330rI had two main requirements for a case – help the PC be as silent as possible and look unassuming (as it’s not locked away but sitting next to the living room). Additionally, as I’d locked myself into an ATX board, I needed an ATX case.

When picking a case, I looked at cooling options as well – by picking a CPU cooler, case fans and a case from the same manufacture, I was comfortable that I could build the system knowing that every thing would fit. I picked the Corsair Carbide 330R Quiet Mid-Tower case which looks nice and has good reviews.

Keeping it Cool

Corsair-H55While a water cooler probably pulls more power than a heat sink/fan combination, I went with water cooling to ensure I could move heat away from the CPU as efficiently as possible.

I chose the Corsair Hydro Series H55 Quiet CPU Cooler. This is a single fan, 120mm radiator but should be enough to sufficiently move any heat generated by the CPU.

The 330R includes 1 x 120mm (rear) and 1 x 140mm fans (front), but I’ve replaced those with Corsair’s Quiet Edition 120mm and 140mm fans with a second 140mm in the front to pull in more air. The case does include room for two 120mm or 140mm fans with a radiator in the top, which requires popping the top plate off to allow for exhaust but would increase the noise coming out of the system.

Snorage

samsung_840_evo_ssd_0To ensure the lowest power consumption for storage devices, I’ve stuck with SSDs. I had a few SSDs already on hand already, but did purchase a couple more to provide more capacity.

In the PC I have installed:

  • 1 x Corsair 64 GB SSD SATA2 (although this drive works under Windows, I can’t mount it under ESXi)
  • 1 x OCZ Vertex 3 120 GB SSD
  • 1 x Samsung 840 EVO Pro 120 GB SSD
  • 2 x Samsung 840 EVO 250 GB SSDs (these are new)

I’ve also got an 8GB USB3 flash drive that ESXi is running from. It will be interesting to see when this drive fails, which is inevitable if I’ve got an OS running from it.

Powering the Beast

Calculating how much power was quite simple, once you find a tool that does it for you. Bryan Chriscoli put me onto eXtreme Power Supply Calculator Lite which does a fantastic job of calculating the minimum PSU wattage required.

Corsair-RM450To power the PC I chose the Corsair RM450 power supply. This is a 450W 80 PLUS Gold Certified (i.e. highly efficient) modular power supply, which is probably more than I need but I found 400W power supplies, that didn’t look like cheap crap, hard to come by.

The RM450 has a fan that will only power on after the PSU reaches a temperature that requires a fan shifting heat, keep noise down during normal operation. It’s a great quality PSU that looks good and feels nice in the hand.

I had originally looked at the Seasonic 400W fanless power supply, but was a little worried that on a really hot day, shifting heat out of the case could be a problem and it’s also a pretty expensive PSU.

Miscellaneous additions

The ASRock mother board has a single Broadcom BCM57781 Gigabit adapter on-board, so I’ve added an Intel Gigabit CT Desktop PCIe adapter. This way I can use the Broadcom adapter for management traffic and the Intel adapter for virtual machine networks.

Something I didn’t realise until I’d put the whole thing together, is that the ASRock Extreme 6 motherboard doesn’t include a GPU. I’d originally focussed on motherboards with on-board Intel graphics and completely forgot to check the ASRock board when ordering it. So I’ve added an EVGA NVIDIA GeForce 201 fanless graphics adapter to the build as well. While this is silent, will have increased the power consumption of the system a bit.

Thoughts on Building this PC

Building this PC took a really long time – it’s been quite a while since I last built a PC and I needed to ensure I got this one right and it was up and running on first go (other than having to source a graphics adapter after completing the build).

Here’s a few thoughts and tips if you’re looking to build your own lab PC:

  • Picking the right hardware and finding out which parts will be suitable is painful. The best site I found that helped me build the right parts as PCPartPicker. It would be nice if PC parts retailers provided this functionality in their web sites, but most of them (at least in Australia) have terrible web sites.
  • The Corsair Carbide 330R is a nice case and performs well for airflow and noise. It would have been nice to have more than 4 x 3.5″ drive bays and there’s not a lot of space between the back of the motherboard and the side panel. This makes putting the side panel back on a challenge, because the routed cables don’t sit completely flat. While I think the 330R is a nice case, I think you could choose a better case.
  • This is the first time I’ve built a water cooled PC – fitting the fan and radiator was fiddly, but fitting the CPU cooler was very simple. Although Corsair recommends fitting the radiator fan so that it sucks air into the case, I’ve fitted it to blow air out of the case – which is a good thing, since…
  • I should have picked low profile DIMMs. The radiator fan is practically sitting right on top of the RAM heat sinks. I don’t think this is an issue, but at least it’s not the radiator touch the heat sinks.
  • Unfortunately the i7 4820K CPU can run at up to 130W, so not ideal for low power.
  • To get the additional SSDs connected to SATA ports on the ASMedia chipset working, I needed to follow this article: How to make your unsupported SATA AHCI Controller work with ESXi 5.5. Without this change to ESXi, the drives connected to the additional SATA ports were just not seen. This wasn’t a problem under Windows Server.
  • The noise level at idle is pretty good, but could be better. I’m now investigating a fan controller and I’m sure that I can run the fans at lower speeds to reduce noise, without affecting cooling performance too much.

Performance

Getting this up and running under Windows Server 2012 R2 was very simple, practically all drivers were picked up by Windows natively or found on Windows Update. Taking a look at Task Manager is a thing of beauty:

hv2-TaskManager-RAM-VanillaInstall

I’m yet to do full performance tests and one thing to attempt is running Login VSI to see what I can get out of it. A simple test of IOmeter simulating a VDI workload on ILIO for Persistent VDI drove an impressive 42K IOPs (with ILIO for Stateless VDI doing about the same number).

Lab-ILIO-Test1

As for power consumption, at idle with ESXi 5.5 booted, it draws up to around 105 watts. I have seen it drop to a low of between 65 and 71 watts as well. As a very basic CPU stress test, I ran HeavyLoad 3.3 on a Windows 8.1 x86 virtual machine with 1 vCPU and 8 cores. Power consumption jumped up to a consistent 128 watts.

128watts

So not as low power as I originally set out to accomplish, but with this configuration it’s a pretty beefy work horse. Some more testing with different settings and under Hyper-V is required.

Spec List

Here’s the final list of components that went into the build and the price in Australian dollars. I’ve colour coded the totals to make it simple to see where the costs went into. It’s not surprising that the most expensive component of this build was the RAM.

PCbuild

Finally, here’s a look inside the finished build:

InsideCustomBuild

Cleaning up and Reducing the Size of your Master Image

Compressed Car

There’s typically not too much that you can do to reduce the size of your master image. You might use application virtualization or layering solutions to reduce the number of master images, but once you work out what needs to go into the core image, that’s going to dictate the size of the image.

Reducing the size of your master image can help reduce the capacity required to store master images and virtual machines (dedupe helps, of course) and spend less cycles transmitting an image across the network for physical PCs.

An easy win, is running the Disk Clean-up tool included in Windows (since Windows XP) and fortunately this tool can be automated to run at the end of a build (e.g from MDT or ConfigMgr). For physical PCs or persistent desktops, this tool could even be run as a scheduled task.

Microsoft released an important update for Windows 7 last year that can result in a significant reduction in disk space: Disk Clean-up Wizard addon lets users delete outdated Windows updates on Windows 7 SP1. The same feature was originally delivered with Windows 8. (Windows 8.1 Update 1 is expected to reduce disk space requirements again).

Here’s an example system where I’ve run the Disk Clean-up tool that has resulted in a 3.4 GB reduction in disk usage – on the left is the before image, on the right is after the cleanup. (I’m cheating a bit here, this is a system that has gone from Windows 7 to Windows 7 SP1, hence the reason for such a large change).

Compare Disk Cleanup Before and After

Disk Clean-up can remove a number of interesting items, most of which will actually be applicable for PCs and persistent desktops post-deployment. Here’s the items that Disk Clean-up can manage on Windows 8.1:

Disk Cleanup options

To automate Disk Clean-up, use the following steps:

  1. Elevate a command prompt
  2. Run CLEANMGR /SAGESET:<number> (where <number is any number between 1 and 65535)
  3. Select each of the items to clean up
  4. Click OK

To run Disk Clean-up from a script run CLEANMGR /SAGERUN:<number> (where <number> is the same number use with SAGESET.

To automate the process of running Disk Cleanup, the following script can be used to enable all of the tool’s options in the registry and then execute the cleanup process. This script should work for both Windows 7 and Windows 8 and would be useful to run at the end of a deployment. This example uses 100 as the configuration setting, so you would run CLEANMGR /SAGERUN:100 to run Disk Cleanup with these settings.

It’s important to note that while Disk Clean-up exists on Windows 7 and Windows 8, it does not exist on Windows Server unless the Desktop Experience feature is installed (i.e. a Remote Desktop Session Host).

If your image is Windows 8.1 or Windows Server 2012 R2, then the following command is available to perform an even more in depth cleanup of the WinSXS folder, making some additional space available:

Running Disk Clean-up and the above DISM command in a script to clean up your master image should result in a smaller image. Don’t forget that this approach is also useful for persistent desktops – unless you’re using some type of dedupe solution, then there’s potentially gigabytes per desktop that can be removed.

There is one more method worth for reducing space worth mentioning – the Uninstall-WindowsFeature PowerShell cmdlet in Windows Server 2012 and Windows Server 2012 R2. This can go a long way too to reducing the disk footprint by completely removing features from Windows (making them unavailable for install).

For instance, if you’re deploying a Remote Desktop Session Host, there’s no need for IIS or Hyper-V to be in the component store. See this blog post article for full details: How to Reduce the Size of the Winsxs directory and Free Up Disk Space on Windows Server 2012 Using Features on Demand