My New Home Lab Setup
After I posted in Twitter that I was building a second ESXi server for my lab I got quite a large number of replies and direct messages on what I used as a lab. Based on the interest I decided to write a blog post on why I choose the gear I did and where do I see upgrading it in the near future.
Needs
I have to say we need to start with the needs first since this will dictate what hardware I will need, what hypervisor is best for the work I will do and will also have an impact on my budget.
I need a lab where I can run:
- Operating Systems:
- Windows XP/2003 to the latest version 8.1/2012 R2 - I have a MSDN Subscription this year that will help me cover the older versions of the OS and allow me to build permanent labs for complex setups since do to time and work I can not be rebuilding every couple of months. For the latest versions of Windows I use trial versions since Microsoft offers 180 days for server versions and 90 days for client version of the OS this allows me to test different types of persistence and weird configs and I just re-deploy from a template.
- Linux - I run all kind of different versions of Linux where I test Bash, Python and Ruby scripts I write plus also test forensics and log management research.
- Oracle Solaris - Currently have customers that run Solaris so I need to be able to run it to test all kinds of configurations, scripts and custom Metasploit modules.
- FreeBSD - To isolate my labs I use PFSense and also I run several VMs with versions of JunOS that is based on FreeBSD.
- OS X - Even though I can run OS X on my MacBook Pro I still prefer to have several copies of the server products and the recent client versions since I have been seeing it more and more in corporate environments and it has always been present in educational ones.
- Nested Hypervisors (VMware, MS Hyper-V, KVM and Xen) - In my day job I do a lot of work on the security of different hypervisors and also I maintain some post-exploitation code to detect when running inside of a VM.
- Support for Virtual Switches - Virtual Switching allows me to create separate networks with different policies so as to isolate traffic and also mimic a real network better. Some virtual switches allow for port mirroring and bandwith throttling so I can use IDS/IPS for testing, Capture traffic and also mimic WAN connections.
- API for VM Management - The ability to automate deployment and configuration of VMs becomes important when one needs to tests code or workflows against different operating systems under different configurations.
Hypervisor
There are currently 2 types of hypervisors to choose from in the market.
- Type 1 Hypervisors where the hypervisors runs in the kernel right on top of the Hardware and the VMs on top. This provides the greatest performance. Examples of this are Citrix XenServer, VMware ESXi
- Type 2 hypervisor sit on top of the operating system and does not have direct access to the HW for the most part.
As we can see the Type 1 Hypervisor is the one that should offer to us the greatest speed and access to resources specially for large configurations sets (Exchange, Oracle, SCCM, SAP). Examples are:
- VMware ESXi
- Hyper-V
- XenServer
- KVM
The Type 2 Hypervisor does offer the flexibility of using the base OS for other stuff, this is great for those starting out with little financial or physical resources. Type 2 Hypervisors are great for mobile workers that need to have small portable labs in their work laptops or just the need to have some none resource intensive VMs. Examples are:
- VMware Workstation and Fusion
- Oracle VirtualBox
- Parallels
In my case I chose ESXi do to Hypervisor Nesting (running other hypervisors inside another) and the flexibility in management and experience I have of year working with it. If money would have been a concern and nested hypervisors would not been as important I would have gone Xenserver do to is greater feature set in other areas. If my main systems where only Windows and I needed better I/O and performance but still wanted a Free Option with great flexibility of management then I would have gone with Microsoft Hyper-V Server for my lab.
Hardware
My lab consists of 2 ESXi servers, one NAS and several Cisco devices I have for network tests that just can't be virtualized. The first server and NAS I built at the end of 2011 and the second server at the end of 2013.
First Server
This server in terms of CPU it out performs the second server do to that it is a Xeon with Hyperthreading, but CPU has not been a constrained so far in my labs when I use Quad Core CPUs. The basic build is:
- Shuttle SH67H3 barebone system
- Segate Barracuda SATA 1TB HDD
- Intel Xeon E3-1230 V2 Ivy Bridge 3.3GHz (3.7GHz Turbo) LGA 1155 69W Quad-Core Server Processor
- 32 GB DDR3 1600 Gskill Memory
- Fanless video card
- EXPI9402PTBLK Intel PRO/1000 PT DP Server Adapter (from Ebay $30,Newegg.com it is over $200)
Second Server
This server uses the latest Haswell technology and runs a little slower in benchmarks compare with the first server I build. The basic build is:
- Shuttle SZ87R6 barebone system.
- Segate Barracuda SATA 1TB HDD
- Intel Core i5-4570 Haswell 3.2GHz
- 32 GB DDR3 1600 Gskill Memory
- EXPI9402PTBLK Intel PRO/1000 PT DP Server Adapter (from Ebay $30,Newegg.com it is over $200)
Each server provides more than adequate performance for most of the labs I configure for research with the exception of high I/O application (WSUS, SCCM, Exchange, MS SQL, SAP ..etc) where the SATA BUS is stressed. I keep the VMs on each server locally since using a NAS I was only getting around 98MB per second on a NFS share while locally I get around 140MB per second speeds.
NAS
I use a NAS for NFS shares that are used as a ISO image repository and for backing up VMs. For this I use a Netgear ReadyNAS Ultra 4 with NFS enabled. Both interfaces are connected to a 100/1000 switch.
Setup
I selected ESXi as the hypervisor for my home lab based on my needs and was running version 5.1 initially and later upgraded to 5.5, when upgraded the embedded RealTek interfaces stopped working since VMware decided to stop supporting and including the drivers for them as stated in this KB
Custom Installer using VMware PowerCLI PowerShell Module
A custom ISO image can be created using VMware PowerCLI PowerShell module, I used the following blog post as a guide
PS C:\> Add-PSSnapin VMware.ImageBuilder
We then add the VMware Online software Depot:
PS C:\> Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml Depot Url --------- https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml PS C:\>
We create a custom profile for the image:
$ProfileInfo = Get-EsxImageProfile ESXi-5.5.0-1331820-standard $CustomProfile = New-EsxImageProfile -CloneProfile $ProfileInfo -Vendor $ProfileInfo.Vendor -Name (($ProfileInfo.Name) + "-customized-Reltek") -Description $ProfileInfo.Description
We add the missing drivers for the network interfaces:
PS C:\> Add-EsxSoftwarePackage -SoftwarePackage net-r8168 -ImageProfile $CustomProfile Name Vendor Last Modified Acceptance Level ---- ------ ------------- ---------------- ESXi-5.5.0-1331820-standard... VMware, Inc. 1/10/2014 1:... PartnerSupported PS C:\> PS C:\> Add-EsxSoftwarePackage -SoftwarePackage net-r8169 -ImageProfile $CustomProfile Name Vendor Last Modified Acceptance Level ---- ------ ------------- ---------------- ESXi-5.5.0-1331820-standard... VMware, Inc. 1/10/2014 1:... PartnerSupported PS C:\>
We can now create a custom ISO and save it in the desktop for our current user, the files will be downloaded from VMware and the image built so this will take a while to download:
Export-EsxImageProfile -ImageProfile $CustomProfile -ExportToISO -FilePath "$([Environment]::GetFolderPath("Desktop"))\ESXi-5.5.0-1331820-standard-customized-reltek.iso" -verbose
Final Result
This is how my home virtual machine lab looks now:
Future Upgrades
In the future I'm looking at getting a managed gigabit switch to use with VLANs, port mirroring and Jumbo Frames. On the NAS side I'm looking at changing to Synology NAS since their management interface is better and provides better performance. On the ESXi server side I plan on upgrading to Samsung 840 Pro or Intel DC S3500 Series solid state drives for datastore since they have a higher amount of data that can be written before failure so they are optimal for use as Datastore to save the VMs.