Home Lab - Networking

in

In this post I will cover the basic set up of the basic building block network which is a simple flat network behind a router acting as Firewall, NAT, DHCP and VPN for the network.

The first action is to create a Virtual Switch that will be connectivity for all the virtual machines in this flat network. Almost all virtual solutions support having a virtual switch under one name or another. Since I choose for my home lab ESXi the commands shown will be for this platform but in general terms can be done the same with Hyper-V or XenServer. 

The main reason why I'm showing all the steps via command line is so that they can be automated later in a script if this is a process that will be repeated several times.

Read More

My New Home Lab Setup

in

After I posted in Twitter that I was building a second ESXi server for my lab I got quite a large number of replies and direct messages on what I used as a lab. Based on the interest I decided to write a blog post on why I choose the gear I did and where do I see upgrading it in the near future.

Needs

I have to say we need to start with the needs first since this will dictate what hardware I will need, what hypervisor is best for the work I will do and will also have an impact on my budget.

I need a lab where I can run:

  • Operating Systems:
    • Windows XP/2003 to the latest version 8.1/2012 R2 - I have a MSDN Subscription this year that will help me cover the older versions of the OS and allow me to build permanent labs for complex setups since do to time and work I can not be rebuilding every couple of months. For the latest versions of Windows I use trial versions since Microsoft offers 180 days for server versions and 90 days for client version of the OS this allows me to test different types of persistence and weird configs and I just re-deploy from a template.
    • Linux - I run all kind of different versions of Linux where I test Bash, Python and Ruby scripts I write plus also test forensics and log management research.
    • Oracle Solaris - Currently have customers that run Solaris so I need to be able to run it to test all kinds of configurations, scripts and custom Metasploit modules.
    • FreeBSD - To isolate my labs I use PFSense and also I run several VMs with versions of JunOS that is based on FreeBSD.
    • OS X - Even do I can run OS X on my MacBook Pro I still prefer to have several copies of the server products and the recent client versions since I have been seeing it more and more in corporate environments and it has always been present in educational ones.
  • Nested Hypervisors (VMware, MS Hyper-V, KVM and Xen) - In my day job I do a lot of work on the security of different hypervisors and also I maintain some post-exploitation code to detect when running inside of a VM.
  • Support for Virtual Switches - Virtual Switching allows me to create separate networks with different policies so as to isolate traffic and also mimic a real network better. Some virtual switches allow for port mirroring and bandwith throttling so I can use IDS/IPS for testing, Capture traffic and also mimic WAN connections.
  • API for VM Management - The ability to automate deployment and configuration of VMs becomes important when one needs to tests conde or workflows against different operating systems under different configurations.
Read More

VI-ToolBox PowerCLI PowerShell Module

Recently I decided to migrate some of the PowerCLI (http://communities.vmware.com/community/vmtn/server/vsphere/automationtools/powercli) scripts I use for managing the lab at work and when I consult in infrastructure in to a module so as to make it easier for me to maintain. Also placed the module in GitHub so I can use git to keep it updated between my machines. The module can be found in https://github.com/darkoperator/VI-ToolBox

For using the module you just download the files in to a folder called VI-Toolbox in any of the that PowerShell v2 and v3 look in to for modules. You can list the folder by looking at the variable inside a PowerShell session:

C:\Users\Carlos> $env:PSModulePath
C:\Users\Carlos\Documents\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules\

Once downloaded the files must be unlocked since Windows tends to block all PowerShell files downloaded from the web. If using PowerShell v2 you will have to right click on each file, going to properties and click on Unblock. In PowerShell v3 we can use the Unblock-File cmdlet with the following command:

Get-ChildItem $env:HOME\Documents\WindowsPowerShell\Modules\VI-Toolbox\* | Unblock-File

Lets start by importing the module and listing the functions we have available:

C:\Users\Carlos> Import-Module VI-ToolBox
C:\Users\Carlos> Get-Command -Module VI-Toolbox

CommandType     Name                                               ModuleName
-----------     ----                                               ----------
Function        Disconnect-VIMSession                              VI-ToolBox
Function        Get-VIMSessions                                    VI-ToolBox
Function        Get-VIUserEvents                                   VI-ToolBox
Function        Get-VMConsoleConnectionCount                       VI-ToolBox
Function        Get-VMEvents                                       VI-ToolBox
Function        Get-VMMountedCDRom                                 VI-ToolBox
Function        Get-VMToolStatus                                   VI-ToolBox
Function        Search-VMIPAddress                                 VI-ToolBox
Function        Search-VMMacAddress                                VI-ToolBox
Function        Search-VMOldSnapshots                              VI-ToolBox

Before we can use the functions we must connect to a vCenter server that manages the ESX/ESXi servers. For this we use the Connect-VIServer cmdlet offered by PowerCLI

C:\Users\Carlos> Connect-VIServer -Server vcenter

Name                           Port  User
----                           ----  ----
vcenter                        443   administrator

Once connected we can start working with the functions in the module.

Finding a VM given its IP Address

Many times in a virtual infrastructure we may find our selfs trying to find a VM with a given IP Address Specially when that VM is mis-behaving. To search for a VM the Search-VMIPAddress is available, we can give this function either a group of VMs thru the pipeline using the Get-VM cmdlet or if no VM Object is given it will search against all. Now vCenter uses the information given by the VMware Tools so they need to be installed and running on the host we are looking for:

C:\Users\Carlos> Search-VMIPAddress -IPAddress 192.168.10.10

VMName                                 VMHost                                IPAddress
------                                 ------                                ---------
ALAB-DC01                              labesxi01.darkoperator.com            192.168.10.10

Finding VM's given their MAC Address

Some times users will not install VMware Tools on their hosts, they might not support them or simply the service is not running. We can look for the VM given its MAC Address with the Search-VMMacAddress Function:

C:\Users\Carlos> Search-VMMacAddress -MAC 00:0c:29:eb:df:67


VMName       : ALB-DC02
VMHost       : labesxi01.darkoperator.com
AddapterName : Network adapter 1
NetworkName  : VM Network
MacAddress   : 00:0c:29:eb:df:67

Working with User Sessions

One of the things I always keep an eye out is for who is connecting to the server and to kill any old sessions for this I wrote 2 functions one call Get-VIMSessions and the other Disconnect-VIMSessions. In the following example I have 2 sessions for the Administrator user, one is our current session the other is a previous one that is Idle:

C:\Users\Carlos> Get-VIMSessions


UserName       : Administrator
FullName       :
Status         : Idle
Key            : 523ce38c-3fe5-d0d5-da47-8354f3a0c8ef
LoginTime      : 4/12/2013 6:35:40 PM
LastActiveTime : 4/12/2013 6:39:45 PM

UserName       : Administrator
FullName       :
Status         : Current Session
Key            : 52dcfcc9-a945-631f-c993-0e72c9e8fd08
LoginTime      : 4/12/2013 6:41:53 PM
LastActiveTime : 4/12/2013 6:41:53 PM

Now we see that in Status we can see what session is our current one in the case of several sessions with the same name. We disconnect sessions using the session key as the way to identify the session we want to kill:

C:\Users\Carlos> Disconnect-VIMSession -Key 523ce38c-3fe5-d0d5-da47-8354f3a0c8ef
True
C:\Users\Carlos> Get-VIMSessions


UserName       : Administrator
FullName       :
Status         : Current Session
Key            : 52dcfcc9-a945-631f-c993-0e72c9e8fd08
LoginTime      : 4/12/2013 6:41:53 PM
LastActiveTime : 4/12/2013 6:43:11 PM

Finding VMs with mounted ISO Images

One of the problems of many VMware Admins that have DRS is when their VM's have mounted CDROM Images and prevents the VMs from moving from one vmhosts to the next. For finding this VMs the Get-VMMountedCDRom function can be used against a given collection of VMs or against all, here is an ugly example:

C:\Users\Carlos> Get-VMMountedCDRom

Name                                                      ISO file
----                                                      --------
v-win2k81
ALAB-DC01                                                 [isostore] Microsoft/OS/en_windows_server_2012_x64_dv...
OEL6                                                      [] /usr/lib/vmware/isoimages/linux.iso
v-win2k82
v-win2k3
NSEC_DOM                                                  [isostore] Microsoft/en_windows_server_2012_x64_dvd_9...
fedora                                                    [isostore] Linux/Fedora-18-x86_64-DVD.iso
ALAB-WinXP01                                              [isostore] Microsoft/OS/en_windows_xp_service_pack_3_...
Win7-tmpl x64
Win2k8                                                    [isostore] Microsoft/en_windows_server_2008_datacente...
vuln-win2k8r2
win2k3-vmtest                                             [isostore] win2k3entsp2.iso
ALAB-LOGSRV                                               [isostore] Microsoft/OS/en_windows_server_2008_r2_sta...
ALAB-Win801                                               [isostore] Microsoft/OS/en_windows_8_enterprise_x64_d...
win2k3-tmpl x86
ALAB-Win802                                               [isostore] Microsoft/OS/en_windows_8_enterprise_x64_d...
ALAB-WinXP02                                              [isostore] Microsoft/OS/en_windows_xp_service_pack_3_...
ESXi5.1-01                                                [isostore] VMWare/VMware-VMvisor-Installer-5.1.0-7997...
MSFRH                                                     [] /usr/lib/vmware/isoimages/linux.iso
NessusScanner                                             [] /usr/lib/vmware/isoimages/linux.iso
vuln-xp
ALB-Exch10                                                [isostore] Microsoft/Exchange/mu_exchange_server_2010...
OSX Lion                                                  [isostore] Apple/OSX.LION.GM.iso
WinXP-tmpl-x86
win2k301                                                  [isostore] win2k3entsp2.iso
win2k302                                                  [isostore] win2k3entsp2.iso
Xenserver                                                 [isostore] XenServer-6.1-install-cd.iso
Win2k8R2-core-tmpl x64
Debian6                                                   [] /usr/lib/vmware/isoimages/linux.iso
Ubuntusrv                                                 [isostore] Linux/ubuntu-12.04.1-server-amd64.iso
ALAB-WSUS                                                 [isostore] Microsoft/OS/en_windows_server_2012_x64_dv...
vuln-7
ALAB-Win702                                               [isostore] Microsoft/OS/en_windows_7_ultimate_with_sp...
win2k81
ALAB-Win701                                               [isostore] Microsoft/OS/en_windows_7_enterprise_x64_d...
CentOS6x64                                                [] /usr/lib/vmware/isoimages/linux.iso
Win2k82
test2centos                                               [isostore] Linux/CentOS-6.3-x86_64-bin-DVD1.iso
vCenter                                                   [isostore] VMWare/VMware-VIMSetup-all-5.1.0-799735.iso
BIND_NS                                                   [] /usr/lib/vmware/isoimages/linux.iso
hvtest                                                    [isostore] Microsoft/OS/en_windows_server_2012_x64_dv...
ALB-DC02                                                  [isostore] Microsoft/OS/en_windows_server_2008_r2_sta...
SL6                                                       [] /usr/lib/vmware/isoimages/linux.iso
win2k8r2tmpl
win2k8_vuln                                               [isostore] Microsoft/en_windows_server_2008_datacente...

Now we can unmount the images from all the machines:

C:\Users\Carlos> get-vm | Get-CDDrive | Set-CDDrive -NoMedia -Confirm:$false

Checking VMware Tools

I wrote a simple function to check if tools are running, outdated or just not installed. The function Get-VMToolStatus can get the status of the tools for a collection of VMs or all if none is specified. Here is an example for finding all the VM's whose tools need attention:

C:\Users\Carlos> Get-VMToolStatus | where {$_.ToolStatus -notin "toolsNotRunning","toolsNotRunning","toolsOK"} | ft -AutoSize

Name                 ToolStatus
----                 ----------
fedora        toolsNotInstalled
win2k3-vmtest toolsNotInstalled
ESXi5.1-01    toolsNotInstalled
Xenserver     toolsNotInstalled
Ubuntusrv     toolsNotInstalled
test2centos   toolsNotInstalled
vCenter                toolsOld

Working with User Login, Permission and VM Console Events

Many times I find my self looking at login/Logoff actions and checking permissions for my customers to make sure nobody was given permissions they do not need or track the actions of a developer. Since this is a repeatable task I wrote a function that lets me filter events for sessions by the type of actions, The function is Get-VIUserEvents and one can filter for the following events:

  • Privilege Management
  • Login/Logoff
  • Connection to VM Console

Many times I need to filter and look in specific time frames so the function supports filtering the search by:

  • Hours
  • Days
  • Months
  • Specific date in the past Also one can filter by user name.

To get a list of the event types we can look at the help information for the parameter:

C:\Users\Carlos> help Get-VIUserEvents -Parameter eventtype

-EventType < string >
    Type of events to filter for. Accepts Permission, Session, Console or Any

    Required?                    false
    Position?                    named
    Default value
    Accept pipeline input?       false
    Accept wildcard characters?  false

Here is an example where we look for session events, in the last 8 hours and filter for the user administrator:

C:\Users\Carlos> Get-VIUserEvents -Hours 8 -EventType Session -UserName administrator


IpAddress            : fe80::6966:22f4:8dc0:b35b%10
UserAgent            : VMware VI Client/4.0.0
Locale               : en_US
SessionId            : 523ce38c-3fe5-d0d5-da47-8354f3a0c8ef
Key                  : 103
ChainId              : 103
CreatedTime          : 4/12/2013 6:35:40 PM
UserName             : Administrator
Datacenter           :
ComputeResource      :
Host                 :
Vm                   :
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : User Administrator@fe80::6966:22f4:8dc0:b35b%10 logged in as VMware VI Client/4.0.0
ChangeTag            :
DynamicType          :
DynamicProperty      :

IpAddress            : 192.168.1.243
UserAgent            : Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.18033)
Locale               : en
SessionId            : 52e52807-1476-783b-f480-d11d4551570c
Key                  : 276
ChainId              : 276
CreatedTime          : 4/12/2013 6:41:52 PM
UserName             : Administrator
Datacenter           :
ComputeResource      :
Host                 :
Vm                   :
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : User Administrator@192.168.1.243 logged in as Mozilla/4.0 (compatible; MSIE 6.0; MS Web
                       Services Client Protocol 4.0.30319.18033)
ChangeTag            :
DynamicType          :
DynamicProperty      :

IpAddress            : 192.168.1.243
UserAgent            : Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 4.0.30319.18033)
Locale               : en
SessionId            : 52dcfcc9-a945-631f-c993-0e72c9e8fd08
Key                  : 277
ChainId              : 277
CreatedTime          : 4/12/2013 6:41:53 PM
UserName             : Administrator
Datacenter           :
ComputeResource      :
Host                 :
Vm                   :
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : User Administrator@192.168.1.243 logged in as Mozilla/4.0 (compatible; MSIE 6.0; MS Web
                       Services Client Protocol 4.0.30319.18033)
ChangeTag            :
DynamicType          :
DynamicProperty      :

Working with VM Power On, Power Off, Deletion, Creation and Console Events

Many times I found myself fishing thru events for monitoring actions taken on events that affected the availability of VM's and to monitor VM Sprawl so I crated in my toolkit Get-VMEvents to help me parse the events. Just like the user event function we can filter by event type and dates. We can see the event types we can filter thru can be seen in the parameter help:

C:\Users\Carlos> help Get-VMEvents -Parameter eventtype

-EventType <string>
    Specific types of event to filter on. Accepts Creation, Deletion, Console, PowerOn, PowerOff or Any

    Required?                    false
    Position?                    named
    Default value
    Accept pipeline input?       false
    Accept wildcard characters?  false

Here is an example of checking for PowerOn events in the last couple of hours:

C:\Users\Carlos> Get-VMEvents -EventType PowerOn -Hours 3


Template             : False
Key                  : 373
ChainId              : 373
CreatedTime          : 4/12/2013 8:02:14 PM
UserName             :
Datacenter           : VMware.Vim.DatacenterEventArgument
ComputeResource      : VMware.Vim.ComputeResourceEventArgument
Host                 : VMware.Vim.HostEventArgument
Vm                   : VMware.Vim.VmEventArgument
Ds                   :
Net                  :
Dvs                  :
FullFormattedMessage : NessusScanner on  labesxi01.darkoperator.com in Lab is powered on
ChangeTag            :
DynamicType          :
DynamicProperty      :