Tag Archives: VMware

VMware Software Download Service error

Posted on by | Leave a reply

Sometimes the download service seems to hang with an error such as “Reading the VMware software depot for the first time. This could take up to several minutes…”. One possible cause is an ongoing corruption issue with some of the underlying depot data files. This is documented in https://communities.vmware.com/thread/568541 and https://communities.vmware.com/thread/553547

In summary, the current issue relates to “corruption” in the dlg_ESXI60U3A.xhtml data file. Reference is made to a missing component (which seem ultimately to be files in the same directory).

The stdout-vapi_server.log file (stored in C:\Users\username\AppData\Roaming\VMware\Software Manager\Download Service) includes reference to the failing items, which can be removed one at a time. Once this is done (with the download service stopped) the service should resume working or report the next failing item. If the above error occurs, keep checking the stdout-vapi_server.log file for references to error items.

IPMI warnings from ASRock C2750D4I with 1.35V RAM – FIXED

Posted on by | Leave a reply

Short bullet points as I may get round to expanding this:

ASRock C2750D4I with 1.35V DRAM modules.

VMware client showed health status as an alert due to low RAM voltage (expected as they are 1.35V but it seems the machine’s BIOS sets the threshold for 1.5V RAM modules)

ESXi 6.0 health indicated VCCM1 (voltage controller for memory 1??) was in a warning state at 1.35V.

 

I found a statically linked copy of ipmitool at https://fattylewis.com/ipmitool-on-esxi-6/ which I uploaded to the ESXi host in question. This worked first time!

./ipmitool chassis status
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : previous
Last Power Event : ac-failed
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false

I checked the sensors:

./ipmitool sensor list
....<snipped>...
ATX+5VSB | 5.040 | Volts | ok | 4.050 | 4.260 | 4.500 | 5.490 | 5.760 | 6.030
+3VSB | 3.440 | Volts | ok | 2.660 | 2.800 | 2.960 | 3.620 | 3.800 | 3.980
Vcore1 | 1.050 | Volts | ok | 0.540 | 0.570 | 0.600 | 1.490 | 1.560 | 1.640
Vcore2 | na | Volts | na | 0.540 | 0.570 | 0.600 | 1.490 | 1.560 | 1.640
VCCM1 | 1.350 | Volts | nc | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810
VCCM2 | na | Volts | na | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810
....<snipped>...

There we have it, the warning is set for 1.38V. This can be changed with IPMI tool:

./ipmitool sensor thresh VCCM1 lnc 1.32

Now we check the result:

./ipmitool sensor list VCCM1|grep VCCM
VCCM1 | 1.340 | Volts | ok | 1.210 | 1.280 | 1.320 | 1.650 | 1.730 | 1.810
VCCM2 | na | Volts | na | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810

And the VMware health status got fixed too!

 

 

For now, I will need to run this command after every reboot or reset of the IPMI management controller. I guess I should automate this for system boot-up….

FOLLOW-UP: I have just upgraded the BMC firmware from v0.27.00 to v0.30.00 and the issue seems to be resolved as the VCCM threasholds have been redefined:

./ipmitool sensor|grep VCCM
VCCM | 1.350 | Volts | ok | 1.090 | 1.120 | na | na | 1.720 | 1.750

Oh well, this was an interesting learning experience. Maybe this ipmitool info will help someone else. Note that this is a useful tool as it allows one to configure the management controller from the host while ESXi is running. Useful perhaps to configure the LAN, change the admin user or reset the controller entirely. For example, we can configure the IP address as follows:

./ipmitool lan set 1 ipsrc static
./ipmitool lan set 1 ipaddr 172.16.1.5
./ipmitool lan set 1 netmask 255.255.255.0
./ipmitool lan set 1 defgw ipaddr 172.16.1.1
./ipmitool lan set 1 arp respond on
./ipmitool lan set 1 arp generate on
./ipmitool lan set 1 arp interval 60

Extra info about using ipmitool and controlling the fans on this motherboard at https://blog.chaospixel.com/linux/2016/09/Fan-control-on-Asrock-C2750D4I-C2550D4I-board.html

Upgrade VM to hardware version other than latest

Posted on by | Leave a reply

Upgrading a VM’s VMware hardware version to the latest version is generally considered best practice. This is easy to do via the C# client or the web-client. If you wish to upgrade to a newer, but not latest, hardware version this can be tricky.

As highlighted in https://www.v-front.de/2014/02/how-to-uprade-your-vms-virtual-hardware.html, one easy way to do so is as follows on the ESXi host in question:

vim-cmd vmsvc/upgrade vmid vmx-10

where vmid is determined using something like:

vim-cmd vmsvc/getallvms|awk '{print $1 "  "$2" "$6}'

The appropriate hardware version can be selected from a useful VMware Virtual Machine Hardware Versions KB. The recent relevant versions are:

vmx-13: ESXi 6.5

vmx-12: Workstation Pro 12.x

vmx-11: ESXi 6.0 / Workstation 11.x

vmx-10: ESXi 5.5 / Workstation 10.x

vmx-9: ESXi 5.1 / Workstation 9.x

vmx-8: ESXi 5.0 / Workstation 8.x

vmx-7: ESXi/ESX 4.x

 

 

 

Manual EVC-like per VM configuration

Posted on by | 1 Reply

Introduction

While working in a lab environment recently I wanted to vMotion a VM between two ESXi hosts. The vMotion failed, which was not entirely unexpected, due to CPU incompatibilities. These particular ESXi hosts are not in a vSphere cluster so enabling EVC (Enhanced vMotion Compatibility), which would resolve the issue, is not an option.

Attempting to vMotion a VM from host A to host B gave errors about:

  • MOVBE
  • 3D!Now PREFETCH and PREFETCHW

After powering the VM down, migrating the VM to host B and powering on, an attempt to vMotion from host B to host A gave errors about:

  • PCID
  • XSAVE
  • Advanced Vector Extensions (AVX)
  • Half-precision conversion instructions (F16C)
  • Instructions to read and write FS and GS base registers
  • XSAVE SSE State
  • XSAVE YMM State

Manual VM CPUID mask configuration

As the error messages indicate, Enhanced vMotion Compatibility (EVC) would enable the vMotion to take place between mixed CPUs within a cluster. Mixing host CPU models within a vSphere cluster is generally considered bad practice and should be avoided where possible. In this instance, the hosts are not in a cluster so EVC is not even an option.

As indicated above shutting down the VM and doing a cold migration is possible. This issue only relates to the case where I want to be able to migrate running VMs between hosts containing processors with different feature sets.

For the two hosts in question, I know (based on the EVC processor support KB and Intel ARK and VMware KB pages) that the Intel “Westmere” Generation baseline ought to be the highest compatible EVC mode; one of the processors is an Intel Avoton C2750 and the other is an Intel i7-3770S Sandy Bridge. The Avoton falls into the Westmere category for EVC. We will come back to EVC later on.

I suspected it would be possible to create a custom CPU mask to enable the vMotion between these to hosts. In general, features supported by a given processor are exposed via a CPUID instruction call. By default, VMware ESXi manipulates the results of the CPUID instructions executed by a VM as part of the virtualisation process. EVC is used to further manipulate these feature bits to hide CPU features from VMs to ensure that “well behaved VMs” are able to run when migrated between hosts containing processors with different features.

In this instance, “well behaved VMs” refers to VMs running code which use the CPUID instruction to determine available features. If the guest OS or application uses the CPUID instruction to determine available processor features then, when moved via vMotion to a different host, that same set of features will be available. If a guest uses some other mechanism to determine processor feature availability (e.g. based on the processor model name) or merely assumes a given feature will be available then the VM or application may crash or have other unexpected errors.

So back to this experiment. Attempting to go from host A to host B indicated only two feature incompatibilities. I turned to the Intel developers manual (64-ia-32-architectures-software-developer-vol-2a-manual.pdf) for the detail about the CPUID instruction. CPUID called with EAX=0x80000001 results in the PREFETCHW capability being exposed at bit 8 in ECX. Similarly with EAX=0x1, the MOVBE capability is exposed at bit 22 in ECX.

As an initial test, I did a cold migration of the VM to host A and edited the VM settings as shown below.

In summary, this is passing the CPUID result of the host via the above mask filter. The mask filter can hide, set or pass through a CPUID feature bit. In this instance I am hiding the two bits identified above through the use of the “0” in those bit positions. There are other options you can use as displayed in the legend.

I chose “0” rather than “R” as I need to hide the feature from the guest OS and I do not care if the destination host actually has that feature or not.

I saved this configuration and powered on the VM. I was able to successfully perform a vMotion from host A to host B. I was also able to vMotion the VM back to host A.

I performed a vMotion back to host B and powered the VM off. I then powered the VM back on on host B. I tried to vMotion to back to host A, which again failed with the same error as shown above. The reason it failed in the reverse direction is that the VM pickups up it’s masked capabilities at power on and maintains that set of capabilities until it is powered off once more. So by powering on the VM on host B, it got a different set of capabilities to when it was powered on on host A. This explains why when attempting to originally perform the vMotion we had two different sets of errors.

To get the masks to enable a vMotion from host B to host A, I took a look at the developers guide and performed some Googlefoo, I identified the CPUID bits needed to mask the unsupported features:

PCID: CPUID EAX=1, result in ECX bit 17
XSAVE: CPUID EAX=1, result in ECX bit 26
AVX: CPUID EAX=1, result in ECX bit 28
F16C: CPUID EAX=1, result in ECX bit 29

FSGSBASE: CPUID EAX=7, result in EBX bit 00

XSAVE SSE: CPUID EAX=0xd, result in EAX bit 01
XSAVE YMM: CPUID EAX=0xd, result in EAX bit 02 (YMM are 256-bit AVX registers)

The first four are easy as the vSphere client allows one to edit the EAX=1 CPUID results. With the below configuration in place, the vMotion from host B to host A only showed the last three errors (FSGSBASE, XSAVE SSE and XSAVE YMM). This is expected as no masking had been put in place.

To put the masking in place for EAX=0x7 and EAX=0xd we need to edit the virtual machine’s .VMX file. We can do this by editing the .vmx file directly or by using the Configuration Parameters dialogue for the VM under Options/Advanced/General in the VM’s settings dialogue. The following two parameters (first one for FSGSBASE and second for the XSAVE) were added:

cpuid.7.ebx = -------------------------------0
cpuid.d.eax = -----------------------------00-

Powering on the VM succeeded, however the vMotion to host A failed with the same error about FS & GS Base registers (but the XSAVE errors were gone). Surprisingly when I checked the .vmx directly and the cpuid.7.ebx line was missing. For some reason it appears that the VI client does not save this entry. So I removed the VM from the inventory, added that line to the .VMX directly and then re-registered the VM.

I was now able to power on the VM on host B and vMotion back and forth. I was not able to do the same when the VM was powered on on host A. I needed to merge the two sets of capabilities.

At this stage we would have the following in the .vmx file:

for host A -> host B:
cpuid.80000001.ecx = "-----------------------0--------"
cpuid.1.ecx = "---------0----------------------"

for host B -> host A:
cpuid.1.ecx = "--00-0--------0-----------------"
cpuid.7.ebx = "-------------------------------0"
cpuid.d.eax = "-----------------------------00-"

(Note that there are some default entries which get added which are all dashes, and one for cpuid.80000001.edx with dashes and a single H).

We merge our two sets of lines to obtain:

cpuid.80000001.ecx = "-----------------------0--------"
cpuid.1.ecx = "--00-0---0----0-----------------"
cpuid.7.ebx = "-------------------------------0"
cpuid.d.eax = "-----------------------------00-"

At this stage we can now power on the VM on either host and migrate in either direction. Success. Using these four lines of config, we have masked the specific features which vSphere was highlighting as preventing vMotion. It has also shown how we can hide or expose specific CPUID features on a VM by VM basis.

Manual EVC Baseline Configuration

Back to EVC. The default EVC masks can be determined by creating a cluster (even without any hosts) and enabling EVC. You can then see the default masks put in place on the host by EVC. Yes, EVC puts a default mask in place on the hosts in an EVC enabled cluster. The masked off CPU features are then not exposed to the guests at power-on and are not available during vMotion compatibility checks.

The default baselines for Westmere and Sandybridge EVC modes are shown below:

 

The differences are highlighted. Leaf1 (i.e. CPUID with EAX=1) EAX result relates to processor family and stepping information. The three Leaf1 ECX flags relate to AES, XSAVE and TSC-Deadline respectively. The three Leafd EAX flags are for x87, SSE and AVX XSAVE state. The three Leafd ECX flags are related to maximum size needed for the XSAVE area.

Anyway I’ve digressed. So the masks which I created above obviously only dealt with the specific differences between my two processors in question. In order to determine a generic “Westmere” compatible mask on a per VM basis we will start with VMware’s ESXi EVC masks above. The EVC masks are showing which feature bits are hidden (zeros) and which features may be passed through to guests (ones). So we can see which feature bits are hidden in a particular EVC mode. So to convert the above EVC baselines to VM CPUID masks I keep the zeros and change the ones to dashes. I selected dashes instead of ones to ensure that the default guest OS masks and host flags still take effect. We get the following for a VM for Westmere feature flags:

cpuid.80000001.ecx = "0000000000000000000000000000000-"
cpuid.80000001.edx = "00-0-000000-00000000-00000000000"
cpuid.1.ecx = "000000-0-00--000---000-000------"
cpuid.1.edx = "-000-------0-0-------0----------"
cpuid.d.eax = "00000000000000000000000000000000"
cpuid.d.ecx = "00000000000000000000000000000000"
cpuid.d.edx = "00000000000000000000000000000000"

I did not map the cpuid.1.eax flags as I did not want to mess with CPU family/stepping flags. Also, the EVC masks listed did not show the cpuid.7.ebc line I needed for the FSGSBASE feature. Sure enough, using only the 7 lines above meant I could not vMotion from host B to host A. So, adding

cpuid.7.ebx = "-------------------------------0"

to the VMX then allowed the full vMotion I was looking for. The ESXi hypervisor must alter other flags apart from only those shown on the EVC configuration page.

TL;DR

To configure a poor man’s EVC on a VM by VM basis for a Westmere feature set, add the following lines to a VM’s .VMX file.

cpuid.80000001.ecx = "0000000000000000000000000000000-"
cpuid.80000001.edx = "00-0-000000-00000000-00000000000"
cpuid.1.ecx = "000000-0-00--000---000-000------"
cpuid.1.edx = "-000-------0-0-------0----------"
cpuid.7.ebx = "-------------------------------0"
cpuid.d.eax = "00000000000000000000000000000000"
cpuid.d.ecx = "00000000000000000000000000000000"
cpuid.d.edx = "00000000000000000000000000000000"

 

Appendix

1

Useful thread -> https://communities.vmware.com/thread/467303

The above thread covers manipulating the guest CPUID. An interesting option is mentioned in post 9 relating to an option to enable further CPUID manipulation than is possible by default. In my tinkering above, I did not need this option.

monitor_control.enable_fullcpuid = TRUE

Note too that the vmware.log of any VM can be used to see the CPUID information of the host, as also mentioned in post 9:

As for extracting the results, you can write a program to query the CPUID function(s) of interest, or you can just look in the vmware.log file of any VM.  All current VMware hypervisors log all CPUID information for both the host and the guest

Post 13, again a user jmattson (exVMware now at Google), reveals a simple way to configure the processor name visible to guests:

cpuid.brandstring = "whatever you want"

2

This thread https://communities.vmware.com/thread/503236, again involving jmattson, discusses cpuid masks – and gives an insight into how EVC masks interact with the VM cpuid masks.

3

This post https://v-reality.info/2014/08/vsphere-vm-version-impact-available-cpu-instructions/ reveals that the virtual hardware version of a given VM also plays a role in the CPUID mask a VM is given. I found this interesting as it does give us another reason to actively upgrade the hardware versions of VMs.

4

A little gem is mentioned at the bottom of https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1029785 – it seems that IBM changed the BIOS default for some of their servers. The option was changed to enable the AES feature by default. This resulted in identical servers configured with BIOS defaults, which were added to a vSphere cluster, having two different sets of CPUID feature bits set (AES enabled on some and disabled on others) resulting in vMotions not being possible between all servers.

VMware Workstation and SATA physical drives

Posted on by | Leave a reply

There are various posts relating to issues with VMware Workstation and the use of SATA physical drives (i.e. passing a physical SATA drive through to the guest VM).

The first challenge is getting past the “Internal Error” error message. To do so, create a VM with a SATA virtual disk. Once you’ve done this, you can try and add a SATA physical drive to the guest. This needs to be as a SATA device, since adding the pass-through drive as a SCSI device works. You will receive the “Internal Error” error message. Note that the .vmdk file is created for the drive in the VM’s directory.

The next step is to edit the .vmx and replace the original SATA device (sata0:0.fileName= line) with the newly created .vmdk file. This will get the SATA pass-through device into the VM. However, I was not able to power on the VM at this stage and got another error message.

Looking in the VM’s log file it was apparent that VMware Workstation was unable to open the raw device,
\\?\Volume{someGUID}

The fix to this is to run VMware Workstation as administrator. So instead of double clicking as you normally would, you need to right click and select “Run as administrator”. This was the step that I did not see mentioned anywhere else.

By doing this, I was able to start the VM and it then worked as expected!

VMware VCAP5-DCA passed – exam experience notes and study aids

Posted on by | 1 Reply

I got an e-mail letting me know that I had passed my VMware VCAP5-DCA exam. Phew! I sat the exam a week before Christmas, so the news after Christmas about my pass was a belated Christmas present!

The exam was pretty much as described by the various other blog postings. The main problem I faced was time. I ended up skipping some questions due to time constraints. I wrote 1-26 on the note board and ticked them off as I went along. I did each question as it popped up unless I was not confident on being able to complete it fairly quickly. This was going well until a question on the vMA.

The question in question was related to something I had not actually done in my prep but I figured I knew enough about the vMA to complete it. I ended up spending about 12 minutes to to get the question completed but didn’t manage to. Looking back, I should have decided to move on much sooner. This wasted time resulted in me not having sufficient time at the end to complete a question I could have.

The other “blunder” I made was related to a cluster configuration. After I read the question I knew what needed to be done. I went through and completed the question and moved on. A few questions later I was back doing something else cluster related and noticed that the previous configuration which I remembered doing was missing!! So, I back tracked the questions (you can in the DCA exam but not in the DCD) and re-did the prior question – this time ensuring I clicked OK and then verified the configuration was there. I guess the first time I must have clicked “cancel”  rather than “OK” in one of the dialogue boxes. Doh! So this effectively cost me another question worth of time.

I went into the exam being aware that I’d not spent enough study time on auto-deploy or image profiles. Needless to say questions related to those topics caused me to use more time than necessary. As I said previously, I ended up missing a few questions out due to time constraints. Had the remote connectivity been quicker and the exam environment been more responsive, I would have been able to do one or two more questions rather than waiting for screen redraws and so on. I’m not saying it was unusable, but more like being on the end of a slow WAN link (oh wait, the exam kit is hosted far away…). The frustrating thing was that for one of the questions my usual troubleshooting method would be to have a couple of windows open and flick between them fairly quickly, diagnosing the problem. Due to the exam environment this was not terribly feasible and I ended up skipping the question to make progress on other questions.

So, my “lessons learned” from this exam for any future DCA exams I might do are:

  • Be time concious and don’t get bogged down with trying to make something work unless you are certain that you have the knowledge needed to complete the question.
  • Know all the content of the blueprint fairly well. Read between the lines of the blueprint to know how the knowledge would be used in a real world situation. So, as an example do know the options for cluster fail over capacity and know how the options would relate to a real world requirements. Try and understand how the topics contained within the blueprint apply to solving day-to-day administrations problems or meeting platform requirements.
  • Read through the supporting documents mentioned in the blueprint. Try to read all the entire documents at least once to get plenty of additional background knowledge.
  • Ensure plenty of hands-on LAB time. Try and use the version of vSphere mentioned within the blueprint. Currently vSphere 5.0!
  • Try and enjoy the exam. It does not feel too much like an exam as you are doing hands on problem solving
  • Be time concious (yes, it is that important I mention it twice!)

 

So, how did I study for the exam? Well, I did hands-on exercises in a nested LAB environment under VMware Workstation (thanks VCP5!!). I covered the blueprint fairly extensively doing tasks based on activities mentioned in the blueprint. I also did “extra credit” type exercises where I tried to apply the knowledge from the topics for some real-world issues I’ve experienced and some hypothetical examples I thought up.

I read the following books:

I read loads of blog posts and followed numerous VMware related twitter peeps. Here are a few of the blogs/postings which I found useful while studying. There were more, but unfortunately I don’t appear to have created bookmarks for them.

And useful for those nested ESXi labs – VMware Tools for ESXi

Hopefully the above will be useful to someone in the future!!

 

VMware VCAP5-DCA sample activities

Posted on by | Leave a reply

I’ve never done the VCAP5-DCA exam before so I don’t know what questions will be asked. That said, looking at the various topics in the blueprint I came up with some activities which I interpret to be within the scope of the exam. I intend to go through these (and other) activities during my study time.

  • Perform all the tasks below without an outage for a particular business critical VM. This VM must have no downtime during these maintenance operations.
  • A client calls you in and asks you to configure auto-deploy for stateless deployment of some 10 new hosts (DHCP has been configured for these hosts as 10.1.15.20 through 10.1.15.29) in an HA cluster called “DEV-TEST-clus” to match their existing three ESXi servers (which are installed to local drives). The 10 new hosts need a non-standard driver for a particular piece of hardware (one can use a driver update for this). The client uses a Windows based vCenter and does not currently have auto-deploy configured.
  • You have been asked to deploy a vMA by your boss to allow him to configure various tasks which he will setup to be run by cron against the vSphere estate. Deploy and configure the vMA so that sample commands can be run from the vMA command line against the vCenter server and the ESXi hosts in the environment.
  • You find that you often need a list of all the VMs, the hosts they are on and their powered up state for a report you write. Create a PowerCLI command/script to provide only this information.
  • Configure a central log host for the ESXi servers for your environment. Use vCLI to configure the hosts to log to this logging host. No more than 20MB of logs should be stored and no log file should be bigger than 2MB.
  • Deploy UMDS. Configure a baseline for ESXi 5.0 hosts for security updates only. Verify this and export the baseline using PowerCLI
  • You need to configure 20 identical hosts with a new vSwitch using two unsed vmnics. You need to create the following port groups: dmzWEB-v10 (VLAN10), dmzAPP-v20 (VLAN20) and dmzCRM-v30 (VLAN30). dmzWEB-v10 is to use load balancing based on originating portID, dmzAPP-v20 is to use source-MAC hash load balancing and dmzCRM-v30 needs to use explicit failover order. The higher number vmnic should be primary and the lower numbered one being standby and failback should be disabled. The dmzWEB-v10 port group needs promiscuous mode enabled due to the way the application works. The dmsAPP-v20 port group should have it’s traffic limited to an average and peak bandwidth of 2Mb/s with a burst size of 5000KB. A VMkernel port should be created on VLAN 50 to be used for FT traffic only (the IP addresses should be 10.10.30.20/24 through 10.10.30.39/24). A portgroup dmzFW should be created with default settings and configured for VGT.
  • Configure PVLANs on an existing dvSwitch, using a command line tool if possible, so that the exact commands can be put in the RFC. VLAN 101 is the promiscuous VLAN, VLAN 102 is the isolated VLAN and 103 and 104 are the community VLANs.
  • Create an HA/DRS cluster (only HA to be enabled and using the default configuration) called “FIN-CLUS1”. Add two hosts, “fin-host1” “fin-host2” to the cluster.
  • Collect IO stats for a VM for approximately 5 minutes from VM power on (to capture boot-up I/O statistics) and export them to a CSV stored on the vCenter server. Maybe boot up into the VM BIOS to ensure all the IOs involved in the bootup process are captured.
  • You notice that a LUN is missing from a host yet the storage admins have confirmed that the SAN and storage array is configured correctly and that the host can see other LUNs on the array. Other hosts can see the LUN. Explain what needs to be checked and then reconfigure the host to be able to use the LUN.
  • Once the LUN is correctly visible to the host, you notice that it is not flagged as an SSD capable LUN. Configure the LUN so that all hosts correctly identify the LUN as an SSD LUN.
  • Migrate the business critical VM mentioned above onto this new SSD LUN. Ensure that this LUN is a preferred LUN for datastore heartbeating.

Is 32GB RAM for a VMware vSphere home lab sufficient these days?

Posted on by | Leave a reply

A recent comment I made on Chris Wahl’s blog seems to have generated a little bit of interest, so I thought I would expand upon my thinking about this.

The comment I wrote in response to the “The New Haswell Fueled ESXi 5.5 Home Lab Build” post is included here:

Great review of your lab kit once more!

I’ve been looking at new lab kit myself and have been considering E3-1200v3 vs E5-2600v2 processors. Obviously an E5 based machine is going to be more expensive than an E3 based one. However- the really big draw for me is the fact the E5 procs can use more than 32GB of RAM.

Looking at some rough costs of 2* E3-1265Lv3 32GB X10SL7-F-0 servers vs 1* E5-2640v2 64GB X9SRH-7F server (same cases and roughly similar cooling components) it seems that the two E3 servers are more expensive.

Do you consider 32GB to still be sufficient for a home lab? And within a year or three time? I’ve not considered the cost-benefit of a E3 now and replacing it (probably similar cost) with the “latest and greatest” equivalent in 2 years time. I guess it depends on one’s expected depreciation time frame.

Who would have thought VCAP-DCD that scale-out vs scale-up questions would be relevant to one’s home lab :)

(In fairness, I was looking at having this “lab” environment also run a couple of “production” VMs for home use concurrently, so the 32GB would not be dedicated to the LAB)

 

Going through various lab scenarios with some VMware software prompted me to consider options as far as home lab or standalone ESXi hardware was concerned. Also, I have some ageing server hardware which still runs vSphere but no nested 64-bit VMs (yes, fairly old E54xx and E5300 CPUs, without EPT and other cool new features) which needs to be replaced.

So, my requirements when considering the hardware for a home lab were

  • modern Intel CPU (I just prefer them… my choice – no debate 🙂  ), so i7, E3-1200v2 or E5-2600v2 are all in the running
  • sufficient RAM
  • remote KVM (i.e. iLO, DRAC, etc), remote media and remote power cycle
  • cost is a factor so high end server kit is probably out
  • power consumption must be considered. Electricity is not cheap these days and less power translates on the most part to less heat too

Now all are fairly self explanatory apart from the RAM. In my particular case, I would want to have the LAB kit also run a “production” workload of a couple of home servers (currently approximately 6-8GB RAM depending on TPS and workload). This presents a couple of challenges. How to separate things out sufficiently? I could go for a “big” server with more than 32GB of RAM or have more than one “smaller” server.

In terms of efficiency and reduced complexity, a single bigger server is probably going to give more scope for expansion in the RAM department. In my experience, RAM is constrained before CPU for most situations (even more so with my VMs ). So, the 32GB limit of the i7 and E3 is definitely something to consider. The E5 gives 64GB and upward, depending on chipset, motherboard, DIMMs in use, etc.

So given I currently need about 7GB RAM for my “production” workload, that would leave 24GB (of a maxed out i7/E3 CPU) for the LAB.  Is 24GB sufficient for my lab purposes? That is a question I am still grappling with before actually purchasing any new kit.

I have managed to run a Site Recovery Manager set up under VMware Workstation on a desktop machine (i7 and 32GB RAM). The performance was “OK” but I noticed a few packets being lost every 15 minutes or so. (I started the ping tests after noticing some network connectivity issues). I attributed this packet loss to the VR traffic causing some bottleneck somewhere, but that is a post for another time.

Clearly 32GB is sufficient for many workloads. Even 8GB would be sufficient for a small two node ESXi cluster with vCenter on a minimalist Windows VM. So – what is the necessary LAB RAM size? Well, to answer that you need to look at the workload you intend to run.

Not only that, you need to factor in how long this lab needs to last and what your expansion plan would be. Do you have space for more than one machine?

So to wrap up with some take-away bullet points to consider when thinking about home/small vSphere labs:

  • 32GB “hosts” (be they ESXi to run nested ESXi and other VMs or Workstation to run nested ESXi and other VMs) are still perfectly viable for the VCP-DCV/VCAP-DCA/VCAP-DCD exams
  • 32GB “hosts” may struggle with cloud lab setups. More VMs doing more things
  • 32GB is a limit imposed by the choice of CPU – i7/E3 and cannot be expanded beyond. Worth bearing in mind if one only has space for a single machine that needs to last for a few years
  • Less RAM and “smaller” CPUs will tend to use less power, create less heat and produce less noise than bigger machines and will be more suited for home use
  • Fewer larger hosts will likely be more “home” friendly
  • More smaller hosts will likely give more lab opportunities – real FT, real DRS, real DPM
  • Scale up vs scale out – factor all options. For instance, my rough costing spreadsheet, as mentioned above, showed a single E5+64GB RAM server was cheaper than two E3 with 32GB servers
  • the i7 and E3 servers tend to be single socket while the E5 can be dual socket capable

Next time I come to replace my lab I will probably lean towards a single E5 with 64GB RAM (if RAM prices have dropped by then) on a SuperMicro motherboard.  Or a E3 with 32GB and a much smaller Intel NUC or Shuttle based box for “production” workloads.

So – yes, 32GB is currently sufficient for many home lab uses… but not all 🙂

 

VCAP5-DCA #vBrownBag video woes – VLC and MediaPlayer have issues but MPC-HC works

Posted on by | Leave a reply

While studying for the VMware VCAP5-DCA exam I’ve been watching a number of the ProfessionalVMware.com #vBrownBag session videos relating to the VCAP5-DCA (and also VCAP5-DCD videos previously) exams.

These are great resources and worth spending some time watching if you are planning on sitting the exams. Notice that, due to time constraints, some of the sessions ended without covering all the scheduled exam objective points. Be aware of this and ensure you cover all the topics included within the blueprints.

That said, I’ve been having some issues watching the source .mp4 files. My normal player of choice is the VLC media player. While watching some of the videos, I noticed that there were often green blocks and patches on the screen. I also noticed some sync issues between the audio and video. I tried changing some VLC settings but this did not help. I switched to the Window’s media player which helped with the green screen issues but did not fix the audio sync problem. I figured it must just be something with the recording itself.

Anyway while watching Josh Atwell’s PowerCLI episode I become very frustrated with the delays and thought I would do something about it. I re-encoded the .mp4 as a .mkv file. This resulting file played perfectly in VLC: no audio-video sync problems, slide transitions worked as expected and the video was perfectly watchable. So- it seems that the source file is OK (possibly slightly corrupted or using some “different” encode settings resulting in odd player behaviour?).

I did some Google-foo and came across UMPlayer. Using this player, the source file plays fine. So, has my default choice of VLC met its match? Only time will tell if UMPlayer can usurp VLC’s role on my PC. It turns out that UMPlayer is not up to the job! See the EDIT: below.

Now, these playback issues might be related to some video driver or other issue on my PC. In which case you can safely ignore this post – unless of course you happen to have the same issue. Anyway, UMPlayer MPC-HC is another tool to consider if having video playback issues 🙂

EDIT: Since using UMPlayer a little bit more, it started to have jittery audio. Sigh! Anyway, I switched to the tried and tested favourite (which I used previously in my tinkering with a media centre PC) of MPC-HC. This little player seems to be working flawlessly so far. I don’t know why I didn’t just use it right away yesterday!

My VCAP5-DCD experience

Posted on by | Leave a reply

vcap-dcd

Well, I managed to pass the “VMware Certified Advanced Professional – Data Center Design” (VCAP5-DCD) exam yesterday! Hurray.

First – a shout out to the various blogs which helped with the studying. Unfortunately, I don’t have a central list thereof to post here, but if I get time to collate the links I will update this post. A very useful summary of the DCD content  is contained here at http://professionalvmware.com/wp-content/uploads/2012/11/VCAP5DCD_StudyOutline.pdf.

In summary, this exam is all about general design processes with an obvious slant towards the VMware virtualisation platform. So you need to know the VMware “base” vSphere offerings along with detail of “general design principles”. This exam is probably not going to be  easy for a day-to-day vSphere admin as this is not about testing technical knowledge of the product set. Having been in a variety of architecture roles for the last number of years I can attest to this exam being a fairly good  representation of the real and thought processes necessary to go from capturing requirements through to implementation and subsequent support. If only we could follow these processes for all projects 🙂

So what to cover in preparation? Well, follow the blueprint! It may seem obvious, but for this exam you need to read all (well at least most of!!) the referenced documentation. I don’t think you need much (if any) hands-on lab time to prepare for this exam. Knowing various available options in the products can be learnt from the documentation. Saying that though, I did do some hands on exercises to reinforce the learning. Various books are incredibly useful too, including “VCAP5-DCD Official Cert Guide” by Paul McSharry, “VMware vSphere Design 2nd Ed” by Forbes Guthrie and Scott Lowe and “VMware vSphere 5 /5.1 Clustering Deep Dive” by Duncan Epping and Frank Denneman. “Mastering VMware vSphere 5” or “Mastering vSphere 5.5” (v5.5, less so for the exam I suppose) by Scott Lowe et al are great books and definitely worth reading, although can  be skipped for the DCD in my opinion if you don’t have the time.

I would point out that the exam is broadly focussed on vSphere 5.0 as opposed to 5.1/5.5. Don’t rule out any technologies  “removed” or “deprecated” by 5.1 and 5.5!

The exam itself. Well 3h45 is a long time for an exam. It flew by for me and I managed to finish with 15 minutes to spare. Somehow I made up time after the half way point which was a pleasant surprise. The 100 questions, of which I had 6 of the “Visio-like” design drawing, all covered the content from the blueprint. I don’t think there was anything which rang alarm bells as “whoa, where did that come from” – just a couple of questions where I though “drat, didn’t cover that in enough detail”. Remember, you cannot go back in this exam – so if you get a subsequent question which shows you answered something incorrectly earlier try not to let it get to you – move forward and stay focussed.

The design drawing questions are fairly straight forward if you can understand what they are trying to test.  That was the first problem I had – I struggled with a couple of them to understand what they were actually trying to get me to draw as I found some of the wording to be a little ambiguous. The rest were fairly straight forward.  Put down a few building blocks and link them together. Ah, and there is the second problem, when you are putting things into other things (for example, say a VM into an ESXi host) sometimes they would not stick and as such I was not sure if the tool “registered” the placement. Anyway – I tried not to get bogged down by this and quickly moved forward regardless. Do practise with the simulation tool on the VMware web site.

The remaining requestions are split between choose 1, choose 2 or choose 3 multiple choice and match the left column to the right column type questions. The multiple choice questions are generally the easier ones, although you need to pay attention to the exact wording and match it to phrases used in the prep material when describing the various terms. Keep an eye on the definitions in the “VCAP5-DCD Official Cert Guide” and “VMware vSphere Design 2nd Ed” books. The match the left to the right would be easy if they were all 1:1 mappings, which they are unfortunately not.  Some are 1:1 some are n:1 and others are 1:n. Tricky stuff! I consider myself pretty good at the requirements, risk, assumption and constraint stuff but some of the terms/phrases they used could be a little ambiguous – let’s hope they accept various answers. In these situations, I tried not to over think the wording and just read it at face value 🙂

So, all in all I think this is a pretty decent exam which does a good job of evaluating a candidate’s understanding of the topics at hand. I don’t think this is one of those exams where one can simply memorise a number of facts and pass.