Enabling LDAPS (i.e LDAPS via SSL/TLS on port 636 or LDAPS via starttls on port 389) on active directory controllers requires a valid certificate to be added to each domain controller. Overall this process is reasonably documented, for example at How to enable LDAP over SSL with a third-party certification authority and Event ID 1220 — LDAP over SSL and LDAP over SSL (LDAPS) Certificate.

So the basic steps are generate a keypair, create a CSR, submit the CSR, obtain a cert and import a certificate into the AD DS personal store. Pretty usual process for certificates.

The missing bit of information is what to do when using OpenSSL to generate the key-pair and CSR. The certificate provided by the CA is likely to be in text .crt/.cer format (i.e. contains —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–). To import into the AD DS personal store we need to use a .pfx which includes the private key, the certificate and CA cert.

To create a .pfx we can do something like (all on one line…):

openssl pkcs12 -export -out servername.pfx -inkey servername- key.pem -in servername-crt.pem -certfile /path/to/rootCAcertificate.pem

We then transfer the .pfx file to the domain controller in question and follow the above documents’ directions.

Foswiki’s LDAPContrib module does not support configuring multiple LDAP servers for redundancy or resiliency purposes. I’m not talking about consuming multiple difference LDAP stores but rather having the ability to configure a primary and backup(s) LDAP server.

I created a patch to enable this in a rudimentary fashion. It is available on GitHub https://github.com/mike2038/LdapContrib/commit/9260ce093b73f09bd6bfefc1d8b455ee1c4e5c02?diff=unified

This relies on the Net::LDAP module’s behaviour when an array is passed to the host entry for new LDAP connections. This is a preference order based array of ldap:// or ldaps:// URIs, host:port pairs or hostnames.

A couple of points to note about this patch

  • No caching of the most recently used LDAP server. For example, in a list of three LDAP servers where the first two are unavailable connection attempts will still be made to them before the third is tried. This will likely cause delays during login or other activities which require working LDAP connections. In my opinion, these delays are preferable to users being unable to login at all.
  • If you need different ports or connection schemes (i.e. ldap vs ldaps) or different Root CA certs then results might be mixed. Assuming you have certificates issued by the same Root CA then Net::LDAP seems to work with different hosts and ports. I suggest testing be carried out.

 

I ran into a problem performing memberOf LDAP queries against an Active Directory controller. Turns out that  while almost everyone is able to read most attributes from user objects, by default memberOf is not visible. The result is that queries using memberOf do not return a result. I was frustrated that the queries worked when tested on a domain controller using the LDP.exe utility, when tested as a domain admin. I was focused on getting the LDAP query correct and had not stopped to consider that the LDAP bind account I was using could not read the memberOf attribute.

Anyway, turns out that “Read Member Of” is a property you can assign via the Active Directory Users and Computers MMC. Right click on the top OU from where you want the permission to be granted (this might be the root of the AD tree or a sub-OU) and select “Properties”. Select the “Security” tab and then click “Advanced”. You can now add a permission for the LDAP bind user (or group as needed) using the “Add” button. Select a principal (i.e. user / group as needed), choose “Allow” for the “Type” and then you’ll probably want “Applies to” to be “This object and all descendant objects”. Scroll the bottom of the screen and choose “Clear all” and then find and select the property “Read Member Of”. Save and close the windows and your LDAP bind account should be able to filter based on group membership.

The below is a simple Perl snippet (based on code from http://search.cpan.org/dist/perl-ldap-0.55/lib/Net/LDAP/Examples.pod) to test the LDAP queries.

use Net::LDAP;

$ldap = Net::LDAP->new ( "dc01.example.com" ) or die "$@";
#if you has LDAPS enabled you can use the following instead
#$ldap = Net::LDAP->new ( "ldaps://dc01.example.com" ) or die "$@";
$mesg = $ldap->bind ( "CN=ldapbind,OU=Services,DC=ad,DC=example,DC=com",
 password => "LongSecretPassword",
 version => 3 );

#@Attrs = (); # to request all attributes
 my @Attrs = ( 'cn','mail','givenName','sn','sAMAccountName');
 my $base = "DC=ad,DC=example,DC=com";
 #find members of the group
 #my $filter = "(memberOf=CN=Some-group,OU=Groups,DC=ad,DC=example,DC=com)";
 #find all objects with a sn attribute
 #my $filter = "sn=*";
 #find active users within the specified group
 my $filter = "(&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(memberOf=CN=Other-group,OU=Groups,DC=ad,DC=example,DC=com))";
 #my $result = LDAPsearch ( $ldap, $query, \@Attrs );
 my $result = $ldap->search ( base => "$base",
 scope => "sub",
 filter => "$filter",
 attrs => \@Attrs
 );

my @entries = $result->entries;
 my $entr;

foreach $entr ( @entries ) {
 print "DN: ", $entr->dn, "\n";

my $attr;
 foreach $attr ( sort $entr->attributes ) {
 # skip binary we can't handle
 next if ( $attr =~ /;binary$/ );
 print " $attr : ", $entr->get_value ( $attr ) ,"\n";
 }
 print "#-------------------------------\n";
 }

 

To factory reset a Polar S725x (possibly S725 or S625 and others?) follow the following steps:

  1. Using a paper clip or some other small, pointed tool press and release the RESET button which is located between the UP and DOWN buttons.
  2. All the LCD segments on the display should light up.
  3. Press and release the bottom left (STOP) button followed by the top right (UP) button. You should now see “Monitor S725” on the screen.
  4. Using the UP and DOWN buttons, scroll until you see “Default SET” displayed.
  5. Press and hold the big bottom middle (OK) button for a few seconds. This should cause “Default SET” to flash, keep holding the button until the display resets to that shown in step 2. Release the button.
  6. Press and release the big bottom middle (START) button

Factory defaults should be applied and any user settings should be cleared.

It’s a shame this procedure is not documented in the Polar S725x manual.

Thanks to “Wayne@PolarUSA”‘s post which gives just enough info to figure out the specifics needed to perform the reset.

Monitoring alerted me to a couple of servers which had lost the ability to replicate SYSVOL using FRS. Microsft KB290762 (https://support.microsoft.com/en-us/help/290762/using-the-burflags-registry-key-to-reinitialize-file-replication-service) provided instructions on how to recover.

In summary, on all members of the replication set (in this instance all DCs) stop the ntfrs service using:

net stop ntfrs

The choose one server which will be the authoritative copy and set BurFlags to 0xd4:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup" /v BurFlags /t REG_DWORD /d 0xd4 /f

On the other servers, set BurFlags to 0xd2:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup" /v BurFlags /t REG_DWORD /d 0xd2 /f

On the authoritative server, start the ntfrs service and watch for event 13516 in the “File Replication Service” event log. Once that event is logged, start the ntfrs service on the next server and again wait for the 13516 event log. Repeat this on the remaining servers.

EDIT: I have since discovered FRS was being used due to upgrades from Windows 2003. Followed https://blogs.technet.microsoft.com/filecab/2008/02/08/sysvol-migration-series-part-1-introduction-to-the-sysvol-migration-process/ to migrate from FRS to DFSR. In future, if replication breaks it can be reinitialised by following this doc: https://support.microsoft.com/en-ie/help/2218556/how-to-force-an-authoritative-and-non-authoritative-synchronization-for-dfsr-replicated-sysvol-like-d4-d2-for-frs

 

Short bullet points as I may get round to expanding this:

ASRock C2750D4I with 1.35V DRAM modules.

VMware client showed health status as an alert due to low RAM voltage (expected as they are 1.35V but it seems the machine’s BIOS sets the threshold for 1.5V RAM modules)

ESXi 6.0 health indicated VCCM1 (voltage controller for memory 1??) was in a warning state at 1.35V.

 

I found a statically linked copy of ipmitool at https://fattylewis.com/ipmitool-on-esxi-6/ which I uploaded to the ESXi host in question. This worked first time!

./ipmitool chassis status
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : previous
Last Power Event : ac-failed
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false

I checked the sensors:

./ipmitool sensor list
....<snipped>...
ATX+5VSB | 5.040 | Volts | ok | 4.050 | 4.260 | 4.500 | 5.490 | 5.760 | 6.030
+3VSB | 3.440 | Volts | ok | 2.660 | 2.800 | 2.960 | 3.620 | 3.800 | 3.980
Vcore1 | 1.050 | Volts | ok | 0.540 | 0.570 | 0.600 | 1.490 | 1.560 | 1.640
Vcore2 | na | Volts | na | 0.540 | 0.570 | 0.600 | 1.490 | 1.560 | 1.640
VCCM1 | 1.350 | Volts | nc | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810
VCCM2 | na | Volts | na | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810
....<snipped>...

There we have it, the warning is set for 1.38V. This can be changed with IPMI tool:

./ipmitool sensor thresh VCCM1 lnc 1.32

Now we check the result:

./ipmitool sensor list VCCM1|grep VCCM
VCCM1 | 1.340 | Volts | ok | 1.210 | 1.280 | 1.320 | 1.650 | 1.730 | 1.810
VCCM2 | na | Volts | na | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810

And the VMware health status got fixed too!

 

 

For now, I will need to run this command after every reboot or reset of the IPMI management controller. I guess I should automate this for system boot-up….

FOLLOW-UP: I have just upgraded the BMC firmware from v0.27.00 to v0.30.00 and the issue seems to be resolved as the VCCM threasholds have been redefined:

./ipmitool sensor|grep VCCM
VCCM | 1.350 | Volts | ok | 1.090 | 1.120 | na | na | 1.720 | 1.750

Oh well, this was an interesting learning experience. Maybe this ipmitool info will help someone else. Note that this is a useful tool as it allows one to configure the management controller from the host while ESXi is running. Useful perhaps to configure the LAN, change the admin user or reset the controller entirely. For example, we can configure the IP address as follows:

./ipmitool lan set 1 ipsrc static
./ipmitool lan set 1 ipaddr 172.16.1.5
./ipmitool lan set 1 netmask 255.255.255.0
./ipmitool lan set 1 defgw ipaddr 172.16.1.1
./ipmitool lan set 1 arp respond on
./ipmitool lan set 1 arp generate on
./ipmitool lan set 1 arp interval 60

Extra info about using ipmitool and controlling the fans on this motherboard at https://blog.chaospixel.com/linux/2016/09/Fan-control-on-Asrock-C2750D4I-C2550D4I-board.html

The Samba module vfs_shadow_copy2 is useful for shares hosted on snapshot capable filesystems/storage. This module allows previous “snapshot” versions of a share to be made visible to users.  This allows for self-service restore of files by end users. On a given share, in smb.conf, you configure something like

[data]
 vfs objects = acl_xattr btrfs shadow_copy2
 path = /btrfs/samba/data
 shadow:basedir = /btrfs/samba/data
 shadow:snapdir = ../data_SNAPS

Depending on your version of Samba, to use the above which includes the “../” link, you might need to add some/all of the below options:

unix extensions = no
wide links = yes
allow insecure wide links = yes

In the 4.3.x and 4.4.x releases there have been a few changes to the vfs_shadow_copy2 module meaning the above three options may or may not have been needed. The jump from 4.4.9 to 4.4.10 addressed Samba BUG 12531. This again involved changes to the vfs_shadow_copy2 module, again breaking the above config (with or without the three options listed). I’ve changed the above config on the share to be

[data]
 vfs objects = acl_xattr btrfs shadow_copy2
 path = /btrfs/samba/data
 shadow:basedir = /btrfs/samba/data
 shadow:snapdir = /btrfs/samba/data_SNAPS

which works without the three options above. Samba 4.4.13, 4.5.8, 4.6.2 also work with the above. Its frustrating since relative (which I assume includes “../”) directories are supposed to be supported with “shadow:snapdir”. Hopefully this configuration now works for all releases going forward.

I find it frustrating that minor releases can break working configs. It makes it difficult to quickly deploy security fixes as the required level of testing is much higher than one would expect for minor releases. This subtle change to the config option is not documented or mentioned in the release notes.  Anyway, more of a rant post than usual… apologies for that!

Upgrading a VM’s VMware hardware version to the latest version is generally considered best practice. This is easy to do via the C# client or the web-client. If you wish to upgrade to a newer, but not latest, hardware version this can be tricky.

As highlighted in https://www.v-front.de/2014/02/how-to-uprade-your-vms-virtual-hardware.html, one easy way to do so is as follows on the ESXi host in question:

vim-cmd vmsvc/upgrade vmid vmx-10

where vmid is determined using something like:

vim-cmd vmsvc/getallvms|awk '{print $1 "  "$2" "$6}'

The appropriate hardware version can be selected from a useful VMware Virtual Machine Hardware Versions KB. The recent relevant versions are:

vmx-13: ESXi 6.5

vmx-12: Workstation Pro 12.x

vmx-11: ESXi 6.0 / Workstation 11.x

vmx-10: ESXi 5.5 / Workstation 10.x

vmx-9: ESXi 5.1 / Workstation 9.x

vmx-8: ESXi 5.0 / Workstation 8.x

vmx-7: ESXi/ESX 4.x

 

 

 

Introduction

While working in a lab environment recently I wanted to vMotion a VM between two ESXi hosts. The vMotion failed, which was not entirely unexpected, due to CPU incompatibilities. These particular ESXi hosts are not in a vSphere cluster so enabling EVC (Enhanced vMotion Compatibility), which would resolve the issue, is not an option.

Attempting to vMotion a VM from host A to host B gave errors about:

  • MOVBE
  • 3D!Now PREFETCH and PREFETCHW

After powering the VM down, migrating the VM to host B and powering on, an attempt to vMotion from host B to host A gave errors about:

  • PCID
  • XSAVE
  • Advanced Vector Extensions (AVX)
  • Half-precision conversion instructions (F16C)
  • Instructions to read and write FS and GS base registers
  • XSAVE SSE State
  • XSAVE YMM State

Manual VM CPUID mask configuration

As the error messages indicate, Enhanced vMotion Compatibility (EVC) would enable the vMotion to take place between mixed CPUs within a cluster. Mixing host CPU models within a vSphere cluster is generally considered bad practice and should be avoided where possible. In this instance, the hosts are not in a cluster so EVC is not even an option.

As indicated above shutting down the VM and doing a cold migration is possible. This issue only relates to the case where I want to be able to migrate running VMs between hosts containing processors with different feature sets.

For the two hosts in question, I know (based on the EVC processor support KB and Intel ARK and VMware KB pages) that the Intel “Westmere” Generation baseline ought to be the highest compatible EVC mode; one of the processors is an Intel Avoton C2750 and the other is an Intel i7-3770S Sandy Bridge. The Avoton falls into the Westmere category for EVC. We will come back to EVC later on.

I suspected it would be possible to create a custom CPU mask to enable the vMotion between these to hosts. In general, features supported by a given processor are exposed via a CPUID instruction call. By default, VMware ESXi manipulates the results of the CPUID instructions executed by a VM as part of the virtualisation process. EVC is used to further manipulate these feature bits to hide CPU features from VMs to ensure that “well behaved VMs” are able to run when migrated between hosts containing processors with different features.

In this instance, “well behaved VMs” refers to VMs running code which use the CPUID instruction to determine available features. If the guest OS or application uses the CPUID instruction to determine available processor features then, when moved via vMotion to a different host, that same set of features will be available. If a guest uses some other mechanism to determine processor feature availability (e.g. based on the processor model name) or merely assumes a given feature will be available then the VM or application may crash or have other unexpected errors.

So back to this experiment. Attempting to go from host A to host B indicated only two feature incompatibilities. I turned to the Intel developers manual (64-ia-32-architectures-software-developer-vol-2a-manual.pdf) for the detail about the CPUID instruction. CPUID called with EAX=0x80000001 results in the PREFETCHW capability being exposed at bit 8 in ECX. Similarly with EAX=0x1, the MOVBE capability is exposed at bit 22 in ECX.

As an initial test, I did a cold migration of the VM to host A and edited the VM settings as shown below.

In summary, this is passing the CPUID result of the host via the above mask filter. The mask filter can hide, set or pass through a CPUID feature bit. In this instance I am hiding the two bits identified above through the use of the “0” in those bit positions. There are other options you can use as displayed in the legend.

I chose “0” rather than “R” as I need to hide the feature from the guest OS and I do not care if the destination host actually has that feature or not.

I saved this configuration and powered on the VM. I was able to successfully perform a vMotion from host A to host B. I was also able to vMotion the VM back to host A.

I performed a vMotion back to host B and powered the VM off. I then powered the VM back on on host B. I tried to vMotion to back to host A, which again failed with the same error as shown above. The reason it failed in the reverse direction is that the VM pickups up it’s masked capabilities at power on and maintains that set of capabilities until it is powered off once more. So by powering on the VM on host B, it got a different set of capabilities to when it was powered on on host A. This explains why when attempting to originally perform the vMotion we had two different sets of errors.

To get the masks to enable a vMotion from host B to host A, I took a look at the developers guide and performed some Googlefoo, I identified the CPUID bits needed to mask the unsupported features:

PCID: CPUID EAX=1, result in ECX bit 17
XSAVE: CPUID EAX=1, result in ECX bit 26
AVX: CPUID EAX=1, result in ECX bit 28
F16C: CPUID EAX=1, result in ECX bit 29

FSGSBASE: CPUID EAX=7, result in EBX bit 00

XSAVE SSE: CPUID EAX=0xd, result in EAX bit 01
XSAVE YMM: CPUID EAX=0xd, result in EAX bit 02 (YMM are 256-bit AVX registers)

The first four are easy as the vSphere client allows one to edit the EAX=1 CPUID results. With the below configuration in place, the vMotion from host B to host A only showed the last three errors (FSGSBASE, XSAVE SSE and XSAVE YMM). This is expected as no masking had been put in place.

To put the masking in place for EAX=0x7 and EAX=0xd we need to edit the virtual machine’s .VMX file. We can do this by editing the .vmx file directly or by using the Configuration Parameters dialogue for the VM under Options/Advanced/General in the VM’s settings dialogue. The following two parameters (first one for FSGSBASE and second for the XSAVE) were added:

cpuid.7.ebx = -------------------------------0
cpuid.d.eax = -----------------------------00-

Powering on the VM succeeded, however the vMotion to host A failed with the same error about FS & GS Base registers (but the XSAVE errors were gone). Surprisingly when I checked the .vmx directly and the cpuid.7.ebx line was missing. For some reason it appears that the VI client does not save this entry. So I removed the VM from the inventory, added that line to the .VMX directly and then re-registered the VM.

I was now able to power on the VM on host B and vMotion back and forth. I was not able to do the same when the VM was powered on on host A. I needed to merge the two sets of capabilities.

At this stage we would have the following in the .vmx file:

for host A -> host B:
cpuid.80000001.ecx = "-----------------------0--------"
cpuid.1.ecx = "---------0----------------------"

for host B -> host A:
cpuid.1.ecx = "--00-0--------0-----------------"
cpuid.7.ebx = "-------------------------------0"
cpuid.d.eax = "-----------------------------00-"

(Note that there are some default entries which get added which are all dashes, and one for cpuid.80000001.edx with dashes and a single H).

We merge our two sets of lines to obtain:

cpuid.80000001.ecx = "-----------------------0--------"
cpuid.1.ecx = "--00-0---0----0-----------------"
cpuid.7.ebx = "-------------------------------0"
cpuid.d.eax = "-----------------------------00-"

At this stage we can now power on the VM on either host and migrate in either direction. Success. Using these four lines of config, we have masked the specific features which vSphere was highlighting as preventing vMotion. It has also shown how we can hide or expose specific CPUID features on a VM by VM basis.

Manual EVC Baseline Configuration

Back to EVC. The default EVC masks can be determined by creating a cluster (even without any hosts) and enabling EVC. You can then see the default masks put in place on the host by EVC. Yes, EVC puts a default mask in place on the hosts in an EVC enabled cluster. The masked off CPU features are then not exposed to the guests at power-on and are not available during vMotion compatibility checks.

The default baselines for Westmere and Sandybridge EVC modes are shown below:

 

The differences are highlighted. Leaf1 (i.e. CPUID with EAX=1) EAX result relates to processor family and stepping information. The three Leaf1 ECX flags relate to AES, XSAVE and TSC-Deadline respectively. The three Leafd EAX flags are for x87, SSE and AVX XSAVE state. The three Leafd ECX flags are related to maximum size needed for the XSAVE area.

Anyway I’ve digressed. So the masks which I created above obviously only dealt with the specific differences between my two processors in question. In order to determine a generic “Westmere” compatible mask on a per VM basis we will start with VMware’s ESXi EVC masks above. The EVC masks are showing which feature bits are hidden (zeros) and which features may be passed through to guests (ones). So we can see which feature bits are hidden in a particular EVC mode. So to convert the above EVC baselines to VM CPUID masks I keep the zeros and change the ones to dashes. I selected dashes instead of ones to ensure that the default guest OS masks and host flags still take effect. We get the following for a VM for Westmere feature flags:

cpuid.80000001.ecx = "0000000000000000000000000000000-"
cpuid.80000001.edx = "00-0-000000-00000000-00000000000"
cpuid.1.ecx = "000000-0-00--000---000-000------"
cpuid.1.edx = "-000-------0-0-------0----------"
cpuid.d.eax = "00000000000000000000000000000000"
cpuid.d.ecx = "00000000000000000000000000000000"
cpuid.d.edx = "00000000000000000000000000000000"

I did not map the cpuid.1.eax flags as I did not want to mess with CPU family/stepping flags. Also, the EVC masks listed did not show the cpuid.7.ebc line I needed for the FSGSBASE feature. Sure enough, using only the 7 lines above meant I could not vMotion from host B to host A. So, adding

cpuid.7.ebx = "-------------------------------0"

to the VMX then allowed the full vMotion I was looking for. The ESXi hypervisor must alter other flags apart from only those shown on the EVC configuration page.

TL;DR

To configure a poor man’s EVC on a VM by VM basis for a Westmere feature set, add the following lines to a VM’s .VMX file.

cpuid.80000001.ecx = "0000000000000000000000000000000-"
cpuid.80000001.edx = "00-0-000000-00000000-00000000000"
cpuid.1.ecx = "000000-0-00--000---000-000------"
cpuid.1.edx = "-000-------0-0-------0----------"
cpuid.7.ebx = "-------------------------------0"
cpuid.d.eax = "00000000000000000000000000000000"
cpuid.d.ecx = "00000000000000000000000000000000"
cpuid.d.edx = "00000000000000000000000000000000"

 

Appendix

1

Useful thread -> https://communities.vmware.com/thread/467303

The above thread covers manipulating the guest CPUID. An interesting option is mentioned in post 9 relating to an option to enable further CPUID manipulation than is possible by default. In my tinkering above, I did not need this option.

monitor_control.enable_fullcpuid = TRUE

Note too that the vmware.log of any VM can be used to see the CPUID information of the host, as also mentioned in post 9:

As for extracting the results, you can write a program to query the CPUID function(s) of interest, or you can just look in the vmware.log file of any VM.  All current VMware hypervisors log all CPUID information for both the host and the guest

Post 13, again a user jmattson (exVMware now at Google), reveals a simple way to configure the processor name visible to guests:

cpuid.brandstring = "whatever you want"

2

This thread https://communities.vmware.com/thread/503236, again involving jmattson, discusses cpuid masks – and gives an insight into how EVC masks interact with the VM cpuid masks.

3

This post https://v-reality.info/2014/08/vsphere-vm-version-impact-available-cpu-instructions/ reveals that the virtual hardware version of a given VM also plays a role in the CPUID mask a VM is given. I found this interesting as it does give us another reason to actively upgrade the hardware versions of VMs.

4

A little gem is mentioned at the bottom of https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1029785 – it seems that IBM changed the BIOS default for some of their servers. The option was changed to enable the AES feature by default. This resulted in identical servers configured with BIOS defaults, which were added to a vSphere cluster, having two different sets of CPUID feature bits set (AES enabled on some and disabled on others) resulting in vMotions not being possible between all servers.

I was doing a clear out of some paper work and came across these write performance tests which I did a few years back on an HP DL380 G5 with a p400 RAID controller.

  • 4 drive RAID10 = 144MB/s (~72MB/s per drive)
  • 4 drive RAID0 = 293MB/s (~73MB/s per drive)
  • 8 drive RAID0 = 518MB/s (~64MB/s per drive)
  • 8 drive RAID5 = 266MB/s (~38MB/s per drive)
  • 8 drive RAID6 = 165MB/s (~28MB/s per drive)
  • 8 drive RAID10 = 289MB/s (~72MB/s per drive)

The “per drive” is data being written per active data drive, excluding any RAID overheads.

I inferred from the above that the p400 RAID controller maxes out at around 518MB/s as it is unable to saturate 8 drives in a RAID0 array (64MB/s vs 72MB/s per drive). Not sure if this is a controller throughput or PCIe bus limitation.

Eitherway, this testing was (I think) done on Ubuntu 10.x or 12.x with pretty bog standard settings using a simple command such as:

dd if=/dev/zero of=/dev/cciss/c0d0 bs=1024k

I thought I’d capture these figures here since I have no where else to save them.