Sometimes the download service seems to hang with an error such as “Reading the VMware software depot for the first time. This could take up to several minutes…”. One possible cause is an ongoing corruption issue with some of the underlying depot data files. This is documented in https://communities.vmware.com/thread/568541 and https://communities.vmware.com/thread/553547

In summary, the current issue relates to “corruption” in the dlg_ESXI60U3A.xhtml data file. Reference is made to a missing component (which seem ultimately to be files in the same directory).

The stdout-vapi_server.log file (stored in C:\Users\username\AppData\Roaming\VMware\Software Manager\Download Service) includes reference to the failing items, which can be removed one at a time. Once this is done (with the download service stopped) the service should resume working or report the next failing item. If the above error occurs, keep checking the stdout-vapi_server.log file for references to error items.

I recently had the need to update the firmware on an HP  LSI22320-HP U320 SCSI card. The HP firmware was resulting in Domain Validation (a SCSI technique to determine appropriate/supported SCSI speed for each SCSI device) errors. I wanted a more up to date version. I turned to the Broadcom LSI22320SE driver site and found an updated firmware download, in a file named Fusion-MPT_IT_FW10334_BIOS_50703pt_FLASH_10304.zip.

I created a USB boot disk with FreeDOS using Rufus (a great tool I only just found). I copied the relevant files (dos4gw.exe, flsh1030.exe, mptps.rom and it_1030.fw) to the USB stick. I booted up and tried to update. I managed to apply the firmware (it_1030.fw) but not the BIOS (mptps.rom). The error displayed was:

 Error: Attempting to download a generic BIOS image to
 a adapter meant for a customer specific BIOS image!

As this is an HP card I was a little disappointed but not surprised. Looking at the help output from “flsh1030.exe -?” I saw no obvious way to override this. Many Google searches later I came across a post on a Russian forum in Google’s cache that gave the hint I needed – the undocumented “-g” flag. Rerunning the following command overwrote the firmware on the card.

flsh1030 -a -g -b mptps.rom

The new flash gave a new set of menus. More importantly the domain validation errors preventing the drives from working were gone.

Brief post, cribbed from http://forum.gns3.net/topic7559.html and https://ask.wireshark.org/questions/5996/how-to-configure-realtek-pcie-gbe-family-controller-to-capture-vlan-tag-packet

  • Update drivers
  • Find reg sub-key under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}
  • Add/update DWORDs:
    • MonitorModeEnabled = 1
    • MonitorMode = 1
    • *PriorityVLANTag = 0
    • SkDisableVlanStrip = 1

Basically, I had a PC running Windows 7 which was on a trunked switch port. I was expecting the native VLAN to work and remaining tagged traffic to not be visible. Turned out that the Realtek driver was stripping off the 802.1q VLAN tags and passing all traffic to the machine. The above settings resolved this.

Enabling LDAPS (i.e LDAPS via SSL/TLS on port 636 or LDAPS via starttls on port 389) on active directory controllers requires a valid certificate to be added to each domain controller. Overall this process is reasonably documented, for example at How to enable LDAP over SSL with a third-party certification authority and Event ID 1220 — LDAP over SSL and LDAP over SSL (LDAPS) Certificate.

So the basic steps are generate a keypair, create a CSR, submit the CSR, obtain a cert and import a certificate into the AD DS personal store. Pretty usual process for certificates.

The missing bit of information is what to do when using OpenSSL to generate the key-pair and CSR. The certificate provided by the CA is likely to be in text .crt/.cer format (i.e. contains —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–). To import into the AD DS personal store we need to use a .pfx which includes the private key, the certificate and CA cert.

To create a .pfx we can do something like (all on one line…):

openssl pkcs12 -export -out servername.pfx -inkey servername- key.pem -in servername-crt.pem -certfile /path/to/rootCAcertificate.pem

We then transfer the .pfx file to the domain controller in question and follow the above documents’ directions.

Foswiki’s LDAPContrib module does not support configuring multiple LDAP servers for redundancy or resiliency purposes. I’m not talking about consuming multiple difference LDAP stores but rather having the ability to configure a primary and backup(s) LDAP server.

I created a patch to enable this in a rudimentary fashion. It is available on GitHub https://github.com/mike2038/LdapContrib/commit/9260ce093b73f09bd6bfefc1d8b455ee1c4e5c02?diff=unified

This relies on the Net::LDAP module’s behaviour when an array is passed to the host entry for new LDAP connections. This is a preference order based array of ldap:// or ldaps:// URIs, host:port pairs or hostnames.

A couple of points to note about this patch

  • No caching of the most recently used LDAP server. For example, in a list of three LDAP servers where the first two are unavailable connection attempts will still be made to them before the third is tried. This will likely cause delays during login or other activities which require working LDAP connections. In my opinion, these delays are preferable to users being unable to login at all.
  • If you need different ports or connection schemes (i.e. ldap vs ldaps) or different Root CA certs then results might be mixed. Assuming you have certificates issued by the same Root CA then Net::LDAP seems to work with different hosts and ports. I suggest testing be carried out.

 

I ran into a problem performing memberOf LDAP queries against an Active Directory controller. Turns out that  while almost everyone is able to read most attributes from user objects, by default memberOf is not visible. The result is that queries using memberOf do not return a result. I was frustrated that the queries worked when tested on a domain controller using the LDP.exe utility, when tested as a domain admin. I was focused on getting the LDAP query correct and had not stopped to consider that the LDAP bind account I was using could not read the memberOf attribute.

Anyway, turns out that “Read Member Of” is a property you can assign via the Active Directory Users and Computers MMC. Right click on the top OU from where you want the permission to be granted (this might be the root of the AD tree or a sub-OU) and select “Properties”. Select the “Security” tab and then click “Advanced”. You can now add a permission for the LDAP bind user (or group as needed) using the “Add” button. Select a principal (i.e. user / group as needed), choose “Allow” for the “Type” and then you’ll probably want “Applies to” to be “This object and all descendant objects”. Scroll the bottom of the screen and choose “Clear all” and then find and select the property “Read Member Of”. Save and close the windows and your LDAP bind account should be able to filter based on group membership.

The below is a simple Perl snippet (based on code from http://search.cpan.org/dist/perl-ldap-0.55/lib/Net/LDAP/Examples.pod) to test the LDAP queries.

use Net::LDAP;

$ldap = Net::LDAP->new ( "dc01.example.com" ) or die "$@";
#if you has LDAPS enabled you can use the following instead
#$ldap = Net::LDAP->new ( "ldaps://dc01.example.com" ) or die "$@";
$mesg = $ldap->bind ( "CN=ldapbind,OU=Services,DC=ad,DC=example,DC=com",
 password => "LongSecretPassword",
 version => 3 );

#@Attrs = (); # to request all attributes
 my @Attrs = ( 'cn','mail','givenName','sn','sAMAccountName');
 my $base = "DC=ad,DC=example,DC=com";
 #find members of the group
 #my $filter = "(memberOf=CN=Some-group,OU=Groups,DC=ad,DC=example,DC=com)";
 #find all objects with a sn attribute
 #my $filter = "sn=*";
 #find active users within the specified group
 my $filter = "(&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(memberOf=CN=Other-group,OU=Groups,DC=ad,DC=example,DC=com))";
 #my $result = LDAPsearch ( $ldap, $query, \@Attrs );
 my $result = $ldap->search ( base => "$base",
 scope => "sub",
 filter => "$filter",
 attrs => \@Attrs
 );

my @entries = $result->entries;
 my $entr;

foreach $entr ( @entries ) {
 print "DN: ", $entr->dn, "\n";

my $attr;
 foreach $attr ( sort $entr->attributes ) {
 # skip binary we can't handle
 next if ( $attr =~ /;binary$/ );
 print " $attr : ", $entr->get_value ( $attr ) ,"\n";
 }
 print "#-------------------------------\n";
 }

A useful summary of LDAP search options supported by AD servers (incl for disabled accounts and nested-group membership) is in an article titled “Active Directory: LDAP Syntax Filters“.

Monitoring alerted me to a couple of servers which had lost the ability to replicate SYSVOL using FRS. Microsft KB290762 (https://support.microsoft.com/en-us/help/290762/using-the-burflags-registry-key-to-reinitialize-file-replication-service) provided instructions on how to recover.

In summary, on all members of the replication set (in this instance all DCs) stop the ntfrs service using:

net stop ntfrs

The choose one server which will be the authoritative copy and set BurFlags to 0xd4:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup" /v BurFlags /t REG_DWORD /d 0xd4 /f

On the other servers, set BurFlags to 0xd2:

reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup" /v BurFlags /t REG_DWORD /d 0xd2 /f

On the authoritative server, start the ntfrs service and watch for event 13516 in the “File Replication Service” event log. Once that event is logged, start the ntfrs service on the next server and again wait for the 13516 event log. Repeat this on the remaining servers.

EDIT: I have since discovered FRS was being used due to upgrades from Windows 2003. Followed https://blogs.technet.microsoft.com/filecab/2008/02/08/sysvol-migration-series-part-1-introduction-to-the-sysvol-migration-process/ to migrate from FRS to DFSR. In future, if replication breaks it can be reinitialised by following this doc: https://support.microsoft.com/en-ie/help/2218556/how-to-force-an-authoritative-and-non-authoritative-synchronization-for-dfsr-replicated-sysvol-like-d4-d2-for-frs

 

Short bullet points as I may get round to expanding this:

ASRock C2750D4I with 1.35V DRAM modules.

VMware client showed health status as an alert due to low RAM voltage (expected as they are 1.35V but it seems the machine’s BIOS sets the threshold for 1.5V RAM modules)

ESXi 6.0 health indicated VCCM1 (voltage controller for memory 1??) was in a warning state at 1.35V.

 

I found a statically linked copy of ipmitool at https://fattylewis.com/ipmitool-on-esxi-6/ which I uploaded to the ESXi host in question. This worked first time!

./ipmitool chassis status
System Power : on
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : previous
Last Power Event : ac-failed
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false

I checked the sensors:

./ipmitool sensor list
....<snipped>...
ATX+5VSB | 5.040 | Volts | ok | 4.050 | 4.260 | 4.500 | 5.490 | 5.760 | 6.030
+3VSB | 3.440 | Volts | ok | 2.660 | 2.800 | 2.960 | 3.620 | 3.800 | 3.980
Vcore1 | 1.050 | Volts | ok | 0.540 | 0.570 | 0.600 | 1.490 | 1.560 | 1.640
Vcore2 | na | Volts | na | 0.540 | 0.570 | 0.600 | 1.490 | 1.560 | 1.640
VCCM1 | 1.350 | Volts | nc | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810
VCCM2 | na | Volts | na | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810
....<snipped>...

There we have it, the warning is set for 1.38V. This can be changed with IPMI tool:

./ipmitool sensor thresh VCCM1 lnc 1.32

Now we check the result:

./ipmitool sensor list VCCM1|grep VCCM
VCCM1 | 1.340 | Volts | ok | 1.210 | 1.280 | 1.320 | 1.650 | 1.730 | 1.810
VCCM2 | na | Volts | na | 1.210 | 1.280 | 1.380 | 1.650 | 1.730 | 1.810

And the VMware health status got fixed too!

 

 

For now, I will need to run this command after every reboot or reset of the IPMI management controller. I guess I should automate this for system boot-up….

FOLLOW-UP: I have just upgraded the BMC firmware from v0.27.00 to v0.30.00 and the issue seems to be resolved as the VCCM threasholds have been redefined:

./ipmitool sensor|grep VCCM
VCCM | 1.350 | Volts | ok | 1.090 | 1.120 | na | na | 1.720 | 1.750

Oh well, this was an interesting learning experience. Maybe this ipmitool info will help someone else. Note that this is a useful tool as it allows one to configure the management controller from the host while ESXi is running. Useful perhaps to configure the LAN, change the admin user or reset the controller entirely. For example, we can configure the IP address as follows:

./ipmitool lan set 1 ipsrc static
./ipmitool lan set 1 ipaddr 172.16.1.5
./ipmitool lan set 1 netmask 255.255.255.0
./ipmitool lan set 1 defgw ipaddr 172.16.1.1
./ipmitool lan set 1 arp respond on
./ipmitool lan set 1 arp generate on
./ipmitool lan set 1 arp interval 60

Extra info about using ipmitool and controlling the fans on this motherboard at https://blog.chaospixel.com/linux/2016/09/Fan-control-on-Asrock-C2750D4I-C2550D4I-board.html

The Samba module vfs_shadow_copy2 is useful for shares hosted on snapshot capable filesystems/storage. This module allows previous “snapshot” versions of a share to be made visible to users.  This allows for self-service restore of files by end users. On a given share, in smb.conf, you configure something like

[data]
 vfs objects = acl_xattr btrfs shadow_copy2
 path = /btrfs/samba/data
 shadow:basedir = /btrfs/samba/data
 shadow:snapdir = ../data_SNAPS

Depending on your version of Samba, to use the above which includes the “../” link, you might need to add some/all of the below options:

unix extensions = no
wide links = yes
allow insecure wide links = yes

In the 4.3.x and 4.4.x releases there have been a few changes to the vfs_shadow_copy2 module meaning the above three options may or may not have been needed. The jump from 4.4.9 to 4.4.10 addressed Samba BUG 12531. This again involved changes to the vfs_shadow_copy2 module, again breaking the above config (with or without the three options listed). I’ve changed the above config on the share to be

[data]
 vfs objects = acl_xattr btrfs shadow_copy2
 path = /btrfs/samba/data
 shadow:basedir = /btrfs/samba/data
 shadow:snapdir = /btrfs/samba/data_SNAPS

which works without the three options above. Samba 4.4.13, 4.5.8, 4.6.2 also work with the above. Its frustrating since relative (which I assume includes “../”) directories are supposed to be supported with “shadow:snapdir”. Hopefully this configuration now works for all releases going forward.

I find it frustrating that minor releases can break working configs. It makes it difficult to quickly deploy security fixes as the required level of testing is much higher than one would expect for minor releases. This subtle change to the config option is not documented or mentioned in the release notes.  Anyway, more of a rant post than usual… apologies for that!

Upgrading a VM’s VMware hardware version to the latest version is generally considered best practice. This is easy to do via the C# client or the web-client. If you wish to upgrade to a newer, but not latest, hardware version this can be tricky.

As highlighted in https://www.v-front.de/2014/02/how-to-uprade-your-vms-virtual-hardware.html, one easy way to do so is as follows on the ESXi host in question:

vim-cmd vmsvc/upgrade vmid vmx-10

where vmid is determined using something like:

vim-cmd vmsvc/getallvms|awk '{print $1 "  "$2" "$6}'

The appropriate hardware version can be selected from a useful VMware Virtual Machine Hardware Versions KB. The recent relevant versions are:

vmx-13: ESXi 6.5

vmx-12: Workstation Pro 12.x

vmx-11: ESXi 6.0 / Workstation 11.x

vmx-10: ESXi 5.5 / Workstation 10.x

vmx-9: ESXi 5.1 / Workstation 9.x

vmx-8: ESXi 5.0 / Workstation 8.x

vmx-7: ESXi/ESX 4.x