Last night I was upgrading some ESX 3.5 VMs from “flexible” NICs to “VMXNET Enhanced” NICs and ran into a problem with on of the servers. As an aside, doing some rudimentary throughput testing using the iperf tool I was surprised to see the significant throughput increase and CPU usage decrease when switching from flexible (i.e. VMXNET) to VMXNET Enhanced (i.e. VMXNET2) NICs. Well worth doing – apart from the gotcha I ran into…
This one particular VM which showed a problem is running Quagga (a BGP daemon) to inject routes for a local AS112 server. Once I switched over the NICs the remote Cisco router started logging the following errors
%TCP-6-BADAUTH: No MD5 digest from
%TCP-6-BADAUTH: Invalid MD5 digest from
and would not bring up the BGP peering session. BGP MD5 authentication is enabled between the router and the Quagga daemon. The use of “debug ip tcp transactions” also shows the invalid MD5 signatures.
I initially suspected it was related to a offloaded checksumming issue which I previously observed (http://communities.vmware.com/thread/250159). Turns out it was not incorrect TCP checksums being calculated (related to the above post’s incorrect UDP checksums).
The VMXNET Enhanced NIC enables additional offloading from the VM to the NIC to enhance performance. This works well for many use cases but causes problems when using TCP MD5 checksums. In my case, turning off “TCP Segmentation Offload” has worked around the problem. Adding a command such as
ethtool -K eth0 tso off
ethtool -K eth0 sg off
to a startup script has worked around the issue to some degree. In an ideal world, the VMXNET driver should allow “tx-checksumming” to be turned off using ethtool aswell.
In fairness to VMware on this one, this issue appears to not be specific to virtual machines but may in fact be observed on physical hardware with NICs providing offload functions.
Having used the above two ethtool commands, to allow the BGP session to be established, I still continue to see the following errors on the Cisco router:
TCP0: bad seg from x.x.x.x -- no MD5 string: port 179 seq
%TCP-6-BADAUTH: No MD5 digest from
%TCP-6-BADAUTH: Invalid MD5 digest from
Interestingly, using the flexible and hence original VMXNET VNIC within the VM (but oddly the same actual VMXNET driver binary!!) tx-checksumming and scatter-gather is enabled for the NIC:
$ ethtool -k eth0
Offload parameters for eth0:
Cannot get device rx csum settings: Operation not supported
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
$
For the record, all the above is using the latest ESX 3.5 VMware tools build 317866 from VMwareTools-3.5.0-317866.tar.gz.
Doing a tcpdump (from a physical server) of the traffic between the router and the VM reveals that packets which leave the VM with a valid MD5 signature (as per tcpdump from within the VM) arrive on the wire with an invalid MD5 signature (tcpdump -s0 -M md5password port 179). This indicates that VMware ESXi may infact be altering the packets between the VM and the wire in some way which is invalidating the MD5 signature 🙁
For now, some errors with an established session is better than lots of errors and no BGP session. Ideally there would be no errors being logged by the Cisco router.
Well, I figured I should put a post on here about the New York Marathon 2010. I managed to get a lottery place and was really looking forward to the race. I have to admit I underestimated how cold it would be and that I also expected it to be flatter than it was.
Going over the Verrazano-Narrows Bridge at the start was amazing – it offers a fantastic view of the Manhattan skyline in the distance. In some ways exhilarating and other ways scary that I still had to get all the way over there. The first 15miles went well – there was fantastic support along the course with many bands playing great music and spectators wanting to slap hands with runners on the way past. I felt like I had a just sub-3h30 time in my legs and was ahead of schedule until mile 25. Around the 15/16mile stage, going over the Queensboro Bridge, I took a bit of a knock. My foot started hurting . Also, going over the bridge there are no supporters to keep one’s spirits up. Add to that my Garmin lost GPS signal so I had no idea of pace – as it happens the pace stayed OK. Going over that bridge is tough mentally.
Turning off the bridge and heading up First Ave is great. The support resumes and the music and festivities commence once more. The only down side to the First Avenue stint is that it is a straight 3.5mile stretch of road heading off into the distance before you. Entering the Bronx (around 20 mile mark) you climb over a short steep (well it certainly felt steep!) bridge which saps the legs. The twists and turns in the Bronx are tough going and stop you getting into a rhythm.
Returning to Manhattan and heading along Fifth Avenue is wonderful – the end is figuratively in sight. Unfortunately the two miles (23rd and 24th) along Fifth Avenue heading to Central Park are uphill – again leg and mind sapping.
It was around here that my foot was in serious agony and my quads had left for the day. The last four miles (two along Fifth Avenue and two in Central Park) were seriously tough and where I lost my time and 3h30 split. The last four miles took roughly 40 minutes and were pure agony. I resorted to a jog/walk routine and managed to limp home to the finish in 3h32:35 with an average pace of 8:07/mile.
I kept to my plan of a gel every 4 miles starting at mile 4. Towards the end of the race I starting taking on the sports drinks which prehaps I should have done sooner.
A great race at the end of the day with superb support and awesome iconic views.
Well yesterday was the 2010 Windsor Half Marathon. I went in aiming for around 1h40 – turned out I managed to do 1h26:47! Yes 1h26:47. Am over the moon with that result.
It was cool and a little chilly but no rain. The wind was gusty and at times annoying. The course is “undulating” but not ridiculously so – apart from the hill just after the 10mile mark.
I headed out during the first mile or so trying to get back on my target time after getting caught up in some runners at the start. In doing so I was running faster than my target pace. I decided to keep at the faster pace and managed to hold on much longer than I expected. After a while I realised that a 1h30 was possible and I fought through the aches and pains and started watching the mile splits anxiously. At 11 miles I realised I was on for the 1h30 and breathed a sigh of relief – all that was left for me to do was not fall over or slow down too much.
I had two gels (4miles and 9miles) washed down with a couple of sips of water – much less hydration than I’d normally do – seemed to work as it was a cool day. Now just to figure out the nutrition and hydration for the marathon in just over 5 weeks time….
I hope my right foot holds up over the next couple of weeks though – the dull ache still persists unfortunately…
So 1h26:47 and 58th overall based on my chip time and 63rd place based on race clock time.
Since my last serious post I have completed a few races – Ironman Swiss 70.3, London Triathlon, and the National Club Relays. All good fun. Managed to PB in Swiss 70.3 and London Triathlon so am pleased with my recent performances.
My training leading up to the Windsor Half-Marathon and the New York Marathon is progressing well. Touch wood my legs are holding out with the increased running volume. This weekend I have the HSBC Standard Distance Triathlon at Dorney Lake. My main goal is sub-2h30 – with the secondary goal of beating my work colleagues who are doing a relay team race.
My swimming is holding steady and doesn’t appear to be improving significantly. I have increased my swim session duration, but have unfortunately reduced my swim frequency. Over the winter I hope to maintain my swimming and hopefully get a bit faster.
My cycling is going well – my last few weekly 10mile time-trials have definitely improved my 30minute (CP30) power output. Over the winter I am looking at trying to not loose too much power while I rest up.
I will hopefully write up some brief race reports over the coming days.
