Well, in short the GIGABYTE G1.Sniper M3 motherboard does support Intel VT-d and both ESXi 5 and 5.1 can use it. I have tested this with BIOS version f9 and “beta” BIOS versions f10c, f10d and f10e and all show VT-d as an option when a compatible processor is installed. Note that this option is not shown unless a suitable (generally Intel i5 or i7 non-k CPU) processor is installed. The “VT-d” option is shown below the “Intel Virtualization Technology” option on the “BIOS Features” page of the BIOS setup.

I have had mixed success with actually passing through devices to VMs. Generally the cards in PCI-E slots and configured for pass through worked as expected within a VM during my testing (USB3, NICs, Hauppauge 1700 cards). However, devices on the motherboard (SATA, Audio, LAN, Video) and PCI-E graphics cards do not work. For the most part, these devices pass through but the devices don’t start under Windows, drivers fail to attach or cause blue screens when accessed (yes, Mr ATI graphics card with your atikmpag.sys driver BSOD).

Until I actually did these tests I was not sure if this motherboard did or did not support VT-d /Intel Virtualization Technology for Directed I/O / VMware VMDirectPath I/O. I already had this motherboard in a HTPC with an Intel i3 (dual core with hyper threading) which, by the way, ran ESXi adequately. I wanted to play with VT-d so  I took a punt on an Intel i7 processor and luckily it worked. If not, my backup plan was to also procure an ASRock motherboard, most of which seem to have working VT-d support.

I had hoped to run a virtual HTPC with an ATI graphics card passed through on this computer. Unfortunately the virtualisation gods do not seem to be happy with this idea at the moment. Still, this box makes a decent whitebox ESXi host, apart from the onboard Intel 82579V NIC which ESXi does not support out the box. A custom driver needs to be injected into the ESXi installation ISO, unless you have a supported PCI-E NIC in which case the driver can be installed post-install.

Note1: While playing with passthrough and various options of internal graphics and PCI-E graphics BIOS configurations I got to the point where I could no longer get graphics from the onboard graphics card. I found a couple of posts on the Internet about this too. Even resetting/clearing CMOS did not resolve this. As per the other posts, I reflashed the BIOS and it sorted it out. Weird behaviour and unexpected – I could not get the BIOS to save the option to use IGFX (Internal graphics) rather than PEG (PCI-E graphics) as the “Init Display First” option.

Note2: The following are the graphics cards I attempted to pass through to the VMs. Note that I tried both VMware ESXi 5.0U2 build 914586 and 5.1 build 799733 and 914609 with motherboard BIOS f9, f10d and f10e.

Asus ATI Radeon HD 5450 – passed through and seen by VM but has atikmpag.sys BSOD 0x0116 “Attempt to reset the display driver and recover from timeout failed.” whenever I connected the monitor to the graphics card or tried to enable the display on the card.

Asus ATI Radeon HD 6450 – exactly as above.

Asus NVIDIA Geforce GT610 – passed through and seen by the VM. However the device driver is unable to start in Windows.

Note3: While trying to get the graphics cards to work properly I tried various combinations of additional/advanced settings including:

On the VM:

pciHole.start 1200
pciHole.end   2200
pciPassthru0.maxMSIXvectors  16
pciPassthru0.msiEnabled   FALSE

On the host, edit of: /etc/vmware/passthru.map by adding

#ATI Radeon HD
1002 ffff bridge false
#tried with flr/d3d0/link/bridge in the third column

Note4: In ESXi5.1 ACS checking is enforced more strictly resulting in quad-port NICs (and other devices apparently) not successfully getting configured for passthrough. After each reboot the devices still show as needing a reboot. The console logs show a message similar to:

WARNING: PCI: ssss: nnn:nnn:nn.n: Cannot change ownership to PASSTHRU 
(non-ACS capable switch in hierarchy)
This can be bypassed (at your own peril) using the host advanced option: disableACSCheck=true
Use the vSphere console: Configuration/Software/Advanced Settings/VMKernel/VMKernel.Boot.DisableAcsCheck
More info can be found at this informative post. This option got the quad port NICs passed through successfully but did not make any difference to the ATI or NVIDIA cards.
Note5: Currently ESXi 5.1 upto and including build 914609 does not seem to allow USB controller passthrough via VMDirectPath I/O. You can select the device for passthrough but once the host is rebooted, the device is unselected. I am not sure if this is a bug or a conscious decision by VMware. A cynic like myself might think this is intentional, as without the ability to pass through a USB controller there is no way to pass through a real keyboard and mouse into a VM and hence no need to get GPUs working with passthrough. (Hmm – maybe a bluetooth USB device passed into a VM and then paired with a BT keyboard and mouse?? Something for another day).

 

 

7 thoughts on “VT-d on GIGABYTE G1.Sniper M3 and ESXi GPU passthrough woes

  1. Hi Mike-
    I’ve been able to get a virtualized HTPC setup going, although not flawlessly, my video card occasionally freezes during playback of Flash/Silverlight content (though not recorded TV), but that is really the only problem. I’m using a Tyan S5512GM4NR motherboard, E3-1230 CPU, and Ati 7750 video card. Did you try using different slots? I’ve read (and experienced) certain slots having issues when others do not.

    The main things that have worked well for me are using the pcihole, 2GB of RAM to the VM, and reserving all the guest memory. Also of course ESXi 5.0, due to the flaws in 5.1. Feel free to email me if you want to follow up, derickso@stanford.edu.

    -David

    Reply
    • Hi,

      Thanks for the feedback. I did try various slots and various releases of ESXi5.0 and 5.1, with various options (eg the pcihole trick), at the time and never got my selection of graphics cards to work. I gave up in the end figuring my MB/chipset/BIOS just wasn’t up to the job. At the time I head good things about compatibility for VT-d with Tyan and Asrock motherboards.

      Kind regards,
      Mike

      Reply
  2. Esxi 5 with ATI 6450 GPU passthrough works on my Asrock B75 PRO-m when I disable HD audio in the Bios (otherwise atikmpag.sys BSOD)
    Also set:
    primary graphics adapter : pci express
    VT-d : enabled
    IGPU Multi monitor: disabled
    Render standby: disabled
    Deep render standby: disabled
    HD audio : disabled

    Passthrough works in Windows7/8

    In esxi 5.5 passthrough of the onboard usb controller #2 works also. Need to use EFI boot with windows8. The ati driver is included in w8 so no installation needed. Extra settings needed for ati 6450 and usb passthrough (otherwise EFI bios crash when init video, even with 2gb of ram):
    pciHole.start=1200
    pciHole.start=2200
    Also set memory to: reserve all guest memory (all locked) Don’t know this is needed.

    Reply
    • Thanks a lot Michel! GPU passthrough finally works on my site thanks to this blog.

      I confirm vt-d support and successful GPU passthrough:
      MB ASUS P8Z68-V LE with two Radeon R7 R250X / R250 (MSI & ASUS) i7-2600 ESXI 5.5 and 6.0. Also succcessful Gt 640 modeed to K1 and ati 4850. I think the most problem with intel hd gpu integrated in cpu, when I use it primary for console always take a BSOD. I think with this combination (intel int gpu-off and pcie GPUs as primary for esxi console and after passthrough it) my ASROCK Z97 Extreme4 100% will work too. Before I use this hardware with xenserver 6.2 passthrough work in any configuration.

      All my successful config on this MOBOs & ESXi hypervisors:
      –In BIOS–
      Primary graphics adapter : PCI Express
      GPU Multi monitor: Disabled
      Render standby: Disabled
      Virtualisation: Enable
      HD audio : Disabled

      –VM settings–
      VM version 8
      I add passthroug GPU using vSphere Client, because web client error this – PCI passthrough device ID (0x-5550) is invalid (01:00:1 and 01:00:2 device added). Anyway web client allows you to add 01:00.1 (or 02:00.1) one device and this would be enough that it worked.
      Reserve all maemory (I add 8Gb).

      Thats all. I upload sysprepared image of Windows 7 x64 and driver for R7 250X.
      No additional config like a pci hole.

      Thank you very much, again, for all in this blog.

      ps Sorry for a lot of details, I hope someone come in handy solution and save spent several months.
      pps Software for remote access to VM with passthough GPU on a very good FPS is Radmin.

      Reply
  3. Thanks a ton! I have a MSI board but the steps to get my Radeon to run where identical:

    Graphic option: PEG
    multi display IGP disabled
    HD Audio disabled

    obvious ESXi boots up with the Radeon, but as it is activated for passthru at 1/3 of bar display “freezes” and once ESXi is up and guest stars, the guest takes over the primary display. Unfortunately the console view is gone but the F12 to shutdown I can type blind :-). Maybe redirect output to serial but didnt look into that yet.

    Reply

Leave a reply

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> 

required