Sign In
Create Account
Hi everyone,
I had an interesting last week trying to get 3970x to work on VMware ESXi, and I decided to post my thoughts here, both to ask for help from smart people, and to provide information for others who want to go down this path.
My goal is to create a system virtualized on the ESXi platform that will allow me to play, track data, develop, test, basically do whatever I want. For this, I chose the AMD 3970x platform.
My technical specifications are as follows:
AMD 3970x 32 core processor
128GB OF RAM Corsair CMT64GX4M4K3600C18
Asus ROG Zenith 2 Extreme w/0702 bios motherboard
H740p (Dell/LSI) raid controller w/8GB NVFlash + BBU
8x 2TB Crucial MX500 SSD in a case with SFF8643 support
3x 1TB Rocket PCIe v4 NVME
4x 12TB Seagate Ironwolf NAS HDD
GTX 2080Ti GPU#1
GTX 1060 GPU#2
2x GT910 auxiliary graphics processor
BeQuiet TR4 Cooler
Boat with Noctua 120mm/140mm Fans
Fractal Design XL R2 Case
4x 1080p BenQ 27" IPS Monitors (models with blue light reduction)
1x Asus ultrawide giant monitor (forgot model)
2x 2K 32" Dell/Acer monitor
There are also many other different parts, such as multiple pcie USB cards, x16 ribbon extenders, Bluetooth USB adapters, and the like.
STATUS SO FAR:
Raid: The H740p Raid controller has exited the Dell server. I love this place. They are configured directly in the Asus UEFI, which is amazing.
I've been using ESXi for years, as well as Dell servers, and I'm pretty comfortable with those platforms. However, I don't claim to be an absolute guru.
I installed an 8x 2TB SSD in RAID6 with about 30% overshoot. RAID6 was chosen for double parity as well as crazy read speed. Note: 30% redundancy (or insufficiency, if you like) has a significant impact on the TDW of the drive. I bought 2x other 2TB disks and made over 3000 disk entries @ 30% OP, and it still works fantastically. The critical MX500 was chosen due to having enough capacitor charge to write to its buffer in the event of a power failure. The speed without disk cache and raid cache is @ 4 GB / s, and without raid/drive cache it is @ 7 GB/s (expands beyond 8 GB write / read).
Initially, there was an issue with the raid controller being in Pcie x4 mode by default, which I fixed with Asus UEFI.
VMware ESXi 6. 7u3 was installed on 1x NVMe pcie4 disks. I decided to put ESXi on these disks, as I will be using it as a staging area to perform vfstool disk conversions, so high disk I / O will be good for this.
The VMware data store is located in a RAID6 array.
PROBLEM:
So far, I have created a 1x VM, Windows 10 LTSB 1603, to test this. 16 GB of RAM, 250 GB of hard disk, 6 cores, thick screen.
I went through the GTX 1060 + GTX 1060 audio through a Win10 VM, and it loads directly to the monitor. Hurray! The only necessary setting was the following:
hypervisor.cpuid.v0 = FALSE
Which allowed the Nvidia GPU to run normally.
THE FIRST PROBLEM: The audio has serious problems. It is extremely choppy and plays videos via Youtube chops/skips/crackles. Fu. I don't know how to fix it. The latest Nvidia drive and audio go directly through the monitor, so it is effectively GPU - > HDMI audio for monitoring.
SECOND PROBLEM: No keyboard or mouse
ESXi seems to filter out all connected HID USB devices. I can't seem to find a way around this. Does anyone know?
You can then run through the entire USB controller to the VM, and then connect whatever you want, like a keyboard/mouse, to that USB device, and it will pop up. But this is not the case.
I enabled the transfer of all USB devices and tried to map them to the VM. From what I can tell, there are 2x USB controllers that are easy to see: ASMedia + AMD.
The AMD USB controller simply won't be “enabled” for end-to-end access. It keeps saying that the ESXi host needs a reboot. So don't go there.
ASMedia one allows me to add it as a pcie-passthru to the VM, however nothing works in the VM. No USB devices are recognized. No keyboard, no mouse, no USB drives, nothing.
Then I added a USB controller card (StarTech USB 3.1 PCIe Card - PEXUS313AC2V) to the system and mapped it. However, nothing is found there either. No keyboard, no mouse, no disks, nothing.
THIRD PROBLEM: Disk Response/Active Time: 100%
