cancel
Showing results for 
Search instead for 
Did you mean: 

Linux KVM and PCIe passthrough

WiFivomFranMan
Level 7
Build,

Zenith Extreme
1950x
64GB ram
6TB storage
GTX 1080
RX 480
GT 720
PCIe USB card.


The goal of my build is to host KVMs. Fedora 26 as the host with two Windows 10 VMs one getting the GTX 1080 and the other getting RX 480. Of course I'd like to them to have their own USB card for hot plugging of USB devices.

I'd like the two powerful GPUs in the 16x slots with the GT720 in the 8x slot 2 for the Fedora host. This causes problems because the primary GPU wants to be slot 1. I heard in 0603 this can be configured? https://www.reddit.com/r/VFIO/comments/71se56/primary_secondary_graphics_card_swap_success_on/

I am also having problems with PCIe passthrough. It is crashing my box when I start my VMs. I am hearing this is a known problem, Do we know when the fix is coming?
12,151 Views
16 REPLIES 16

Korth
Level 14
Which KVM hypervisor? Just asking in case you're not using this. Can never tell with Fedora users, lol.

Are the (Win10) GTX1080, (Win10) RX480, and (Fedora) GT720 each outputting on separate monitors?

I don't understand your need for discrete USB cards ... unless each of your (three) operating systems runs its own array of USB hardware. Or your mobo's dozen USB ports just aren't enough, lol. I thought the KVM software could partition the USB controllers/ports (just as it does with main memory and other hardware resources). I might be wrong.
"All opinions are not equal. Some are a very great deal more robust, sophisticated and well supported in logic and argument than others." - Douglas Adams

[/Korth]

Korth wrote:
Which KVM hypervisor? Just asking in case you're not using this. Can never tell with Fedora users, lol.

Are the (Win10) GTX1080, (Win10) RX480, and (Fedora) GT720 each outputting on separate monitors?

I don't understand your need for discrete USB cards ... unless each of your (three) operating systems runs its own array of USB hardware. Or your mobo's dozen USB ports just aren't enough, lol. I thought the KVM software could partition the USB controllers/ports (just as it does with main memory and other hardware resources). I might be wrong.


3 monitors.

When you apply a USB device to a VM you can't remove it or add a new one while the VM is running. If you pass the whole USB pci card to the VM you can plug and unplug devices on the fly. Also when using USB headsets you can get cracking sounds.

I should be able to pass the 3.1 USB hub from the mobo to one of the Windows 10 VM and my PCI usb card to the second windows 10 VM. Basically they'll act like two stand alone windows 10 boxes.

WiFivomFranMan wrote:
When you apply a USB device to a VM you can't remove it or add a new one while the VM is running. If you pass the whole USB pci card to the VM you can plug and unplug devices on the fly. Also when using USB headsets you can get cracking sounds.

Ah, I understand now. Not something I've ever done. I tend to minimize my USB tangle and I think I would've just used hubs to branch out if needed.

Does it work properly with any USB controllers/ports/hubs embedded (dedicated, not pass-through) in KVM-switched monitors and keyboards?

Crackling audio probably indicates signal latencies or overprocessing (from software, mostly).

Not sure how to help if your BIOS doesn't present any option to allocate PCIEX8_2 GPU card to primary/boot, aside from hoping @ASUS provides this option in an updated BIOS. I think it should be possible to configure in firmware (even if the option isn't yet available) because the slot is "hardwired" for use in 3-way and 4-way multi-GPU configurations, but I don't really *know* (while ASUS does).
"All opinions are not equal. Some are a very great deal more robust, sophisticated and well supported in logic and argument than others." - Douglas Adams

[/Korth]

550 bucks for the motherboard, doesn't work and can't even get a reply.... ****.

WiFivomFranMan wrote:
550 bucks for the motherboard, doesn't work and can't even get a reply.... ****.


[I believe] PCI passthrough is not really a supported use case. [Supported in the sense that you can call tech support and they can help. The hardware may be there but you're on your own.] While it works in a lot of Intel systems it's not an advertised feature [on this board]. I wouldn't expect anyone at ASUS to have a schedule for fixing something like that.

I haven't been keeping up (so this may have changed) but I remember at one point Wendel from L1Techs had tried passthrough and found it didn't work on Ryzen. [Specifically the IOMMU groups seem to be sub-optimal on this platform.] So this could be an AMD problem and not something ASUS can fix.

EDIT: Post on 2nd page indicates potential confusion of personal opinion as policy. I do not have any involvement in ASUS products and don't receive any inside information. Clarifying text added in [brackets] above.
A bus station is where a bus stops. A train station is where a train stops. On my desk, I have a work station…

Are you crazy? AMD-Vi is supported. Who would need 32 threads for a gaming machine? Of course VMs are a use case.

If it is an Asus or AMD problem I don't care. I expect them them to talk to each other and figure it out. I gave both companies lot of money for a working solution. And yes I am *****ing at AMD as well. At least they reply.

I knew I should have stuck with ASRock and Intel

Key feature "Virtualization"

http://www.amd.com/en/products/cpu/amd-ryzen-threadripper-1950x


32 threads and virtualization isn't a use case? What would be the use case? Some video editing? Why make a gaming motherboard if you can't use all the core. I should be able to run 4 gaming boxes off of this machine.

WiFivomFranMan wrote:
Key feature "Virtualization"

http://www.amd.com/en/products/cpu/amd-ryzen-threadripper-1950x


32 threads and virtualization isn't a use case? What would be the use case? Some video editing? Why make a gaming motherboard if you can't use all the core. I should be able to run 4 gaming boxes off of this machine.


the options are in the BIOS to enable it.....

xeromist
Moderator
"Virtualization" is a broad term that covers a mixture of features. You can have virtualization without PCI-E passthrough. Even AMD's page on virtualization is a bit vague. It does mention I/O passthrough but doesn't specify if that means any PCI-E device or a limited set of built in components.
https://www.amd.com/en-us/innovations/software-technologies/processors-for-business/virtualization

Anyway, if you've seen something that explicitly touted PCI passthrough as a feature of Threadripper, that's awesome. I don't have any official information so if you do, ignore me. That means they may actually prioritize fixing it. I just haven't seen anything as such and it has been an issue since Ryzen launched so I'm not optimistic about the timeline.
A bus station is where a bus stops. A train station is where a train stops. On my desk, I have a work station…