Page 1 of 2 1 2 LastLast
Results 1 to 10 of 17
  1. #1
    New ROGer Array
    Join Date
    Sep 2017
    Reputation
    10
    Posts
    9

    Linux KVM and PCIe passthrough

    Build,

    Zenith Extreme
    1950x
    64GB ram
    6TB storage
    GTX 1080
    RX 480
    GT 720
    PCIe USB card.


    The goal of my build is to host KVMs. Fedora 26 as the host with two Windows 10 VMs one getting the GTX 1080 and the other getting RX 480. Of course I'd like to them to have their own USB card for hot plugging of USB devices.

    I'd like the two powerful GPUs in the 16x slots with the GT720 in the 8x slot 2 for the Fedora host. This causes problems because the primary GPU wants to be slot 1. I heard in 0603 this can be configured? https://www.reddit.com/r/VFIO/commen...ap_success_on/

    I am also having problems with PCIe passthrough. It is crashing my box when I start my VMs. I am hearing this is a known problem, Do we know when the fix is coming?

  2. #2
    ROG Guru: Black Belt Array Korth PC Specs
    Korth PC Specs
    MotherboardASUS X99 R5E (BIOS2101/1902)
    ProcessorHaswell-EP E5-1680-3 SR20H/R2 (4.4GHz)
    Memory (part number)Vengeance LPX 4x8GB SS DDR4-3000 (CMK32GX4M4C3000C15)
    Graphics Card #1NVIDIA Quadro GP100GL/16GB, 16xPCIe3, NVLink1 (SLI-HB)
    Graphics Card #2NVIDIA Quadro GP100GL/16GB, 16xPCIe3, NVLink1 (SLI-HB)
    Sound CardJDS Labs O2+ODAC (RevB), USB2 UAC1
    MonitorASUS PG278Q
    Storage #1Samsung 850 PRO 512GB SSDs, 4xSATA3 RAID0
    Storage #2Comay BladeDrive E28 3200GB SSD, 8xPCIe2
    CPU CoolerRaijintek NEMESIS/TISIS, AS5, 2xNH-A14
    CaseObsidian 750D (original), 6xNH-A14
    Power SupplyZalman/FSP ZM1250 Platinum
    Headset Pilot P51 PTT *modded*
    OS Arch, Gentoo, Win7x64, Win10x64
    Network RouterActiontec T3200M VDSL2 Gateway
    Accessory #1 TP-Link AC1900 Archer T9E, 1xPCIe
    Accessory #2 ASUS/Infineon SLB9635 TPM (TT1.2/FW3.19)
    Accessory #3 ASUS OC Panel I (FW0501)
    Korth's Avatar
    Join Date
    Mar 2015
    Reputation
    152
    Posts
    2,719

    Which KVM hypervisor? Just asking in case you're not using this. Can never tell with Fedora users, lol.

    Are the (Win10) GTX1080, (Win10) RX480, and (Fedora) GT720 each outputting on separate monitors?

    I don't understand your need for discrete USB cards ... unless each of your (three) operating systems runs its own array of USB hardware. Or your mobo's dozen USB ports just aren't enough, lol. I thought the KVM software could partition the USB controllers/ports (just as it does with main memory and other hardware resources). I might be wrong.
    "All opinions are not equal. Some are a very great deal more robust, sophisticated and well supported in logic and argument than others." - Douglas Adams

    [/Korth]

  3. #3
    New ROGer Array
    Join Date
    Sep 2017
    Reputation
    10
    Posts
    9

    Quote Originally Posted by Korth View Post
    Which KVM hypervisor? Just asking in case you're not using this. Can never tell with Fedora users, lol.

    Are the (Win10) GTX1080, (Win10) RX480, and (Fedora) GT720 each outputting on separate monitors?

    I don't understand your need for discrete USB cards ... unless each of your (three) operating systems runs its own array of USB hardware. Or your mobo's dozen USB ports just aren't enough, lol. I thought the KVM software could partition the USB controllers/ports (just as it does with main memory and other hardware resources). I might be wrong.
    3 monitors.

    When you apply a USB device to a VM you can't remove it or add a new one while the VM is running. If you pass the whole USB pci card to the VM you can plug and unplug devices on the fly. Also when using USB headsets you can get cracking sounds.

    I should be able to pass the 3.1 USB hub from the mobo to one of the Windows 10 VM and my PCI usb card to the second windows 10 VM. Basically they'll act like two stand alone windows 10 boxes.

  4. #4
    ROG Guru: Black Belt Array Korth PC Specs
    Korth PC Specs
    MotherboardASUS X99 R5E (BIOS2101/1902)
    ProcessorHaswell-EP E5-1680-3 SR20H/R2 (4.4GHz)
    Memory (part number)Vengeance LPX 4x8GB SS DDR4-3000 (CMK32GX4M4C3000C15)
    Graphics Card #1NVIDIA Quadro GP100GL/16GB, 16xPCIe3, NVLink1 (SLI-HB)
    Graphics Card #2NVIDIA Quadro GP100GL/16GB, 16xPCIe3, NVLink1 (SLI-HB)
    Sound CardJDS Labs O2+ODAC (RevB), USB2 UAC1
    MonitorASUS PG278Q
    Storage #1Samsung 850 PRO 512GB SSDs, 4xSATA3 RAID0
    Storage #2Comay BladeDrive E28 3200GB SSD, 8xPCIe2
    CPU CoolerRaijintek NEMESIS/TISIS, AS5, 2xNH-A14
    CaseObsidian 750D (original), 6xNH-A14
    Power SupplyZalman/FSP ZM1250 Platinum
    Headset Pilot P51 PTT *modded*
    OS Arch, Gentoo, Win7x64, Win10x64
    Network RouterActiontec T3200M VDSL2 Gateway
    Accessory #1 TP-Link AC1900 Archer T9E, 1xPCIe
    Accessory #2 ASUS/Infineon SLB9635 TPM (TT1.2/FW3.19)
    Accessory #3 ASUS OC Panel I (FW0501)
    Korth's Avatar
    Join Date
    Mar 2015
    Reputation
    152
    Posts
    2,719

    Quote Originally Posted by WiFivomFranMan View Post
    When you apply a USB device to a VM you can't remove it or add a new one while the VM is running. If you pass the whole USB pci card to the VM you can plug and unplug devices on the fly. Also when using USB headsets you can get cracking sounds.
    Ah, I understand now. Not something I've ever done. I tend to minimize my USB tangle and I think I would've just used hubs to branch out if needed.

    Does it work properly with any USB controllers/ports/hubs embedded (dedicated, not pass-through) in KVM-switched monitors and keyboards?

    Crackling audio probably indicates signal latencies or overprocessing (from software, mostly).

    Not sure how to help if your BIOS doesn't present any option to allocate PCIEX8_2 GPU card to primary/boot, aside from hoping @ASUS provides this option in an updated BIOS. I think it should be possible to configure in firmware (even if the option isn't yet available) because the slot is "hardwired" for use in 3-way and 4-way multi-GPU configurations, but I don't really *know* (while ASUS does).
    "All opinions are not equal. Some are a very great deal more robust, sophisticated and well supported in logic and argument than others." - Douglas Adams

    [/Korth]

  5. #5
    New ROGer Array
    Join Date
    Sep 2017
    Reputation
    10
    Posts
    9

    550 bucks for the motherboard, doesn't work and can't even get a reply.... ****.

  6. #6
    TeamROG Moderator Array xeromist PC Specs
    xeromist PC Specs
    Laptop (Model)Dell Inspiron 15 7567
    MotherboardMSI x470 Gaming Plus
    ProcessorAMD 2600X
    Memory (part number)16GB Crucial Ballistix Elite 3600
    Graphics Card #1ASUS GTX 1080 Strix
    MonitorBenQ BL3200PT
    Storage #1Intel 600p NVMe
    CPU CoolerWraith
    Casecustom Antec 900
    Power SupplyCorsair HX1000
    Keyboard Logitech Orion Spark
    Mouse Logitech MX500
    Headset Plantronics 777 with Oregon Aero upgrade
    xeromist's Avatar
    Join Date
    Jul 2010
    Reputation
    294
    Posts
    7,439

    Quote Originally Posted by WiFivomFranMan View Post
    550 bucks for the motherboard, doesn't work and can't even get a reply.... ****.
    [I believe] PCI passthrough is not really a supported use case. [Supported in the sense that you can call tech support and they can help. The hardware may be there but you're on your own.] While it works in a lot of Intel systems it's not an advertised feature [on this board]. I wouldn't expect anyone at ASUS to have a schedule for fixing something like that.

    I haven't been keeping up (so this may have changed) but I remember at one point Wendel from L1Techs had tried passthrough and found it didn't work on Ryzen. [Specifically the IOMMU groups seem to be sub-optimal on this platform.] So this could be an AMD problem and not something ASUS can fix.

    EDIT: Post on 2nd page indicates potential confusion of personal opinion as policy. I do not have any involvement in ASUS products and don't receive any inside information. Clarifying text added in [brackets] above.
    Last edited by xeromist; 09-28-2017 at 04:46 AM.
    * Support disease research with Folding@Home *

    < < < Click the drop-down above my avatar for my PC specs!

  7. #7
    New ROGer Array
    Join Date
    Sep 2017
    Reputation
    10
    Posts
    9

    Are you crazy? AMD-Vi is supported. Who would need 32 threads for a gaming machine? Of course VMs are a use case.

    If it is an Asus or AMD problem I don't care. I expect them them to talk to each other and figure it out. I gave both companies lot of money for a working solution. And yes I am *****ing at AMD as well. At least they reply.

    I knew I should have stuck with ASRock and Intel

  8. #8
    New ROGer Array
    Join Date
    Sep 2017
    Reputation
    10
    Posts
    9

    Key feature "Virtualization"

    http://www.amd.com/en/products/cpu/a...adripper-1950x


    32 threads and virtualization isn't a use case? What would be the use case? Some video editing? Why make a gaming motherboard if you can't use all the core. I should be able to run 4 gaming boxes off of this machine.

  9. #9
    TeamROG Moderator Array xeromist PC Specs
    xeromist PC Specs
    Laptop (Model)Dell Inspiron 15 7567
    MotherboardMSI x470 Gaming Plus
    ProcessorAMD 2600X
    Memory (part number)16GB Crucial Ballistix Elite 3600
    Graphics Card #1ASUS GTX 1080 Strix
    MonitorBenQ BL3200PT
    Storage #1Intel 600p NVMe
    CPU CoolerWraith
    Casecustom Antec 900
    Power SupplyCorsair HX1000
    Keyboard Logitech Orion Spark
    Mouse Logitech MX500
    Headset Plantronics 777 with Oregon Aero upgrade
    xeromist's Avatar
    Join Date
    Jul 2010
    Reputation
    294
    Posts
    7,439

    "Virtualization" is a broad term that covers a mixture of features. You can have virtualization without PCI-E passthrough. Even AMD's page on virtualization is a bit vague. It does mention I/O passthrough but doesn't specify if that means any PCI-E device or a limited set of built in components.
    https://www.amd.com/en-us/innovation...virtualization

    Anyway, if you've seen something that explicitly touted PCI passthrough as a feature of Threadripper, that's awesome. I don't have any official information so if you do, ignore me. That means they may actually prioritize fixing it. I just haven't seen anything as such and it has been an issue since Ryzen launched so I'm not optimistic about the timeline.
    * Support disease research with Folding@Home *

    < < < Click the drop-down above my avatar for my PC specs!

  10. #10
    New ROGer Array
    Join Date
    Sep 2017
    Reputation
    10
    Posts
    9

    Quote Originally Posted by WiFivomFranMan View Post
    Key feature "Virtualization"

    http://www.amd.com/en/products/cpu/a...adripper-1950x


    32 threads and virtualization isn't a use case? What would be the use case? Some video editing? Why make a gaming motherboard if you can't use all the core. I should be able to run 4 gaming boxes off of this machine.
    the options are in the BIOS to enable it.....

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •