{{tag>Brouillon Virt CPU Hardware}}

= KVM LibVirt

Voir :
* [[Notes LibVirt]]
* https://github.com/kholia/OSX-KVM


Votre CPU gère t-il **VT** ou **AMD-V** ?

<code bash>
egrep --color '^flags.*(vmx |svm )' /proc/cpuinfo
</code>

Ou 
<code bash>
apt-get install cpu-checker
kvm-ok
</code>

[[https://docs.openstack.org/devstack/latest/guides/devstack-with-nested-kvm.html|Nested KVM]] ?
<code bash>
cat /sys/module/kvm_intel/parameters/nested
cat /sys/module/kvm_amd/parameters/nested
</code>


Parfois il faut activer l'option dans le BIOS


<code ->
# virt-host-validate
...
   LXC: Checking if device /sys/fs/fuse/connections exists                   : FAIL (Load the 'fuse' module to enable /proc/ overrides)
   
echo fuse >> /etc/modules
modprobe fuse
</code>

Sous RedHat désactiver le NetworkManager
<code bash>
systemctl stop NetworkManager
systemctl disable NetworkManager
</code>

et ajouter la ligne suivante dans /etc/sysconfig/network-scripts/ifcfg-*

''/etc/sysconfig/network-scripts/ifcfg-*''
<code ini>
NM_CONTROLLED=no
</code>


== Install

<code bash>
apt-get install --no-install-recommends qemu-kvm libvirt-clients libvirt-daemon-system
</code>

<code bash>
adduser jean libvirt
#adduser jean kvm
</code>


''~/.bash_aliases''
<code bash>
alias virsh="virsh -c qemu:///system"
</code>

ou 

''~/.bashrc''
<code bash>
export LIBVIRT_DEFAULT_URI="qemu:///system"
</code>

<code bash>
virsh net-autostart default
virsh net-start default
</code>


''/etc/default/libvirt-guests''
<code ini>
#ON_BOOT=ignore
ON_BOOT=start

#START_DELAY=0
START_DELAY=30

#PARALLEL_SHUTDOWN=0
PARALLEL_SHUTDOWN=1

#SHUTDOWN_TIMEOUT=300
SHUTDOWN_TIMEOUT=180
</code>

Pour virt-sysprep (après clonage de VM)
<code bash>
apt-get install libguestfs-tools
</code>


=== UEFI

<code bash>
apt-get install ovmf
</code>

== Virt-install

Exemple : 
* https://raymii.org/s/articles/virt-install_introduction_and_copy_paste_distro_install_commands.html
* http://serverfault.com/questions/481244/preseed-command-string-fail-with-newline-character-using-virt-install-initrd-inj

<code bash>
virt-install \
--name vm1 \
--ram 512 \
--location=http://ftp.us.debian.org/debian/dists/stable/main/installer-amd64/ \
--initrd-inject=/tmp/preseed.cfg \
--extra-args="console=ttyS0,115200n8 serial locale=fr_FR console-setup/ask_detect=false keyboard-configuration/layoutcode=fr hostname=virtual domain=unassigned-domain interface=auto" \
--vcpu=1 \
--vnc \
--vnclisten=0.0.0.0 \
--os-type=linux \
--os-variant=debianwheezy \
-w bridge=virbr0 \
-w bridge=br1 \
-w network=default \
--console pty,target_type=serial \
--disk format=qcow2,size=8,bus=virtio,path=/var/lib/libvirt/images/vm1.qcow2
</code>

<code ->
#--connect qemu:///system
#--location=http://archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/ \
#--noautoconsole \
#--autostart \
#--cdrom=/var/lib/libvirt/images/test.iso \
</code>

''--extra-args'' ne marche que si ''--location'' est défini.


== Réseaux

KVM Réseaux Bridge TUN 

http://doc.fedora-fr.org/wiki/Discussion:Virtualisation_:_KVM,_Qemu,_libvirt_en_images

== PCI Passthrough - IOMMU

Voir :
* https://blog.zerosector.io/2018/07/28/kvm-qemu-windows-10-gpu-passthrough/
* https://www.deltasight.fr/pci-passthrough-nvidia-gtx-1050-kvm/
* https://jugulaire.github.io/GPU-passthrough-KVM-QEMU/
* https://www.debian-fr.org/t/kvm-et-passthrough-afin-de-se-passer-du-dual-boot/60794/5
* https://pve.proxmox.com/wiki/Pci_passthrough
* https://davidwin93.github.io/vfio_guide/
* https://developers.redhat.com/articles/2024/04/05/enable-gpu-acceleration-kernel-module-management-operator

BIOS/UEFI

Première étape, faire un tour dans le BIOS/UEFI pour activer les choses suivantes :

    VT-D (cas d’un CPU intel)
    AMD-Vi (cas d’un CPU AMD)

Pour vérifier :

<code ->
# dmesg | grep -e "Directed I/O"
[    1.693161] DMAR: Intel(R) Virtualization Technology for Directed I/O
</code>

Pour du AMD :
<code bash>
dmesg | grep AMD-Vi
</code>

Dans le BIOS activer "SR-IOV" ?


<code ->
# ls /sys/class/iommu/*/
/sys/class/iommu/dmar2/:
devices  intel-iommu  power  subsystem  uevent

/sys/class/iommu/dmar4/:
devices  intel-iommu  power  subsystem  uevent

/sys/class/iommu/dmar5/:
devices  intel-iommu  power  subsystem  uevent

/sys/class/iommu/dmar6/:
devices  intel-iommu  power  subsystem  uevent

/sys/class/iommu/dmar7/:
devices  intel-iommu  power  subsystem  uevent
</code>

''/etc/modprobe.d/blacklist-nouveau.conf''
<code ->
blacklist nouveau
options nouveau modeset=0
</code>

''/etc/default/grub''
<code bash>
#GRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nouveau modprobe.blacklist=nvidiafb"
#GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt rd.driver.pre=vfio-pci video=efifb:off vfio_iommu_type1.allow_unsafe_interrupts=1"

#GRUB_CMDLINE_LINUX="kvm_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt pci-stub.ids=10de:1b81,10de:10f1 video=efifb:off"
</code>

<code bash>
update-grub
</code>

<code bash>
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
reboot
dmesg | grep 'remapping'
</code>

Pour nvidia
<code bash>
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
</code>

<code ->
# lspci -nn | grep -i nvidia
3b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104GL [Quadro P5000] [10de:1bb0] (rev a1)
3b:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
af:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104GL [Quadro P5000] [10de:1bb0] (rev a1)
af:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
</code>

''/etc/modprobe.d/vfio.conf''
<code baSh>
#options vfio-pci ids=10de:1bb0,10de:10f0
options vfio-pci ids=10de:1bb0
options vfio-pci disable_vga=1
</code>

<code bash>
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
update-initramfs -u
</code>

<code ->
# dmesg |grep "Directed I/O"
[    1.705358] DMAR: Intel(R) Virtualization Technology for Directed I/O
</code>

<code bash>
#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
</code>


<code bash>
dmesg | grep -i vfio
</code>

<code ->
# lspci -nnk |grep -i -A 2 nvidia
3b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104GL [Quadro P5000] [10de:1bb0] (rev a1)
        Subsystem: Dell GP104GL [Quadro P5000] [1028:11b2]
        Kernel driver in use: vfio-pci
--
3b:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
        Subsystem: Dell GP104 High Definition Audio Controller [1028:11b2]
        Kernel driver in use: snd_hda_intel
--
af:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104GL [Quadro P5000] [10de:1bb0] (rev a1)
        Subsystem: Dell GP104GL [Quadro P5000] [1028:11b2]
        Kernel driver in use: vfio-pci
--
af:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
        Subsystem: Dell GP104 High Definition Audio Controller [1028:11b2]
        Kernel driver in use: snd_hda_intel
</code>

<code ->
# echo "0000:3b:00.0" | tee /sys/bus/pci/drivers/vfio-pci/unbind
0000:3b:00.0
# echo "0000:3b:00.0" | tee /sys/bus/pci/drivers/vfio-pci/unbind
0000:3b:00.0
tee: /sys/bus/pci/drivers/vfio-pci/unbind: No such device
</code>

<code bash>
cd /sys/bus/pci/devices/0000\:3b\:00.0
echo 1 > rom
cat rom > /tmp/gpu.rom
echo 0 > rom
</code>

''/etc/libvirt/qemu/vm1.xml''
<code xml>
   <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x3b' slot='0x00' function='0x0'/>
      </source>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0xaf' slot='0x00' function='0x0'/>
      </source>
      <rom bar='on'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
</code>