Error creating libvirt domain: Permission denied

[stew3254]

When trying to set up any basic vm with a disk I created with Terraform, I run into this error

error creating libvirt domain: internal error: process exited while connecting to monitor: 2025-03-03T11:25:49.961773Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/cloud-noble.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/cloud-noble.qcow2': Permission denied

I noticed Terraform doesn't like to follow the usual file permissions of libvirt vms. I've tried changing it, but to no avail. I have no idea why it keeps saying permission denied. I am using upstream Terraform version v1.11.0. Here is my Terraform

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
      version = "0.8.2"
    }
  }
}

provider "libvirt" {
  uri = "qemu:///system"
}

### Network Configuration ###
resource "libvirt_network" "my_net" {
  name = "lv-net"
  mode = "nat"
  addresses = ["192.168.123.1/24"]
  autostart = true
}

### Image Configuration ###
resource "libvirt_volume" "noble" {
  name = "cloud-noble.qcow2"
  source = "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
}

resource "libvirt_domain" "test" {
  name = "test"
  description = "Test image"
  vcpu = 2
  memory = "2048"
  disk {
    volume_id = libvirt_volume.noble.id
  }
  network_interface {
    network_id = libvirt_network.my_net.id
    wait_for_lease = true
  }
}

[stew3254]

Also, when it fails it doesn't clean up the resource. I have to manually destroy the virsh domain. You should probably mark that as tainted instead of error. That way it should destroy and re-create it upon a re-run

[GregDomzalski]

I'm actually running into this same exact issue right now too. And I'm stuck.

I am using a pre-existing storage pool that's a mounted iSCSI LUN.

When I create VMs through virt-manager, I see disks are created with libvirt-qemu:kvm permissions. However it seems that any disk created with virsh are created as root:root.

After some experimentation, it looks like using virsh vol-create-as exhibits this same behavior, seeming like this is the default. But if I do a virsh vol-create and pass the XML down directly with the /volume/target/permissions/{owner,group} set explicitly, then it creates the disk with the correct permissions.

My environment

Operating system

OS: Ubuntu 24.04 noble
Kernel: x86_64 Linux 6.8.0-53-generic

Terraform

Terraform v1.10.5 on linux_amd64

terraform {
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = ">= 0.8.1"
    }
  }
}

KVM / libvirt

Compiled against library: libvirt 10.0.0
Using library: libvirt 10.0.0
Using API: QEMU 10.0.0
Running hypervisor: QEMU 8.2.2

Storage pool

This was an existing pool created through virt-manager

<pool type='dir'>
  <name>vmstorage</name>
  <uuid>c9e20d41-b851-4660-85ff-e65ba1d849bc</uuid>
  <capacity unit='bytes'>2163348520960</capacity>
  <allocation unit='bytes'>847078973440</allocation>
  <available unit='bytes'>1316269547520</available>
  <source>
  </source>
  <target>
    <path>/mnt/vmstorage</path>
    <permissions>
      <mode>0655</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

$ ll /mnt/vmstorage
total 827225592
drw-r-xr-x+ 3 root         root         4096 Mar  4 14:02 ./
drwxr-xr-x  3 root         root         4096 Aug  1  2024 ../
-rw-r-xr-x  1 libvirt-qemu kvm  536953094144 Mar  3 17:49 csbuild-aos.qcow2*
-rw-r-xr-x  1 libvirt-qemu kvm  274920177664 Mar  4 14:23 cshost.qcow2*
-rw-r--r--  1 root         root       372736 Mar  4 13:59 csio-app-server-commoninit.iso
-rw-r--r--  1 root         root       197408 Mar  4 14:02 csio-app-server-main.qcow2
-rw-r-xr-x  1 libvirt-qemu kvm  386697330688 Mar  4 14:24 dockerhost.qcow2*
-rw-r--r--  1 root         root       372736 Mar  4 13:59 home-net-server-commoninit.iso
-rw-r--r--  1 root         root       197408 Mar  4 14:02 home-net-server-main.qcow2
-rw-r-xr-x  1 libvirt-qemu kvm  549840027648 Mar  4 14:24 homestack.qcow2*
drw-r-xr-x  2 root         root        16384 Aug  1  2024 lost+found/
-rw-r--r--  1 root         root    610110464 Mar  4 14:02 ubuntu-server-24.04-base.qcow2
-rw-r--r--  1 root         root    614146048 Mar  4 14:01 ubuntu-server-24.10-base.qcow2

Yes, the pool and folder permissions and are root:root. I have changed the directory ownership to libvirt-qemu:kvm and did a virsh pool-refresh which did update the owner/group, but that did not change the overall behavior.

What I've tried

1. I've tried setting the folder permissions for /mnt/vmstorage to libvirt-qemu:kvm and ran virsh pool-refresh. I verified that the pool XML reflected the correct permissions. I then ran Terraform and the volumes were still created as root:root.

2. So, the plugin documentation leads me to believe that you're sending down the XML. Great! So I tried using the xml feature of the libvirt_volume resource to try and modify / set the permissions ahead of time (which worked when using virsh CLI). I used:

<?xml version="1.0" ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:output omit-xml-declaration="yes" indent="yes"/>
    
    <!-- Identity template to copy everything as-is -->
    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*"/>
        </xsl:copy>
    </xsl:template>
    
    <!-- Modify owner to be "libvirt-qemu" -->
    <xsl:template match="/volume/target/permissions/owner/text()">
        <xsl:text>64055</xsl:text>
    </xsl:template>
    
    <!-- Modify group to be "kvm" -->
    <xsl:template match="/volume/target/permissions/group/text()">
        <xsl:text>994</xsl:text>
    </xsl:template>
    
</xsl:stylesheet>

Unfortunately, this did not work, and the volumes were still created as root:root.

3. Using the various AIs along with QEMU/KVM/libvirt documentation, I tried to make sure that I had everything configured properly so that libvirt would create resources as libvirt-qemu:kvm by default. It appeared like that should be the case, but it's possible I missed something. I had changed a few things based on suggestions from the AIs (after reading the docs to make sure it make some sense) but it ended up breaking the existing VMs. After I reset everything back to the way it was, things still didn't work right - so chowning everything to root:root actually fixed things. But now, somehow, after getting those original VMs running, their permissions are back to libvirt-qemu:kvm. No idea how that happened.

Examples of disks

Disk created via virt-manager:

<volume type='file'>
  <name>cshost.qcow2</name>
  <key>/mnt/vmstorage/cshost.qcow2</key>
  <capacity unit='bytes'>274877906944</capacity>
  <allocation unit='bytes'>25289981952</allocation>
  <physical unit='bytes'>274920177664</physical>
  <target>
    <path>/mnt/vmstorage/cshost.qcow2</path>
    <format type='qcow2'/>
    <permissions>
      <mode>0655</mode>
      <owner>64055</owner>
      <group>994</group>
    </permissions>
    <timestamps>
      <atime>1741097773.526252052</atime>
      <mtime>1741098045.137521955</mtime>
      <ctime>1741098045.137521955</ctime>
      <btime>0</btime>
    </timestamps>
    <compat>1.1</compat>
    <clusterSize unit='B'>65536</clusterSize>
    <features>
      <lazy_refcounts/>
    </features>
  </target>
</volume>

Disk created via Terraform / dmacvicar/libvirt

This disk was created using the XSL shown above and the following Terraform resource definition:

# Ubuntu Server 24.10
resource "libvirt_volume" "ubuntu_server_24_10_base" {
  name   = "ubuntu-server-24.10-base.qcow2"
  pool   = var.storage_pool_name
  source = "https://cloud-images.ubuntu.com/oracular/current/oracular-server-cloudimg-amd64.img"
  format = "qcow2"

  xml {
    xslt = file("${path.module}/disk_permissions.xsl")
  }
}

Result:

<volume type='file'>
  <name>ubuntu-server-24.10-base.qcow2</name>
  <key>/mnt/vmstorage/ubuntu-server-24.10-base.qcow2</key>
  <capacity unit='bytes'>3758096384</capacity>
  <allocation unit='bytes'>614150144</allocation>
  <physical unit='bytes'>614146048</physical>
  <target>
    <path>/mnt/vmstorage/ubuntu-server-24.10-base.qcow2</path>
    <format type='qcow2'/>
    <permissions>
      <mode>0644</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
    <timestamps>
      <atime>1741096904.674849908</atime>
      <mtime>1741096904.673849906</mtime>
      <ctime>1741096904.673849906</ctime>
      <btime>0</btime>
    </timestamps>
    <compat>1.1</compat>
    <clusterSize unit='B'>65536</clusterSize>
    <features/>
  </target>
</volume>

I suspect this is more of a libvirt configuration issue than an issue with this provider... but still, any help or insight on how to make things work via Terraform would be greatly appreciated! Or maybe someone else has already encountered this and has figured out how to fix it.

[GregDomzalski]

Looks like #1142 and #1024 are also related / same.

The workaround by disabling AppArmor suggested here: #1024 (comment) does seem to work. But that seems like quite a heavy hammer solution. Is there a way to craft a more narrow rule / exception, I wonder?

[stew3254]

Thanks for the great writeup @GregDomzalski! I didn't know virsh that well. That was a good primer to learn more about internals. I just wanted to say I am also running Ubuntu 24.04 with the same kernel

[fuenfgek]

@stew3254, @GregDomzalski,

I encountered a similar issue when I created a pool in a directory where the parent directories hadn’t been granted chmod o+x permissions. Here's a little example of why I think this was causing the issue.

The pool and image are automatically created as root:
ubuntu@manager-ubuntu:~/overlay_pool$ sudo ls -lh total 196K -rw-r--r-- 1 root root 193K Mar 9 09:44 test.qcow2

When the VM is started, the owner changes:
ubuntu@manager-ubuntu:~/overlay_pool$ sudo ls -lh total 1.1M -rw-r--r-- 1 libvirt-qemu kvm 366K Mar 9 09:45 commoninit.iso -rw-r--r-- 1 libvirt-qemu kvm 768K Mar 9 09:45 test.qcow2

When the VM is shut down, the image reverts to root permissions:
sudo ls -lh total 248M -rw-r--r-- 1 libvirt-qemu kvm 366K Mar 9 09:45 commoninit.iso -rw-r--r-- 1 root root 248M Mar 9 09:48 test.qcow2

The overlay pool was located in /parent1/parent2/overlay_pool, and the permission error disappeared once I added chmod o+x to the parent1 and parent2 directories. I believe this was because the libvirt-qemu process wasn’t allowed to enter the directories due to restrictive permissions.

So in my case, the parent directories needed to be accessible prevent the permission issue.

[dylanf9797]

I had this issue when having AppArmor installed and enabled and i found solutions and it worked for me.
For my case:

• libvirt + qemu + kvm
• OS: Ubuntu server
• OS version: 24.04
• arch: amd64
• target disk images dir: /var/lib/images/mypool/debian12.qcow2

libvirt-qemu.txt

libvirt create new AppArmor profile when creating new domain (vm) and each profile uses file /etc/apparmor.d/abstractions/libvirt-qemu and it does not have write access to images dir /var/lib/libvirt/images .

• Solution 1: Disable AppArmor (bad for security)

sudo systemctl disable --now apparmor

• Solution 2: Create file in AppArmor config dir

  Notes:
  By default, dirs, sub dirs, files and sub files in /etc/apparmor.d/ have these permissions:

  • For dir:s 0755
  • For files: 0644

  • Check file /etc/apparmor.d/abstractions/libvirt-qemu, it should be like this

    less /etc/apparmor.d/abstractions/libvirt-qemu

    See the uploaded file libvirt-qemu.txt. (I added extension .txt just for upload but it should not have this)
    At the end of file you should have.

    Notes:
    There are 2 spaces at the beginning of each line.

      include if exists <abstractions/libvirt-qemu.d>
    

  • Create dir if no exists: /etc/apparmor.d/abstractions/libvirt-qemu.d

    sudo mkdir -p /etc/apparmor.d/abstractions/libvirt-qemu.d

  • Create this file /etc/apparmor.d/abstractions/libvirt-qemu.d/override.

    sudo vim /etc/apparmor.d/abstractions/libvirt-qemu.d/override

    # /etc/apparmor.d/abstractions/libvirt-qemu.d/override
    
     /var/lib/libvirt/images/** rwk,
    

  • Restart AppAmor

    sudo systemctl restart apparmor