Merge pull request #2814 from OSInside/support_live_http

Update live boot remote boot features

Author: schaefi

doc/source/working_with_images/network_live_iso_boot.rst

@@ -28,67 +28,39 @@ the network:
 
    .. code:: bash
 
-       $ mount {exc_image_base_name}.x86_64-{exc_image_version}.iso /mnt
-       $ cp /mnt/boot/x86_64/loader/initrd /srv/tftpboot/boot/initrd
-       $ cp /mnt/boot/x86_64/loader/linux /srv/tftpboot/boot/linux
-       $ umount /mnt
+      $ mount {exc_image_base_name_live}.x86_64-{exc_image_version}.iso /mnt
+      $ cp /mnt/boot/x86_64/loader/initrd /srv/tftpboot/boot/initrd
+      $ cp /mnt/boot/x86_64/loader/linux /srv/tftpboot/boot/linux
+      $ umount /mnt
 
    .. note::
 
-       This step must be repeated with any new build of the live ISO image
+      This step must be repeated with any new build of the live ISO image
 
 2. Export Live ISO To The Network
 
-   Access to the live ISO file is implemented using the AoE protocol
-   in {kiwi}. This requires the export of the live ISO file as remote
-   block device which is typically done with the :command:`vblade`
-   toolkit. Install the following package on the system which is
-   expected to export the live ISO image:
+   Access to the live ISO file must be provided by either `ftp`,
+   `http`, `https` or `dolly`. The most simple method is to setup a FTP server
+   e.g. `vsftpd` and copy the live ISO file to the data directory:
 
    .. code:: bash
 
-       $ zypper in vblade
+       $ mkdir -p /srv/ftp/image
+       $ cp {exc_image_base_name_live}.x86_64-{exc_image_version}.iso /srv/ftp/image
 
    .. note::
 
-       Not all versions of AoE are compatible with any kernel. This
-       means the kernel on the system exporting the live ISO image
-       must provide a compatible aoe kernel module compared to the
-       kernel used in the live ISO image.
-   
-   Once done, export the live ISO image as follows:
-
-   .. code:: bash
-
-       $ vbladed 0 1 INTERFACE {exc_image_base_name}.x86_64-{exc_image_version}.iso
-
-   The above command exports the given ISO file as a block storage
-   device to the network of the given INTERFACE. On any machine
-   except the one exporting the file, it will appear as
-   :file:`/dev/etherd/e0.1` once the :command:`aoe` kernel module
-   was loaded. The two numbers, 0 and 1 in the above example, classifies
-   a major and minor number which is used in the device node name
-   on the reading side, in this case :file:`e0.1`. The numbers given
-   at export time must match the AOEINTERFACE name as described in
-   the next step.
-
-   .. note::
-
-       Only machines in the same network of the given INTERFACE
-       can see the exported live ISO image. If virtual machines
-       are the target to boot the live ISO image they could all
-       be connected through a bridge. In this case INTERFACE
-       is the bridge device. The availability scope of the live
-       ISO image on the network is in general not influenced
-       by {kiwi} and is a task for the network administrators.
+       Check if the image can be downloaded via:
+       `wget ftp://IP/image/{exc_image_base_name_live}.x86_64-{exc_image_version}.iso`
+       before the next step.
 
 3. Setup live ISO boot entry in PXE configuration
 
    .. note::
 
-       The following step assumes that the pxelinux.0 loader
-       has been configured on the boot server to boot up network
-       clients
+      The following step assumes that the pxelinux.0 loader
+      has been configured on the boot server to boot up network
+      clients
 
    Edit the file :file:`/srv/tftpboot/pxelinux.cfg/default` and create
    a boot entry of the form:
@@ -97,22 +69,91 @@ the network:
 
       LABEL Live-Boot
           kernel boot/linux
-          append initrd=boot/initrd rd.kiwi.live.pxe root=live:AOEINTERFACE=e0.1
+          append initrd=boot/initrd ip=dhcp root=live:ftp://IP/image/{exc_image_base_name_live}.x86_64-{exc_image_version}.iso
 
-   * The boot parameter `rd.kiwi.live.pxe` tells the {kiwi} boot process to
-     setup the network and to load the required :mod:`aoe` kernel module.
+   * The `ip=` parameter controls how the dracut network module sets up
+     the network. Several options exists to control how the network
+     interface should be setup. Please consult the dracut manual
+     for further information.
 
-   * The boot parameter `root=live:AOEINTERFACE=e0.1` specifies the
-     interface name as it was exported by the :command:`vbladed` command
-     from the last step. Currently only AoE (Ata Over Ethernet)
-     is supported.
+   * The boot parameter `root=live:PROTOCOL://IP/PATH...` specifies the
+     remote endpoint and protocol to find the live ISO file. So far
+     `ftp`, `http`, `https` and `dolly` are supported.
 
 4. Boot from the Network
 
    Within the network which has access to the PXE server and the
-   exported live ISO image, any network client can now boot the
+   IP in the root= option, any network client can now boot the
    live system. A test based on QEMU is done as follows:
 
    .. code:: bash
 
-      $ sudo qemu -boot n
+      $ qemu -boot n
+
+Available Remote Boot Options
+-----------------------------
+
+There are the following kernel boot options available to control
+the behavior of the remote boot process.
+
+rd.kiwi.live.curl_options=
+  Options passed along to the curl call
+
+rd.kiwi.live.dolly_options=
+  Options passed along to the dolly call
+
+rd.kiwi.live.system=
+  Block device to use for the system. By default this is set to `/dev/ram0`
+
+ramdisk_size=bytes
+  Size of the ramdisk in bytes. By default this is set to 2097152 (2G)
+
+rd.kiwi.live.reset
+  Force reset of the live system. This option only makes sense if
+  the live system device (rd.kiwi.live.system) points to a persistent
+  storage device. In this case {kiwi} loads the system only once
+  and does not overwrite it unless a reset is requested.
+
+Persistent Live System
+----------------------
+
+The remote boot process of a live ISO image, places the ISO file
+into a ramdisk by default. This means all data lives in memory and
+is not persistent. In order to boot up the live system from a remote
+location and keep it on a persistent storage, it's required to
+pass the `rd.kiwi.live.system` boot option with the device name
+pointing to that persistent storage.
+
+.. warning::
+
+    All data on the device given via `rd.kiwi.live.system` will be wiped
+
+A test based on QEMU is done as follows:
+
+Edit the file :file:`/srv/tftpboot/pxelinux.cfg/default` and create
+a boot entry of the form:
+
+.. code:: bash
+
+   LABEL Live-Boot
+       kernel boot/linux
+       append initrd=boot/initrd ip=dhcp root=live:ftp://IP/image/{exc_image_base_name_live}.x86_64-{exc_image_version}.iso ramdisk_size=3545728 rd.kiwi.live.system=/dev/sda rd.live.overlay.persistent rd.live.overlay.cowfs=xfs
+
+* The `rd.live.overlay.persistent` and `rd.live.overlay.cowfs=xfs`
+  options are standard {kiwi} live ISO options to control if and how a
+  persistent write partition should be created. The options only take
+  an effect when booting into a persistent storage device.
+
+Next create a persistent storage disk of 3G and attach it to the
+QEMU instance.
+
+.. code:: bash
+
+   $ qemu-img create mydisk.raw 3G
+   $ qemu -boot n -hda mydisk.raw
+
+The live system will be deployed once to the locally attached disk and
+boots from it. Any subsequent boot process will not modify the local
+disk unless `rd.kiwi.live.reset` is passed on the kernel command line.
+If deployed there is also no need for the network anymore and the
+system could also boot standalone.

dracut/modules.d/90kiwi-live/kiwi-generator.sh

@@ -22,6 +22,10 @@ case "${liveroot}" in
         root="${root//\//\\x2f}"
         root="live:aoe:/dev/etherd/${root#AOEINTERFACE=}"
         rootok=1 ;;
+    live:ftp:*|live:http:*|live:https:*|live:dolly:*) \
+        root="${root#live:}"
+        root="live:net:${root}"
+        rootok=1 ;;
 esac
 
 [ "${rootok}" != "1" ] && exit 0

dracut/modules.d/90kiwi-live/kiwi-live-genrules.sh

@@ -1,5 +1,8 @@
 #!/bin/bash
 
+# shellcheck disable=SC1091
+type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
+
 declare root=${root}
 
 case "${root}" in
@@ -22,4 +25,16 @@ case "${root}" in
         } >> /etc/udev/rules.d/99-live-aoe-kiwi.rules
         wait_for_dev -n "${root#live:aoe:}"
         ;;
+    live:net:*)
+        system=$(getarg rd.kiwi.live.system=)
+        if [ -z "${system}" ];then
+            system=/dev/ram0
+        fi
+        {
+        printf 'KERNEL=="%s", RUN+="/sbin/initqueue %s %s %s"\n' \
+            "${system#/dev/}" "--settled --onetime --unique" \
+            "/sbin/kiwi-live-root" "${root}"
+        } >> /etc/udev/rules.d/99-live-net-kiwi.rules
+        wait_for_dev -n "${system}"
+        ;;
 esac

dracut/modules.d/90kiwi-live/kiwi-live-lib.sh

@@ -15,6 +15,60 @@ function getOverlayBaseDirectory {
     echo "${overlay_base}"
 }
 
+function ProvideIso {
+    local root=$1
+    local iso_url=${root#live:net:}
+    local system
+    local rd_size
+    local custom_rd_size
+    local modfile=/etc/modprobe.d/99-brd.conf
+    system=$(getarg rd.kiwi.live.system=)
+    if [ ! -b "${system}" ];then
+        system=/dev/ram0
+    fi
+    ptable=$(blkid -s PTTYPE -o value "${system}" 2>/dev/null)
+    case "${iso_url}" in
+        ftp:*|http:*|https:*|dolly:*) \
+            if [ -z "${ptable}" ] || getargbool 0 rd.kiwi.live.reset; then
+                custom_rd_size=$(getarg ramdisk_size=)
+                if [ -n "${custom_rd_size}" ];then
+                    rd_size="${custom_rd_size}"
+                else
+                    # set minimum default ramdisk size: 2G
+                    rd_size="2097152"
+                fi
+                mkdir -p /etc/modprobe.d
+                if [ -n "${rd_size}" ];then
+                    echo "options brd rd_size=${rd_size}" > ${modfile}
+                fi
+                modprobe --remove brd
+                modprobe brd
+                udev_pending
+                fetch_file "${iso_url}" | dd bs=32k of="${system}" &>/dev/null
+            fi
+            if [[ ${system} =~ ^/dev/ram ]];then
+                echo "${system}"
+            else
+                udev_pending
+                # This is not 100% safe as in theory the volume ID
+                # of the live ISO could be configured in the kiwi image
+                # description and doesn't have to be set to: CDROM.
+                # The value could be obtained as follows:
+                # label=$(
+                #     isoinfo -j utf-8 -d -i ${system} |\
+                #     grep "Volume id:" | cut -f2 -d:
+                # )
+                # but this would require an additional tool as part of the
+                # initrd. Thus this is subject to a followup pull request
+                echo "/dev/disk/by-label/CDROM"
+            fi
+        ;;
+        *) \
+            echo "${root}"
+        ;;
+    esac
+}
+
 function lookupIsoDiskDevice {
     local root=$1
     local iso_label=${root#/dev/disk/by-label/}
@@ -276,9 +330,73 @@ function runMediaCheck {
     fi
 }
 
+function fetch_file {
+    # Fetch file from remote location
+    local source_url=$1
+    local source_uncompressed_bytes=$2
+    local fetch_info=/tmp/fetch.info
+    local fetch
+    local curl_options
+    local dolly_options
+    curl_options=$(getarg rd.kiwi.live.curl_options=)
+    dolly_options=$(getarg rd.kiwi.live.dolly_options=)
+    if [ -n "${curl_options}" ]; then
+        curl_options=$(str_replace "${curl_options}" "," " ")
+        fetch="curl ${curl_options} -f ${source_url}"
+    else
+        fetch="curl -f ${source_url}"
+    fi
+    if [ -n "${dolly_options}" ]; then
+        dolly_options=$(str_replace "${dolly_options}" "," " ")
+        fetch="dolly ${dolly_options} -f ${source_url}"
+    fi
+    if _is_dolly "${source_url}";then
+        fetch="dolly"
+    fi
+    if _is_xz_compressed "${source_url}";then
+        fetch="${fetch} | xz -d"
+    fi
+    if [ -z "${source_uncompressed_bytes}" ];then
+        eval "${fetch}" 2>${fetch_info}
+    else
+        eval "${fetch}" 2>${fetch_info} |\
+            pv --size "${source_uncompressed_bytes}" --stop-at-size -n
+    fi
+    return "${PIPESTATUS[0]}"
+}
+
+function show_log_and_quit {
+    local text_message="$1"
+    local log_file="$2"
+    local timeout
+    timeout=$(_dialog_timeout)
+    if [ "${timeout}" = "off" ];then
+        _run_dialog --backtitle "\"${text_message}\"" \
+            --textbox "${log_file}" 15 80
+    else
+        _run_dialog --timeout "${timeout}" --backtitle "\"${text_message}\"" \
+            --textbox "${log_file}" 15 80
+    fi
+    if getargbool 0 rd.debug; then
+        die "${text_message}"
+    else
+        reboot -f
+    fi
+}
+
 #=========================================
 # Methods considered private
 #-----------------------------------------
+function _is_xz_compressed {
+    local source_url=$1
+    [[ ${source_url} =~ .xz$ ]]
+}
+
+function _is_dolly {
+    local source_url=$1
+    [[ ${source_url} =~ ^dolly ]]
+}
+
 function _setup_interactive_service {
     local service=/run/systemd/system/dracut-run-interactive.service
     mkdir -p /run/systemd/system
@@ -312,9 +430,29 @@ function _run_interactive {
     systemctl start dracut-run-interactive.service
 }
 
+function _dialog_timeout {
+    local timeout=60
+    custom_timeout=$(getarg "rd.kiwi.dialog.timeout=")
+    [ -n "${custom_timeout}" ] && timeout="${custom_timeout}"
+    echo "${timeout}"
+}
+
 function _run_dialog {
-    echo "dialog $*" >/run/dracut-interactive
+    # """
+    # Run the dialog program via the systemd service either in a
+    # framebuffer terminal or on the console. The return code of
+    # the function is the return code of the dialog call. The
+    # output of the dialog call is stored in a file and can be
+    # one time read via the get_dialog_result function
+    # """
+    local dialog_result=/tmp/dialog_result
+    local dialog_exit_code=/tmp/dialog_code
+    {
+        echo "dialog $* 2>$dialog_result"
+        echo "echo -n \$? >$dialog_exit_code"
+    } >/run/dracut-interactive
     _run_interactive
+    return "$(cat $dialog_exit_code)"
 }
 
 function _partition_count {