Skip to content

Run package scripts in apt bootstrap phase #2661

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 4, 2024
Merged

Run package scripts in apt bootstrap phase #2661

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ad50bc7
Run package scripts in apt bootstrap phase
schaefi Sep 27, 2024
15b4bbc
Support older apt versions for bootstrap
schaefi Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 91 additions & 33 deletions kiwi/package_manager/apt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
import os
import glob
import logging
from textwrap import dedent
import pathlib
from typing import (
List, Dict
)
Expand Down Expand Up @@ -222,6 +222,13 @@ def _process_install_requests_bootstrap(self) -> CommandCallT:
# the essential debs and tell us their full in the apt cache without
# actually installing them.
try:
# TODO: Drop once apt 2.5.4 is widely available.
pathlib.Path(f'{self.root_dir}/var/lib/dpkg').mkdir(
parents=True, exist_ok=True
)
pathlib.Path(f'{self.root_dir}/var/lib/dpkg/status.kiwi').touch()
pathlib.Path(f'{self.root_dir}/var/lib/dpkg/available').touch()

if 'apt' not in self.package_requests:
self.package_requests.append('apt')
update_cache = [
Expand All @@ -236,53 +243,50 @@ def _process_install_requests_bootstrap(self) -> CommandCallT:
'Apt update: {0} {1}'.format(result.output, result.error)
)
package_names = Temporary(prefix='kiwi_debs_').new_file()
package_extract = Temporary(prefix='kiwi_bootstrap_').new_file()
download_bootstrap = [
'apt-get'
] + self.apt_get_args + self.custom_args + [
'install',
'-oDebug::pkgDPkgPm=1',
f'-oDPkg::Pre-Install-Pkgs::=cat >{package_names.name}',
'?essential',
'?exact-name(usr-is-merged)',
'base-passwd'
'?essential'
] + self.package_requests
# Download solved bootstrap packages
result = Command.run(
download_bootstrap, self.command_env
)
log.debug(
'Apt download: {0} {1}'.format(result.output, result.error)
)
script = dedent('''\n
set -e
while read -r deb;do
echo "Unpacking $deb"
dpkg-deb --fsys-tarfile $deb | tar -C {0} -x
done < {1}
while read -r deb;do
pushd "$(dirname "$deb")" >/dev/null || exit 1
if [[ "$(basename "$deb")" == base-passwd* ]];then
echo "Running pre/post package scripts for $(basename "$deb")"
dpkg -e "$deb" "{0}/DEBIAN"
test -e {0}/DEBIAN/preinst && chroot {0} bash -c "/DEBIAN/preinst install"
test -e {0}/DEBIAN/postinst && chroot {0} bash -c "/DEBIAN/postinst"
rm -rf {0}/DEBIAN
fi
popd >/dev/null || exit 1
done < {1}
''')

script.format(self.root_dir, package_names.name)

with open(package_extract.name, 'w') as install:
install.write(
script.format(self.root_dir, package_names.name)
# Extract bootstrap packages
with open(package_names.name) as packages:
solved_debootstrap_packages = [p.rstrip() for p in packages]
self.command_env['PATH'] = '$PATH:/usr/bin:/bin:/usr/sbin:/sbin'
for package in solved_debootstrap_packages:
Command.run(
[
'bash', '-c',
'dpkg-deb --fsys-tarfile {0} | tar -C {1} -x'.format(
Copy link
Collaborator

@IsaacJT IsaacJT Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a specific reason you're using tar here? You can use dpkg -e <.deb path> to extract the scripts from a .deb

Copy link
Collaborator Author

@schaefi schaefi Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This extraction is not for the scripts. This extraction "installs" the downloaded bootstrap .deb package into the new root-tree. From the man page of dpkg-deb

--fsys-tarfile archive
           Extracts the filesystem tree data from a binary package and
           sends it to standard output in tar format.  Together with
           tar(1) this can be used to extract a particular file from a
           package archive.

So this call is not related to the script part of things, which comes later.

Copy link
Collaborator Author

@schaefi schaefi Sep 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the process to bootstrap a debian based system follows the following logic

  1. apt-get _with_call_options_to_resolve_packages_and_only_download_them
  2. dpkg-deb --fsys-tarfile to extract (install) the set of resolved and downloaded packages
  3. dpkg -e extract the scripts
  4. run the scripts (chrooted)

Copy link
Collaborator

@IsaacJT IsaacJT Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. You could instead just use dpkg -x <.deb> <dest> to skip the intermediate tar stage

package, self.root_dir
)
], self.command_env
)
result = Command.run(
['bash', package_extract.name], self.command_env
# Run package scripts. Unfortuantely Debian based systems
# requires special sauce for bootstrap. See the exceptions
# we have to apply below:
#
# * manual order is required to make sure users(root) exists
# * the usr-merge strategy only works after bootstrap
#
# 1. Pass: Run package scripts, manual order
self._run_bootstrap_scripts(
solved_debootstrap_packages,
only_for=['base-passwd'], skip=['usrmerge']
)
log.debug(
'Apt extract: {0} {1}'.format(result.output, result.error)
# 2. Pass: Run package scripts in apt order
self._run_bootstrap_scripts(
solved_debootstrap_packages,
skip=['usrmerge']
)
self.cleanup_requests()
return Command.call(
Expand Down Expand Up @@ -513,3 +517,57 @@ def _package_requests(self) -> List:
items = self.package_requests[:]
self.cleanup_requests()
return items

def _run_bootstrap_scripts(
self, solved_debootstrap_packages: List[str],
only_for: List[str] = [], skip: List[str] = []
):
# TODO: this should not be needed but without setting
# the following environment variables no package pre/post
# script completes its task. I leave it up to the Debian
# experts to provide a fix if needed.
self.command_env['DPKG_MAINTSCRIPT_NAME'] = 'true'
self.command_env['DPKG_MAINTSCRIPT_PACKAGE'] = 'libc6'
Copy link
Collaborator Author

@schaefi schaefi Sep 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@IsaacJT As we discussed I created a refactor and implementation for the apt based bootstrap such that all package scripts from the resolved apt list are taken into account.

At this point however I needed two weird environment variables and I had to set libc6 as DPKG_MAINTSCRIPT_PACKAGE otherwise I got usr-merge complains. I really have no idea what these variables really do and my knowledge on the inner works of Debian/Ubuntu packaging is really bad.

I wanted to ask if you can give this PR a try if it would also fix the issue you found and maybe check if the above is ok as a permanent in code setup or if this is all wrong. I believe the way how I set the variables is not a good way to handle the issue that comes up if you disable them. Feel free to run the process without these variables so you can see the errors from apt.

Thanks much

Copy link
Collaborator

@IsaacJT IsaacJT Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK DPKG_MAINTSCRIPT_NAME should be the name of the script (e.g. postinst, etc.) and DPKG_MAINTSCRIPT_PACKAGE should be the name of the package that the scripts belong to. I would have to double check this though...

Haven't had time to a have a proper look through yet.

Copy link
Collaborator Author

@schaefi schaefi Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok

Copy link
Collaborator Author

@schaefi schaefi Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we run the scripts manually the setting of just true would then be ok ?

Copy link
Collaborator Author

@schaefi schaefi Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also note that libc6 was the only package in the collection of all packages that complains about the missing variables DPKG_MAINTSCRIPT_*

Copy link
Collaborator Author

@schaefi schaefi Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I simply do not understand the concept behind them

Copy link
Collaborator Author

@schaefi schaefi Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the comment for this variable settings. I could not find any bad or negative side effect of setting them during the bootstrap of a Debian based system. I tested Ubuntu and Debian and the integration tests pass and boots. Thus I leave it up to the experts if this needs to be changed or can harm the bootstrap procedure.

Copy link
Collaborator

@IsaacJT IsaacJT Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe DPKG_MAINTSCRIPT_PACKAGE should be set in the for block below so that it represents the name of the package that the script is currently being run for? And DPKG_MAINTSCRIPT_NAME could be set
to the script name immediately before calling the script, e.g.

if os.path.exists(script_pre):
    self.command_env['DPKG_MAINTSCRIPT_PACKAGE'] = 'preinst'
    ...

I think this would more closely emulate the environment that the scripts would normally be run in


post_script_dir = Temporary(
prefix='kiwi_debpost.', path=self.root_dir
).new_dir()
for package in solved_debootstrap_packages:
package_base_name = os.path.basename(package)
go_ahead = False if only_for else True
for name in only_for:
if name in package_base_name:
go_ahead = True
break
for name in skip:
if name in package_base_name:
go_ahead = False
break
if not go_ahead:
continue
log.debug(
f'Running pre/post scripts for: {package_base_name}'
)
package_metadata_dir = \
f'{post_script_dir.name}/{os.path.basename(package)}'
Command.run(
['dpkg', '-e', package, package_metadata_dir]
)
script_pre = f'{package_metadata_dir}/preinst'
script_post = f'{package_metadata_dir}/postinst'
# 1. preinst
if os.path.exists(script_pre):
Command.run(
[
'chroot', self.root_dir, 'bash',
f'{script_pre.replace(self.root_dir, "")}', 'install'
], self.command_env
)
# 2. postinst
if os.path.exists(script_post):
Command.run(
[
'chroot', self.root_dir, 'bash',
f'{script_post.replace(self.root_dir, "")}', 'configure'
], self.command_env
)
Loading
Loading