Add tool to compare built ipa and master ipa initramfs #1058

peppi-lotta · 2025-07-29T12:06:56Z

This PR introduces a script ipa_debug_tools.sh that provides a set of utilities to assist with debugging and comparing IPA (ironic-python-agent) initramfs builds. The script is particularly useful for identifying differences between locally built IPA images and the official master images.

This tool is meant especially for situations where built IPA size get over a certain limit limit and two file systems need to be compared.

🛠️ What's Included

set_up_debug_dirs
- Sets up a local debug environment under /tmp/debug-initramfs
- Builds the local IPA using jenkins/scripts/dynamic_worker_workflow/build_ipa.sh
- Downloads and extracts the master IPA image from upstream
compare_dir_sizes [path]
- Compares sizes of files/directories between the local build and master IPA
- Provides a colored, sorted table of size differences
- Highlights large discrepancies (10MB+, 1MB+, etc.)
- Lists files exclusive to either initramfs tree

compare_rpm_packages
- Extracts and compares installed RPMs between local and master IPA environments
- Outputs side-by-side view of mismatched or missing packages
- Helps identify unintended package changes

clean_up_debug_dirs
- Deletes all temporary directories and files created during debugging

Rozzii

In general LGTM,
It is a very useful tool as I have mentioned offline.
I have one improvement proposal but it can be a later addition if you don't feel like implementing it now.

Rozzii · 2025-07-31T06:15:14Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


It would be nice if the upstream IPA URI and tar file name could be customized same as in IPA downloader, you could actually use IPA downloader here instead of your own download logic to keep it simple .

For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

I am fine either way, but the url and the name of the tar file has to be customizable because otherwise only a specific master on a specific base image can be tested. I suggested to use the downloader because that has all the customization options implemented but I am fine with doing it with the one liners too.

Rozzii · 2025-07-31T06:22:40Z

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

metal3-io-bot · 2025-07-31T06:22:45Z

@Rozzii: Overrode contexts on behalf of Rozzii: metal3-ubuntu-e2e-integration-test-main

Details

In response to this:

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tuminoid

This is great addition to debug tools, thanks @peppi-lotta.

Some cleanup to do, some minor syntax things to fix and some suggestions to make it more readable and maintainable.

hack/ipa_debug_tools.sh

tuminoid · 2025-08-05T06:19:00Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

hack/ipa_debug_tools.sh

Rozzii · 2025-09-29T17:56:01Z

hack/ipa_debug_tools.sh

+# -----------------------------------------------------------------------------
+clean_up_debug_dirs() {
+    # Clean up debug directories
+    sudo rm -rf "${BASE_DEBUG_DIR}" /tmp/dib ipa-file-injector.service 2> /dev/null


I get "${BASE_DEBUG_DIR}" and /tmp/dib but how does ipa-file-injector.service end up here?
I don't see any other reference to that file, and AFAIK that is the systemd unit file of the similarly named DIB module. It could be that I am missing something here but looks like this is some leftover reference maybe.

ipa-file-injector.service appears in the /tmp directory after running the build-ip.sh. That is why I've added that to the clean up as well.

okay, that might be then some sort of residue from the template rendering process. I might remove it later, but then it is fine to remove it here.

tuminoid

IMO colors are fine, but emojis not.

hack/ipa_debug_tools.sh

Rozzii

/lgtm

peppi-lotta · 2025-10-20T07:04:53Z

@tuminoid I've addressed all comment can you approve and override test?

tuminoid · 2025-10-20T07:39:00Z

It doesn't work?

2025-10-20 07:35:48.405 | + diskimage_builder/lib/common-functions:tmpfs_check:29                            :   return 0
2025-10-20 07:35:48.409 | + diskimage_builder/lib/common-functions:cleanup_image_dir:236                     :   timeout 120 sh -c 'while ! sudo umount -f /tmp/dib_image.Jc5gaKZ3; do sleep 1; done'
2025-10-20 07:35:48.426 | + diskimage_builder/lib/common-functions:cleanup_image_dir:241                     :   rm -rf --one-file-system /tmp/dib_image.Jc5gaKZ3
2025-10-20 07:35:48.431 | + diskimage_builder/lib/img-functions:trap_cleanup:37                              :   exit 1
Command '['disk-image-create', '-o', 'ironic-python-agent', 'ironic-python-agent-ramdisk', 'centos', 'dynamic-login', 'journal-to-console', 'devuser', 'openssh-server', 'extra-hardware', 'ipa-module-autoload', 'ipa-add-buildinfo', 'ipa-cleanup-dracut', 'simple-init', 'override-simple-init', 'ipa-file-injector', 'cleanup-package', '-x']' returned non-zero exit status 1.
build_ipa.sh failed, will check for artifact...
Required file '/tmp/dib/ironic-python-agent.initramfs' not created, exiting.

Running the test with using default options (centos10 master, opendev dib url).

tuminoid

Not exactly fond of that it just asks for sudo password and then goes and installs stuff on the system. Installing anything should be another "subcommand" that user must run explicitly.

The sudo usage in general is also very liberal (please don't run sudo wget and sudo tar .... Making a temporary directory with sudo is not typical either.

Sudo chroot also is not needed.

hack/ipa_debug_tools.sh

metal3-io-bot · 2025-12-10T12:31:56Z

New changes are detected. LGTM label has been removed.

Copilot

Pull request overview

This PR adds a comprehensive debugging tool ipa_debug_tools.sh to help developers compare locally built IPA (ironic-python-agent) initramfs images against official master images, particularly useful when troubleshooting size limit issues.

Key changes:

New shell script with four main functions for IPA initramfs comparison and debugging
Automated environment setup that builds local IPA and downloads master image
Visual comparison utilities with color-coded output for size differences and RPM package variations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-12T06:58:08Z

hack/ipa_debug_tools.sh

+
+# set_up_debug_dirs [file] [url]
+#
+# This ceates a minimal set up in /tmp/debug-initramfs directory for


Corrected spelling of 'ceates' to 'creates'.

Suggested change

# This ceates a minimal set up in /tmp/debug-initramfs directory for

# This creates a minimal set up in /tmp/debug-initramfs directory for

Copilot · 2025-12-12T06:58:08Z

hack/ipa_debug_tools.sh

+#   ./ipa_debug_tools.sh set_up_debug_dirs ipa-centos10-master.tar.gz
+#   ./ipa_debug_tools.sh set_up_debug_dirs ipa-centos10-master.tar.gz https://tarballs.opendev.org/openstack/ironic-python-agent/dib/
+#
+# - [file]: Name of tar file o be downloaded [default: ipa-centos10-master.tar.gz]


Corrected spelling of 'o' to 'to'.

Suggested change

# - [file]: Name of tar file o be downloaded [default: ipa-centos10-master.tar.gz]

# - [file]: Name of tar file to be downloaded [default: ipa-centos10-master.tar.gz]

Copilot · 2025-12-12T06:58:08Z

hack/ipa_debug_tools.sh

+#   ./ipa_debug_tools.sh set_up_debug_dirs ipa-centos10-master.tar.gz
+#   ./ipa_debug_tools.sh set_up_debug_dirs ipa-centos10-master.tar.gz https://tarballs.opendev.org/openstack/ironic-python-agent/dib/


The documentation examples reference 'ipa-centos10-master.tar.gz', but line 27 uses 'ipa-centos9-master.tar.gz' as the default value. This inconsistency could confuse users. Consider either updating the examples to match the default or updating the default to match the examples.

Copilot · 2025-12-12T06:58:09Z

hack/ipa_debug_tools.sh

+    export TEST_IN_CI="false"
+
+    # Build ipa with build_ipa.sh. Search for the script in a path that is
+    # relative to this files location. 


Corrected spelling of 'files' to 'file's'.

Suggested change

# relative to this files location.

# relative to this file's location.

Copilot · 2025-12-12T06:58:09Z

hack/ipa_debug_tools.sh

+
+# After debugging and comparison operations, temporary directories can be cleaned up.
+# This function removes the build and master filesystems. If this operation is run
+# the set_up_debug_dirs needs to be run again to create the compareble filesystems.


Corrected spelling of 'compareble' to 'comparable'.

Suggested change

# the set_up_debug_dirs needs to be run again to create the compareble filesystems.

# the set_up_debug_dirs needs to be run again to create the comparable filesystems.

Copilot · 2025-12-12T06:58:09Z

hack/ipa_debug_tools.sh

+}
+
+# Print a rpm package list. The print omits packages that are identical in both
+#the built ipa and master.


Missing space after comment marker '#'. Add a space for consistency with other comments in the file.

Suggested change

#the built ipa and master.

# the built ipa and master.

tuminoid

Just fix the typo nits by the bot.

/approve
/hold

metal3-io-bot · 2025-12-12T07:21:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tuminoid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [tuminoid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: peppi-lotta <[email protected]>

tuminoid · 2025-12-16T14:20:28Z

/override metal3-ubuntu-e2e-integration-test-main
/cc @Rozzii

metal3-io-bot · 2025-12-16T14:20:32Z

@tuminoid: Overrode contexts on behalf of tuminoid: metal3-ubuntu-e2e-integration-test-main

Details

In response to this:

/override metal3-ubuntu-e2e-integration-test-main
/cc @Rozzii

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

metal3-io-bot requested review from Rozzii and smoshiur1237 July 29, 2025 12:07

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 29, 2025

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch 2 times, most recently from 4c2b687 to 9e9d4b4 Compare July 30, 2025 06:15

Rozzii reviewed Jul 31, 2025

View reviewed changes

Rozzii added this to Metal3 - Roadmap Jul 31, 2025

Rozzii moved this to MISC WIP in Metal3 - Roadmap Jul 31, 2025

tuminoid reviewed Aug 5, 2025

View reviewed changes

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch 2 times, most recently from 00054eb to 7a7af56 Compare September 29, 2025 09:51

Rozzii reviewed Sep 29, 2025

View reviewed changes

tuminoid reviewed Sep 30, 2025

View reviewed changes

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 7a7af56 to 27494cb Compare October 3, 2025 07:16

Rozzii reviewed Oct 7, 2025

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 7, 2025

tuminoid reviewed Oct 20, 2025

View reviewed changes

hack/ipa_debug_tools.sh Show resolved Hide resolved

hack/ipa_debug_tools.sh Show resolved Hide resolved

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 27494cb to 7aa7038 Compare December 10, 2025 12:31

metal3-io-bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 10, 2025

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 7aa7038 to aba76fb Compare December 10, 2025 12:36

tuminoid requested a review from Copilot December 12, 2025 06:54

Copilot started reviewing on behalf of tuminoid December 12, 2025 06:54 View session

Copilot AI reviewed Dec 12, 2025

View reviewed changes

tuminoid reviewed Dec 12, 2025

View reviewed changes

metal3-io-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 12, 2025

metal3-io-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 12, 2025

Add tool to compare built ipa and master ipa initramfs

46501da

Signed-off-by: peppi-lotta <[email protected]>

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from aba76fb to 46501da Compare December 16, 2025 14:04

metal3-io-bot requested a review from Rozzii December 16, 2025 14:20

	# This ceates a minimal set up in /tmp/debug-initramfs directory for
	# This creates a minimal set up in /tmp/debug-initramfs directory for

	# - [file]: Name of tar file o be downloaded [default: ipa-centos10-master.tar.gz]
	# - [file]: Name of tar file to be downloaded [default: ipa-centos10-master.tar.gz]

		# ./ipa_debug_tools.sh set_up_debug_dirs ipa-centos10-master.tar.gz
		# ./ipa_debug_tools.sh set_up_debug_dirs ipa-centos10-master.tar.gz https://tarballs.opendev.org/openstack/ironic-python-agent/dib/

	# relative to this files location.
	# relative to this file's location.

	# the set_up_debug_dirs needs to be run again to create the compareble filesystems.
	# the set_up_debug_dirs needs to be run again to create the comparable filesystems.

Add tool to compare built ipa and master ipa initramfs #1058

Are you sure you want to change the base?

Add tool to compare built ipa and master ipa initramfs #1058

Uh oh!

Conversation

peppi-lotta commented Jul 29, 2025

🛠️ What's Included

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

Rozzii Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuminoid Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii commented Jul 31, 2025

Uh oh!

metal3-io-bot commented Jul 31, 2025

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tuminoid Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rozzii Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peppi-lotta Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

Rozzii Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

peppi-lotta commented Oct 20, 2025

Uh oh!

tuminoid commented Oct 20, 2025

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

metal3-io-bot commented Dec 10, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Rozzii Jul 31, 2025 •

edited

Loading

Rozzii Sep 29, 2025 •

edited

Loading