Document how to configure kdump on CoreOS #28164

kelvinfan001 · 2020-12-16T17:08:04Z

RHCOS 4.7 includes kexec-tools (required for kdump) so
investigating kernel crashes through kdump is now supported.

travier

Thanks, this looks good.

As we expect administrators to use MCs instead of relying on SSH for node setup, I'm wondering if we should first provide a full example using a MachineConfig with a filter to select only one or some of the nodes and keep the manual setup instructions in a second section.

See the docs for kargs changes for MC examples.

cgwalters · 2020-12-17T17:07:58Z

As we expect administrators to use MCs instead of relying on SSH for node setup,

Yes...but I think it will be common for admins to want to enable kdump on just one node (or just a subset), and we don't support machine specific machineconfigs yet.

I think this is probably OK for now; an admin who actually wants to enable kdump on multiple nodes could indeed use a MachineConfig, and it'd probably be worth at least mentioning that.

kelvinfan001 · 2020-12-17T18:30:36Z

an admin who actually wants to enable kdump on multiple nodes could indeed use a MachineConfig, and it'd probably be worth at least mentioning that.

I'll mention this, but since it's unlikely that admins would want to enable kdump on all nodes, I'll omit the fulll example of using MachineConfigs for now, especially since there will soon be better kdump support through FCC.

travier · 2020-12-18T14:05:38Z

This is planned for 4.7.

cynepco3hahue · 2020-12-30T15:22:41Z

I tried to enable it and got an error:

Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com systemd[1]: Starting Crash recovery kernel arming...
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com kdumpctl[59248]: No kdump initial ramdisk found.
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com kdumpctl[59248]: Rebuilding /boot(hd0,gpt3)/ostree/rhcos-912fd9f507a1f7eb885c1c86689d8df3a72d383dcaada7254789a43fe1d7be87/initramfs-4.18.0-240.8.1.el8_3.x86_64kdump.img
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com kdumpctl[59248]: /boot(hd0,gpt3)/ostree/rhcos-912fd9f507a1f7eb885c1c86689d8df3a72d383dcaada7254789a43fe1d7be87 does not have write permission. Can not rebuild /boot(hd0,gpt3)/ostree/rhcos-912fd9f507a1f7eb885c1c86689d8df3a72d383dcaada7254789a43fe1d7be87/initramfs-4.18.0-240.8.1.el8_3.x86_64kdump.img
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com kdumpctl[59248]: Starting kdump: [FAILED]
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com systemd[1]: kdump.service: Main process exited, code=exited, status=1/FAILURE
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com systemd[1]: kdump.service: Failed with result 'exit-code'.
Dec 30 13:55:08 cnfdd3.clus2.t5g.lab.eng.bos.redhat.com systemd[1]: Failed to start Crash recovery kernel arming.

kelvinfan001 · 2021-01-04T18:48:22Z

@cynepco3hahue Did you set the KDUMP_BOOTDIR variable in /etc/kdump.conf manually?

BOOT_LOC=/boot$(cat /proc/cmdline | egrep -o "/ostree/.*/vmlinuz" | sed -e "s|/vmlinuz||g")
sudo sed -i "s|^#KDUMP_BOOTDIR=\"/boot\"|KDUMP_BOOTDIR=\"${BOOT_LOC}\"|" /etc/sysconfig/kdump

This is required on RHCOS, currently. It will no longer be required once kexec-tools-2.0.20-35.el8 or after lands. Related BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1866611

cynepco3hahue · 2021-01-04T19:30:05Z

@kelvinfan001 Thanks for information!

travier · 2021-01-05T16:01:39Z

@cynepco3hahue Can you confirm that the documented steps work for you? Thanks

kelvinfan001 · 2021-01-05T16:25:42Z

@openshift/team-documentation

modules/investigating-kernel-crashes.adoc

bmcelvee · 2021-01-12T15:57:40Z

modules/investigating-kernel-crashes.adoc

+
+.  Ensure that `kdump` has loaded a crash kernel by checking that `kdump.service` has started and exited successfully and that `cat /sys/kernel/kexec_crash_loaded` prints `1`.
+
+=== Enabling kdump on day-1


I would consider making this a new module, especially if it ends up with additional procedures.

modules/investigating-kernel-crashes.adoc

bmcelvee · 2021-01-12T15:57:48Z

modules/investigating-kernel-crashes.adoc

+== Testing the kdump configuration
+
+ifdef::openshift-enterprise[]
+Please refer to the link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/system_design_guide/installing-and-configuring-kdump_system-design-guide#testing-the-kdump-configuration_installing-and-configuring-kdump["Testing the kdump configuration" section] over at the {op-system-base} documentation for `kdump`.


The following apply throughout:

s/Please refer to the/See

s/over at the/in the

No quotations needed for titles

support/troubleshooting/troubleshooting-operating-system-issues.adoc

bmcelvee · 2021-01-12T16:06:38Z

@kelvinfan001 thank you, this is looking great! I did a first pass review and left some suggestions. Also tagging @bobfuru since he works directly with CoreOS content.

bobfuru · 2021-01-12T17:59:56Z

Added a few more comments and agree with suggestions from @bmcelvee. This is great work, @kelvinfan001 - thank you!

kelvinfan001 · 2021-01-14T05:30:05Z

Thank you @bmcelvee and @bobfuru for the review! I've made the suggested changes.

bobfuru · 2021-01-14T16:23:27Z

modules/investigating-kernel-crashes.adoc

+.  Ensure that `kdump` has loaded a crash kernel by checking that `kdump.service` has started and exited successfully and that `cat /sys/kernel/kexec_crash_loaded` prints `1`.
+
+== Enabling kdump on day-1
+`kdump` is intended to be enabled per-node to debug kernel problems. It is not recommended to enable `kdump` on all of your nodes in the cluster. Although machine-specific `MachineConfigs` are not yet supported, it is possible to do the above through a systemd unit in a `MachineConfig` object on day-1 and have kdump enabled on all nodes in the cluster. You can create a `MachineConfig` object and inject that object into the set of manifest files used by Ignition during cluster setup. See "Customizing nodes" in the _Installing -> Installation configuration_ section for more information and examples on how to use Ignition configs.


A few suggestions in this paragraph:

So that the sentence doesn't start lowercase: s/kdump is intended/The kdump service is intended/

s/Although machine-specific MachineConfigs/Although machine-specific machine configs/ (do not pluralize an object ref, according to docs guidelines)

s/it is possible to do the above through a systemd unit/you can perform the previous step through a systemd unit/

bobfuru · 2021-01-14T16:25:47Z

modules/investigating-kernel-crashes.adoc

+
+[source, terminal]
+----
+sudo rpm-ostree kargs --append='crashkernel=256M'


Add $ prompts at the beginning of terminal commands (or # as root)

To reserve memory for the crash kernel during the first kernel booting, provide kernel arguments by entering the following command:

bobfuru · 2021-01-14T16:27:05Z

modules/investigating-kernel-crashes.adoc

+
+.Procedure
+
+The following steps are needed to enable `kdump` on {op-system}.


Let's make this active voice. s/The following steps are needed to enable kdump on {op-system}./Perform the following steps to enable kdump on {op-system}:/

bobfuru · 2021-01-14T16:29:55Z

modules/investigating-kernel-crashes.adoc

+sudo rpm-ostree kargs --append='crashkernel=256M'
+----
+
+.  By default, the path in which the vmcore will be saved is `/var/crash`. It is also possible to write the dump over the network or to some other location on the local system by editing `/etc/kdump.conf`. For example, assuming `/var/usrlocal/cores` exists, enter the following command to edit `/etc/kdump.conf` to save the vmcore to `/var/usrlocal/cores`:


Tiny nit to single-space after . for list items:
s/. By default/. By default

Ah, I wonder where I got the idea to put two spaces.

bobfuru · 2021-01-14T16:31:01Z

modules/investigating-kernel-crashes.adoc

+sed -i "s/^path.*/path \/var\/usrlocal\/cores/" /etc/kdump.conf
+----
+
+For additional information, see `kdump.conf`, a manual page for the `/etc/kdump.conf` configuration file containing the full documentation of available options, and the comments in `/etc/kdump.conf` and `/etc/sysconfig/kdump`.


Maybe s/and the comments/and note the comments/ ?

bobfuru · 2021-01-14T16:31:53Z

modules/investigating-kernel-crashes.adoc

+sudo systemctl reboot
+----
+
+.  Ensure that `kdump` has loaded a crash kernel by checking that `kdump.service` has started and exited successfully and that `cat /sys/kernel/kexec_crash_loaded` prints `1`.


s/that kdump.service/that the kdump.service

bobfuru · 2021-01-14T16:32:59Z

modules/investigating-kernel-crashes.adoc

+endif::[]
+* link:https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html[Linux kernel documentation for kdump]
+* kdump.conf(5) — a manual page for the `/etc/kdump.conf` configuration file containing the full documentation of available options
+* kexec(8) — a manual page for kexec


s/kexec/kexec

bobfuru · 2021-01-14T16:34:45Z

Thanks for the updates, @kelvinfan001 - just a few more minor nits and otherwise LGTM.

RHCOS 4.7 includes `kexec-tools` (required for `kdump`) so investigating kernel crashes through `kdump` is now supported.

kelvinfan001 · 2021-01-14T16:50:58Z

Thanks again, @bobfuru. I've updated the PR with your additional suggestions.

bobfuru · 2021-01-14T16:54:33Z

LGTM!! 👍

bobfuru · 2021-01-18T18:15:14Z

/cherrypick enterprise-4.7

openshift-cherrypick-robot · 2021-01-18T18:15:31Z

@bobfuru: new pull request created: #28657

In response to this:

/cherrypick enterprise-4.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 16, 2020

travier reviewed Dec 17, 2020

View reviewed changes

kelvinfan001 force-pushed the kfan-rhcos-kdump branch from 41f3e03 to 57a4edc Compare December 17, 2020 19:18

travier approved these changes Dec 18, 2020

View reviewed changes

travier mentioned this pull request Dec 18, 2020

OCP 4.7 Release Notes Tracker #26801

Closed

miabbott reviewed Jan 5, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Outdated Show resolved Hide resolved

kelvinfan001 force-pushed the kfan-rhcos-kdump branch from 57a4edc to 3f6848b Compare January 5, 2021 21:02

bmcelvee reviewed Jan 12, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Outdated Show resolved Hide resolved

bmcelvee reviewed Jan 12, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Outdated Show resolved Hide resolved

bmcelvee reviewed Jan 12, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Show resolved Hide resolved

bmcelvee reviewed Jan 12, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Outdated Show resolved Hide resolved

bmcelvee reviewed Jan 12, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Outdated Show resolved Hide resolved

bmcelvee reviewed Jan 12, 2021

View reviewed changes

modules/investigating-kernel-crashes.adoc Outdated Show resolved Hide resolved

bmcelvee reviewed Jan 12, 2021

View reviewed changes

support/troubleshooting/troubleshooting-operating-system-issues.adoc Show resolved Hide resolved

bmcelvee added the branch/enterprise-4.7 label Jan 12, 2021

bmcelvee added this to the Future Release milestone Jan 12, 2021

kelvinfan001 force-pushed the kfan-rhcos-kdump branch from 3f6848b to f41b230 Compare January 14, 2021 05:28

bobfuru reviewed Jan 14, 2021

View reviewed changes

kelvinfan001 force-pushed the kfan-rhcos-kdump branch from f41b230 to 71782fb Compare January 14, 2021 16:40

Document how to configure kdump on CoreOS

ad48071

RHCOS 4.7 includes `kexec-tools` (required for `kdump`) so investigating kernel crashes through `kdump` is now supported.

kelvinfan001 force-pushed the kfan-rhcos-kdump branch from 71782fb to ad48071 Compare January 14, 2021 16:43

bobfuru added the peer-review-done Signifies that the peer review team has reviewed this PR label Jan 14, 2021

bobfuru merged commit 8320005 into openshift:master Jan 18, 2021

openshift-cherrypick-robot mentioned this pull request Jan 18, 2021

[enterprise-4.7] Document how to configure kdump on CoreOS #28657

Merged

vikram-redhat modified the milestones: Future Release, OCP 4.7 GA Feb 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document how to configure kdump on CoreOS #28164

Document how to configure kdump on CoreOS #28164

kelvinfan001 commented Dec 16, 2020

travier left a comment

cgwalters commented Dec 17, 2020

kelvinfan001 commented Dec 17, 2020

travier commented Dec 18, 2020

cynepco3hahue commented Dec 30, 2020

kelvinfan001 commented Jan 4, 2021

cynepco3hahue commented Jan 4, 2021

travier commented Jan 5, 2021

kelvinfan001 commented Jan 5, 2021

bmcelvee Jan 12, 2021

bmcelvee Jan 12, 2021

bmcelvee commented Jan 12, 2021

bobfuru commented Jan 12, 2021

kelvinfan001 commented Jan 14, 2021

bobfuru Jan 14, 2021 •

edited

Loading

bobfuru Jan 14, 2021

bobfuru Jan 14, 2021

bobfuru Jan 14, 2021

bobfuru Jan 14, 2021

kelvinfan001 Jan 14, 2021

bobfuru Jan 14, 2021

bobfuru Jan 14, 2021

bobfuru Jan 14, 2021

bobfuru commented Jan 14, 2021

kelvinfan001 commented Jan 14, 2021

bobfuru commented Jan 14, 2021

bobfuru commented Jan 18, 2021 •

edited by openshift-ci-robot

Loading

openshift-cherrypick-robot commented Jan 18, 2021 •

edited by openshift-ci-robot

Loading


		. Ensure that `kdump` has loaded a crash kernel by checking that `kdump.service` has started and exited successfully and that `cat /sys/kernel/kexec_crash_loaded` prints `1`.

		=== Enabling kdump on day-1


		.Procedure

		The following steps are needed to enable `kdump` on {op-system}.

Document how to configure kdump on CoreOS #28164

Document how to configure kdump on CoreOS #28164

Conversation

kelvinfan001 commented Dec 16, 2020

travier left a comment

Choose a reason for hiding this comment

cgwalters commented Dec 17, 2020

kelvinfan001 commented Dec 17, 2020

travier commented Dec 18, 2020

cynepco3hahue commented Dec 30, 2020

kelvinfan001 commented Jan 4, 2021

cynepco3hahue commented Jan 4, 2021

travier commented Jan 5, 2021

kelvinfan001 commented Jan 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmcelvee commented Jan 12, 2021

bobfuru commented Jan 12, 2021

kelvinfan001 commented Jan 14, 2021

bobfuru Jan 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobfuru commented Jan 14, 2021

kelvinfan001 commented Jan 14, 2021

bobfuru commented Jan 14, 2021

bobfuru commented Jan 18, 2021 • edited by openshift-ci-robot Loading

openshift-cherrypick-robot commented Jan 18, 2021 • edited by openshift-ci-robot Loading

bobfuru Jan 14, 2021 •

edited

Loading

bobfuru commented Jan 18, 2021 •

edited by openshift-ci-robot

Loading

openshift-cherrypick-robot commented Jan 18, 2021 •

edited by openshift-ci-robot

Loading