parser: Add KTAP parser for handling kernel selftests #6597

schlad · 2025-07-21T09:41:08Z

Progress: https://progress.opensuse.org/issues/182360
VR: https://rmarliere-openqa.qe.prg2.suse.org/tests/1332

This patch introduces a dedicated parser for Linux kernel selftests using the KTAP (Kernel Test Anything Protocol) format.

While KTAP is based on the TAP (Test Anything Protocol) standard, it includes significant extensions tailored for kernel testing. This parser builds on TAP::Parser to handle common TAP elements while providing additional logic to interpret KTAP-specific constructs such as grouped subtests and prefixed comment lines.

For more details on KTAP, see: https://docs.kernel.org/dev-tools/ktap.html

Example output of the ktap test run:

  TAP version 13
  1..13
  # overriding timeout to 300
  # selftests: cgroup: test_core
  # ok 1 test_cgcore_internal_process_constraint
  # ok 2 test_cgcore_top_down_constraint_enable
  # ok 3 test_cgcore_top_down_constraint_disable
  # ok 4 test_cgcore_no_internal_process_constraint_on_threads
  # ok 5 test_cgcore_parent_becomes_threaded
  # ok 6 test_cgcore_invalid_domain
  # ok 7 test_cgcore_populated
  # ok 8 test_cgcore_proc_migration
  # ok 9 test_cgcore_thread_migration
  # ok 10 test_cgcore_destroy
  # ok 11 test_cgcore_lesser_euid_open
  # ok 12 test_cgcore_lesser_ns_open
  ok 1 selftests: cgroup: test_core

There are group of tests with nested subtests. Each group starts as follow:

  # selftests: cgroup: test_cpu

After that individual tests are listed with results ok | not ok:

  # ok 1 test_cgcore_internal_process_constraint

however such subtests are prepended with # which - from the TAP specification standpoint - are just comments. Moreover KTAP allows for more comments like:

  # overriding timeout to 300

Each group of tests ends with:

  ok 2 selftests: cgroup: test_cpu

or

  not ok 10 selftests: cgroup: test_zswap # exit=1

The KTAP parser correctly identifies these groups and subtests, associates results appropriately, and exports them in a structured format compatible with OpenQA.

perlpunk

Could you add a test in t/30-test_parser.t along with an example output?
If you have trouble with the test, let us know, but an example output would be helpful

schlad · 2025-07-21T10:20:36Z

Could you add a test in t/30-test_parser.t along with an example output? If you have trouble with the test, let us know, but an example output would be helpful

absolutely, I will try to add such test and if I find this difficult I will use your offer. Thanks for a quick check! I appreciate it

lib/OpenQA/Parser/Format/KTAP.pm

codecov · 2025-07-21T10:28:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.12%. Comparing base (606fa3c) to head (7af7bb3).
Report is 19 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6597   +/-   ##
=======================================
  Coverage   99.11%   99.12%           
=======================================
  Files         399      400    +1     
  Lines       40723    40809   +86     
=======================================
+ Hits        40364    40450   +86     
  Misses        359      359

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lib/OpenQA/Parser/Format/KTAP.pm

Martchus

I only have some style nitpicks and would also suggest to use signatures.

We probably also want 100 % test coverage for newly introduced code so unit tests need to be extended.

lib/OpenQA/Parser/Format/KTAP.pm

Martchus · 2025-07-21T11:42:54Z

Looks like the CI check for patch coverage is passing. This is probably because this change introduces a completely new file which Codecov doesn't seem to handle, e.g. the new file is not present in the listing on https://app.codecov.io/gh/os-autoinst/openQA/pull/6597/tree/lib/OpenQA/Parser/Format?dropdown=coverage. (Making the URL manually leads to https://app.codecov.io/gh/os-autoinst/openQA/pull/6597/blob/lib/OpenQA/Parser/Format/KTAP.pm?dropdown=coverage showing no coverage, though.)

schlad · 2025-07-21T11:48:46Z

Could you add a test in t/30-test_parser.t along with an example output? If you have trouble with the test, let us know, but an example output would be helpful

sub ktap_parse added. I admit its the first time I write unit tests in perl let alone openQA project :-)

This looks ok:
prove -l t/30-test_parser.t t/30-test_parser.t .. ok All tests successful. Files=1, Tests=17, 1 wallclock secs ( 0.10 usr 0.02 sys + 1.04 cusr 0.20 csys = 1.36 CPU) Result: PASS

and if I assert against not expected values I get a proper error, so I assume this is OKish. But certainly I would appreciate your review and scrutiny. Thx

schlad · 2025-07-21T11:51:17Z

I only have some style nitpicks and would also suggest to use signatures.

We probably also want 100 % test coverage for newly introduced code so unit tests need to be extended.

ah 100% I see; I added some test coverage, let me know if this is sufficient or I shall add more.

schlad · 2025-07-21T11:53:11Z

thx @okurz for detailed review. I will wait to see if the unit tests are considered good and fix those first if needed and then will start to address your comments taking the advantage of the newly introduced unit tests.

schlad · 2025-07-21T11:58:25Z

Looks like the CI check for patch coverage is passing. This is probably because this change introduces a completely new file which Codecov doesn't seem to handle, e.g. the new file is not present in the listing on https://app.codecov.io/gh/os-autoinst/openQA/pull/6597/tree/lib/OpenQA/Parser/Format?dropdown=coverage. (Making the URL manually leads to https://app.codecov.io/gh/os-autoinst/openQA/pull/6597/blob/lib/OpenQA/Parser/Format/KTAP.pm?dropdown=coverage showing no coverage, though.)

I would guess this is something what I could, and probably should, "fix" in my PR, so the KTAP.pm would be also checked.

Martchus · 2025-07-21T12:10:24Z

You've already fixed that as far as your PR is concerned by having some tests at all for the newly introduced file. This is enough for the file to be considered by codecov. You can now use the report to see what code paths still need to be covered.

(Whether we want to improve the previous situation where a newly introduced file without any tests still gets a passing coverage CI check is out of scope for this PR. Maybe this is a limitation of codecov we cannot do much about anyway.)

lib/OpenQA/Parser/Format/KTAP.pm

okurz

LGTM. This will be merged automatically as soon as the patch coverage reports 100%. Maybe codecov later re-evaluates and this fixes itself. Otherwise I wouldn't rush it and wait for any additional idea from any reviewer to find out what's missing here.

schlad · 2025-07-21T17:14:37Z

LGTM. This will be merged automatically as soon as the patch coverage reports 100%. Maybe codecov later re-evaluates and this fixes itself. Otherwise I wouldn't rush it and wait for any additional idea from any reviewer to find out what's missing here.

FYI: I'm learning more about the unit tests in perl, so I'm pushing some other tests to get the 100% coverage.

Thx for the review @okurz

lib/OpenQA/Parser/Format/KTAP.pm

This patch introduces a dedicated parser for Linux kernel selftests using the KTAP (Kernel Test Anything Protocol) format. While KTAP is based on the TAP (Test Anything Protocol) standard, it includes significant extensions tailored for kernel testing. This parser builds on TAP::Parser to handle common TAP elements while providing additional logic to interpret KTAP-specific constructs such as grouped subtests and prefixed comment lines. For more details on KTAP, see: https://docs.kernel.org/dev-tools/ktap.html Example output of the ktap test run: TAP version 13 1..13 # overriding timeout to 300 # selftests: cgroup: test_core # ok 1 test_cgcore_internal_process_constraint # ok 2 test_cgcore_top_down_constraint_enable # ok 3 test_cgcore_top_down_constraint_disable # ok 4 test_cgcore_no_internal_process_constraint_on_threads # ok 5 test_cgcore_parent_becomes_threaded # ok 6 test_cgcore_invalid_domain # ok 7 test_cgcore_populated # ok 8 test_cgcore_proc_migration # ok 9 test_cgcore_thread_migration # ok 10 test_cgcore_destroy # ok 11 test_cgcore_lesser_euid_open # ok 12 test_cgcore_lesser_ns_open ok 1 selftests: cgroup: test_core There are group of tests with nested subtests. Each group starts as follow: # selftests: cgroup: test_cpu After that individual tests are listed with results ok | not ok: # ok 1 test_cgcore_internal_process_constraint however such subtests are prepended with # which - from the TAP specification standpoint - are just comments. Moreover KTAP allows for more comments like: # overriding timeout to 300 Each group of tests ends with: ok 2 selftests: cgroup: test_cpu or not ok 10 selftests: cgroup: test_zswap # exit=1 The KTAP parser correctly identifies these groups and subtests, associates results appropriately, and exports them in a structured format compatible with OpenQA.

perlpunk · 2025-07-21T17:52:21Z

lib/OpenQA/Parser/Format/KTAP.pm

+
+has [qw(test)];
+
+sub _testgroup_init ($self, $line, $test_ref, $steps_ref, $m_ref) {


I am wondering if instead of having to pass all those references, they can be made members of $self

good point. I will try to do that.

huh really merged instantly once all checks are green.

I will not take it easy @perlpunk and will try it regardless.

Two files with examplary output are added, one with version and plan and the other one without version. Examples contain the group of tests with pass, fail and skip statuses

schlad force-pushed the KTAP_parser_kernel_selftests branch 3 times, most recently from d3ce481 to e2f36ba Compare July 21, 2025 10:10

perlpunk reviewed Jul 21, 2025

View reviewed changes

okurz reviewed Jul 21, 2025

View reviewed changes

lib/OpenQA/Parser/Format/KTAP.pm Outdated Show resolved Hide resolved

schlad mentioned this pull request Jul 21, 2025

kselftests: Use KTAP parser os-autoinst/os-autoinst-distri-opensuse#22701

Open

schlad force-pushed the KTAP_parser_kernel_selftests branch from e2f36ba to fb69c04 Compare July 21, 2025 10:39

okurz requested changes Jul 21, 2025

View reviewed changes

Martchus requested changes Jul 21, 2025

View reviewed changes

lib/OpenQA/Parser/Format/KTAP.pm Outdated Show resolved Hide resolved

schlad requested a review from perlpunk July 21, 2025 11:48

schlad force-pushed the KTAP_parser_kernel_selftests branch 2 times, most recently from 63923d6 to 9c935f5 Compare July 21, 2025 13:11

b10n1k reviewed Jul 21, 2025

View reviewed changes

lib/OpenQA/Parser/Format/KTAP.pm Show resolved Hide resolved

schlad force-pushed the KTAP_parser_kernel_selftests branch 4 times, most recently from 15cb7ae to 3ddcc08 Compare July 21, 2025 14:18

Martchus approved these changes Jul 21, 2025

View reviewed changes

schlad requested a review from okurz July 21, 2025 14:36

okurz approved these changes Jul 21, 2025

View reviewed changes

perlpunk reviewed Jul 21, 2025

View reviewed changes

lib/OpenQA/Parser/Format/KTAP.pm Outdated Show resolved Hide resolved

schlad force-pushed the KTAP_parser_kernel_selftests branch from 3e4c084 to 0ec6538 Compare July 21, 2025 17:37

perlpunk reviewed Jul 21, 2025

View reviewed changes

unit_tests: Add unit tests for KTAP parser

7af7bb3

Two files with examplary output are added, one with version and plan and the other one without version. Examples contain the group of tests with pass, fail and skip statuses

schlad force-pushed the KTAP_parser_kernel_selftests branch from 0ec6538 to 7af7bb3 Compare July 21, 2025 18:01

b10n1k approved these changes Jul 21, 2025

View reviewed changes

mergify bot merged commit b59d7bf into os-autoinst:master Jul 21, 2025
51 checks passed


		has [qw(test)];

		sub _testgroup_init ($self, $line, $test_ref, $steps_ref, $m_ref) {

parser: Add KTAP parser for handling kernel selftests #6597

parser: Add KTAP parser for handling kernel selftests #6597

Conversation

schlad commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

perlpunk left a comment

Choose a reason for hiding this comment

Uh oh!

schlad commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Martchus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Martchus commented Jul 21, 2025

Uh oh!

schlad commented Jul 21, 2025

Uh oh!

schlad commented Jul 21, 2025

Uh oh!

schlad commented Jul 21, 2025

Uh oh!

schlad commented Jul 21, 2025

Uh oh!

Martchus commented Jul 21, 2025

Uh oh!

Uh oh!

okurz left a comment

Choose a reason for hiding this comment

Uh oh!

schlad commented Jul 21, 2025

Uh oh!

Uh oh!

perlpunk Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

schlad Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

schlad Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

schlad commented Jul 21, 2025 •

edited

Loading

schlad commented Jul 21, 2025 •

edited

Loading

codecov bot commented Jul 21, 2025 •

edited

Loading