Skip to content

Cherry-pick to master: [sw,cov] Add on device coverage framework #28240 #28478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Cherry-pick to master: [sw,cov] Add on device coverage framework #28240 #28478

wants to merge 13 commits into from

Conversation

sasdf
Copy link
Contributor

@sasdf sasdf commented Oct 14, 2025 •
edited
Loading

This is a manual cherrypick of:

In addition to resolving the merge conflicts, this PR introduces the following new commits for master branch compatibility:

  • [sw,cov] Enable UartConsole matching mode for coverage builds
    It renames UartConsole's uses_regex to matching_mode, which will also enable byte-by-byte matching mode under coverage builds even if no exit regex is specified.
  • [sw,cov] Disable segment allocation for coverage bss sections #28480
    This fixes the segment allocation utilized by the qemu flashgen tool.
  • [sw,cov] Clippy lint fix for coverage_lib.rs

There's no coverage test in CI currently, and it's tested manually with the following commands:

./bazelisk.sh coverage \
  --test_output=streamed \
  --cache_test_results=no \
  --config=ot_coverage \
  --test_env=VERBOSE_COVERAGE=1 \
  //sw/device/tests:uart_smoketest_fpga_cw340_rom_ext

genhtml -o /tmp/$USER/coverage \
  --ignore-errors inconsistent \
  bazel-out/_coverage/_coverage_report.dat

sasdf added 12 commits October 14, 2025 16:28
@sasdf
[sw,cov] Cleanup old coverage utils
cca4388 
Removes the legacy coverage collection utilities.

The new framework will be integrated with the `bazel coverage` subcommand.

Change-Id: I1123e9a6ea293130e6a72d95f91b7c6864228446
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit 304e6ca)
@sasdf
[sw,cov] Cleanup old coverage runtime
d4489d5 
This change removes code for the old LLVM compiler_rt based coverage runtime on
the device, which will be replaced by a in-house custom runtime developed for
OpenTitan.

Change-Id: Ic35f61ad941b64f54602a78f1aaa90a61c0aa98a
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit b1db997)
@sasdf
[bazel,cov] Add transition to disable coverage collection
a692ab0 
This change allows users to disable coverage collection for particular Bazel
targets. This can be useful for targets like metal ROM, which are sensitive
to changes in code size.

Support for ROM/ROM_EXT coverage will be added in later PRs;
to ensure the minimal framework works, ROM/ROM_EXT coverage collection is
explicitly disabled in this commit.

Change-Id: Icbfa8f64fa8a580d6f595046035a25731d5dc6af
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit f319bd6)
@sasdf
[bazel,cov] Strip initial coverage values sections
7f7a5ef 
Initial coverage values should typically be set to "uncovered."
Consequently, the array storing these initial values is not required in the
final firmware, and can be initialized programmatically.

This change introduces a check to verify that a section is entirely
"uncovered," and then removes it from the final firmware.

Change-Id: I0f4da48a1eb0ceac726d70e3843e299d3d692fa5
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit 65ae885)
@sasdf
[sw,cov] Add on-device OTTF coverage runtime
8431088 
This change adds a new on-device coverage runtime with OTTF transport.

When coverage is enabled, the runtime can collect profile counter data from the device and send it over the console.
On the host, these raw counter values can be combined with static profile data to recreate the full LLVM profile.

The coverage runtime consists of the following components:
* `api.h`: Provides a set of shared C APIs implemented by transport runtimes.
* `api_asm.h`: Provides APIs for assembly code.
* `printer.h`/`printer.c`: Implements the shared logic for different runtimes.
* Linker scripts (`bss.ld`, `info.ld`, `rodata.ld`): Define special sections for LLVM profile data, counters, and build ID.

OTTF transport implementation is also included:
* `ottf_runtime.c`: Integrates the coverage printer with the OTTF console for data output.
* `ottf_runtime_skip.c`: A no-op version of the runtime for builds where coverage is enabled but not instrumented.

Change-Id: I2e4b135a58b93a257fc37880700e23f4b99defc4
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit 7d450f8)
@sasdf
[sw,cov] Disable segment allocation for coverage bss sections
0965a6b 
The `__llvm_prf_cnts` section is stripped during conversion to `.bin` files
to save firmware size, as it's always filled with `0xff`.

However, unlike standard BSS, this section contains non-zero initial contents,
leading the compiler to mark it as `PROGBITS` and loadable. This PR modifies
the linker script to explicitly allocate this section to `NONE` segment,
preventing the `qemu flashgen` tool from loading it.

The special `NONE` segment tells the linker to not put the section in any
segment at all.

Change-Id: I917a3e97e0afcd17c3231b0cfb5ce61c870ce292
Signed-off-by: Yi-Hsuan Deng <[email protected]>
@sasdf
[sw,cov] Use new coverage runtime in OTTF tests
7e9e043 
This change integrates the new coverage runtime into the OTTF framework.

Change-Id: I507a1b40b2b76e5fe4309607ede1288a46a23835
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit ca1dee4)
@sasdf
[sw,cov] Shrink heap size under coverage mode
f0ac3f8 
Heap size is reduced in coverage mode to free up RAM for the
coverage profile counters.

Change-Id: Ifc9d1ba43be26e8a04572d47adc381e625da365f
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit 3617cba)
@sasdf
[sw,cov] Ignore LLVM coverage sections in extract_sw_logs tool
7daf319 
The LLVM sections like `__llvm_prf_cnts` contain profile counter information.
Ignore them in `extract_sw_logs` to avoid parsing errors.

Change-Id: I61a36e75aba2c1ffd22b798b169edc659be2e132
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit 3e35c44)
@sasdf
[sw,cov] Save coverage report in opentitantool UartConsole
ab2705a 
This change adds a feature to opentitantool's `UartConsole` that allows saving
device-side coverage reports.

The new implementation parses coverage data embedded in the console output,
validates it using CRC, and then saves it to a `.xprofraw` file.

This is part of Bazel coverage integration to collect reports for tests running
directly on the OpenTitan device. It follows LLVM's convention for saving to the
path specified by the `LLVM_PROFILE_FILE` environment variable, which Bazel will
set.

Additionally, a new `wait_for_coverage` function is introduced, allowing
`opentitantool` to block until coverage data is received or skipped.

Change-Id: I682655aeeb3c16228036ad58fb41187f7eb932cb
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit 550cb50)
@sasdf
[sw,cov] Enable UartConsole matching mode for coverage builds
ee6c2cb 
The `UartConsole` has a `uses_regex` method which controls whether
`UartConsole` enters a character-by-character read loop or not. The
intent for this character-by-character read loop is to avoid consuming
more output than necessary when a regular expression matches.

However, the `uses_regex` method only checks for `exit_success` and
`exit_failure` regexes. If the coverage mode is enabled, the
`UartConsole` also needs to read character-by-character to avoid
consuming output past coverage markers.

This commit renames `uses_regex` to `matching_mode` and makes it also
check for `ot_coverage_enabled` feature. This change ensures that
`UartConsole` will correctly enter a character-by-character read loop
when coverage is enabled.

Change-Id: I6544cf7e9a7b9fdf26fb8ff72fa684cf71867738
Signed-off-by: Yi-Hsuan Deng <[email protected]>
@sasdf
[bazel,cov] Add bazel coverage integration to opentitan_test
51feb63 
This change implements the `collect_cc_coverage` tool for `opentitan_test`.

The coverage collection process now works as follows:

1. A new `collect_cc_coverage` tool (written in Rust) is invoked for
   each test that produces coverage data. This tool is responsible for
   converting the compressed coverage counters (`.xprofraw`) saved by
   opentitantool's UartConsole into the standard LLVM profraw format.
   It achieves this by correlating the `.xprofraw` data with the metadata
   extracted from the ELF files of the device's firmware.

2. The `collect_cc_coverage` tool then uses llvm tools to export an LCOV
   (`.dat`) file, which is the format expected by Bazel's coverage
   reporting.

This integration allows OpenTitan's C++ code coverage to be collected
and reported consistently within the Bazel build system, making it
possible to combined with other test coverage (unit test / OTBN test).

Change-Id: I8717ec315e8bf30e0b09bdc72a9ff86857034151
Signed-off-by: Yi-Hsuan Deng <[email protected]>
(cherry picked from commit c862ac3)
@sasdf
[sw,cov] Clippy lint fix for coverage_lib.rs
58c9b40 
Change-Id: Iaa71348e6700f3a78eb65c9bd0f3170f3385dce8
Signed-off-by: Yi-Hsuan Deng <[email protected]>
@sasdf sasdf marked this pull request as ready for review October 14, 2025 14:37
@sasdf sasdf requested review from a team as code owners October 14, 2025 14:37
@nbdd0121
Copy link
Contributor

I think this way of adding new features to UartConsole is not sustainable. Any reason it cannot be a separate utility and has to be UartConsole?

@sasdf
Copy link
Contributor Author

sasdf commented Oct 14, 2025

The current system merges coverage and log data via stream multiplexing, requiring demultiplexing within UartConsole for correct exit regex matching and log suppression.

@nbdd0121 How about using a middleware plugin architecture to decouple these functions from UartConsole?
This would allow UartConsole to just handle byte reading, and pass the bytes through the middleware chain,
where dedicated middleware modules manage coverage, logging, and exit status processing.

@nbdd0121
Copy link
Contributor

nbdd0121 commented Oct 14, 2025 •
edited
Loading

Do you think this should be an filtering middleware or should it be broadcasting?

I currently have some WIP code that allows a dyn Uart to connected to multiple sinks, with each being able to receive all data (i.e. broadcasting). But middleware makes sense too..

EDIT: I guess if the the coverage is to be suppressed then this should be a middleware.

@sasdf
Copy link
Contributor Author

sasdf commented Oct 14, 2025

Yes, it would be filtering middleware in order to hide the report from both logging and exit regex modules.

@sasdf
Copy link
Contributor Author

sasdf commented Oct 15, 2025

Created #28488 for the UartConsole refactoring. Let's track the UartConsole-related stuff there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pamaury pamaury Awaiting requested review from pamaury

@cfrantz cfrantz Awaiting requested review from cfrantz cfrantz is a code owner

@nbdd0121 nbdd0121 Awaiting requested review from nbdd0121

@timothytrippel timothytrippel Awaiting requested review from timothytrippel

@AlexJones0 AlexJones0 Awaiting requested review from AlexJones0

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sasdf @nbdd0121