Test case fail when refactoring some common code (expect, ignore, ...) into utility function outside of test file scope

[silabs-theophilel]

Version / environemnt
MacOS Sonoma 14.2
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin23]
Ceedling version v1.0.0
CMock => 2.6.0
Unity => 2.6.1
CException => 1.3.4

I have multiple test files:

test/test_foo.c
test/test_faa.c
test/test_fuu.c
utility/common.c
utility/common.h

The content of utility files are some helpers function that contains expect ignore and the associated test logic to simplify readability of the test code in original test_foo.c file:

void expect_timer_running_check(bool *running)
{
  sl_sleeptimer_is_timer_running_ExpectAnyArgsAndReturn(SL_STATUS_OK);
  sl_sleeptimer_is_timer_running_ReturnThruPtr_running(running);
}

void expect_invalid_timeout_check(uint32_t *timeout_ms)
{
  // relock will first convert ms to tick
  sl_sleeptimer_ms32_to_tick_ExpectAndReturn(*timeout_ms, NULL, SL_STATUS_INVALID_PARAMETER);
  sl_sleeptimer_ms32_to_tick_IgnoreArg_tick();
}

void expect_valid_timeout_check(uint32_t *timeout_ms, uint32_t *timeout_tick)
{
  // relock will first convert ms to tick
  sl_sleeptimer_ms32_to_tick_ExpectAndReturn(*timeout_ms, NULL, SL_STATUS_OK);
  sl_sleeptimer_ms32_to_tick_IgnoreArg_tick();
  sl_sleeptimer_ms32_to_tick_ReturnThruPtr_tick(timeout_tick);
}

void expect_stop_timer(void)
{
  sl_sleeptimer_stop_timer_ExpectAnyArgsAndReturn(SL_STATUS_OK);
}
...

sometimes even other functions inside utility module can be nested with other utility function:

void expect_transition_to_inactive(zpal_pm_power_lock_t *lock)
{
  bool is_not_running = false;
  expect_timer_running_check(&is_not_running);

  sl_power_manager_em_t em;
  pm_type_to_em(lock->type, &em);
  sli_power_manager_update_em_requirement_Expect(em, false);
}

PROBLEM:
I have realized that my tested where passing on my laptop and in my environment, BUT not on other people environment. The error message is that some function are called more than expected, sometimes less than expected... it depends per test cases.
After putting all my utility code in the original test filetest_foo.c without utility helpers my test now pass on all platforms.

Have you already seen such a behavior when we refactor CMock code into other function , sometime in other files scope, the test result start to become random, passing, sometimes not?

For now I will stick to the basic solution on putting everything in the test file even if that leads to code duplication. But I wanted to report this behavior if not seen already. I can share an exerpt of my project.myl conf below:

:project:
  :use_mocks: TRUE
  :compile_threads: :auto
  :test_threads: :auto
  :use_exceptions: FALSE
  :use_test_preprocessor: :all
  :use_auxiliary_dependencies: TRUE
  :use_backtrace: :simple
  :build_root: build
#  :release_build: TRUE
  :test_file_prefix: test_
  :which_ceedling: gem
  :ceedling_version: 1.0.0
  :default_tasks:
    - test:all
  :use_decorators: :none
  :verbosity: 4

:test_build:
  :use_assembly: TRUE

:environment:

:extension:
  :executable: .out

:paths:
  :test:
    - +:test/**
    - -:test/support
  :source:
    - ../../src/**
  :include:
    .. Long list of path ...
  :support:
    - test/support
  :libraries: []

:defines:
  # in order to add common defines:
  #  1) remove the trailing [] from the :common: section
  #  2) add entries to the :common: section (e.g. :test: has TEST defined)
  :common: &common_defines []
  :test:
    - *common_defines
    - TEST
  :test_preprocess:
    - *common_defines
    - TEST
 
 :gcov:
  :reports:
    - HtmlDetailed
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90


:cmock:
  :mock_prefix: mock_
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :plugins:
    - :ignore
    - :ignore_arg
    - :return_thru_ptr
    - :callback
    - :expect
    - :expect_any_args
  :includes:
    - sl_sleeptimer.h
 :libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

:plugins:
  :load_paths: []
  :enabled:
    - report_tests_pretty_stdout
    - module_generator

[Letme]

This utility/common.c should be in test/support based on your project.yml paths. So why is it not there? It is support for test files.

[silabs-theophilel]

Yes sorry, my mistake I wrote a bad information in original post. The actual file tree is like this, so that my utility module is under test/support path:

[Letme]

Still include should not be relative but just the header file... keep in mind that those are separate compilation units so you need to properly expose global variables as well. Do nit bunch it all up in single file and copy/paste it around multiple files just because you did not properly expose the used variables.

[mvandervoord]

I've not run into this problem, however, I see that you have :use_test_preprocessor set to :all. Do your tests and/or mocked headers include #ifdef calls that vary from test to test? It's possible that you've run into a series of issues that we just fixed in version 1.0.1 (not yet released, but you can manually download and install the gem from releases on github if you'd like to give it a try).

Inconsistencies between development machines are often the result of undefined behavior, particularly if there are uninitialized variables, etc. These can show up in mysterious ways (a counter variable, for example, might change the number of times a particular mock gets called by the release code). Also, differences in toolchain might contribute?

None of these have obvious answers... I'm just throwing out things that are worth investigating.