R package test fails for Windows

[jakirkham]

In the R 4.4 + UCRT migration ( #196 ), Windows was enabled for R, which ran into a few issues largely addressed in that migration PR.

However one outstanding issue was the tests fails sporadically on the Windows. As a result the test was skipped for that build

xgboost-feedstock/recipe/meta.yaml

Line 268 in 8a5e464

- Rscript test-r-xgboost.r # [not win]

Here is some relevant discussion in the PR starting with this comment: #196 (comment)

A snippet of the test failure is shown below this is taken from this Windows CUDA CI job (also attached log for posterity)

(base) %SRC_DIR%>conda activate "%PREFIX%" 
Loading required package: xgboost
List of 2
 $ data :Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
  .. ..@ i       : int [1:143286] 2 6 8 11 18 20 21 24 28 32 ...
  .. ..@ p       : int [1:127] 0 369 372 3306 5845 6489 6513 8380 8384 10991 ...
  .. ..@ Dim     : int [1:2] 6513 126
  .. ..@ Dimnames:List of 2
  .. .. ..$ : NULL
  .. .. ..$ : chr [1:126] "cap-shape=bell" "cap-shape=conical" "cap-shape=convex" "cap-shape=flat" ...
  .. ..@ x       : num [1:143286] 1 1 1 1 1 1 1 1 1 1 ...
  .. ..@ factors : list()
 $ label: num [1:6513] 1 0 0 1 0 0 0 1 0 0 ...
[1] 6513  126
[1] 1611  126
[1] "dgCMatrix"
[1] "numeric"
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1] 1611
[1] 0.28520852 0.92405337 0.28520852 0.28520852 0.05161287 0.92405337
[1] 0 1 0 0 0 1
[1] "test-error= 0.0217256362507759"
[1]	train-logloss:0.233156	test-logloss:0.226460 
[2]	train-logloss:0.136542	test-logloss:0.137763 
[1]	train-error:0.046522	train-logloss:0.233156	test-error:0.042831	test-logloss:0.226460 
[2]	train-error:0.022263	train-logloss:0.136542	test-error:0.021726	test-logloss:0.137763 
WARNING: Tests failed for r-xgboost-2.1.3-cuda126_r44h979e81a_1.conda - moving package to D:\bld\broken
Traceback (most recent call last):
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3484, in test
    utils.check_call_env(
  File "D:\Miniforge\Lib\site-packages\conda_build\utils.py", line 406, in check_call_env
    return _func_defaulting_env_to_os_environ("call", *popenargs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Miniforge\Lib\site-packages\conda_build\utils.py", line 382, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['C:\\Windows\\system32\\cmd.exe', '/d', '/c', 'D:\\bld\\xgboost-split_1737432643462\\test_tmp\\conda_test_runner.bat']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Miniforge\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(execute())
             ^^^^^^^^^
  File "D:\Miniforge\Lib\site-packages\conda_build\cli\main_build.py", line 622, in execute
    api.build(
  File "D:\Miniforge\Lib\site-packages\conda_build\api.py", line 211, in build
    return build_tree(
           ^^^^^^^^^^^
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3671, in build_tree
    test(pkg, config=metadata.config.copy(), stats=stats)
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3498, in test
    tests_failed(
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3545, in tests_failed
    raise CondaBuildUserError("TESTS FAILED: " + os.path.basename(pkg))
conda_build.exceptions.CondaBuildUserError: TESTS FAILED: r-xgboost-2.1.3-cuda126_r44h979e81a_1.conda

A snippet of the test failure is shown below this is taken from this Windows CPU CI job (also attached log for posterity)

(base) %SRC_DIR%>conda activate "%PREFIX%" 
Loading required package: xgboost
List of 2
 $ data :Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
  .. ..@ i       : int [1:143286] 2 6 8 11 18 20 21 24 28 32 ...
  .. ..@ p       : int [1:127] 0 369 372 3306 5845 6489 6513 8380 8384 10991 ...
  .. ..@ Dim     : int [1:2] 6513 126
  .. ..@ Dimnames:List of 2
  .. .. ..$ : NULL
  .. .. ..$ : chr [1:126] "cap-shape=bell" "cap-shape=conical" "cap-shape=convex" "cap-shape=flat" ...
  .. ..@ x       : num [1:143286] 1 1 1 1 1 1 1 1 1 1 ...
  .. ..@ factors : list()
 $ label: num [1:6513] 1 0 0 1 0 0 0 1 0 0 ...
[1] 6513  126
[1] 1611  126
[1] "dgCMatrix"
[1] "numeric"
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1]	train-logloss:0.233156 
[2]	train-logloss:0.136542 
[1] 1611
[1] 0.28520852 0.92405337 0.28520852 0.28520852 0.05161287 0.92405337
[1] 0 1 0 0 0 1
[1] "test-error= 0.0217256362507759"
[1]	train-logloss:0.233156	test-logloss:0.226460 
[2]	train-logloss:0.136542	test-logloss:0.137763 
[1]	train-error:0.046522	train-logloss:0.233156	test-error:0.042831	test-logloss:0.226460 
[2]	train-error:0.022263	train-logloss:0.136542	test-error:0.021726	test-logloss:0.137763 
WARNING: Tests failed for r-xgboost-2.1.3-cpu_r44h8cb75e7_2.conda - moving package to D:\bld\broken
Traceback (most recent call last):
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3484, in test
    utils.check_call_env(
  File "D:\Miniforge\Lib\site-packages\conda_build\utils.py", line 406, in check_call_env
    return _func_defaulting_env_to_os_environ("call", *popenargs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Miniforge\Lib\site-packages\conda_build\utils.py", line 382, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['C:\\Windows\\system32\\cmd.exe', '/d', '/c', 'D:\\bld\\xgboost-split_1737698263317\\test_tmp\\conda_test_runner.bat']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Miniforge\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(execute())
             ^^^^^^^^^
  File "D:\Miniforge\Lib\site-packages\conda_build\cli\main_build.py", line 622, in execute
    api.build(
  File "D:\Miniforge\Lib\site-packages\conda_build\api.py", line 211, in build
    return build_tree(
           ^^^^^^^^^^^
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3671, in build_tree
    test(pkg, config=metadata.config.copy(), stats=stats)
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3498, in test
    tests_failed(
  File "D:\Miniforge\Lib\site-packages\conda_build\build.py", line 3545, in tests_failed
    raise CondaBuildUserError("TESTS FAILED: " + os.path.basename(pkg))
conda_build.exceptions.CondaBuildUserError: TESTS FAILED: r-xgboost-2.1.3-cpu_r44h8cb75e7_2.conda

[jakirkham]

@danielnachun @h-vetinari , please let us know if there are other details we should be tracking here

[danielnachun]

One important and somewhat confusing detail here is that there were some runs where this test did work. The only thing that changed between the failing and successful runs was that the feedstock was rerendered. Hopefully we'll get to the bottom of this soon!

[h-vetinari]

Let's open a PR that re-enables this test (and skips everything but win+CUDA) so we can keep working on it, and keep it visible.

[jakirkham]

Well it failed on Windows CPU on main. I went ahead and extended the test skip to Windows. Have updated the OP with the added details if someone wants to dig in

This helps us get the last Windows package out. Perhaps another way to explore this issue will be to install the Windows package and debug locally

Also folks are welcome to open a PR and debug on CI if that is better. It has certainly been a value way to collaborate in conda-forge ever since this project begun

Personally am a little tied down with other commitments at the moment. So cannot dig into it myself

[jakirkham]

The R package test was entirely rewritten as part of the XGBoost 3 upgrade: #226

Let's see if things look better now

[jakirkham]

@conda-forge-admin , please re-render

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-webservice.

I just wanted to let you know that I started rerendering the recipe in #232.

[jakirkham]

Adding back the test with PR: #232

Marking as draft as I would like to see CI pass reliably after restarting it a few times before including it