SCIP tests fail when using HiGHs as the solver

[orthorhombic]

Summary

I'm unable to get the SCIP tests to pass when compiling SCIP with HiGHS set as the solver. Specifically I tested this with SCIP 9.1.0 and HiGHS 1.7.2 (and 1.7.1). It finished compiling without error, but there are a series of tests that fail when HiGHS is set as the solver. All the HiGHS tests pass, so I figured I'd reach out here first. I didn't have any problem when setting -DLPS=spx.

I'm unsure if this is related to #89, though I didn't see the errors specified there. @svigerske, in the same issue you mentioned being able to compile "the SCIP/HiGHS interface of SCIP 9.1.0 with HiGHS 1.7.1." Is the below configuration what you meant and were you able to successfully get the tests to pass?

I apologize in advance if I'm missing something with how I'm compiling. Solutions or next steps for troubleshooting are welcome.

Tests that fail

Specifically this is the list of failing tests from when I was trying to trying to compile the whole suite and not just SCIP. Just to speed up testing, I randomly selected one of these tests for the Dockerfile.

The following tests FAILED:
     538 - Indicator-indicatorlogicorsepa-mcf128-4-1.lp (Failed)
     547 - Indicator-indicatorlogicorsepa-mcf64-4-1.lp (Failed)
     565 - Indicator-indicatorlogicorsepa-mcf64-4-1.zpl (Failed)
     602 - MIP-heuristics_off-blend2.mps (Failed)
     612 - MIP-convertinttobin_mostinf-blend2.mps (Failed)
     615 - MIP-oddcyclelift_randomhybrid-blend2.mps (Failed)
     627 - MIP-solvingphases-blend2.mps (Failed)
     630 - MIP-tolerance-blend2.mps (Failed)
     1040 - MIP-treemodel_svts_nofilter-vpm2.fzn (Failed)
     1041 - MIP-oddcycleliftheur_feaspump20-vpm2.fzn (Failed)
     1050 - MIP-tolerance-vpm2.fzn (Failed)
     1310 - BendersQP-benders-qp-classical_30_0.mps (Failed)
     1350 - MIP-readertest-mps-blend2.mps (Failed)
     1354 - MIP-readertest-fzn-blend2.mps (Failed)
     1546 - unittest-cons-cons (Failed)
     1624 - unittest-cons-superindicator-indicator-bugfix (Failed)
     1626 - unittest-cons-superindicator-indicator-generic (Failed)
     1874 - applications-miniisc-prob.10.30.100.0 (Failed)
     1875 - applications-miniisc-prob.15.40.100.1 (Failed)
     1876 - applications-miniisc-prob.20.50.100.0 (Failed)
     1877 - applications-miniisc-prob.5.030.100.0 (Failed)
     1926 - GCG-BPP-default-N1C1W4_M.BPP.lp (Failed)
     1927 - GCG-BPP-default-N1C2W2_O.BPP.lp (Failed)
     1928 - GCG-BPP-default-N1C3W1_A.lp (Failed)
     1929 - GCG-CPMP-default-p1250-2.lp (Failed)
     1930 - GCG-CPMP-default-p1650-2.txt.lp (Failed)
     1931 - GCG-CPMP-default-p2050-1.txt.lp (Failed)
     1932 - GCG-CS-default-TEST0055.lp (Failed)
     1933 - GCG-CS-default-TEST0059.lp (Failed)
     1934 - GCG-GAP-default-gap4_2.txt.lp (Failed)
     1935 - GCG-GAP-default-gap8_4.txt.lp (Failed)

Error from trying to build the Dockerfile

After spot checking a few of the tests failing above, all seemed to have the same error as below.

[lp.c:12197] ERROR: (node 11) error or unknown return status of primal simplex in LP 136 (internal status: 0)
[lp.c:12257] ERROR: Error <-6> in function call
[lp.c:12511] ERROR: Error <-6> in function call
[scip_probing.c:752] ERROR: Error <-6> in function call
[scip_probing.c:830] ERROR: Error <-6> in function call
[scip_var.c:3181] ERROR: Error <-6> in function call
[scip_var.c:3587] ERROR: Error <-6> in function call
[branch_relpscost.c:1417] ERROR: Error <-6> in function call
[branch_relpscost.c:2050] ERROR: Error <-6> in function call
[branch.c:1590] ERROR: Error <-6> in function call
[branch.c:2591] ERROR: Error <-6> in function call
[scip_branch.c:1241] ERROR: Error <-6> in function call
[cons_integral.c:106] ERROR: Error <-6> in function call
[cons.c:3471] ERROR: Error <-6> in function call
[solve.c:3522] ERROR: Error <-6> in function call
[solve.c:4427] ERROR: Error <-6> in function call
[solve.c:5109] ERROR: Error <-6> in function call
[scip_solve.c:2657] ERROR: Error <-6> in function call
[scipshell.c:110] ERROR: Error <-6> in function call
[scipshell.c:527] ERROR: Error <-6> in function call
[scipshell.c:594] ERROR: Error <-6> in function call
SCIP Error (-6): error in LP solver

Sample Dockerfile to try to compiling:

I saved the below as Dockerfile_test and tried building it with the command below. The specified test was one randomly selected from the list above.
docker build -t build_test --file Dockerfile_test --progress=plain . &> build_test.log

FROM python:3.12.4-slim-bookworm AS builder

ENV WORKSPACE=/builds \
    INSTALLS=/installs \
    SCIP_TAG=910 \
    HIGHS_VERSION=1.7.2 

RUN apt-get update && apt-get install -y \
    wget \
    cmake \
    g++ \
    m4 \
    xz-utils \
    unzip \
    zlib1g-dev \
    libtbb-dev \
    libreadline-dev \
    pkg-config \
    git \
    flex \
    bison \
    libcliquer-dev \
    gfortran \
    libopenblas-dev \
    file \
    dpkg-dev \
    rpm \
    libopenmpi-dev \
    libboost1.81-dev \
    libgmp-dev \
    libtbb-dev \
    libmetis-dev \
    metis \
    libcriterion-dev


RUN mkdir $WORKSPACE && mkdir $INSTALLS
WORKDIR $WORKSPACE


RUN cd ${WORKSPACE} && \
    wget https://github.com/ERGO-Code/HiGHS/archive/refs/tags/v${HIGHS_VERSION}.tar.gz && \
    tar xzf v${HIGHS_VERSION}.tar.gz && \
    cd ${WORKSPACE}/HiGHS-${HIGHS_VERSION} && \
    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${INSTALLS}/lib/ cmake -S . -B build -DCMAKE_INSTALL_PREFIX=${INSTALLS} -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DALL_TESTS=1 && \
    cmake --build build -j 3 && \
    cd build && ctest --parallel --timeout 300 --output-on-failure && cd .. && \
    cmake --install build

RUN cd ${WORKSPACE} && \
    wget https://github.com/scipopt/scip/archive/refs/tags/v${SCIP_TAG}.tar.gz && \
    tar xfz v${SCIP_TAG}.tar.gz && \
    cd ${WORKSPACE}/scip-${SCIP_TAG} && \
    mkdir build && \
    cd ${WORKSPACE}/scip-${SCIP_TAG}/build && \
    PATH=$PATH:${INSTALLS}/bin/ cmake .. \
    -DCMAKE_INSTALL_PREFIX=${INSTALLS} \
    -DCMAKE_BUILD_TYPE=Release \
    -DLPS=highs \
    -DSYM=snauty \
    -DBoost_USE_STATIC_LIBS=on \
    -DPAPILO=false \
    -DLAPACK=false \
    -DZIMPL=false \
    -DGMP=false \
    -DREADLINE=false \
    -DIPOPT=false \
    -DIPOPT_DIR=${INSTALLS} \
    -DHIGHS_DIR=${INSTALLS}/lib/cmake/highs \
    -DCMAKE_C_FLAGS="-s -I${INSTALLS}/include/ -I${INSTALLS}/include/highs -L${INSTALLS}/lib/" \
    -DCMAKE_CXX_FLAGS="-s -I${INSTALLS}/include/ -I${INSTALLS}/include/highs -L${INSTALLS}/lib/" \
    -DTPI=tny && \
    make -j6
RUN cd ${WORKSPACE}/scip-${SCIP_TAG}/build && \
    ctest -R MIP-tolerance-vpm2.fzn --output-on-failure

[svigerske]

I only made it compile again.

But also our internal tests started to fail after upgrading to a more recent HiGHS, 1.7.2 in particular.
The SCIP/HiGHS link developers @ambros-gleixner @alexhoen @GioniMexi have been made aware of this and I hope they will look into it soon.

You may have to go back to an older HiGHS version for now.

[orthorhombic]

Just following up on this, I tried iterating on a variety of configurations where I iterated on the dockerfile I shared above, but have continued to run into issues getting all the tests to pass. I was able to get more tests working with SCIP 9.1.0 with HiGHS 1.6.0 as long as I did cp ${INSTALLS}/lib/libhighs.a ${INSTALLS}/lib/liblibhighs.a (results below)/

You mention HiGHS 1.7.2 in particular giving you trouble on the internal tests. Are you aware of any "known good" SCIP/HiGHS version pairings that would be recommended as a baseline? How much weight should I put on getting passing tests? Any other details you can shat that might help?

In other words, if a pairing of versions seems to successfully solve problems I throw at it, do you think I should I trust the results?

Tests Failed with SCIP 9.1.0/ HiGHS 1.6.0:

The following tests FAILED:
800 - MIP-readertest-lp-blend2.mps (Failed)
802 - MIP-readertest-pip-blend2.mps (Failed)
805 - MIP-readertest-rlp-blend2.mps (Failed)
1225 - unittest-sepa-gauge (Failed)

Error (all same):

[lpi_highs.cpp:496] ERROR: HiGHS terminated with model status <Unknown> (15) after simplex strategy <Dual (serial)> (1)
[lpi_highs.cpp:1486] ERROR: Error <-6> in function call

[GioniMexi]

The error happens when solving an LP in strong branching.

The model_status_ of highs changes from kIterationLimit to kNotSet when turning presolve back on after the LP solve

https://github.com/scipopt/scip/blob/master/src/lpi/lpi_highs.cpp#L519
HIGHS_CALL( lpi->highs->setOptionValue("presolve", "on") );

@ambros-gleixner this happens in the codeblock:

   /* if basis factorization is unavailable, this may be due to presolving; then solve again without presolve */
   HIGHS_CALL( lpi->highs->getOptionValue("presolve", presolvestring) );
   assert(presolvestring == "on" || presolvestring == "off"); /* values used in SCIPlpiSetIntpar() */
   if( !lpi->highs->hasInvert() && presolvestring == "on" )
   {
      SCIP_RETCODE retcode;

      SCIPdebugMessage("No inverse: running HiGHS again without presolve . . .\n");
      HIGHS_CALL( lpi->highs->setOptionValue("presolve", "off") );
      retcode = lpiSolve(lpi);
      if( retcode != SCIP_OKAY )
      {
         HighsModelStatus model_status = lpi->highs->getModelStatus();
         SCIPerrorMessage("HiGHS terminated with model status <%s> (%d) after trying to recover inverse\n",
            lpi->highs->modelStatusToString(model_status).c_str(), (int)model_status);
      }
      HIGHS_CALL( lpi->highs->setOptionValue("presolve", "on") );
      SCIP_CALL( retcode );
   }

where presolving is turned off and on again.

There was a recent change in highs where the model_status is reset in setOptionValue() if the solution is not primal or dual feasible. In our case, the LP was solved by dual simplex up to the iteration limit and the solution was not primal feasible.

@ambros-gleixner One workaround could be to set the model status to the previous value after calling setOptionValue(), or change the logic in lp.c? which is more involved, or ask Julian if it is really necessary to reset the model_status when changing a parameter.

[ambros-gleixner]

We should not change the logic in lp.c, and I would not count on Highs changing this behavior.

Another workaround, that may be easier to implement and safer:

• add a bool flag presolve to struct SCIP_Lpi that stores whether presolving should be used (from SCIPlpiSetIntpar).
• call HIGHS_CALL( lpi->highs->setOptionValue("presolve", ...) ); if necessary in lpiSolve()
  Then we can remove the last setOptionValue in the above code block and the model status should be kept by Highs.

Sounds good?

[GioniMexi]

Sounds good, I will prepare a MR for this

[svigerske]

@GioniMexi implemented the workaround that was suggested by @ambros-gleixner and this has been included in SCIP 9.2.0.

[DominikKamp]

...though MIP-readertest-mps-blend2.mps will still fail as described in #102 (comment) since LP errors are handled more verbose than necessary but these LP results are withdrawn safely as usual (you can trust the results).

[adbuerger]

@svigerske When using SCIP 9.2.1 with a recent version of HiGHS (I use https://github.com/ERGO-Code/HiGHS/tree/ccf22bd1f98de41505e1e8af9543f58dfe6d906d where an issue in the latest released version 1.10.0 that hinders compiling in this context is already resolved, cf. ERGO-Code/HiGHS#2283) I can see multiple failing tests which I wanted to inform you about:

99% tests passed, 8 tests failed out of 1693

[...]

The following tests FAILED:
	601 - MIP-convertinttobin_mostinf-bell5.mps (Failed)
	631 - MIP-convertinttobin_mostinf-blend2.mps (Failed)
	952 - MIP-separating_fast-rgn.mps (Failed)
	1077 - FastMIP-cgmip-bell5.mps (Failed)
	1078 - FastMIP-cgmipdirect-bell5.mps (Failed)
	1079 - FastMIP-cgmipstrong-bell5.mps (Failed)
	1080 - FastMIP-cgmipviol-bell5.mps (Failed)
	1148 - FastMIP-cgmipviol-p0033.osil (Failed)
Errors while running CTest

To gather more details, I re-ran the failed tests using ctest --rerun-failed, this is the output:

Test project /path/to/scipoptsuite-9.2.1/build
    Start  601: MIP-convertinttobin_mostinf-bell5.mps
1/8 Test  #601: MIP-convertinttobin_mostinf-bell5.mps ....***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/convertinttobin_mostinf.set> not found]  2.24 sec
    Start  631: MIP-convertinttobin_mostinf-blend2.mps
2/8 Test  #631: MIP-convertinttobin_mostinf-blend2.mps ...***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/convertinttobin_mostinf.set> not found]  2.49 sec
    Start  952: MIP-separating_fast-rgn.mps
3/8 Test  #952: MIP-separating_fast-rgn.mps ..............***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/separating_fast.set> not found]  1.35 sec
    Start 1077: FastMIP-cgmip-bell5.mps
4/8 Test #1077: FastMIP-cgmip-bell5.mps ..................***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/cgmip.set> not found]  3.06 sec
    Start 1078: FastMIP-cgmipdirect-bell5.mps
5/8 Test #1078: FastMIP-cgmipdirect-bell5.mps ............***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/cgmipdirect.set> not found]  7.15 sec
    Start 1079: FastMIP-cgmipstrong-bell5.mps
6/8 Test #1079: FastMIP-cgmipstrong-bell5.mps ............***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/cgmipstrong.set> not found]  3.75 sec
    Start 1080: FastMIP-cgmipviol-bell5.mps
7/8 Test #1080: FastMIP-cgmipviol-bell5.mps ..............***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/cgmipviol.set> not found]  2.07 sec
    Start 1148: FastMIP-cgmipviol-p0033.osil
8/8 Test #1148: FastMIP-cgmipviol-p0033.osil .............***Failed  Error regular expression found in output. Regex=[ERROR|user parameter file </path/to/scipoptsuite-9.2.1/scip/check/coverage/settings/cgmipviol.set> not found]  3.58 sec

For comparison, I ran SCIP with SoPlex and I cannot see any failing tests then (0 tests failed out of 1693).

If I can provide any more information that helps to sort out these problems please let me know.

[svigerske]

Yes, we're aware that there are more issues left, in particular with recent versions of HiGHS. There hasn't been the time to look into it so far. If @ambros-gleixner or another lpi_highs dev doesn't get to it first, then I hope @Opt-Mucca will find time to look into it in a few months.