Install Request: CONVERGE for Brunel on Young

[heatherkellyucl]

https://myservices-arc.uk.4me.com/requests/1560016

Restricted application group and modules in brunel-modules.

Installer provided.

License server access from Young has been set up at their side - test.

[heatherkellyucl]

Package Convergent_Science_Full_Package-3.1.12.tar.gz is on Young in the pkg-store.

[heatherkellyucl]

Run the Install script found in this directory (./INSTALL)
Follow the on-screen prompt to choose an install location.

The installation places more documentation in the doc/ directory under your
installation root. Upon completion, the installer will notify you of your
installation location.

Can run ./INSTALL for the prompts or ./INSTALL $install_root in which case it will proceed in quiet mode.

[heatherkellyucl]

Making reserved application group lgconbru, adding ccspap2.

(Ought to be replacing these soon with longer-named AD groups).

[heatherkellyucl]

Group exists, installed centrally on Young. Need to check on what we do or don't need to put in our modulefile because it comes with a whole pile of modulefiles itself at /shared/ucl/apps/Convergent_Science/3.1.12/Convergent_Science/Environment/modulefiles

[ikirker]

To use the CONVERGE modules as-is (i.e. without any additions to make it locally more convenient), some notes:

• the modules seem intended to use the module that just includes the MPI name and the version label, e.g. CONVERGE/CONVERGE-IntelMPI/3.1.12 and then this loads the other necessary modules
• unfortunately this results in the modules conflicting with each other: I suspect this was designed for a previous version of the modules system where it didn't check conflicts of already-loaded modules against modules being loaded
• this can be worked around by using the -f option to module load, but it's not ideal

So e.g.:

$ module use /shared/ucl/apps/Convergent_Science/3.1.12/Convergent_Science/Environment/modulefiles
$ module load -f CONVERGE/CONVERGE-IntelMPI/3.1.12
Loading CONVERGE-IntelMPI/cvg/3.1.12
  WARNING: Conflicting CONVERGE/CONVERGE-IntelMPI/3.1.12 is loading

Loading CONVERGE-IntelMPI/mpi/3.1.12
  WARNING: Conflicting CONVERGE/CONVERGE-IntelMPI/3.1.12 is loading

Loading CONVERGE-IntelMPI/utilities/3.1.12
  WARNING: Conflicting CONVERGE/CONVERGE-IntelMPI/3.1.12 is loading

Loading CONVERGE-IntelMPI/udf/3.1.12
  WARNING: Conflicting CONVERGE/CONVERGE-IntelMPI/3.1.12 is loading

Loading CONVERGE/CONVERGE-IntelMPI/3.1.12
  WARNING: Conflicting CONVERGE-IntelMPI is loaded
  Loading requirement: CONVERGE-IntelMPI/cvg/3.1.12 CONVERGE-IntelMPI/mpi/3.1.12 CONVERGE-IntelMPI/utilities/3.1.12 CONVERGE-IntelMPI/udf/3.1.12
$ export RLM_LICENSE=[license server details omitted, but it's $port@$hostname]
$ converge-intelmpi
[0] MPI startup(): I_MPI_DEVICE environment variable is not supported.
[0] MPI startup(): To check the list of supported variables, use the impi_info utility or refer to https://software.intel.com/en-us/mpi-library/documentation/get-started.
WARNING: Unable to obtain 1 parent license. Aborting.
Please contact licensing@convergecfd.com for help troubleshooting this problem.

FATAL_ERROR: License error code -17. Communications error with license server (-17)
Connection refused at server (-111)
Stack trace(s) written to abort_trace_rank[#].out
Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

So... maybe the license server isn't actually even running?

[heatherkellyucl]

Hmm, I think we have the correct util node IPs specified but the wrong one for the login nodes (needs the xlate one). Checking.

[heatherkellyucl]

No, I get the same thing on a compute node which ought to be correct.

[heatherkellyucl]

The xlate address does need adding instead of the login node IPs, but there's still some issue at the license server end since the compute nodes can't reach it via the allowed util node IPs either.

[heatherkellyucl]

Firewall updated at their end - tried again on login02 but the same error, currently waiting for a qrsh session to try the compute nodes.

[heatherkellyucl]

Still also fails in a job.

become ccspap2
qrsh -l mem=1G,h_rt=0:45:0 -pe mpi 40 -P Test -A Test_allocation -now no
module use /shared/ucl/apps/Convergent_Science/3.1.12/Convergent_Science/Environment/modulefiles
module load -f CONVERGE/CONVERGE-IntelMPI/3.1.12
export RLM_LICENSE=<port>@<host ip>
converge-intelmpi

[0] MPI startup(): I_MPI_DEVICE environment variable is not supported.
[0] MPI startup(): To check the list of supported variables, use the impi_info utility or refer to https://software.intel.com/en-us/mpi-library/documentation/get-started.
WARNING: Unable to obtain 1 parent license. Aborting.
Please contact licensing@convergecfd.com for help troubleshooting this problem.

FATAL_ERROR: License error code -17. Communications error with license server (-17)
Connection refused at server (-111)
Stack trace(s) written to abort_trace_rank[#].out
Abort(1) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

If you use the license server FQDN instead, it isn't recognised - this is likely expected from outside Brunel.

FATAL_ERROR: License error code -43. Bad server hostname in license file or port@host (-43)
Bad hostname for network connect (-131)
Stack trace(s) written to abort_trace_rank[#].out

[heatherkellyucl]

Info from CONVERGE support:

(i) Ensure that the port x and y (both UDP and TCP, inbound and outbound rules) are open.
(ii) Since the license is superbase, the keyword super should be added after the executable name, for example, converge-intel super.

(That last is now -l super as super on its own is deprecated).

We certainly won't have inbound for the first of those. Opening ticket with Networks.