./Configure hangs during g++ configuration on OpenBSD

[jkeenan]

In the course of testing the branch developed in #21949 to address the g++/clang++ build-time failures reported in #21948, I had occasion to try to configure and build perl on OpenBSD-6.9 using first g++, then clang++ as the C-compiler. Running ./Configure with these arguments:

$ sh ./Configure -des -Dusedevel -Dcc=g++

... configuration halted here:

...
Do you wish to use dynamic loading? [y]  
Source file to use for dynamic loading 

... and had to be killed with Ctrl-C. This freeze occurred in both the terminal and in ./smokecurrent.sh being run in a screen session.

Configuration and build with clang++, however, proceeded to completion (albeit with a ton of warnings).

The default shell on OpenBSD is ksh.

$ echo $KSH_VERSION 
@(#)PD KSH v5.2.14 99/07/13.2

I do not believe the problem is with ksh, because I invoked bash, then configured with g++ as above, and configuration stopped at exactly the same point. Configuration with ksh on FreeBSD proceeded successfully. Hence, this problem seems to be some interaction between the OS, g++ and Configure.

I concede that this is an older version of OpenBSD; I am unable to install VMs of the more recent versions on the host to which I have access.

$ uname -mrs
OpenBSD 6.9 amd64

I further concede that I rarely attempt to build perl on OpenBSD with g++ so I don't know whether this is a recently developed problem or not.

Here is the relevant section of Configure:

 8348 : determine which dynamic loading, if any, to compile in
 8349 echo " "
 8350 dldir="ext/DynaLoader"
 8351 case "$usedl" in
 8352     $define|y|true)
 8353     dflt='y'
 8354     usedl="$define"
 8355     ;;
 8356     $undef|n|false)
 8357     dflt='n'
 8358     usedl="$undef"
 8359     ;;
 8360     *)
 8361     dflt='n'
 8362     case "$d_dlopen" in
 8363         $define) dflt='y' ;;
 8364     esac
 8365     : Does a dl_xxx.xs file exist for this operating system
 8366     $test -f $rsrc/$dldir/dl_${osname}.xs && dflt='y'
 8367     ;;
 8368 esac
 8369 rp="Do you wish to use dynamic loading?"
 8370 . ./myread
 8371 usedl="$ans"
 8372 bin_ELF="$undef"
 8373 case "$ans" in
 8374     y*) usedl="$define"
 8375     case "$dlsrc" in
 8376         '') if $test -f $rsrc/$dldir/dl_${osname}.xs ; then
 8377             dflt="$dldir/dl_${osname}.xs"
 8378         elif $test "$d_dlopen" = "$define" ; then
 8379             dflt="$dldir/dl_dlopen.xs"
 8380         else
 8381             dflt=''
 8382         fi
 8383         ;;
 8384         *)  dflt="$dldir/$dlsrc"
 8385         ;;
 8386     esac
 8387     echo "The following dynamic loading files are available:"
 8388     : Can not go over to $dldir because getfile has path hard-coded in.
 8389     tdir=`pwd`; cd "$rsrc"; $ls -C $dldir/dl*.xs; cd "$tdir"
 8390     rp="Source file to use for dynamic loading"
 8391     fn="fne"
 8392     gfpth="$src"
 8393     . ./getfile
 8394     usedl="$define"
 8395     : emulate basename
 8396     dlsrc=`echo $ans | $sed -e 's%.*/\([^/]*\)$%\1%'`

Calling all ./Configure and/or OpenBSD specialists!

[Leont]

@afresh1

[tonycoz]

It is prompting for which DynaLoader module to use (aka dlsrc).

If you enter "dl_dlopen.xs" and hit Enter Configure will continue to completion.

This is happening because dlopen() isn't being detected, but I don't know why at this point.

[jkeenan]

It is prompting for which DynaLoader module to use (aka dlsrc).

If you enter "dl_dlopen.xs" and hit Enter Configure will continue to completion.

This is happening because dlopen() isn't being detected, but I don't know why at this point.

On OpenBSD-6.9, I logged the output of these two invocations:

sh ./Configure -d -Dusedevel -Dcc=gcc

sh ./Configure -d -Dusedevel -Dcc=g++

... where the gcc configuration completed, but the g++ configuration paused for input at the location identified. Here is an extract of the diff:

211c211
< Use which C compiler? [gcc]  
---
> Use which C compiler? [g++]  
223c223
< Your C compiler needs -std=gnu99 to compile C99 code
---
> Your C compiler doesn't need any special flags to compile C99 code
228,229c228,229
< Maybe "gcc -E" will work...
< Nope...maybe "gcc -E -" will work...
---
> Maybe "g++ -E" will work...
> Nope...maybe "g++ -E -" will work...
318c318
< [-std=gnu99 -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include]  
---
> [-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include]  
321c321
< They appear to be: -std=gnu99 -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include
---
> They appear to be: -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include
408c408
< You are not using a C++ compiler.
---
> You are using a C++ compiler.
596c596
< dlopen() found.
---
> dlopen() NOT found.
607,2172c607
< Source file to use for dynamic loading [ext/DynaLoader/dl_dlopen.xs]  
< 

... g++ build pauses here for input; Skip 1500+ lines of output from Configure using gcc

< 
< Press return or use a shell escape to edit config.sh:  
---
> Source file to use for dynamic loading 
\ No newline at end of file

Most obvious difference: with g++, dlopen() NOT found.

[jkeenan]

Let's hack up ./Configure with some echo statements:

$ git diff | cat
diff --git a/Configure b/Configure
index 5a7b447649..287923ebe5 100755
--- a/Configure
+++ b/Configure
@@ -8370,20 +8370,26 @@ rp="Do you wish to use dynamic loading?"
 . ./myread
 usedl="$ans"
 bin_ELF="$undef"
+echo "FFF: usedl: $usedl | bin_ELF: $bin_ELF | define: $define | dlopen: $dlopen | d_dlopen: $d_dlopen";
 case "$ans" in
     y*) usedl="$define"
 	case "$dlsrc" in
 	    '') if $test -f $rsrc/$dldir/dl_${osname}.xs ; then
+            echo WWW
 		    dflt="$dldir/dl_${osname}.xs"
 		elif $test "$d_dlopen" = "$define" ; then
+            echo "XXX"
 		    dflt="$dldir/dl_dlopen.xs"
 		else
+            echo "YYY"
 		    dflt=''
 		fi
 		;;
 	    *)  dflt="$dldir/$dlsrc"
+            echo "ZZZ"
 		;;
 	esac
+    echo "BBB: dflt: <$dflt>"
 	echo "The following dynamic loading files are available:"
 	: Can not go over to $dldir because getfile has path hard-coded in.
 	tdir=`pwd`; cd "$rsrc"; $ls -C $dldir/dl*.xs; cd "$tdir"

Now, configuring with gcc and then with g++, the diff shows:

$ diff -w $P5P_DIR/out.gcc $P5P_DIR/out.g++ 
211c211
< Use which C compiler? [gcc]  
---
> Use which C compiler? [g++]  
223c223
< Your C compiler needs -std=gnu99 to compile C99 code
---
> Your C compiler doesn't need any special flags to compile C99 code
228,229c228,229
< Maybe "gcc -E" will work...
< Nope...maybe "gcc -E -" will work...
---
> Maybe "g++ -E" will work...
> Nope...maybe "g++ -E -" will work...
318c318
< [-std=gnu99 -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include]  
---
> [-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include]  
321c321
< They appear to be: -std=gnu99 -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include
---
> They appear to be: -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include
408c408
< You are not using a C++ compiler.
---
> You are using a C++ compiler.
596c596
< dlopen() found.
---
> dlopen() NOT found.
601,603c601,603
< FFF: usedl: y | bin_ELF: undef | define: define | dlopen:  | d_dlopen: define
< XXX
< BBB: dflt: <ext/DynaLoader/dl_dlopen.xs>
---
> FFF: usedl: y | bin_ELF: undef | define: define | dlopen:  | d_dlopen: undef
> YYY
> BBB: dflt: <>
610,761c610
< Source file to use for dynamic loading [ext/DynaLoader/dl_dlopen.xs]  
< 

So, with g++, the 'failure' to find dlopen means that the elif $test at line 8378 of ./Configure (in blead) fails; the case goes on to the else at line 8380 and dflt is set to an empty string; and we pause for user entry.

[jkeenan]

We have previously encountered the problem wherein Configure fails to find dlopen() when building with g++. We've provided workarounds in hints programs for other OSes; we should consider doing so for OpenBSD as well.

Linux: hints/linux.sh modified back in 2006:

commit c3faea48927502248be7f55e1cfb047fff817fa8
Author:     Jarkko Hietaniemi <jhi@iki.fi>
AuthorDate: Tue Oct 10 17:00:40 2006 +0300
Commit:     Rafael Garcia-Suarez <rgs@consttype.org>
CommitDate: Tue Oct 10 11:20:19 2006 +0000

    Linux and Solaris hints: C++ vs dlerror

FreeBSD: hints/freebsd modified (via 2 commits) back in 2017:

commit 21a33adc37856aaedd4bf756d5dca47bdc4f7b50
Author:     James E Keenan <jkeenan@cpan.org>
AuthorDate: Mon May 22 21:25:18 2017 -0400
Commit:     James E Keenan <jkeenan@cpan.org>
CommitDate: Sat Jun 3 18:22:46 2017 -0400

    When building with g++ on FreeBSD, explicitly set 'usedl' and 'dlsrc'.
    
    For: https://rt.perl.org/Ticket/Display.html?id=131337
    
commit 2c8efe4079b75c61cf34425054539a9c24913e9f
Author:     Andy Dougherty <doughera@lafayette.edu>
AuthorDate: Mon Jun 12 08:02:10 2017 -0400
Commit:     Andy Dougherty <doughera@lafayette.edu>
CommitDate: Mon Jun 12 15:06:53 2017 -0400

    Simpler hints fix for [perl #131337].
    
    The Configure scan fails to find dlopen() with g++.  Explicitly making
    it availble allows Configure to default to using dynamic loading, but
    still allows the user to override and use static loading.

We could patch hints/openbsd.sh in the way we patched hints/freebsd.sh:

$ git diff -w blead..jkeenan/force-d_dlopen-gh-21958-20240214 -- hints
diff --git a/hints/openbsd.sh b/hints/openbsd.sh
index dbe29187dd..c02609fabe 100644
--- a/hints/openbsd.sh
+++ b/hints/openbsd.sh
@@ -167,4 +167,14 @@ ccflags="$ccflags -DLIBC_HANDLES_MISMATCHED_CTYPE"
 # shared objects, so we need to disable it. See GH #19109
 d_thread_local=undef
 
+# If using g++, the Configure scan for dlopen() fails.
+# Easier for now to just to forcibly set it.
+# (As we did for FreeBSD in https://github.com/Perl/perl5/issues/15984)
+# For: https://github.com/Perl/perl5/issues/21958
+case "$cc" in
+*g++*)
+  d_dlopen='define'
+  ;;
+esac
+
 # end

See this branch: https://github.com/jkeenan/perl5/tree/force-d_dlopen-gh-21958-20240214

This has the following results:

• On OpenBSD (I only have old version 6.9), we can choose to build with g++ (I only have old version 4.2.1 from 2007); sh ./Configure -des -Dusedevel -Dcc=g++ will run to completion. GOOD

• If, however, we remove the -s switch and configure simply with sh ./Configure -de -Dusedevel -Dcc=g++, we will get output like this:

dlopen() NOT found.
Hmm.  Based on the hints in hints/openbsd.sh,
the recommended value for $d_dlopen on this machine was "define"!
    Keep the recommended value? [y]
...
Source file to use for dynamic loading [ext/DynaLoader/dl_dlopen.xs]

FAIRLY GOOD; workaround our imperfect understanding of g++'s interaction with dlopen().

• However, while we can now call make with g++ on OpenBSD, blead quickly fails:

$ make test_harness
echo @`sh  cflags "optimize='-O2'" opmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
@g++ -c -DPERL_CORE -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -O2 -Wall -Werror=pointer-arith -Wextra -Wno-long-long -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
...
g++ -c -DPERL_CORE -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -O2 -Wall -Werror=pointer-arith -Wextra -Wno-long-long -Wwrite-strings class.c
class.c:575: error: expected primary-expression before '.' token
class.c:576: error: expected primary-expression before '.' token
class.c:577: error: expected primary-expression before '.' token
class.c:1041: error: expected primary-expression before '.' token
class.c:1042: error: expected primary-expression before '.' token
class.c:1043: error: expected primary-expression before '.' token
class.c:1045: error: expected primary-expression before '.' token
class.c:1046: error: expected primary-expression before '.' token
class.c:1047: error: expected primary-expression before '.' token
*** Error 1 in /home/jkeenan/gitwork/perl (makefile:260 'class.o': @`sh  cflags "optimize='-O2'" class.o`  class.c)

... which, on my machine, appears to be the same build failure I get if, when configuring and building with g++ in blead, I manually enter dl_dlopen.xs at the prompt as suggested by @tonycoz above. BAD, except that it forces us to look more carefully at the relatively new code in class.c -- but that should be discussed in a separate ticket.

And we of course will have to see how this flies with the OpenBSD project.

@tonycoz, @afresh1, @doughera88, @Tux, your thoughts?

[Leont]

We have previously encountered the problem wherein Configure fails to find dlopen() when building with g++. We've provided workarounds in hints programs for other OSes; we should consider doing so for OpenBSD as well.

Sounds like the csym logic is broken and should be fixed instead?

However, while we can now call make with g++ on OpenBSD, blead quickly fails:

What version of g++? Designated initializers are a C99 feature and has only been allowed in C++20 (though AFAIK g++ allowed it longer), so C++ compilers may trip over it.

[tonycoz]

The test for dlopen() is failing due to a linker error, I added some debug output to Configure:

extern "C" void *dlopen(); void *(*(p()))() { return &dlopen; } int main(int argc, char **argv) { if(p() && p() != (void *)argv[0]) return(0); else return(1); }
g++ -o try -O2 -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -shared -Wl,-E -fstack-protector-strong -L/usr/local/lib try.c -lpthread -lm -lutil -lc
ccflags=-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -shared
ldflags=-Wl,-E -fstack-protector-strong -L/usr/local/lib
libs=-lpthread -lm -lutil -lc
ld: error: relocation R_X86_64_PC32 cannot be used against symbol p(); recompile with -fPIC
>>> defined in /tmp//ccL0t6LO.o
>>> referenced by try.c
>>>               /tmp//ccL0t6LO.o:(main)
collect2: ld returned 1 exit status

The probe for dlopen() specifically adds -shared to ccflags when probing:

: see if dlopen exists
xxx_runnm="$runnm"
xxx_ccflags="$ccflags"
runnm=false
: with g++ one needs -shared to get is-in-libc to work for dlopen
case "$gccversion" in
'')	;;
*Clang*)	;;
*)	case "$d_cplusplus" in
	"$define") ccflags="$ccflags -shared" ;;
	esac
	;;
esac
set dlopen d_dlopen
eval $inlibc
runnm="$xxx_runnm"
ccflags="$xxx_ccflags"

That appears to be the cause of the problem, if I remove it then dlopen() is successfully detected and Configure runs to completion.

Adding -fPIC also fixes the problem.

WRT to the C++ build errors with the old g++, as far as I know we build with C++ to:

• ensure the headers build as C++
• allow improved error checking done by C++ (I know C++ complaining about bypassing initializations has caught some problems)

To be able to use C99 features, we require a C++20 compiler if you do want to build perl with C++, though we do need to ensure that headers don't use such features (since C++ 11 or 14 is still pretty common.)

So I don't think the g++ compilation errors are a bug that needs to be fixed, we do allow use of the form of designated initializers, a C++ compiler that doesn't support them isn't supported for the specific case of building perl itself.

[jkeenan]

The test for dlopen() is failing due to a linker error, I added some debug output to Configure:

extern "C" void *dlopen(); void *(*(p()))() { return &dlopen; } int main(int argc, char **argv) { if(p() && p() != (void *)argv[0]) return(0); else return(1); }
g++ -o try -O2 -DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -shared -Wl,-E -fstack-protector-strong -L/usr/local/lib try.c -lpthread -lm -lutil -lc
ccflags=-DNO_LOCALE_NUMERIC -DNO_LOCALE_COLLATE -DNO_LOCALE_MONETARY -DNO_LOCALE_TIME -DNO_LOCALE_MESSAGES -DLIBC_HANDLES_MISMATCHED_CTYPE -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -shared
ldflags=-Wl,-E -fstack-protector-strong -L/usr/local/lib
libs=-lpthread -lm -lutil -lc
ld: error: relocation R_X86_64_PC32 cannot be used against symbol p(); recompile with -fPIC
>>> defined in /tmp//ccL0t6LO.o
>>> referenced by try.c
>>>               /tmp//ccL0t6LO.o:(main)
collect2: ld returned 1 exit status

The probe for dlopen() specifically adds -shared to ccflags when probing:

: see if dlopen exists
xxx_runnm="$runnm"
xxx_ccflags="$ccflags"
runnm=false
: with g++ one needs -shared to get is-in-libc to work for dlopen
case "$gccversion" in
'')	;;
*Clang*)	;;
*)	case "$d_cplusplus" in
	"$define") ccflags="$ccflags -shared" ;;
	esac
	;;
esac
set dlopen d_dlopen
eval $inlibc
runnm="$xxx_runnm"
ccflags="$xxx_ccflags"

That appears to be the cause of the problem, if I remove it then dlopen() is successfully detected and Configure runs to completion.

Adding -fPIC also fixes the problem.

WRT to the C++ build errors with the old g++, as far as I know we build with C++ to:

* ensure the headers build as C++

* allow improved error checking done by C++ (I know C++ complaining about bypassing initializations has caught some problems)

To be able to use C99 features, we require a C++20 compiler if you do want to build perl with C++, though we do need to ensure that headers don't use such features (since C++ 11 or 14 is still pretty common.)

So I don't think the g++ compilation errors are a bug that needs to be fixed, we do allow use of the form of designated initializers, a C++ compiler that doesn't support them isn't supported for the specific case of building perl itself.

Well, I am currently unable to install a more recent VM for OpenBSD, nor can I upgrade the g++-4.2.1. So what I think we need is something that DTRT on current OpenBSDs and does no harm on older versions. Suggestions welcome. Thanks.

[tonycoz]

You can use clang++

Or not build as C++, we build as C++ for specific reasons with specific requirements, if the C++ compiler you have doesn't meet those requirements then there no more point in testing with it than with building with a K&R era C compiler.

Building perl itself as C++ isn't an end-user option, 90% of the XS on CPAN isn't going to build, we don't need perl to build as C++ on every platform.

I do think we could test the headers for C++ compatibility in better ways, but I ran into problems with doing that with EU::CB.