local_zig still downloads from upstream

[dch]

I did a minimal zigler nif https://git.sr.ht/~dch/ziggle using hex published 0.7.2 & zig 0.7.1 & also tried latest master.

• zig still downloads from upstream instead of using local version
• then blows out spectacularly, I'm not yet clear whats happening but I think we're missing some includes/headers
• BTW can you publish the tags you're using to publish to hex, also to the zigler github repo?

Any suggestions?

 mix compile
Compiling 1 file (.ex)

15:54:39.817 [debug] compiling nif for module Ziggle in path /tmp/.zigler_compiler/dev/Elixir.Ziggle

15:54:39.880 [debug] wrote build.zig to /tmp/.zigler_compiler/dev/Elixir.Ziggle/build.zig

== Compilation error in file lib/ziggle.ex ==
** (CompileError)  this zig compiler warning hasn't been incorporated into the parser.
Please file a report at:
https://github.com/ityonemo/zigler/issues
error(compilation): clang failed with stderr: In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/libunwind.cpp:12:
In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/include/libunwind.h:18:
In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libcxx/include/stdint.h:106:
/tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libcxx/include/__config:247:12: fatal error: 'sys/endian.h' file not found

error(compilation): clang failed with stderr: In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/Unwind-seh.cpp:13:
/tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/config.h:16:10: fatal error: 'assert.h' file not found

error(compilation): clang failed with stderr: In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/UnwindLevel1.c:21:
/tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found

error(compilation): clang failed with stderr: In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/UnwindLevel1-gcc-ext.c:12:
/tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found

error(compilation): clang failed with stderr: In file included from /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/Unwind-sjlj.c:14:
/tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found

error(compilation): /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/libunwind.cpp:1:1: unable to build C object: clang exited with code 1

error(compilation): /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/Unwind-seh.cpp:1:1: unable to build C object: clang exited with code 1

error(compilation): /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/UnwindLevel1.c:1:1: unable to build C object: clang exited with code 1

error(compilation): /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/UnwindLevel1-gcc-ext.c:1:1: unable to build C object: clang exited with code 1

error(compilation): /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/Unwind-sjlj.c:1:1: unable to build C object: clang exited with code 1

error: unable to build libunwind: BuildingLibCObjectFailed
Elixir.Ziggle...The following command exited with error code 1:
/tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/zig build-lib /tmp/.zigler_compiler/dev/Elixir.Ziggle/Elixir.Ziggle.zig --single-threaded --cache-dir /tmp/.zigler_compiler/dev/Elixir.Ziggle/zig-cache --global-cache-dir /home/dch/.cache/zig --name Elixir.Ziggle --version 0.1 -dynamic -target x86_64-freebsd-gnu -isystem /usr/local/lib/erlang23/erts-11.1.7/include -isystem /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/libc/musl/include -isystem /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/libc/include/x86_64-freebsd-musl --enable-cache 
error: the following build command failed with exit code 1:
/tmp/.zigler_compiler/dev/Elixir.Ziggle/zig-cache/o/4c955e129d544a6d30f2b514bffab757/build /tmp/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/zig /tmp/.zigler_compiler/dev/Elixir.Ziggle /tmp/.zigler_compiler/dev/Elixir.Ziggle/zig-cache /home/dch/.cache/zig

    (zigler 0.7.2) lib/zig/parser/error.ex:54: Zig.Parser.Error.parse/2
    (zigler 0.7.2) lib/zig/command.ex:36: Zig.Command.compile/2
    (zigler 0.7.2) lib/zig/compiler.ex:102: Zig.Compiler.compilation/2
    (zigler 0.7.2) expanding macro: Zig.Compiler.__before_compile__/1
    lib/ziggle.ex:1: Ziggle (module)

Using a full local zig with the same hack as last time:

$ mkdir -p deps/zigler/zig/zig-freebsd-x86_64-0.7.1 \
  && ln -s /usr/local/bin/zig deps/zigler/zig/zig-freebsd-x86_64-0.7.1/zig \
  && ln -s /usr/local/lib deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib
dch@z01 /u/h/d/s/z/ziggle (main)> tree -Fap deps/zigler/zig
deps/zigler/zig
├── [drwxr-xr-x]  zig-freebsd-x86_64-0.7.1/
│   ├── [lrwxr-xr-x]  lib -> /usr/local/lib/
│   └── [lrwxr-xr-x]  zig -> /usr/local/bin/zig
└── [-rw-r--r--]  zig-freebsd-x86_64-0.7.1.tar.xz

2 directories, 2 files
$  mix do compile, test
Compiling 1 file (.ex)

16:39:49.757 [debug] compiling nif for module Ziggle in path /tmp/.zigler_compiler/dev/Elixir.Ziggle

16:39:49.821 [debug] wrote build.zig to /tmp/.zigler_compiler/dev/Elixir.Ziggle/build.zig
Compiling lib/ziggle.ex (it's taking more than 10s)

== Compilation error in file lib/ziggle.ex ==
** (CompileError)  this zig compiler warning hasn't been incorporated into the parser.
Please file a report at:
https://github.com/ityonemo/zigler/issues
error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/Unwind-sjlj.c:14:
/usr/local/lib/zig/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found

error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/Unwind-seh.cpp:13:
/usr/local/lib/zig/libunwind/src/config.h:16:10: fatal error: 'assert.h' file not found

error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/UnwindLevel1.c:21:
/usr/local/lib/zig/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found

error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/UnwindLevel1-gcc-ext.c:12:
/usr/local/lib/zig/include/inttypes.h:21:15: fatal error: 'inttypes.h' file not found

error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/libunwind.cpp:12:
In file included from /usr/local/lib/zig/libunwind/include/libunwind.h:18:
In file included from /usr/local/lib/zig/libcxx/include/stdint.h:106:
/usr/local/lib/zig/libcxx/include/__config:247:12: fatal error: 'sys/endian.h' file not found

error(compilation): /usr/local/lib/zig/libunwind/src/Unwind-sjlj.c:1:1: unable to build C object: clang exited with code 1

error(compilation): /usr/local/lib/zig/libunwind/src/Unwind-seh.cpp:1:1: unable to build C object: clang exited with code 1

error(compilation): /usr/local/lib/zig/libunwind/src/UnwindLevel1.c:1:1: unable to build C object: clang exited with code 1

error(compilation): /usr/local/lib/zig/libunwind/src/UnwindLevel1-gcc-ext.c:1:1: unable to build C object: clang exited with code 1

error(compilation): /usr/local/lib/zig/libunwind/src/libunwind.cpp:1:1: unable to build C object: clang exited with code 1

error: unable to build libunwind: BuildingLibCObjectFailed
Elixir.Ziggle...The following command exited with error code 1:
/usr/local/bin/zig build-lib /tmp/.zigler_compiler/dev/Elixir.Ziggle/Elixir.Ziggle.zig --single-threaded --cache-dir /tmp/.zigler_compiler/dev/Elixir.Ziggle/zig-cache --global-cache-dir /home/dch/.cache/zig --name Elixir.Ziggle --version 0.1 -dynamic -target x86_64-freebsd-gnu -isystem /usr/local/lib/erlang23/erts-11.1.7/include -isystem /usr/home/dch/src/zig/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/libc/musl/include -isystem /usr/home/dch/src/zig/ziggle/deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/libc/include/x86_64-freebsd-musl --enable-cache 
error: the following build command failed with exit code 1:
/tmp/.zigler_compiler/dev/Elixir.Ziggle/zig-cache/o/200da5397a70b244375bad7e52bbfd99/build /usr/local/bin/zig /tmp/.zigler_compiler/dev/Elixir.Ziggle /tmp/.zigler_compiler/dev/Elixir.Ziggle/zig-cache /home/dch/.cache/zig

    (zigler 0.7.2) lib/zig/parser/error.ex:54: Zig.Parser.Error.parse/2
    (zigler 0.7.2) lib/zig/command.ex:36: Zig.Command.compile/2
    (zigler 0.7.2) lib/zig/compiler.ex:102: Zig.Compiler.compilation/2
    (zigler 0.7.2) expanding macro: Zig.Compiler.__before_compile__/1
    lib/ziggle.ex:1: Ziggle (module)

which is ... equally bad

[dch]

build.zig.log

[ityonemo]

aw crap. I'm so sorry. I think what is happening is that I disabled implicit libc and zig doesn't currently have an obvious way to load the libc headers without loading libc. ziglang/zig#7687 and if I'm not mistaken freebsd requires libc on account of it being the official OS syscall entry point (*bsd does not guarantee kernel syscall stability). Do you have any suggestions on how to set up freebsd CI pipelines? I want this to not be a problem moving forward.

[ityonemo]

can you try including link_libc: true in your use Zig options and see if that makes it go away?

[dch]

per chat, that doesn't seem to help either - it does add this to the generated build.zig

defmodule Ziggle do
  use Zig,
    system_include_dirs: ["/usr/include", "/usr/lib/include"],
    link_libc: true
...

    lib.linkSystemLibrary("c");
    lib.addSystemIncludeDir("/usr/include");
    lib.addSystemIncludeDir("/usr/lib/include");  

and shows

error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/libunwind.cpp:12:
In file included from /usr/local/lib/zig/libunwind/include/libunwind.h:18:
In file included from /usr/local/lib/zig/libcxx/include/stdint.h:106:
/usr/local/lib/zig/libcxx/include/__config:247:12: fatal error: 'sys/endian.h' file not found

error(compilation): clang failed with stderr: In file included from /usr/local/lib/zig/libunwind/src/Unwind-seh.cpp:13:
/usr/local/lib/zig/libunwind/src/config.h:16:10: fatal error: 'assert.h' file not found

[zigster64]

Reporting the same issue here - will try to fix on my system and post an update if I get it working.

defmodule LiveCounter.Count do
  use GenServer
  use Zig,   local_zig: true, link_libc: true

error

error(compilation): clang failed with stderr: In file included from ...deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/libunwind.cpp:12:
In file included from ...deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/include/libunwind.h:18:
In file included from ...deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libcxx/include/stdint.h:106:
...deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libcxx/include/__config:247:12: fatal error: 'sys/endian.h' file not found

error(compilation): clang failed with stderr: In file included from ...deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/Unwind-seh.cpp:13:
...deps/zigler/zig/zig-freebsd-x86_64-0.7.1/lib/zig/libunwind/src/config.h:16:10: fatal error: 'assert.h' file not found

...

(file paths shortened above for brevity)

[zigster64]

odd finding : in lib/zig/command.ex. line 18

def compile(compiler, zig_tree) do
  zig_executable = if compiler.module_spec.local_zig do
    Logger.debug("Using local zig")
    System.find_exectuable("zig")
  else
    Logger.debug("Not using local zig")
     ...

Regardless of the setting of local_zig in my app code, the variable isnt getting set by the time it gets to compiler.ex

If I force hack the code to use the system zig, that does seem to work.

Just leaves the weird include errors - having a look at that now

[zigster64]

OK - in lib/zig/builder.ex, which I think has the template for generating the build.zig file

I changed the target definition from
const target = b.standardTargetOptions(<%= to_structdef target %>);

to a generic
const target = b.standardTargetOptions(.{});

And now it compiles without all the include errors.

No idea if this is a fix - Im just hacking, so treat with caution :)

... a few moments later :
With a little bit of fiddling around with my nifs code (Im a complete Elixir noob here, so fumbling around a bit)
and it works !

[ityonemo]

this should be resolved now in d1afc54 (note this is currently in devel and will be merged as part of 0.8.1 release)