Size of build artifacts

[max-sixty]

Hi there — we're trying to upgrade to 0.8.1. Unfortunately we're getting failures in CI about files being too big.

I see there's Windows CI on this repo, so I'm unsure why it's hitting PRQL but not here.

For context

• We first saw an error on the msvc toolchain

  D:\a\prql\prql\target\x86_64-pc-windows-msvc\debug\deps\integration-bada33bd8ffe8f7b.exe : 
  fatal error LNK1180: insufficient disk space to complete link
  

• After seeing https://stackoverflow.com/a/68596301, we tried switching to the gnu toolchain, but also see an error

  warning: C:\Users\RUNNER~1\AppData\Local\Temp\ccXJpLZD.s: 
  Fatal error: D:\a\prql\prql\target\x86_64-pc-windows-gnu\debug\build\libduckdb-sys-e63aefa3250cb202\out\duckdb/third_party/fastpforlib/bitpacking.o: 
  file too big
  

Do you have any thoughts? Happy to wait to see whether other dependencies see similar errors.

[wangfenjin]

We are using msvc for the CI in this repo.

Have you ever retry?

[max-sixty]

We've tried msvc — that's what we've traditionally been using. CI has failed > 5 times.

It's possibly not the fault of this repo though — unless you have any immediate ideas, we can see whether other projects are successful at using 0.8.1 in CI over the next few days?

[max-sixty]

I think this is actually just an issue with disk space on Windows GitHub Actions Runners. We're now getting an error on 0.8.01

A couple of links:

• https://github.com/servo/servo/pull/27865/files#diff-7829468e86c1cc5d5133195b5cb48e1ff6c75e3e9203777f6b2e379d9e4882b3R25
• Running out of disk space on Windows 2019 actions/runner-images#1341

So unless there's some way to reduce the amount of disk space that duckdb uses in its build, we can close this. Larger projects such as PRQL might be unable to use duckdb-rs in Windows GHA.

Footnotes

1. (this run has less disk space available because it pulls a cache but also needs to rebuild because the cargo command changed) ↩

[wangfenjin]

Disk should be very cheap, don't understand why github have these such a limitation 😂

[wangfenjin]

Or you guys can follow the the issue and use the C disk?

[max-sixty]

Or you guys can follow the the issue and use the C disk?

This looks more complicated than we're hoping for — we're not really testing duckdb, we're testing PRQL. So I think it's easier for us to disable using duckdb on Windows.

[max-sixty]

FWIW — and very possibly this is necessary, duckdb is a big project — but duckdb-rs does use lots of space!

In PRQL, after cargo clean && cargo build --features=test-dbs:

• The liblibduckdb_sys-.rlib file within target/debug/deps is 684MB; the next largest dependency is libsqlparser at 57MB
• The libduckdb-sys- path within target/debug/build is 1.3GB; the next largest dependency is libsqlite3-sys- at 8.0M (!)

---

Renaming this to represent the size of the cache — but if that's not really flexible, please close!

[max-sixty]

FYI we're getting disk space errors in GHA on ubuntu now:

warning: ar: /home/runner/work/prql/prql/target/x86_64-unknown-linux-gnu/debug/build/libduckdb-sys-fe973d02cdbe21fe/out/libduckdb.a: No space left on device

Running the command locally:

CARGO_INCREMENTAL=0 cargo doc --features=test-dbs-external --no-deps

...leaves 2.3GB of files, o/w 1.3GB is target/debug/build/libduckdb...

...so it is using much more space than anything else.

(TBC, thank you very much for the project, and sorry to only appear when there's a problem!)

[mlafeldt]

Dealing with huge build artifacts is a known issue with Rust. You can tweak a compiler knob here and there, but a target/ directory with even 10s of gigabytes is entirely normal in Rust projects.