Skip to content

Conversation

@cj-zhukov
Copy link
Contributor

Which issue does this PR close?

Rationale for this change

This PR is for consolidating all the udf examples into a single example binary. We are agreed on the pattern and we can apply it to the remaining examples

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@cj-zhukov
Copy link
Contributor Author

High-Level Overview

This PR consolidates all udf (simple, advanced, async) examples into a single example binary.
Previously, each example had its own file, but now they can be executed via subcommands using:

cargo run --example udf -- [adv_udaf|adv_udf|adv_udwf|async_udf|udaf|udf|udtf|udwf]

Copy link
Member

@xudong963 xudong963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I checked the original issue and the PR implements the proposal.

Could you please do some verification to see if the PR can reduce the binary size?

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cj-zhukov

@xudong963 per testing:

on this branch, the example binary is 174M

cargo run --profile=ci --example udf -- adv_udaf`
$ du -h target/ci/examples/udf
174M	target/ci/examples/udf

On main, the equivalent 6 binaries are:

cargo run --profile=ci --example advanced_udaf 
cargo run --profile=ci --example simple_udaf 
cargo run --profile=ci --example advanced_udf 
cargo run --profile=ci --example  simple_udf 
cargo run --profile=ci --example advanced_udwf 
cargo run --profile=ci --example simple_udwf

And they each take 173M

$ du -s -h target/ci/examples/*
173M	target/ci/examples/advanced_udaf
173M	target/ci/examples/advanced_udf
173M	target/ci/examples/advanced_udwf
173M	target/ci/examples/simple_udaf
173M	target/ci/examples/simple_udf
173M	target/ci/examples/simple_udwf

So by my calculations this PR saves 173*6 - 124 = 914MB !

@cj-zhukov -- one thing I noticed is that by consolidating the examples, we are likely no longer actually execute them as part of CI:

example_name=`basename $filename ".rs"`
# Skip tests that rely on external storage and flight
if [ ! -d $filename ]; then
cargo run --profile ci --example $example_name
fi

Could you make a new PR that also runs them all? Maybe something like add an ExampleKind::All type to the examples and then pass in all as an argument 🤔

@cj-zhukov
Copy link
Contributor Author

@alamb @xudong963 Thanks a lot for the review and helpful feedback!
I’ll continue with consolidating the remaining examples next.

I’m also planning to open a follow-up PR to update part of the CI in rust_example.sh - this was originally pointed out by @Jefffrey in this comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants