Lazy loading of NCCL symbols

[sogartar]

I ran into this issue where I want to run a program that does not use collective operations. I have an older NCCL version that does not have the function ncclCommInitRankConfig. This function appears in NCCL version 2.14.3. Whem loading the CUDA driver I get this error

c/runtime/src/iree/base/internal/dynamic_library_posix.c:315: NOT_FOUND; symbol 'ncclCommInitRankConfig' not found in library

In the code if the library is not found it is not an error and will fail only later on channel creation. Isn't it more correct that a symbol load failure be also silenced here? If lazy-like loading is intended, not having the right NCCL version is the same as not having it at all.

[sogartar]

@okkwon

[okkwon]

Yes, reporting an error at runtime is intended, since we don't ship the library with the IREE runtime. There is a version check so it is interesting to see that you didn't get the error message.

[okkwon]

Oh, yeah, the error happens even before checking the version, because the dynamic loading happens earliler.

[sogartar]

@benvanik

[sogartar]

Here is a PR #13403

[ScottTodd]

This looks better suited for an issue than a discussion. Can we close in favor of #13422? (Even though this was started first)