llama : add thread safety test

[slaren]

Basic thread safety tests that loads a copy of the model on each GPU and CPU, and runs inference with multiple contexts in different threads.

[ggerganov]

Maybe we can use an even smaller model for this test:

https://huggingface.co/ggml-org/models/tree/main/tinyllamas

[slaren]

The SYCL ggml-ci does not seem to have libcurl installed yet.

[ggerganov]

Should be installed now.

[slaren]

There is some issue with this model (stories15M-q4_0.gguf) on CPU, but I don't think it is a threading issue. Only seems to happen on CPUs with AVX512.

test-thread-safety: /home/ggml/work/llama.cpp/ggml/src/ggml-cpu/ops.cpp:2934: void ggml_compute_forward_silu_f32(const ggml_compute_params*, ggml_tensor*): Assertion `!isnan(x)' failed.

[ggerganov]

I looked into it a bit and it does not seem to happen if OpenMP is disabled. Think it is something related to the repacking, but didn't confirm. I'll take an extra look now.

[ggerganov]

Pretty sure this is a data-race because the chunk counter will be shared by all contexts:

llama.cpp/ggml/src/ggml-cpu/llamafile/sgemm.cpp

Lines 395 to 398 in 487a5e0

	template <int RM, int RN, int BM>
	NOINLINE void gemm(int64_t m, int64_t n, int64_t BN) {
	static std::atomic<int64_t> current_chunk;

If I disable GGML_LLAMAFILE on ggml-2 the test works correctly even with OpenMP enabled.

@Djip007 Could you take a look and propose a fix?

[slaren]

19: 0.00.068.420 E common_download_file_single: invalid http status code received: 429

429 is "too many requests". @ngxson do you know if it is a temporary issue with huggingface, or are we being throttled?

[ngxson]

HF backend currently has a problem, the team is investigating, should be back very soon

[slaren]

@0cc4m @jeffbolznv The Vulkan backend is crashing on this test. It happens even with a single context per model (-np 1), which is not great because it would prevent, for example, evaluating a draft model simultaneously with the main model. I can hold merging this if you think it could be fixed in the near future, otherwise it might be better to disable the Vulkan CI tests for now.

[0cc4m]

It is known that the Vulkan backend is not thread-safe yet, yes.