Running on top of MACbook 2019 (Intel i9 + AMD Radeon Pro 5500)

[mineirinh0]

Situation: It's running on CPU; when trying to run on GPU (after compiling correctly with Metal + OpenBlas), i'm reaching a point of break and the model does not run.

GGML_ASSERT: ggml-metal.m:766: false && "not implemented"
fish: Job 1, './main -m models/ggml-vicuna-7b…' terminated by signal SIGABRT (Abort)

My machine:

uname -a                                                                                                                                               (amd_gpu) 18:07
Darwin host 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64 x86_64

Do someone have a clue of what I'm missing?

[slaren]

The assert suggests that you may be using a type of quantization that is not supported yet in metal:

llama.cpp/ggml-metal.m

Lines 757 to 766 in 31cfbb1

	switch (src0->type) {
	case GGML_TYPE_F16: [encoder setComputePipelineState:ctx->pipeline_get_rows_f16]; break;
	case GGML_TYPE_Q4_0: [encoder setComputePipelineState:ctx->pipeline_get_rows_q4_0]; break;
	case GGML_TYPE_Q4_1: [encoder setComputePipelineState:ctx->pipeline_get_rows_q4_1]; break;
	case GGML_TYPE_Q2_K: [encoder setComputePipelineState:ctx->pipeline_get_rows_q2_K]; break;
	case GGML_TYPE_Q3_K: [encoder setComputePipelineState:ctx->pipeline_get_rows_q3_K]; break;
	case GGML_TYPE_Q4_K: [encoder setComputePipelineState:ctx->pipeline_get_rows_q4_K]; break;
	case GGML_TYPE_Q5_K: [encoder setComputePipelineState:ctx->pipeline_get_rows_q5_K]; break;
	case GGML_TYPE_Q6_K: [encoder setComputePipelineState:ctx->pipeline_get_rows_q6_K]; break;
	default: GGML_ASSERT(false && "not implemented");

[mineirinh0]

Thank you slaren, what does "type of quantization" means? (I'm kind novice at this field)
How to use 'other' type of quantization?

[slaren]

Quantization is the technique that ggml and llama.cpp uses to reduce the size of the models. There are many types, but some of them may not be fully supported yet in every backend. I would suggest looking for a model that is in one of the types listed there.

[ianscrivener]

I'm pretty sure that your Intel Mac with AMD GPU does not support Metal. The specs list Listed APIs are DirectX, OpenGL OpenCL and Vulcan

I'd suggest you try a CLBlast build... which uses OpenCL ie make LLAMA_CLBLAST=1

Good news is that that the Vicuna model you have should work with CLBlast.

[mineirinh0]

Thanks for this.... Yes, compiling with CLBlast works redirection workload to my Intel GPU... not for the AMD Radeon...
I need some more tries...

[ianscrivener]

Great!
don't forget to run with the -ngl 1 command line flag - this tells llama.cpp to use GPUs (not CPUs)

-ngl N, --n-gpu-layers N - number of layers to store in VRAM

[ianscrivener]

Also, check out the main README... it mentions

GGML_OPENCL_PLATFORM=AMD GGML_OPENCL_DEVICE=1 ./main ...

[guinmoon]

Situation: It's running on CPU; when trying to run on GPU (after compiling correctly with Metal + OpenBlas), i'm reaching a point of break and the model does not run.

GGML_ASSERT: ggml-metal.m:766: false && "not implemented"
fish: Job 1, './main -m models/ggml-vicuna-7b…' terminated by signal SIGABRT (Abort)

My machine:

uname -a                                                                                                                                               (amd_gpu) 18:07
Darwin host 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64 x86_64

Do someone have a clue of what I'm missing?

Hi i run llama.cpp on hackintosh Ventura with intel xeon E5-2640 v3 @ 2.60GHz 16GB RAM and AMD RX Vega 56 8GB.
When i run orca-mini-7b.ggmlv3.q3_K_M.bin everything works fine, but even a little slower than on the CPU, despite the fact that metal 3 support is indicated in the system information.

But when i run orca-mini-13b.ggmlv3.q4_1.bin i get :

ggml_metal_graph_compute: command buffer 0 failed with status 5
GGML_ASSERT: ggml-metal.m:969: false

I think this happens because in my system, like on old Macs, RAM and video memory are separated and 8GB video memory for metal is not enough to run a model 13B model. New Macs has unified memory and on Mac with 16GB Ram you can run 13B model with metal.

[ianscrivener]

You have an Intel Mac (x86_64)... and you are trying run Metal... that will not work.
Only Apple M1 or M2 chips can run Metal
Intel Macs usually working MUST with run OpenBLAS or CUBlas (Nvidia cards)

[mqy]

You have an Intel Mac (x86_64)... and you are trying run Metal... that will not work.
Only Apple M1 or M2 chips can run Metal

My MacBook Pro 2018 can run metal. Apple metal doc says:

Metal 3 is supported on the following hardware:
iPhone and iPad: Apple A13 Bionic or later
Mac: Apple silicon (M1 or later), AMD Radeon Pro Vega series, AMD Radeon Pro 5000/6000 series, Intel Iris Plus Graphics series, Intel UHD Graphics 630

[ianscrivener]

Thanks for the fact check.. I stand corrected.

Perhaps it is a Hackintosh issue?

You can check that Metal is working;

1. install pytorch for Mac - pip3 install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cpu
   .. this has the Metal drivers

2. check for Metal in python / pytorch

import torch
print(f"PyTorch version: {torch.__version__}")
print(f"Is MPS (Metal Performance Shader) built? {torch.backends.mps.is_built()}")
print(f"Is MPS available? {torch.backends.mps.is_available()}")
device = "mps" if torch.backends.mps.is_available() else "cpu"
print(f"Using device: {device}")

This is what it looks like when Metal is available via Pytorch

PyTorch version: 2.1.0.dev20230608
Is MPS (Metal Performance Shader) built? True
Is MPS available? True
Using device: mps

[khcd]

Dead thread revive but similar specs to you except a W5700x, examples run but ggml_metal_init seems to label all floating point operations as not supported and it falls back to the CPU

you can try the vulkan build with:
cmake -B build -DLLAMA_CURL=1 -DGGML_METAL=OFF -DGGML_VULKAN=1 -DVulkan_INCLUDE_DIR=~/VulkanSDK/1.3.296.0/macOS/include -DVulkan_LIBRARY=/usr/local/lib/libvulkan.1.3.296.dylib cmake --build build --config Release

I can confirm this does use the GPU but example models output is bugged, and I tried with Mistral-7B-Instruct-v0.3-Q4_K_M.gguf which outputs spanish. See issue: #10984

Browsing through numerous threads and issues it does not seem like anyone has gotten it to work with mac/intel/amd combo and actually utilising the gpu? Could be mistaken.