feat(llama.cpp): Vulkan, Kompute, SYCL

[mudler]

Tracker for: ggml-org/llama.cpp#5138 and also ROCm

• Vulkan: feat(vulkan): add vulkan support to the llama.cpp backend #2648 (upstream Vulkan Implementation ggml-org/llama.cpp#2059 )
• Kompute: (upstream Nomic Vulkan backend ggml-org/llama.cpp#4456 )
• sycl: feat(sycl): Add support for Intel GPUs with sycl (#1647) #1660 ( upstream Feature: Integrate with unified SYCL backend for Intel GPUs  ggml-org/llama.cpp#2690 )
• ROCm: Build docker container for ROCm #1595 (feat(llama.cpp): enable ROCm/HIPBLAS support #1100)

[RiQuY]

The merge requests linked on this issue appears to be merged upstream. Does that mean LocalAI already supports Vulkan or there are any additional tasks to do before that?

[mudler]

The merge requests linked on this issue appears to be merged upstream. Does that mean LocalAI already supports Vulkan or there are any additional tasks to do before that?

Only kompute is missing as for now

[jim3692]

The merge requests linked on this issue appears to be merged upstream. Does that mean LocalAI already supports Vulkan or there are any additional tasks to do before that?

Only kompute is missing as for now

It looks like kompute is also merged

[KhazAkar]

So.. what's missing in LocalAI to support vulkan? Or compilation of in-tree llama.cpp to support vulkan would be enough to use it?

[arenekosreal]

It seems that we need to set BUILD_TYPE=vulkan to let llama.cpp use vulkan. But does this mean we have to rebuild the program if we want to use CPU backend? I do not see any option to let me switch between them at runtime.

I have a pretty old GTX 1050Ti Mobile GPU, which means that I can only run some mini models on it, but it is still much faster than run those small models on CPU. With the ability to switch dynamically, I can run mini models on my GPU while can also try some larger models with my CPU. The reason why I do not use cuda is that vulkan is much smaller than cuda runtime. Although using vulkan is slower than using cublas, I think it is acceptable on my old buddy.

My idea is that we can also let vulkan backend optional like cuda backend, so LocalAI will prefer vulkan backend while can also fallback to CPU backend. At this moment, I find that LocalAI cannot turn to use CPU if it meets large models failed to load into GPU when building with BUILD_TYPE=vulkan.

[KhazAkar]

It seems that we need to set BUILD_TYPE=vulkan to let llama.cpp use vulkan. But does this mean we have to rebuild the program if we want to use CPU backend? I do not see any option to let me switch between them at runtime.

I have a pretty old GTX 1050Ti Mobile GPU, which means that I can only run some mini models on it, but it is still much faster than run those small models on CPU. With the ability to switch dynamically, I can run mini models on my GPU while can also try some larger models with my CPU. The reason why I do not use cuda is that vulkan is much smaller than cuda runtime. Although using vulkan is slower than using cublas, I think it is acceptable on my old buddy.

My idea is that we can also let vulkan backend optional like cuda backend, so LocalAI will prefer vulkan backend while can also fallback to CPU backend. At this moment, I find that LocalAI cannot turn to use CPU if it meets large models failed to load into GPU when building with BUILD_TYPE=vulkan.

You can technically don't pass ngl or set it to 0 to not use GPU for offload

[arenekosreal]

It seems that we need to set BUILD_TYPE=vulkan to let llama.cpp use vulkan. But does this mean we have to rebuild the program if we want to use CPU backend? I do not see any option to let me switch between them at runtime.
I have a pretty old GTX 1050Ti Mobile GPU, which means that I can only run some mini models on it, but it is still much faster than run those small models on CPU. With the ability to switch dynamically, I can run mini models on my GPU while can also try some larger models with my CPU. The reason why I do not use cuda is that vulkan is much smaller than cuda runtime. Although using vulkan is slower than using cublas, I think it is acceptable on my old buddy.
My idea is that we can also let vulkan backend optional like cuda backend, so LocalAI will prefer vulkan backend while can also fallback to CPU backend. At this moment, I find that LocalAI cannot turn to use CPU if it meets large models failed to load into GPU when building with BUILD_TYPE=vulkan.

You can technically don't pass ngl or set it to 0 to not use GPU for offload

I found a very hacky way to achieve that: Build llama-cpp-fallback with BUILD_TYPE=openblas, while build other parts with BUILD_TYPE=vulkan. Then LocalAI will try vulkan first, and fallback to openblas. But this means that I have to enable other optimizations on llama-cpp-fallback. I am not sure if this usage is valid because there do not have any documentation about the mixture of types of backends.

[KhazAkar]

Interesting, thanks for that @arenekosreal !
Technically llama.cpp is able to be compiled with multiple backends backed in, so this distinction in LocalAI is interesting to see