Skip to main content ↑ ↓
Topic: SOLVED: Seeking Advice on Setting Up NVIDIA CUDA (Read 414 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

SOLVED: Seeking Advice on Setting Up NVIDIA CUDA

Hi all.

I recently started experimenting with running various AI LLM chatbots from huggingface locally via llama.cpp. I'm having fun and found a model with settings that produces decent code for a beginner like me. The problem is that right now I'm running llama.cpp solely with my Ryzen 5 3600 CPU, which runs it fine for a 7B parameter model, but at 13B or above, the text output from the chat bot starts to lag bad. Since I have a NVIDIA 2060 for gaming, tutorials online say that to enable GPU processing for llama.cpp to install CUDA. Upon looking this up though, it appears it may not be as simple as installing the package.

When I pacman -S cuda-toolkit, it looks like the world repo indeed has what I need. But, from what I've been reading, there also seems to be some set up involved  with pointing CUDA within my PATH environment variable?

Normally I'd be happy to just wing it and configure it based off of docs and tutorials, but this CUDA set up seems a tiny bit in depth. I could just be overthinking things, but I wanted to reach out here to the community to see if anyone has successfully set up CUDA on their Artix systems and if there's any gotchas or caveats to doing this. Anyone got any advice? Is it as simple as install CUDA and  adjust your PATH variable?

Any and all help/advice is of course, greatly appreciated, and thanks as always for taking the time. ☮️

Re: Seeking Advice on Setting Up NVIDIA CUDA

Reply #1 –
You want the package 'cuda'.
The package installs /etc/profile.d/cuda.sh which will set up $PATH (log out and in) as long as you use Bash as your login shell.

Re: Seeking Advice on Setting Up NVIDIA CUDA

Reply #2 –
Quote from: gripped – on
You want the package 'cuda'.
The package installs /etc/profile.d/cuda.sh which will set up $PATH (log out and in) as long as you use Bash as your login shell.

Thank you, I'm glad it's relatively simple. I'm assumming that I'll need to adjust $PATH manually if I use Zsh?
 

Re: SOLVED: Seeking Advice on Setting Up NVIDIA CUDA

Reply #4 –
Thanks @gripped for all the help. I just needed to know how easy/not easy this would be. Turns out very simple thanks to the work the Arch/Artix devs put into maintenance. Just pacman -S cuda, adjust the PATH variable according to the archwiki's article on CUDA, and voila, done and done. Recompiled llama.sh with the appropriate flag, set the -ngl flag to use up appropriate amount of layers, and damn is llama much snappier feeding into my GPU! Thanks again! Super helpful as always around here. ☮️