46
Advice - Getting started with LLMs
(beehaw.org)
Rumors, happenings, and innovations in the technology sphere. If it's technological news or discussion of technology, it probably belongs here.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
Ok, so using my "older" 2070 Super, I was able to get a response from a 70B parameter model in 9-12 minutes. (Llama 3 in this case.)
I'm fairly certain that you're using your CPU or having another issue. Would you like to try and debug your configuration together?
I think I fucked up my docker setup and will wipe and start over.
Good luck! I'm definitely willing to spend a few minutes offering advice/double checking some configuration settings if things go awry again. Let me know how things go. :-)
My setup is Win 11 Pro ➡️ WSL2 / Debian ➡️ Docker Desktop (for windows)
Should I install the nvidia drivers within Debian even though the host OS already has drivers?
I think there was a special process to get Nvidia working in WSL. Let me check... (I'm running natively on Linux, so my experience doing it with WSL is limited.)
https://docs.nvidia.com/cuda/wsl-user-guide/index.html - I'm sure you've followed this already, but according to this, it looks like you don't want to install the Nvidia drivers, and only want to install the cuda-toolkit metapackage. I'd follow the instructions from that link closely.
You may also run into performance issues within WSL due to the virtual machine overhead.
I did indeed follow that guide already, thank you for the respect; I am an idiot and installed both the nvidia WSL driver on top of the host OS driver _as well as the Cuda driver. So I'll try again with only that guide and see what breaks.
We all mess up! I hope that helps - let me know if you see improvements!