this post was submitted on 25 Jul 2024
656 points (100.0% liked)

196

16566 readers
2111 users here now

Be sure to follow the rule before you head out.

Rule: You must post before you leave.

^other^ ^rules^

founded 1 year ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 31 points 4 months ago (2 children)

You're running a 405b param model on 24gb of VRAM, no shit it's not gonna work

[–] [email protected] 1 points 4 months ago (1 children)
[–] [email protected] 1 points 3 months ago

Yeah I'm sure that's how they got it to run at all lol, luckily they've fixed a lot of the issues with earlier versions of model runners, I had blue screen running 7b models back then. One of this size might've literally started a fire on my computer