I doubt this person actually had a computer than could run the 405b model. You need over 200gb of ram, let alone having enough vram to run it with gpu acceleration.
Hot damn that's a lot! They ain't messing around with that requirement.
My current server has 32 MB of VRAM. Yes, MB not GB. Once I finish the hardware migration it's going to 8GB but that's not even a drop in the bucket compared to that requirement.
I doubt this person actually had a computer than could run the 405b model. You need over 200gb of ram, let alone having enough vram to run it with gpu acceleration.
In terms of RAM it's not impossible, my current little server has 192GB of RAM installed.
Pic from TrueNAS
The VRAM would be quite the hurdle though, I'm curious on it's requirements for VRAM
Edit: Moving data in anticipation of a hardware migration ATM so basically none of the services are running.
VRAM would be 810Gb/403Gb/203Gb for FP16/FP8/INT4 for interferrence, according to their website.
Hot damn that's a lot! They ain't messing around with that requirement.
My current server has 32 MB of VRAM. Yes, MB not GB. Once I finish the hardware migration it's going to 8GB but that's not even a drop in the bucket compared to that requirement.