Wikipedia

Because that would basically mean building a brand new plane. Which they kind of did: the Mcdonald Douglas F4 Phantom II replaced it not long after.
The cause for the high landing speed of the F104 is its obvious stubby wings. However, they were designed that small for a reason: Doctrine at the time called for speed above anything else, even sacrificing maneuverability. F104 ended up being insanely fast, but had some of the worst turning radii out there, simply because the lifting surfaces are so small.

EDIT: I found a good video of one landing: https://youtu.be/5xTXPo4gIys
Notice how fast it's going and the high angle of attack compared to other aircraft. It was built to go high and fast. And as you can probably guess, most landings are done low and slow.

A contributing factor was that the ejector seat of the F104 ejected the pilot down instead of up. This meant that the ejector seat was only useful if at altitude. I believe I read somewhere that the reason for this design choice was to not have the pilot hit the tailfin, but I cannot confirm this. HOWEVER, it is also worth noting that at the time, there were no ejector seats thaylt could safely eject the pilot close to the ground, no matter the direction. A parachute needs a lot of altitude to properly unfold, and seats at the time simply didn't launch high/far enough, so ejecting near the ground wasn't much of an option.
Later revisions of F104 had an upward firing ejector seats, once seats powerful enough had been developed.

Fun fact: the first successful 0/0 ejection (no velocity, no altitude) was an accident. A Soviet pilot had just started to power up his plane when a design flaw unexpectedly caused the ejection seat to launch. As the hydraulics came live, the elevator moved to neutral position, and the stick of course moved with it. The strap for the ejector seat happened to be around the stick, so the stick effectively pulled the strap and ejected the pilot. Lucky for him, the seat turned out to be powerful enough to provide a safe 0/0. Despite having an emotional moment, he was safe and sound.

This incident is why that particular plane model has two pieces of wood connected with a string slung over the tail to support the elevator when parked. That way the elevator remains in neutral position even when without hydraulic power. Sadly I do not remember which plane it is, but it's one of the older models that are still in use in russia.
EDIT: It was/is Su-25

And just for the record, despite its flaws, I think the F104 is a pretty cool aircraft.