The original Xbox shipped with 64MB of RAM, which was quite a lot for its time in 2001 but seems small by modern standards. A modder called Prehis2oricman has quadrupled the amount of available RAM to 256MB using a custom RAM module that consists of two parts: a RAM chip and an interposer allowing use of all eight memory slots including those typically empty on the console's motherboard.
This modification was achieved through months of reverse engineering, hardware engineering, software hacking, and perseverance. However, there's little reason to install this mod currently as existing applications don't utilize it. The potential for new BIOS updates that unlock additional features could change the situation in the future.
Do you think modifying console hardware like the Xbox to quadruple its original RAM is worth pursuing, considering the current state of software support?
Nah, your thinking 128, which matches what the devkit system would have had.
With that mod, you had to add additional RAM chips in the unpopulated traces and installing a new BIOS.
The new mod requires you to remove the old RAM and install new RAM chips that have an custom adapter board on them. Plus an even newer BIOS.
The problem now is that no software is built to make use of it.