I love my Anbernic handhelds. Pretty much any of them, but I draw the line at the Android ones. Might as well just get a controller for a phone at that point. The Linux ones are neat though - currently my daily driver is the 351V. Debated getting a 351P, partially because I mostly play GBA and the screen is made for that but also because I collect them.

IMO Steam Deck is the best overall option, as it's beefy enough to run PS3 or even Switch games.

If it's too heavy, the Retroid Pocket 3+ is a decent emulation competitor with a lighter, smaller form factor.

I find myself constantly switching between them all and can’t settle down to actually play a game.

This sounds like one of those problems where buying more hardware doesn't help. Ruminating about the "best" handheld choice is an easy way to forget about playing the games themselves. I've been there. The solution is to just pick a device at random, and go outside to play it. There will be pros and cons, but if you're not even getting to the point where you can actually play the game, then the pros and cons don't matter either.

I think a modded PSP would be the best bank foe the buck.

It can emulate tons of retro classics (and of course the original PlayStation)

And the PSP catalog is full of hidden gems as well.

RG35XX

PlayStation Vita is fantastic for handheld retro gaming, it has Retro Arch and can handle pretty much anything up to GBA. It also runs PS1 and PSP games almost perfectly.

Plus the slim model has Micro USB charging