This seems like a non-issue if you use FDE and proper file permissions. And even if you don't want to type your password at boot, you can use TPM for the FDE itself.

I prefer use Yubikey/Canokey instead of TPM. SSH key may be used outside or in other environment (such as my laptop, Android or iPad). And TPM support less algorithms than Yubikey, especially Ed25519 and RSA4096.