Something like Zoho is only $12 a year per hosted email address.
The cheapest CyberPower/APC that is pure sine wave should be just fine.
Lots of cheep SFF/thin client machines on eBay.
Sorry guys, it's because I just bought another domain today. This really sucks, Google Domains was easy to use and cheap.
Assuming you don't want to expose these services directly to the internet (I don't recommend it) then you want to set up a VPN to connect back to your home network. Wireguard or OpenVPN are the most commonly used. As far as guides that will depend where/how you want to run it.