TLDR~ Networking is deceptively easy to grasp conceptually and infuriatingly fiddly to implement in the real world. I am looking for help and advice to design a solution that fits my needs but done 'the right way'.
The Hardware and Physical Network:
The main server is living in my home, it is an intel NUC running Ubuntu.
I rent a tiny VPS (linode) running Debian with a public facing static IP (hello internet!).
My networking is fairly standard consumer grade hardware with most things wired into my main gigabit switch. I have more than one wifi access point but all that is managed by my router. The router is connected to my ISP router in a way that creates a double NAT situation.
Before you comment - I can't change ISP, I can't open ports, I can't change the ISP router, my ISP doesn't hand out static IP addresses, if you have any questions about my ISP the worst possible answer is probably correct.
(The connection however is fiber and I'm getting about 800Mbps down / 80Mbps up.)
The Software and Setup:
I now have about 65 docker containers running all the usual services with plenty of extra apps that are somewhat useful but also just for fun (the number of containers is a bit misleading because I often have separate containers for databases or cron jobs and the like).
The greatest hits include: Nextcloud, Homeassistant, Jellyfin, Photoprism, Vaultwarden, Pihole, Mailu and more. I also have some services setup natively (apt install): tailscale, wireguard and mergefs. About half of my services are 100% local only where I can access them on an assigned port number (e.g. jellyfin would be "server_name:port_number") and I can access those services in a pinch either through tailscale or via wireguard which bring us nicely onto the VPS: The VPS is runnng a wireguard 'server' and I have wireguard client configs for my devices and for my home server so that if I connect to the wireguard VPN I can access my server and also route traffic to the internet just like any other VPN provider.
Reverse Proxy
Now this is the really tricky part. I also have my own domain and I have configured a series of subdomains for services that I want to be able to access seamlessly from anywhere. I don't want to use the VPS/VPN unnecessarily when I'm at home and I don't want to have to remember to toggle wireguard/tailscale whenever go out or come home. For the most part I have solved this but I am sure I have done this in an amateur way: I run two duplicate nginx reverse proxy containers, one on the VPS and one on the home server. The VPS is able to request and obtain https certificates from letsencrypt and these allow either reverse proxy to terminate the encrypted web traffic. When connecting from the public internet, nginx (on the VPS) then proxy forwards the connection unencrypted down through the wireguard tunnel (so still encrypted) back to my server at home. At home I am running a Pihole DNS server so that when I request the same domain but from my local network the request goes instead to the duplicate nginx reverse proxy. When connecting nginx (on my home server) the https termination is handled there and then forwarded on the wireguard network but now this is all happening on the server and not across the internet.
Happy days? Yes for the most part this all works great but it does seem like the wrong way to do it.
Mailu (but not really)
Currently my Mailu setup is limited as I am using the fetchmail service to get incoming emails from my existing email providers (gmail etc) and I am using a SMTP relay to send email. Mailu is not actually doing the job of a fully independent email server which I'm fine with for now and it gives me the possibility to expand its functionality in the future.
So really everything up to this point has been the back-story to explain my situation with Mailu now. Currently I have Mailu setup exactly as the developers suggest: Mailu is a collection of docker containers that make up the individual parts of an email server and the stack is all wrapped up nicely by using its own specially configured nginx reverse proxy (if you've been keeping score that makes 3). This reverse proxy manages the https connections and then routes them to the back-end servers (containers). I was able to make this work for the webmail service because I configured it to serve unencrypted over http and then use my existing nginx (VPS and home server) the same way as for all my other services. However I wanted to have access to the IMAP and the SMTP services and for this I took a different approach. I used the nginx "stream" block in the configuration to forward the IMAP and the SMTP connections from the public internet via the VPS but on my local network the connection is just direct to the home server. So I can connect Thunderbird from my laptop and K-9 from my phone and send and receive email. And with the magic of having my own DNS server this actually works totally fine because all the connections require authentication and I don't need to know the source IP address of the client connecting for this to work. So when I setup my mail clients I use my own domain for both the IMAP and the SMTP servers - very satisfying!
Sending and Receiving Email Properly:
Firstly, if the reports about actually sending email from your own sever are true then I can safely say its not for me. I am happy to use a relay I'm not a masochist....
I do want to properly receive emails to my email server and I think it should be much easier than sending. I did manage to receive some emails. I configured my domain MX records correctly and then by opening port 25 on my VPS and having nginx stream that connection to Mailu I was able to receive email directly (yay!) but with one major problem: the source IP for all the connections were from my internal wireguard IP and not the actual public IP of the server sending me the email. This was a huge problem for my spam filter as it needs the source IP when trying to identify spam. The number of false positives just skyrocketed.
Issues with Proxy Forwarding
I am obviously not the first person to face this issue so I know that technically this can be configured correctly. The issues I am facing is that if I try and enable any of the proxy protocols in nginx to send the client IP in the header then Mailu spits an error (like 500 bad command) because we are interfering with the email protocol and the connecting server doesn't like that. I have tried changing settings in the Mailu reverse proxy but everything I have tried so far has no effect. Alternatively we can go up a level and use some IP forwarding rules to rewrite the destination IP of the incoming packets (maybe using iptables instead of nginx) and this will transparently preserve the 'from IP' packet header but then we need to setup my home server to relay everything back to the VPS otherwise the return packets will come from my home IP and not the VPS IP which will fail to correctly establish the connection. My current thinking is to create another wireguard client (in a container this time) that is part of the Mailu stack and that I can assign its own virtual NIC and IP address. Then I can bind port 25 to a unique IP and then have a routing rule to redirect all this traffic from that IP back to the VPS.
Cry for help:
How do I get this to work without breaking everything!?
Have I made this more complicated than it needs to be?
Have I just reached the edge of what these systems were deigned to do?
Thank you!
Few weeks late to pitch in now but I can +1 docker-mailserver.
It has almost everything included and the configuration files are quite straightforward and flexible enough that you can drop little edits into the individual services if you need to tweak something.
My setup is very close to what you want: I use fetchmail to pull in from my old gmail and yahoo inboxes, I also have my own domain so I configured the MX records so that emails go straight to my server, with a fallback to my email provider (any mail that doesn't make it directly to my mailserver will still get pulled into my inbox with fetchmail when it comes back online).
Docker-mailserver allows you to set the SMTP of your instance to use your provider. This is important because it means that they do all the reputation stuff so that your emails work properly (and both my home ISP and my VPS provider don't do sending over pot 25 anyway).
So when I need to connect a new client (like Thunderbird) to my email I don't need to manually config anything as docker-mailserver has all the auto config messages so its really seamless. At the same time my risk is low because even if my sever is off my provider will still receive anything on my behalf. I can only send using the username I have paid for from my provider and switching between gmail and yahoo is not possible without rewriting configs and restarting services but its not something I want anyway. On the receiving side I can have any number of aliased usernames that will all be received by my server (but only when its on so i use them rarely and for disposable addresses).
Big downsides are:
Most of my complaints stem from the fact that I'm not very good at this but in the end it has been very satisfying to drop the occasional: "I host my own email BTW"
Good luck! Let us know how you get along!