Yep that's the new idea. The sad part is that with this method there's no way to get historical data. Only new posts. So if a server goes down, gets DDOSd etc... I'll lose posts forever.
Also building an ActivityPub implementation from scratch isn't trivial either. So that'll take some time.
I've got a few other ideas I'm playing with as well. Like just assuming that internal post IDs are all sequential and literally fetching them one by one. Or maybe some combination of both?
A couple of options in my opinion, as I just did this myself:
You can use the CLI tool to "upload" them. You can even do this from the server itself. So upload times would be as fast as your network card can process or however fast your server is, whichever is slower. It does require that you create an API key for the user in question though.
Otherwise you can create an external library and link that to your account. Now Immich will still index this library but it won't move or manage the actual files. I'm not sure though if it looks at those files for duplicates (i.e. if you try and upload the same photo from your phone to the server). This external library will also prevent deleting photos as well, FYI.
There might be other options that I'm not aware of, as I've only been using Immich for about a month now.
Edit: link to the CLI documentation: https://immich.app/docs/features/command-line-interface/