Hey thank you for the help.
I have run a jellyfin off my gaming PC and that worked fine out of the box - no stuttering. This is over wifi on the same network, which makes me think that there is something specific to my new set up that is going wrong.
I will definitely check the network speeds though and run a few more tests.
The data is on an SATA SSD but I could try moving some over to the small M2 drive to see if there is a difference. Jellyfin is the only process accessing it.