Update: With some difficulty, switching to Wayland fixes it. I guess another side effect of Xorg slow death spiral?
Aside from that, I could have sworn enabling Wayland on nVidia was a lot easier in the past, seems to have regressed. Not super surprising though.