They said it's mostly about the amount of moderation action they've had to take against users from those instances. Maybe lemmy.ml has less users who behave badly outside their server?

I could also see there being a reticence to defederate from the "canonical" instance so to speak.

lemmy.ml is significantly smaller because they chose to point users to other instances and even closed registration at one point to ensure that they could continue to keep up with demand.

This was my first thought, as well.