I think you're right that only a tiny minority are directly responsible for the negative interactions, but as someone within academic science, there's also a much larger chunk of people who don't challenge the assholes or the systemic fuckery when they see it.

Minorities who face oppression are much more likely to be ignored if they report inappropriate or offensive behaviour; I directly know people who have been made to feel like they are the problem for highlighting a problem. This is especially common if it's an established and respected academic who makes the iffy comments, because there's a tendency to them like a senile grandparent at Christmas. If they're a professor emeritus, there's a sense of them not really being relevant anymore, even if they're still respected, but it can feel tremendously isolating to see no-one step in to challenge the comments, either at an individual or institutional level.

It's understandable to not want to rock the boat, but abstaining is easier for some than others.