It sounds like you're describing the Paradox of Tolerance.
The paradox of tolerance states that if a society is tolerant without limit, its ability to be tolerant is eventually ceased or destroyed by the intolerant. Karl Popper described it as the seemingly self-contradictory idea that in order to maintain a tolerant society, the society must retain the right to be intolerant of intolerance.
I don't really have a good answer other than follow your heart, I guess.