As a general rule I just try to work out if an activity adds more to my life than it takes away when deciding how to spend my time. Is there something you'd rather be doing? Is reading manga affecting your mental state?

If only some manga makes you feel bad, consider asking for recommendations and if something bothers you just stop. If you feel bad because of social pressure or something, another general rule for me is as long as you aren't hurting yourself or others then do what makes you happy. In the interest of free speech, others can state their opinion if they want and piss off if they become bullies about it.