If you have an individual or group philosophy that says people are either with you or against you, there is a very good chance that people who agree with you will end up against you.

Follow your morals and ideals. If one of those morals isn’t inclusion, you might want to look into that. Paraphrasing Voltaire, don’t let the perfect be the enemy of the good.