You probably just get a bounty to pay off. You can clear that at any bounty hunters kiosk, you can even build one at your settlement.
But I don't think you can screw things up permanently. Last save I killed everyone on the UC flagship. All it does is give you a bounty and fails the betrayal quest. Also the pirates think you're a badass and reach out to recruit you.