I 100% understand where you're coming from, and sometimes you just don't want to deal with people! And I think that's okay.
Can you guys make a deal that you'll leave early? And decide beforehand what time 'early' means. Like, the party starts at 7, we'll get there at 7 and leave by 8:30, or whatever. Say you have another party or you're not feeling well or something like that? Otherwise, I think it's fine to not go at all.