Send event invites to select groups not by visibility

I have events that are visible to multiple groups, but I only want to send calendar invites to a subset - e.g., visible to all cohort alum, but only send invites to the active cohort/select groups.
Currently to avoid event invite spam- I have to either:

1. By default, show the event to "all users" but not send any notifications (ie, send a separate follow-up email to just the active group)
   OR
2. Choose to limit visibility to the active cohort/group and select send invites