Ability to turn off invite Admin to Voice Chat

I currently have voice chat enabled for my community because I'm encouraging members to use this feature for Matchups. However, today one of my members tried to invite me into a voice chat, which I definitely don't want people doing. If I want to open up office hours or something like that, I'll create a channel for it. I don't want people thinking they can just randomly have meetings with me without any prior prompting or conversation. I would imagine most admins would initiate the chat if needed, not be wanting members to message them in this way, but this could be a toggle option in the user permissions area.