You are under no obligation to engage someone whose goal is to irritate you. Never be bullied into having a conversation. Have a conversation because you want to, not because you are being harrassed for not speaking to someone. Consent applies to all participants.
This is a message that sounds so natural and self-evident to me. Compare it with a more aggressive message that "nobody owes you teaching" (alternatively, that it's exploitative to ask members of oppressed groups to explain their position) that is in wide circulation among the woke left (on which I already ranted here).