NOT having it can be a net negative.
Maybe in the short term :) in the long run, you're filtering out unnecessary people.
Did you try to say you don't have WhatsApp when you meet someone? Doesn't that make you interesting in front of them? I never had the experience that someone insisted on using WhatsApp when we met. Only when we used WhatsApp for a long time already.