Wearing those in my country aren't really an issue. I've never worn one myself but I've seen many people wear them even tough they are from different religious groups.
And in my time in the Middle East and North Africa people were very relaxed as it’s a practical item (mine have always been a neutral colour). Try it in New York or London though…
reply