I get the biggest glares if I ever wear my Inspiral Carpets "Cool as Fuck" shirt or my Ned's Atomic Dustbing "Did you miss Ned's?" on the front and the "then you FUCKED UP" on the back. I couldn't really care less though, I try not to wear them during daylight hours when I'm more likely to run into small kids.
I usually just tell people that I'm a music geek and try to listen to as many types of bands as possible. I explain that I like everything from quiet acoustic folk stuff (Elliott Smith) to more electronic-based dance (Daft Punk) to hardcore punk (Minor Threat) to classic rock (AC/DC, the Who) to more "modern" rock (PJ Harvey). If they are polite they'll just mutter something like "cool", but on occasion people get more interested, especially when I tell them that I've been to over 500 shows and that I've been keeping a concert log from day one. They think I'm nuts, but they're intrigued because most people probably go to one concert a year, if that.