Remember - perception becomes reality. If 6'2" 260lb football playing amish dude feels unsafe in the area, then that becomes 'his' reality. The truth is I don't think anyone is going to attack a 6'2" 260lb'er....horse and cart or not!
Down here there's sharks just off the beach so my perception is I'm going to get eaten if I venture into the ocean...the reality is I see tons of surfers out there every day so the true reality is it's there's a very small chance I'd get bitten, but you'll never convince me of that.
I don't think I've ever felt unsafe in any area of DC. What was that venue called in the middle of S.E.? I thought that was okay too.
IF you're not used to being in an urban environment then any city could be a bit threatening to some people...it's not a weakness, just a foriegn envioronment. Muggers etc are basically cowards so if you walk around scared then you're prey, if you walk around with an air of confidence then they won't come near you.