Two best are within walking distance of each other in the east village...
Other Music - E. 4th (just east of broadway)
Kim's - St. Marks (between 2nd and 3rd ave)
both are better than anything in DC...Other Music has better vinyl and maybe slightly lower prices on some import and experimental stuff, Kim's has a larger selection of new indie, and more in stock.
Earwax in Brooklyn (on Bedford...around north 6) is also really well stocked, although really small. But they always seem to have the hard to find stuff.
oh it's a heck of a town...