Loofa-away my man, what happens in the shower should stay in the shower.
And I'm betting that as long as your shower/bath is clean pretty much anything goes (short of well things I won't mention, but I shouldn't have to mention, because well they're unmentionable) with the ladies.
As far as I can tell, clean trumps most other transgressions, though both my bathrooms are loofa free, not out of prejudice toward your spongy friend, but rather because my customary and traditional wash methods seem to work just fine.
In closing have a Loofa-Palooza and ignore the haters.