If we're going by biblical Christ anecdotes; my co-worker, who is a Jehovah's Witness, claims the reason they don't hold celebrations, while believing in Christ, is because the Bible makes no claim of Jesus' birth date or references a celebration of such within text. So in his words December 25th is just some arbitrary date.
I personally feel that it's safe to say that we live in a country where most Christmas revelers' primary image of the holiday is: fat man, red coat, and delicately wrapped (if you're my SO) expensive gifts (do people really gift cars for Christmas... 1995 Toyota Corolla here so what do I know) and a pine tree or reasonable facsimile. None of which, I think, existed at some time early AD in the middle east.
To me, "non-believers" celebrating Christmas is like a family that chooses to root for a sports team that they have no (or at best a tenuous) affiliation with. It's their choice, it's fun to have something that you like to do.
Maybe we should just roll pagan Christmas celebrations in with Thanksgiving, they're already playing the songs by then. That way the true believers can have the whole day for quiet contemplation or religious ceremony.
Also, I grew up baptized Catholic with a lapsed Jewish father. . . So I celebrate what I know. . . Presents, gelt and Chinese food.