First, you need to get an absolute commitment regarding the second job. The strongest they will go for (an employment contract is the gold standard, but those are hard to come by these days). You don't want to be in the position of bowing out of your new job, only to find out that the second company was front-running anticipated budget release/contract win/executive approval or something else that wasn't actually in the bag when you heard the offer. It does happen. In the process of establishing that certainty, ask your champion at the second company whether you can have a little flexibility on start date.
If you can, then you have a much better position as far as leaving the first company without completely burning bridges. You never can tell when somebody you burned in the past will show up as a roadblock in your future--plus there's your conscience, karma, and soft stuff like that. With some start date flexibility, you can offer to help recruit/train your replacement at the first company. Most likely the offer will be declined, but you'll score enough good-guy points to moderate the harsh feelings your quick flip might otherwise trigger.
