Just remember that there's a big difference between "claimed" and actual. Just because a gasser returned 40 mpg once with the wind at its back going downhill in neutral doesn't mean 40 mpg in the real world. This despite EPA estimates.
The TDI on the other hand very seldom returns less than EPA estimates and usually beats them by a good margin. Again despite the EPA and all their facts and figures.
The biggest reasons I see for choosing a gasser are the lesser upfront costs and less maintenance being required. Basically an owner can fill the tank, turn the key, and drive without much thought to what's going on under the hood. That is, of course, until nothing happens when the key is turned. And let's face it, there are certain brands that run forever despite the abuse they take.
The TDI is more personal if you will. I'd never recommend the TDI to a habitually negligent owner who has no concept of maintenance. For that matter I wouldn't recommend to anyone unwilling to spend a lot of money at a shop or unable to perform basic maintenance on a TDI.
Then there comes something we really can't quantify. To some (maybe many) a car is simply a means of getting between points. It's a combination people mover, cafeteria, and trash bin. To others a car is something to be enjoyed and not abused. While I'm not in the camp of "loving" a hunk of metal, I think it fair to say that there's something about the TDI that draws me toward the diesel.
So... No cafeteria, no trash, and no drinks in our cars. We (family) all have TDI's and we perform our own maintenance. We do this not only because we have the facilities but because we want to. Maintenance is meticulous. In return we're rewarded with a car that gives back in driving pleasure and economy. VW and the TDI in particular isn't for everyone. It's more a vehicle made for a driver who appreciates and is willing to pay a premium in return for something we TDI owners feel is special.
Perhaps this is where VW marketing fails. It seems to me that VW fails to understand that their strength lies not in being a Honda or Toyota but in being a very Teutonic and different approach to transportation all the way from the most basic models right on up the line. The TDI being perhaps the most different.
I've driven the gas editions of Beetle, Jetta, and NMS. They're all nice cars. All seem pretty well built. The thing is that each is very little different from the offerings of other manufacturers. Why would anyone purchase one of the VW offerings when there's a Honda or some such store on virtually every corner and the nearest VW dealer is sixty miles distant?
Then there's the TDI. In my opinion the TDI is what makes the car. It makes an already nice car something special. Some appreciate and some don't.