Around here, most (new) hybrid buyers won't ever keep their car car as long as many TDI buyers will. So I think that draws in a very different demographic, as the possible "payoff" of driving something more fuel efficient may not ever be gratified in the owner's stewardship of the car.
I service a lot of 250,000+ mile TDIs. And many of these owners are happy to spend north of $3000 to get everything back in shape. A Prius owner will balk at spending $500, and that very night be shopping for a new car instead.
My boss loves his Prius. It is his 3rd one. He would not dare drive anything past about 50k miles... which is what I do in one year. He'll tell you how great they are if you ask him. But his opinion is of buying one wholesale slightly used (maybe 20k miles or so at the most), driving it a couple years to about 50k and selling it off retail. So at that rate, he essentially drives for hardly anything, does hardly anything to the car, and usually sells for almost what he paid. So yeah, he would love the car! And he's tiny, too, so he can fit in them to boot.
The other thing to consider is that many non-hybrid gas cars are available with very good fuel economy now. No, maybe not as good as some hybrids or TDIs, but still good enough to easily see the advantage in purchase given how much less they cost. My sister bought a new Civic sedan in 2004. It was not the top of the line model, but it was pretty well-equipped, with A/C, power windows/locks, slushbox, etc. It was still a full $8k less than the hybrid version, and it still was able to get 35 MPG mixed, versus the 45 for the hybrid. $8000 buys a LOT of gas, and if you don't drive all that much, it would hardly be worth the extra money.