Think of it this way: The idea behind an internal combustion engine is to convert chemical energy into mechanical energy. Heat energy is an unwanted byproduct. (Except in winter!) Any heat produced as a result of the combustion is "wasted" energy.
Diesel fuel does contain more available chemical energy than an equal volume of gasoline, but the difference is not nearly as much as I always thought it was. Someone pointed out to me a while back that the difference is only about 5%. But a diesel engine uses about 40% less fuel to produce the same power output as a gasoline engine with the same number of cylinders and total displacement. More mechanical energy from the same amount of fuel means less heat, since the chemical energy you've spent has to accounted for and the equation balanced. If it's not power, it must be heat! A major reason that a diesel engine is so much more efficient is the extremely high compression ratio. There is a lot more oxygen crammed into the cylinder at the top of the compression stroke, so the fuel is able to burn more thoroughly. A diesel engine is able to accomplish this because the fuel is not injected until the piston reaches the top of the compression stroke. It is only compressing air, not fuel. A gasoline engine compresses the fuel/air mix, so the compression ratio is limited by the fuel's ability to avoid self detonation, or "pre-ignition." (That's the "pinging" a gas motor makes when the octane is too low, or something else goes wrong.) By the way, many manufacturers are currently working on Direct Injection designs for gasoline engines which will greatly increase their efficiency and improve their emissions performance.
Both gasoline and diesel engines like to be "hot" in order to work efficiently. But with a gasoline engine the problem is "How do I get rid of all the EXCESS heat the engine produces?" With a diesel, the problem is just the opposite: "How do I conserve heat in the engine so that the fuel burns more completely?" On a cold day there's hardly any coolant circulating through your TDI, and the cooling fans will never even turn on! The problem of conserving heat also partially explains why the fuel mileage drops so much in the winter. ("Winterized" fuels containing less energy are also a problem.)
The "clattering" sound of a diesel has nothing to do with the amount of heat produced. A gasoline engine is burning fuel vapor mixed with air. A "flame front" moves outward from the spark plug. It is more of a rapid, smooth burn than an explosion. A diesel engine, on the other hand, burns very finely atomized droplets of liquid fuel. Ideally it would begin to burn instantaneously as it enters the combustion chamber from the nozzle of the injector, but in practice there is always a little bit of "delay" and a tendency for the fuel to explode all at once. Hence the "clatter" you hear. Many diesels, including TDIs, use Pilot Injection to help control the combustion noise. A tiny, metered quantity of fuel is injected several degrees before top dead center in order to start an "early" burn which helps ignite the main injection more rapidly. That also helps lower emissions. The new HEUI injector designs (High efficiency Electronically controlled Unit Injectors) use extremely high fuel pressures (about 10 times higher than most DI diesels)to more precisely meter the pilot injection, and to more thoroughly atomize the fuel spray for more efficient combustion. More complete burning means more power from less fuel, accompanied by lower emissions.
-mickey