These are the quality control issues you're seeing with the naked eye.. imagine if you did some measuring with a mike.
How accurate do you suppose the bearing clearances are for proper oiling? What do you suspect the actual hardness of the lobs is?
Not to be a Debbie Downer... but feel the need to confirm that you understand that if the cam goes (or if it prematurely wears out the timing belt that in your case sounds like is being driven at an angle) the head will grenade. If you're lucky it will not damage the bottom end of the engine as well when it does so, but it could.
It's also not going to go south the first time you start the car, or even the 10th...but it's a high stress component, particularly on a PD engine, and so time will take its toll.
Your car your rules of course, just making doubly-sure you're making an informed decision.
Per your other question: the rockers do not float... they all pivot on a fixed axis.