I don't think the cam or the bearings are the problem. The cam dimension is probably accurate, although the cam lobe wear issue have been historically, a problem.
Also, you realize, if you removed the usual bolts that are used in the PD, they are not reusable? We have a set of reusable chrome/ moly bolts for this engine. Some think you cannot reuse any bolts. We have been using this bolt for 10 years with no issues.
When installing the bearings, cleanliness counts. The bearings look dirty, but it may be torn bearing material instead. The first bearing pictured is scored. The second bearing pictured is off-set from it's proper location.
The other important thing to note is cam caps can only be installed in their intended location. The caps are marked 1-5, with #1 closest to the timing belt. The caps are align bored to fit only their respective locations. Do not mix up the cam caps.
Start with new bolts, as your old ones cannot be used unless they are like our reusable ones. The stock bolts are ZXN heads and will break off if reused. Ours are allen headed screws.
Install the cam followers. Install the cam bearings in the cylinder head saddles. Apply assembly lube to the cam followers and cam bearings. CLEANLINESS COUNTS!
Install the bearings into the cam caps and apply assembly lube to the bearings.
Start with cam caps 2 and 4, tightening the bolts incrementally until the cam caps are seated. Install the number 1, 3 and 5 cap. We usually install the cam seal now and use a piece of electrical tape over the woodruff key slot to protect the seal. Tighten all cam cap bolts to 48 inch lbs.
In order to properly center the bearings on the journals, we use a plastic tear drop mallet and strike the cam caps hard. The plastic mallet will not harm the aluminum caps, but will vibrate the cam bearings to center them. Install the cam sprocket (74 ft lbs) and use a counterhold tool to turn the cam. It should turn smoothly and at certain points, the cam will snap over from valve spring pressure.
If using the stock bolts, tighten to 72 in lbs + 1/4 turn. Strike caps at each increase in tension. Turn cam to see it runs smoothly in the journals. Finish with 1/4 turn.
At any time, if the cam becomes hard to turn, loosen cam cap bolts until the offending bearing cap is noted. See if it is dirt, a burr or if it's just a poor fit from cam to bearing.
Our reusable bolts are tightened to 144 in lbs;`12 ft lbs. Since they are not TTY bolts, no need for the 1/4 turn.
Good luck.