From the best I can hear, that sound in the video is an intake thump, very similar to the noise the PD engines make when they wipe a cam lobe or chew a hole in a follower, or what some other engines do if they bend a pushrod or wear a rocker arm out. I think your original focus on the valvetrain as the source of this issue is correct, I don't hear anything in your video that sounds like an injector knock to me, or an injection pump noise, etc.
That intake pop/whump/thump noise is usually an exhaust valve not opening enough, and/or closing too early, due to the flat cam lobe or failed follower. Then when the intake valve then opens for the intake stroke, there is residual exhaust pressure inside the cylinder that burps back into the intake, causing the popping noise. If it gets bad enough, you can also get MAF plausibility codes, find soot in the air filter, other signs like that. Have you seen any of those?
I suppose you could also get a similar noise from other valvetrain malfunctions, but I also hear what sounds like some lifter tap at the same time the thump happens. So I think insufficient lift or duration from excessive follower clearance is more or less still what you'd want to go after.
The possibility of excessive clearance in the lifter bore and resulting oil pressure loss and lifter bleed-down could be something to investigate, like you said. But in that case I would probably expect the issue to get worse when warm, not better. Still, maybe the lifter clearance gets tighter when the engine warms up and that offsets the thinner hot oil. I think you're right to keep this possibility on the list.
Remind us once more what kind of work was done on the cylinder head when you had the engine apart? You said new cam and lifters, what about the rest of the valvetrain? Did it get a valve job?
Any chance it could have one or two valves with an installed stem height that is shorter than spec, due to a new seat or due to the stem end being ground down or some other cause? If that were the case, then the hydraulic lash adjuster in the lifter would be working overtime to make up the excess clearance and maybe it wouldn't be able to keep up when the oil is cold, leading to excess clearance and the two noises (tap and intake thump) that you're hearing.
If you got creative and could get hands on the right measuring equipment, I bet you could check both the lifter bore clearance and the valve stem height with the head still on the motor and the cam and lifters pulled.
Other than those two possibilities, hard to think of much else.