I wouldn't worry about them being switched either. I switched mine so the buttons would be on the inside rather than the outside and never noticed a problem. There's a little metal tang on one of the shafts. This tang is always on the drivers side of the vehicle. (Unless you have a right hand drive version. Not likely in Philly, but I'll trade if it is!) So going by what you've told us, this tang is not going into the shaft when you press the button. That's why the button side shaft moves a bit but the other side does not. If the tang isn't moving, forcing the headrest will only break something. Like compu_85 said, make sure you are depressing the button the whole way. Compare the depth with the working drivers side unit. One thing you could try is pushing the button in and out repeatedly while working the headrest side to side, front to back, and up and down. (But gently!) If the tang is just caught on something this may get it unstuck. Barring that, fixing it may necessitate disassembling the headrest by undoing the stitching on the bottom and removing the foam surrounding the frame and mechanism inside. It sounds awful I know but it really isn't that hard to sew one back up.
I wish you luck!
Bill K.