"Would having Lighter rims make any difference?"
IMO, an emphatic yes! Personally I think the sweet spot is 16" rims.
Both rims and tires are lighter than the "performance" bigger wheels.
Unsprung weight makes a big difference in driveability and response. My current combo, check my sig, comes in at ~35# @.
I lucked out and found my XXIOs on Craigslist for $200.
I lust after a set of Enkei RPF1 racing rims at 13.7# @, but circa $900,
is pricey.
For a factory rim, dig around for a set of Le Castellets, they run about
20#s. Tirerack has good data, including weights of tires and rims.
My $.02,
R+2