AC analysis uses a unit amplitude (unless told otherwise) and small signal
models, while tran uses what you give it for amplitudes and large signal.
If there is anything even a bit away from DC OP, that changes the transfer
function you would see it in tran and not in ac. I expect s-param analyses
follow ac analysis in this regard.
So I would first look to scaling factors of input sources and maybe try a
couple of input amplitudes to see if there's that kind of "different angles
make different pictures" kind of thing going on.