hi,
its really hard to estimate the difference you will see between layout and circuit simulations! its really depends on the complexity of a layout. there is no equation to predict that!
5dB looks normal to me. your statement saying you didnt find any way to decrease it sounds weird to me! are you using lumped component in your design? if yes, all you need to do is optimize them and again to get same results a circuit in schematic. you may end up in different component values for layout and schematics. its normal and remember that its as a results of cancelling parasitics caused layout.