The IF filtering always stays constant.
The local oscillator is tuned to centre the received signal within the IF bandwidth (so that it can be heard).
But in order to correctly demodulate the SSB back into audio, a beat frequency oscillator is required to replace the missing carrier.
For USB the BFO will be placed at the low end of the IF bandwidth,
For LSB the BFO will be at the high end.
Now the actual frequency readout on the receiver dial corresponds to where the missing carrier would be. So the readout frequency on the receiver dial must be moved by the amount the BFO frequency changes when switching between sidebands.
In older equipment with a mechanical frequency readout, there is usually a movable graticule line that you can use to correct for this slight change in frequency when switching sidebands.
In modern equipment with digital readout there will be a microprocessor that adds or subtracts the slight frequency offset required automatically.