Shifting the audio is the problem. You substitute one obstacle for another.
With DSP it is possible but if you want to do it with analog circuits, you are essentially adding a second mixer stage into the design, one to up-convert the voice spectrum prior to the RF mixer. Think of how you swap sidebands too, you need to spectrally invert the audio frequencies which means mixing with a frequency just above or just below the audio range. For example, to retain the 'normal' audio direction, the audio up-mixer LO has to be 17KHz (using your figures) which would place it within the LPF range. To reverse the spectrum it would have to be at 20.3KHz which would imply a very steep roll off in the LPF between 20KHz and 20.3KHz which would be difficult to achieve.
Brian.