Yes, forgot to mention about voltage drop - but that's quite obvious.
One solution for that may be to place a "big" cap in parallel with your circuit.
Yet there will always be some different between the voltage sources, so there will always be a little change in voltage when switching between them.
A transistor may also provide a little "jump" in the voltage when switching, only it happens faster though. I guess it's better having a cap over the circuit any way, and using another cap between the transistors emitter/base in order to get it conducting gradualy and smooth.
Problem you might face here is that it will also turn off smooth, and then you will have a short lived situation where both the power supply and the battery is conected in parallel - just what you want to avoid.
You might think out a relatively simple solution for that, a diode/cap/resistor coupling to get it slow on, but fast off.
You could also put a drossel in series between the switch and the circuit in aditional.