You could try a NMOS transistor connected to the low side of the batteries with the gate wired to your power source and a resistor (~10k) pulling the gate to the ground. This should effectively isolate the batteries when the system is off. However, I haven't tried this, so YMMV. A NMOS transistor with low Rds(on) should be selected, around 10 to 100mohm, and it must be capable of handling the charging current.
You may want to consider a relay in a similar configuration, if your application is not too cost or size sensitive, because those are pretty guaranteed to turn the batteries on or off.
And of course a Schottky diode if you can use it will only drop 0.5 volts or so.