Do you have 2 a/d channels? If you do, you can skip the instrumentation amp and just subtract Bat+ and Bat-
Or, you can use a really neat characteristic of the differential amplifier.
The equation of the differential amp that people usually use is: Vout =(Va-Vb)*(Rf/Ri). However, it is actually Vout =((Va-Vb)*(Rf/Ri))+Vc where Vc is the "ground voltage" of the Rf resistor on the non-inverting input. Normally Vc is ground, but it doesn't have to be. The ground voltage pins are 1 and 5 of the INA117. So if you put pins 1 and 5 at say, 0.25V, then a voltage of "0" on the battery will be 0.25V on the PIC, positive battery voltages will range from 0.25V to 5V and negative battery voltages will be less than 0.25V. Easy to detect in software. OK, now the caveat: the impedance of the resistor divider that sets the 0.25V (or 0.1V, or whatever you choose) must be very low. An opamp is a good choice, or maybe 950 ohm and 50 ohm resistors. TI also shows it in Figure 11 of the INA117 data sheet. Pin 8 still goes to ground.
---------- Post added at 18:41 ---------- Previous post was at 18:12 ----------
A couple more comments. 1) Your boost converter is pretty spooky, I bet. The output range of say 3-50V is a really wide range (duty cycle from 1 down to 0.05). As you know, you have practically no current limit available for low batteries. Have you considered putting an RC on the gate of the MOSFET charger switch and PWM'ing the gate drive when you start so you can use the MOSFET as a poor man's load resistor? It would have to only run for a couple minutes to bring a battery up, and it may save burning off a pc board trace. 2) If you are charging lead acid batteries, your a/d converter doesn't have near enough resolution to support float charge operation (-2mV/deg/cell characteristic, for example). Be careful. 3) Your accuracy is also affected by the resistor dividers feeding the INA117. Since the INA117 can go to a couple hundred volts common mode voltage, why don't you take the (+) input of the INA117 directly to Bat+ and use one divider from Bat- to Bat+ (referenced to Bat+ not to ground). Then there's only a single resistor ratio that sets the value. If you want to maintain a decent untrimmed divider ratio, use identical value surface mount resistors (surface mount resistors on a single strip are nearly identical since they come from one batch), or even better a dip resistor network. Matching on networks is something like 0.1% over temperature (which you won't get even with two different value 0.1% resistors). So to get a 13:1 divider ratio (60V-ish full scale), put 4 10k's in series (40k) and 3 10k's in parallel (3.333k).