Strictly spoken, a setting of VRef- = 3.400 , VRef+ = 3.600 violates the datasheet specification A22 for VRef-, which effectively requires a minimum reference span of 2.0 V rather than 0.2V. But even if the datasheet won't explicitely specify a minimum span, it should be obvious that a too low span won't get you anywhere. You can simply assume, that most ADC error terms are effectively absolute voltage errors. So if you achieve e.g. 1 LSB accuracy with 5V Vref span, you can't expect more than 25 LSB with a span of 0.2 V, which is actually in the order of magnitude you observed. The only way to measure a limited voltage span without loosing accuracy is to use an external amplifier. Some PICs have also a PGA option for the ADC, which at least offers some improvement for reduced voltage ranges, altough it can't compete with an external amplifier.