I've not used that MCU but looking at the data sheet, the power-on-default for the PRI_SD bit in OSCCON2 is to have the oscillator drive circuit turned on. I suggest you turn it off.
Also, RA4 has analog capability and this needs to be switched to 'digital' mode with the ANSEL register. If not then it will always read as '0'.
With 'interrupt on change', you need to read the PORT to do 2 things: work out which pin has changed and also to reset the IOC detection (see Section 9.1 of the data sheet). If you don't read the the PORT then you will only get 1 interrupt.
Susan