As FvM noted, you need to remove C1 and also C3. You never want to intentionally add capacitance to the summing junction of an op amp, since it can cause instability and ringing in the circuit response.
If you want to filter noise, then add a capacitor across R4. But the capacitance tolerance difference between that capacitor and C2 will still degrade the common-mode rejection.
If you really need high common-mode rejection (greater than about 40dB) then use an integrated circuit instrumentation or differential amp designed specifically for that purpose. They have closely matched internal components to give a high CMRR.