I see the problem that the A5303 input circuit operation isn't even remotely explained in the datasheet. According to the scarce description, the gain is set by capacitors rather than resistors. The purpose of the 560 ohm series resistor isn't mentioned, why did you place 560k?
It's surprising at first sight that a photo diode is bypassed with a relative low resistor value, but the circuit operates dynamically, so I presume it's required.
Without a detailed design guide for the input circuit, you have to start from the suggested application circuit, take care that the IR-LED and photodiode are corresponding to the specification and the chamber has the expected optical characteristic. You didn't tell about the used LED and photo diode type (beyond showing photos), are you even sure that the "detector" is a photodiode and connected correctly?
I guess that A5303 is mainly applied by smoke detector manufacturers with additional FAE support. Did you ask at Allegro if they can provide supporting documents?