Hi,
Is it possible to supply the display with the power left side of the 1N4007? This may be an easy solution without any additional device.
***
The 4s delay probably is caused by the adapter´s inside capacitor. So the voltage drops slowly. Then the threshold voltage to detect "LOW" is too low.
Consider to use a supply voltage monitoring circuit. Sounds difficult, but it is not. A single three-pin device instead of your two resistors.
Look for LM809 or MAX 809. There are different threshold levels you can choose.
Hope this helps
Klaus