Here is how this boost converter works:
Phase 1:
When V1 is High, M1 turns on, pulling the voltage at node 2 low.
This reverse biases the diode and isolates the two halves of the circuit.
During this phase, the current going through the inductor increases, and additional
energy is stored in the inductor in the form of magnetic flux.
The output voltage / current is supplied by the Capacitor during this phase.
Phase 2:
V1 goes low. M2 turns off. node 2 increases in voltage, and the diode D2 becomes foward biased. The energy from the 5 volt Vdc and the energy stored in the inductor is injected to the right half of the circuit, causing the voltage to rise beyond 5 V.
Now to answer your question, if the Duty Cycle is too close to 1. Then there will be very little time for the Inductor to inject its energy to the right half of the circuit. and Vout will drop.
Thats my best guess.