There are a few possible issues:
1.) Clock to out time of the source. This comes up in RAMs and shift-registers in some cases. Sometimes in other hard-ip. This is also important for IO. Many FPGA designs will use the IO registers in addition to one or more routing registers if desired.
2.) Combinatorial logic. This gets harder to estimate as clock rate decreases and allowed complexity greatly increases. Put another way, it is easy to estimate if logic might run at 500MHz but difficulty to estimate if something is so complex it would not run at 100MHz.
3.) Routing delays. This is the delays in connections between logic elements. In some cases this is due to physical constraints -- for example, routing to IO pins means the logic must be somewhat near that IO.
4.) Congestion. This is related to the above -- in this case, the logic could work much faster, but some elements are placed too far apart. This is a case where routing delays could be lower in a smaller design. In some cases, this can be fixed by more efficient use of BRAM/DMEM/etc...
IIRC, you should also provide a clock constraint. I forget if the tools will take too much effort to optimize your design otherwise.