I think it's an opinion based on smaller parts, for the largest parts in the Xilinx family ISE was very slow due to the extremely large memory requirements and the database design that wasn't very scalable (mentioned by someone from Xilinx). Vivado was a rewrite and it has much lower memory requirements. The performance is improved over ISE from what I've experienced.
e.g we had a Virtex (6 I think) design in ISE that would take >5 hrs and >8GB to to implement. It took ~3 hrs and <<8GB to run implementation on Vivado. Of course at the time due to the newness (first release of the tools) of Vivado the QoR suffered due to the lack of any kind of re-timing features in the synthesis, which that design needed to meet timing. This was done on a dual Xeon (8 core) workstation with 48GB of RAM, so it wasn't slow for the ISE due to any disk swapping.
Now for Altera (now Intel) I always noticed their tools always seemed to be much faster at least they were a decade ago.