The delay functions are calculated and configured by the compiler using the information on clock speed from boards.txt. In the Arduino environment, under the tools menu, the board selected has a clock speed associated with it. Each board that appears in that menu has a stanza in boards.txt, that defines this.
The bootloader needs to be configured for the proper frequency so that it is using an accurate baud rate, to upload the program.
So, yes, the delay functions and baud rates would be too slow if just the crystal were changed.