It boils down to a trade-off between speed/performance and power. A lower resistance pull-up will have higher current, consume more power but switch faster than a larger resistor.
I2C is a great example of a bus whose performance (in terms of speed) suffers due to the use of passive pull-ups on its lines.
However you must also be careful not to dammage your active components by using pull-ups or pull-downs that are too small. Check the datasheet for the chips that you are using and pay careful attention to the maximum currents that your output stages can sink/source. These impose certain hard limits on how small your resistors can be without causing dammage.