A lot of the choice for transmission line type is determined by the application. If you running differential signals over long distances (and probably using a connector on the box), then you'd go with twisted pair. For unbalanced RF signals, you typically use coax so that you retain the ground reference at both ends of the cable (i.e. not differential signaling).
The frequency, signal power level and signal type (system impedance) will often drive the type of transmission medium. Most RF systems are 50 or 75 ohm and are well-suited to drive coaxial lines. High speed digital drivers may specify a 300 ohm transmission line, which is readily supported by twisted pair or other differential style transmission mediums.
The answers are mostly given by experience and looking at implementations of similar systems, or (hint hint) vendor recommended implementations.