Unfortunately there isn't a general rule for mixed digital&RF PCB design, except maybe a good ground plane (gold plated) and many vias near the signals as someone said. Sometimes the number of vias does not improve the signal quality as long the line impedance even was correct computed is modified during the factory process using different epsilon for the substrate as it was requested...
Splitting ground planes and connecting those in one point could help but also could
produce interesting behaviour for the noise path. So, when you don't have differential analog signals (180 degree phase signal) comming from DA outputs, always keep at least two or more possible connection points between the RF ground and digital ground and use the one which minimise better the noise.
and so on, things hard to find in theory...