You have to sense the voltage on the line. If the modem is using the line the voltage will be low 11-17VDC. If the line is available, the voltage will be around 48VDC. So your circuit could easily lockout the phone by activating a relay when the modem is in use.