As begginer you can study
the 3rd 4rth chapter from the book by the sklar fundamental of digital communication
for advanced study use the book from schmit and messer lee
The best book for modem design is Digital Fundamental by Sklar, Digital Analog and communication by Lathi,,,and Digital communication by FOROUZAN especially, which has the comprehensive material about MODEM.