Modem basics
The best way to start is to get 2 external RS232-modems, connect them using a phone line (*) and connect them to 2 PCs (**), run a terminal program on each PC. Type on one of them
ATD123456 (replace 123456 by the real telephone number of the other side)
you should hear how the modem "picks up", waits until dialtone, then dials, and you should hear the ringtone. At that moment, on the other side, the modem should send to terminal:
RING
and if a phone is attached, it should ring. Type on the other side:
ATA
you shold hear how the modems are negotiating (scratchy noises). After a while of whistling, both modems should write
CONNECT xxxxx
where xxxxx is some sort of speed (but usually with the factory setting they report only the speed how they are communicating to the PC, not to each other).
Now the modems are connected and in DATA MODE (as opposed to COMMAND MODE, in which the modem is after switching it on) and everything typed on one side is transmitted to the other and vice versa. You can have fun conversating with some friend...
If you want to hang up, you need to escape from DATA MODE to COMMAND MODE. Don't type anything for at least 1 second, then type rapidly 3 pluses (+++), then wait again for at least 1 second. The modem should write OK as a confirmation it is in COMMAND MODE. Then type ATH to hangup.
If you get the feeling, try playing with the AT command set - try the effect of each command. If you get lost, either switch the modem off and on, or try typing AT&F (factory reset).
Later you should also read more also on the handshaking (HW:RTS/CTS or SW:XON/XOFF) - while only humans are typing on a terminal, it is not important, but if computers start transmitting data, it becomes to be increasingly important. Also it is good to understand the meaning of other handshaking signals, it might e.g. spare some parsing problems (using RI instead of detecting the "RING" stream etc.).
And now, when you understand how the modem works, you can start implementing the routines on a mcu.
Have fun!
wek
(*) There are special (and usually more expensive) modems capable of connecting on a direct line (2 wires), these are called leased-line modems. However, they usually need some tweaking to get them into the leased-line mode, so it is generally not a good idea to start with these. Standard modems DO NEED the signals provided by the central, so you cannot simply connect 2 standard modems using 2 wires.
(**) Of course it can be a single PC with two COM ports and two terminal programs running; but it's quite a mess, not recommended for beginners.