I've read some papers about this topic. This topic is broad : Blind Source Separation. In general, people use more than 1 channel to separate mix signals. This is the "party cocktail problem". There're several methods but I prefer ICA (Independent Component Analysis).
I have read a paper recently, about sources separation using 1 channel only. That's really interested but I haven't tested with it . You can search with the keyword "Logmax algorithm". Hope you enjoy.