It depends which point you begin from. At least several mathematical books should be in your bookshelf such as matrix, stachastic process, further reading contains the signal an system, etc.
A basic understanding of LTI systems probably wouldn't hurt, either, depending on what type of systems you're studying. Specifically, knowing the basics of elementary ideal LPFs are useful when you're learning about modulation systems.