Math for DSP can be very difficult!!! It depends on applications, some of them do not require math but some of them do.
If you want to design, for example, system with adaptive filters or aplication in image processing with wavelets theory, you need to know funtional analysis, complex analysis, matrix theory, numerical analysis, fourier and wavelets theory which are based on many math fields, and etc.
So,you do not have to be an expert for math but if you think that is undergraduate math it is wrong.