Hi,
There are a multitude of possibilities for ISE throwing an error as such.
i) Are you using a core function corresponding to an ip?
ii) As you mentioned the design might be too large to compile..a factor 1.3 doesn't seem to be much but how many filters are you implementing, polyphase?
iii) there is a semantic error in your code, causing the synthesizer to go insane.
Suggestion, i) try othr values of filter coefficients, ii) reduce the number of coefficients to say by a factor of 1.01 or 1.2..etc.
Good luck!