You can try free softwares like Scilab, Octave, LastWave, ObjectPro DSP, packges at Ptolemy etc. You can find more linux or open source based in this link : https://www.linux-sound.org/dsp.html
Lab View is also another choice. Or atleast there are many good libraries available in C++ for DSP.