Well that leaves only software flow control, XON and XOFF.
Here's a link to a AT89C51 Serial Port Interface:
AT89C51 Serial Port Interface
Discusses many of the issues concerning AT89C51 and Serial Interface.
You most likely can implement software flow control (XON, XOFF), the receiver issues XON when ready and XOFF when not ready. Another alternative is using external interrupts tied to the CTS line of RS-232 to implement your own hardware flow control. You can always roll your own flow control as well, have the MCU send a set size block of data and then wait for a response from the PC side before sending another block. You should also be able to increase the buffer size on the Matlab side.
The last alternative is to upgrade your MCU, the newer AVRs have hardware flow control, SPI to write data to flash card, etc.