i need to know how to assign the rst pin and psen pin to rs232 in c code for the handshake protocol. i cannot start the coding because i cannot assign it.
I'm not sure what you mean by "isp mode" and RS-232. First of all, an 8051 doesn't HAVE an isp mode (unless you are using some other variant--WHICH ONE?). Secondly, RS-232 is a communications protocol, it has nothing to do with isp. Thirdly, an 8051 doesn't explicitly implement RS-232, it has a UART which handles the data transmit and receive lines; you'll have to create the handshaking lines in your software.
i'm using the p89c51rd2h which has the in system(ISP)/in application(IAP) programming function. the ISP in the datasheet are about manually reset the uC by bringing the RESET pin low and manually bring the PSEN pin high circuits. i want to do a c code where i can trigger it using software so i do not have to manually bring it to ISP mode. yes,i have done it referring to assembly language. i'm doing it in c language. i'll put the code for you to check if it can work or not because i cannot test it without knowing how to toggle the RESET pin and PSEN pin.
I'm not really familiar with this device, but from what I can see I don't think you can reprogram the entire device (like ISP), you can only use IAP to program sections of the ROM.
I think you've got a chicken-and-egg situation: if your application code asserts RESET, then your application will stop running.
But I think you can use the bootloader to reprogram the device on startup.