I'm looking for all kind of information about serial protocol analysis method (reverse engineering) based on serial port software sniffer.
If you know good books or good websites, i'll be glad to know.
there are many freewares and sharewares around. Try searching for serial port monitor. There has been some discussion on this forum also. Also have a look at ethereal. Though built for ethernet, it also looks for serial port PPP protocol.
1) start portmon
2) select the comport to monitor (which port your application under test is using)
3) you can monitor the activity on serial port
4) You can also log the activity
Data transmitted/received/baud rate settings every activity will be shown
Also AGG Software's Advanced Serial Port Monitor can be useful for you, because allows not only capture data in the sniffer mode, but allows to send commands to the RS232 device in the manual mode.
Take a look at the products of Eltimasoft - they offer the serial port sharing tool/driver - it would help to monitor the actual data exchange between the 2 serial agents using any of the tools mentioned in the other posts.
Good luck!