I am confused with data transfer of application hex file from host to target. Where shall i exactly add the checksum in the application hex file and which algorithm should i use? Please suggest.
I'd say there are many different protocols.
You need to say which protocol you are using.
At least which microcontroller, and which bootloader you mean.
I am using dspic 16 bit bootloader using uart. I am saying that in case application hex file is faulty how to recognize it? Should i first be storing all the uart hex file data and start flashing or block by block should i flash.
If we are talking about intel hex, then there is a specific format. There is a checksum in every line, so you must check line by line. Also you know how many bytes exist in each line because there is a corresponding byte that informs you about this. You can get the end of file (last hex line) from the record type byte. A google search will give you plenty of results.