You first need to receive some documentation about the algorithm you are going to implement. Study them well and if you already have a code (C implemented on DSP), you should view its documentation. Then try to take the already implemented-in-DSP blocks and start re-implementing them as RTL. This may imply moving from a certain level of abstraction to another lower level of abstraction. for example, C doesn't care about clocks, while RTL cares. C doesn't take into concisderation synchronization, RTL cares. C doesn't interact much with hardware, while RTL is hardware.
You need also to partition your system , maybe you will still need a processing to be available in your design. After partitioning your system, go for partitioning your blocks to sub-blocks and so on, till you reach a certain level of abstraction where you feel you are comfortable with describing it in RTL. At this moment start writing your State Machines.
Hope this helped.