VLSI is a very much vast domain. In my limited knowledge, it includes all of the things that are involved in designing, manufacturing, fabricating and verifying a CMOS chip containing thousands and millions of transistor on a single chip. The designing of a chip to its logic verification then its manufacturing and testing etc, all of them have a lot of scope available for research in them.
Now coming to your topic of interest, I think that in FSMs, currently there are researches going on for State Minimization Algorithms and Incompletely Specified State Machines. Further, I have seen on web that some research is being done on efficient coding of FSMs in different languages e.g. Verilog and VHDL. Integration of FSMs with AI can also be a topic of research. There are various avenues available and are not limited. You can explore a variety of research areas available.
Thanks,
MSBR