You probably want to read about non-blocking assignments. In this case you want Y to only use blocking assignments. IIRC, some version of Verilog or maybe SystemVerilog allow reg's to be declared local to the always block. Normally blocking assignments in clocked always blocks is discouraged as the use of the blocking assigned reg outside of that always block is not well specified.
This should synthesize to some small logic. However, I've not tested this coding style. You can also generate a state update matrix and an output matrix and then do matrix-vector multiplies. This can be done by having the matrices be wires that are assigned in for-generate, or by having them be localparams assigned by a function. The way this is done is basically the same as what you have done, just in a way that makes it more clear that small constants can be constructed. I've also seen this all done in an initial statement, but it seems some tools will remove that logic and only issue a warning.
for the matrix-vector multiply, "y[n] <= |(x & a[n])" can be useful.
https://zipcpu.com/dsp/2017/11/13/lfsr-multi.html <-- this is one example of this using the generate statement and wires.
I wrote a version that used a function and a localparam, but it is on another computer. And I tried a handful of new things out so the file is kinda ugly.