Since each AVR´s input voltage refers to the AVR_GND .... for sure both reference points (both GNDs) need to be connected.
Otherwise it´s like you have a voltmeter and connect just one input. It will show "nothing" in best case, or nonsense. Take care about ESD.
Btw: it´s no good idea for longer distance.
For longer distance: Either use differential signaling like RS485, or galvanically isolated interface (or both).