I'm not aware of any equation or formulae that you can use to calculate the spacing of stitching vias.
The distance between vias would be a function of the frequencies involved, the dielectric, and the distance between the ground plane and the signal source. It would require some trial-and-error simulation in a 3D simulation program to find the optimum spacing.
Many designers avoid the simulation by simply placing stitching vias with random spacing, and taking a chance that it will work. If it doesn't, they respin the board. In my view, that is a sloppy way to work. Often, stitching vias are used in cases where none are necessary by folks too lazy to do the pre-routing analysis.