Did you check to see if the memphy on your device supported DDR4?
Normally the high bandwidth memory uses 4:1 clocking. This means the logic runs at 312.5MHz, and the IO at 625MHz DDR. This works for DDR4 as it is burst oriented, and commands can be issued at a lower rate. If the controller has an 8:1 mode, this would be 156.25Mhz to get 625MHz.
That said, the calibration procedure and signaling requirements may be different enough to matter if you attempt to use a non-DDR4 controller with DDR4 devices.