debugging RGMII interface

[el00]

Hello, I developed a board with an FPGA and a 1GB PHY. The 1GB autonegotiation is OK, and the board is only transmitting data in streaming, there is no need to receive in this application.
The stack is not a standard TCP one, but it is a reduced customized stack that only sends packets blindly, without checksum.
What happens is that we get some errors (10% of data are corrupted) in the transmission, therefore we think that the problem is in the RGMII interface.
Now, my qyestion is: what hardare tools are necessary to debug such an hardware? Suppose that we want to buy a 6GHz scope, not on purpose for this, but also for other reasons so that we invest the money in something that is useful for other lab jobs, would it be the best choice or there might be some more dedicated hardware?
What is normally done in these sistuations? It's the first time we develop for Gigabit, and give the shortage we cannot make too many tests, and evaluation boards are out of stock.
However, for this particular chip we have some stock, but before repeating the same mistakes we need to understand what is going on. Thanks.

 

[Taihung]

Hi,
Your data link connected through ... it mean all wire/net connect right pins.
Now, you have to take care 02 effects: Power & Clock.
Normally, design using switching power supply like BUCK converter. Sometime has pre-regulator/ LDO reduce noise. PHY often has digital and analog VDD. If these power have too high ripple or internal equipvalent resistor too high or step respond too slow, all of them make wrong logic level for PHY operating (under 1V). Exspecial if switching radiate too much EMI. This case is easily to to see when you ping short message like 4k is OK, but 4M have more failure rate.
Next CLOCK is very important. If clock jitter or not stability, the PLL can not synschronous timming, and data will lost. For 100Base request lower spec compare to 1000Base. So, check clock source: jitter (50%), tolerance (ppb), stability (ppb).
Also check carefully impedance matching, jitter/skew matching for all diff-pair.

 

[FvM]

I suspect a timing or signal integrity issue. 10 % data corruption suggests that the problem can be visualized with an oscilloscope, but it should be recognizeable in a design review as well.

Die you verify FPGA data generation in a simulation? Also, did you implement the RGMII clock to data relation as specified in the PHY datasheet?

 

[dpaul]

The stack is not a standard TCP one, but it is a reduced customized stack that only sends packets blindly, without checksum.

That means the FPGA transmits TCP packets but does not wait for a reply from the sender....ok I take that with a pinch of salt!
One question why didn't you use UDP stack?

Back to the main question regarding RGMII transmit interface debug....

My guess is that the RGMII clock and data transmit signals are not always properly aligned leading to a 10% loss.
You must be aware about the RGMII clock skew that can be added to the rgmii_tx_clk at 3 different stages.
1. Added by the FPGA
2. Added via PCB traces
3. Added at the PHY by configuring a register commonly called RGMII Transmit Delay Register.

Now you must tell us how in your design is this delay being added. Also is the question as to whether you are adding the correct delay value in nano-seconds?

Then have your TCP stack send out frames one by one and I guess at the receive side you are using something like Wireshark to verify the frames where you observe a 10% loss?

Basically out of the 3 delay sources you must turn off 2 sources and rely only on one. Taking the hypothetical case that 1. and 2. are adding no delays, you do the delay setting at the PHY register and observe the Wireshark data. If you still see some % error, change the delay value again at the PHY and observe the Wireshark results again. At some point of time you will get 0% packet loss.

When I was working with RGMII interface, I kept the RGMII delay value fixed at the PHY. I did not care what amount of delay the PCB traces were adding to the rgmii_tx_* signals (as I had no way of finding it out). I was playing around with my FPGA design constraints file such that I was shifting all the non clock RGMII signals to get the required oversall shift relative to the clock.

Other people might suggest you other methods, but the basic thing is, you must at some part of your design add the skew between the RGMII transmit clock and the data signals. Then to get 0% packet loss you have to find out experimentally (because you have no idea what delay is inserted by the PCB traces) which delay value suits the best!

My assumption for this reply - You have verified through simulation that your RGMII interface is working as expected. If not done, do it as a 1st priority without even thinking of anything else.