iteration limit reached

[BartlebyScrivener]

I am reaching an iteration limit with my design under certain conditions using Modelsim. I have been looking on line and the only way I see people debugging this is to manually read through code and look for errors. My code is monsterous. Is there no other way to find out what is causing the failure?

I am simulating 64 routers and 64 nodes connected in a torus, producing random traffic. It only fails when I put the network under extreme load for a long time. To debug without knowing where to look is daunting!

 

[TrickyDicky]

Iterations usually get reached when you get inside a long loop. Are there any while loops that are not hitting an exit condition? Why are you using while loops anyway?
You could also reach the interation limit if you have an asynchronous logic loop (another bad thing to have)
IS the code fully synchronous?

 

[BartlebyScrivener]

Sorry, I'm not sure about fully synchronous meaning lol. As in, do I use combinational logic? Yes. Or as in, do combinational logics feed themselves? I do have combinational feedback paths for arbiters. I don't have any while loops, I have hundreds of for loops. The simulation works absolutely fine over thousands and thousands of stimulus in all but the harshest conditions. Without knowing where to look, I don't know how to start debugging it.

 

[FvM]

Are the for loops intended to be synthesizable? In this case the loop bounds must be statically computable and the iteration count will be fixed.

 

[BartlebyScrivener]

hmm. It synthesizes, so perhaps the problem is in my test bench

- - - Updated - - -

I found a magic button, looks like a blue arrow pointing down to a spot. Run my simulation to just before the fail condition, then press the blue button, it steps through the code line by line.

Hopefully I will spot it repeating somewhere.

- - - Updated - - -

ok, the blue button isn't going to work, I can't step through every single fifo for every single router and node in the network lol

- - - Updated - - -

is there no way to get the simulator to say where it is looping? It seems like the kind of thing that would be easy to work out if I were a computer program going through steps.

- - - Updated - - -

or perhaps to say how many iterations it is doing? Or any info other than simply 'it's looping somewhere in your thousands of lines'!?

There has to be a way to debug this other than just thinking where the problem might be! If it's happening once in hundreds of thousands of cycles!?

- - - Updated - - -

I feel like jumping out of a window!

 

[mrflibble]

ROFL!!! Your updates read like an XKCD comic. Keep up the good work!

On a more serious note, you probably have to change your testbench methodology just a tad... If your design is so complicated that you don't know where the hell to look during debugging, then you probably need to split things up. Testbench smaller chunks, and by the sound of it use random constraints (a la systemverilog) to increase your test coverage.

 

[BartlebyScrivener]

LOL thanks. I have tested everything idividually with systemverilog, everything works fine. Inputting random data to individual modules and testing the output.

I am testing it as an entire network now to get network results, and it also works fine, in loads of conditions and for hundreds of thousands of cycles. I can not imagine what situation is happening this cycle, that hasn't happened before because despite the amount of parts involved, it's mostly repetitive. read packet/arbitrate/send * a few hundred times per cycle

- - - Updated - - -

Surely the simulator can go, hang on, I keep repeating x, the point I entered this loop was x. Like how hard can that be!?

 

[TrickyDicky]

how are you creating these for loops? do they have constant bounds?
synchonous design means all logic runs off a clock of some sort or another - usually a single system clock.
Having combinatorial feedback paths worrys me. It means your timing specs will be pretty poor and be more prone to glitching.

 

[pbernardi]

Well, you cannot simulate the universe quark-by-quark.

My suggestion is check the real word. Is it possible to mount a test situation with real FPGA? This would be the best. Once you get a failure situation, you could thy to dump the registers/IO/whatever values and simulate it on FPGA.

Simulation only works up to certain stage, after that checking FPGA behavior usually is better.

 

[mrflibble]

https://joshishantanu.wordpress.com/2012/11/19/debugging-error-iteration-limit-reached-in-vhdl/

Frustrated at not knowing a good way to debug these errors, I beseeched Google to shed light on my ignorance, and Google did not disappoint.

Rather, embarrassingly the search result was an old version of the lecture slides for either ECE 337 or ECE 437 at Purdue !

The method (with some modifications I made) is as follows :

1. Under Simulation —> Runtime Options in Modelsim increase your Iteration Limit

2. Set Break points where you suspect the Combinational loop exists .

3. Step through your code at clock cycle (or half clock cycle to be safe) granularity

4. Now every line you step through is a part of the combinational loop.

Also, another Pro Tip my TA gave me was that, once you find the signal that is causing the loop, simply make it a register.

While that will get rid of your loop, it may throw of your timing so beware !

 

[BartlebyScrivener]

Everything runs from the system clock, but there are combinatorial blocks between registers. So for instance, I have part of the data stored at the front of a fifo being read by an always_comb block that computes where the data wants to go. This is read by another synchronous block which will send the data.

The combinatorial feedback path I have is the following, which is an arbitration module.

---

Code Verilog - [expand]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
input  logic clk,
  input  logic ce,
  input  logic reset_n,
  
  input  logic [0:N-1] i_request,         // Active high Request vector
  
  output logic [0:N-1] o_grant);          // One-hot Grant vector
  
         logic [0:N-1] l_priority;        // One-hot Priority selection vector
         logic [0:N-1] l_carry;           // Carry-bit between arbiter slices
         logic [0:N-1] l_intermediate;    // Intermediate wire inside arbiter slice
 
  // Variable priority iterative arbiter slice generation.  Final slice carry loops round.
  // ------------------------------------------------------------------------------------------------------------------
  genvar i;
  generate
    for (i=0; i<N-1; i++) begin : PPE_SLICES
      or  gate1  (l_intermediate[i], l_carry[i], l_priority[i]);
      and gate2  (o_grant[i], l_intermediate[i], i_request[i]);
      and gate3  (l_carry[i+1], l_intermediate[i], ~i_request[i]);
    end
    or  gate1  (l_intermediate[N-1], l_carry[N-1], l_priority[N-1]);
    and gate2  (o_grant[N-1], l_intermediate[N-1], i_request[N-1]);
    and gate3  (l_carry[0], l_intermediate[N-1], ~i_request[N-1]);   
  endgenerate
  
  // Round-Robin priority generation.
  // ------------------------------------------------------------------------------------------------------------------
  always_ff@(posedge clk) begin
    if(~reset_n) begin
      l_priority[0] <= 1'b1;
      l_priority[1:N-1] <= 0;
    end else begin
      if(ce) begin
        l_priority <= |o_grant ? {o_grant[N-1], o_grant[0:N-2]} : l_priority;
      end
    end
  end

---

A simple example of a for loop I use would be

---

Code Verilog - [expand]
1
2
3
4
5
always_comb begin
    for(int i=0; i<NODES; i++) begin
      s_i_data_val[i] = s_i_data[i].valid;
    end
  end

---

- - - Updated - - -

Thanks pbernadi, I will get it into an FPGA, and it synthesizes fine, but ideally, I would like to get test bench results for the networks so that I can look at packet latency etc.

I actually am getting all the results I 'need', the simulation is only failing when technically, the network has all ready failed. However, I am worried this indicates a deeper problem so don't want to ignore it.

mrfibble, thanks for finding that! I have just increased my iterations, and then gone simulate>break when delta goes crazy. I will start stepping, but my computer grinds to a halt for 10 seconds after every 2 steps, so this may take some time!

- - - Updated - - -

could anyone explain what the step buttons do? The user manual doesn't really elaborate much!

 

[K-J]

I am reaching an iteration limit with my design under certain conditions using Modelsim. I have been looking on line and the only way I see people debugging this is to manually read through code and look for errors. My code is monsterous. Is there no other way to find out what is causing the failure?

You have to step through the code, but you don't have to go through all of your 'monsterous code'. At the point where the iteration limit has been reached the only code that will still be executing is the code that is part of the problem. So...
- Run until the iteration limit is reached
- Single step. The code that was just executed is part of the problem. Write down the file and line number that you're at. Maybe take a look now at your design to see how the inputs to this logic are generated to see if you now see the combinatorial feedback loop. If not...
- Single step again. Maybe now you'll have another clue to where the problem lays. If not...
- Single step again (getting the hint?). Eventually, you will end up back at the first instruction that you single stepped and you will recognize it because you wrote down the file and line number, right? Presumably you will have figured out your loop before getting back to the first instruction, but if not now you know every instruction that is involved in creating the loop.

Another approach is to synthesize it and peruse the following:
- Look for warnings about a latch being generated
- Look for warnings about a combinatorial timing loop in the static timing analysis report

Kevin Jennings

 

[BartlebyScrivener]

Thanks for everyones help. Some really helpful posts. Now I worked out how to cycle through I realised exactly how much combinational logic there is. The cycle didn't work because my computer is too naff. Each step took around 10 seconds so when I hit loops that were going to execute 60 odd times I just felt like crying.

But. I found the problem using the old trusty white board and thinking about it! It was a peculiarity of how I had some modules connected during a certain network config made the input enable of a fifo dependent on its output enable.

It has made me think though, is it necessarily bad to have a lot of combinational logic in a design? SHould I be avoiding it, or should I just be more careful?!

Thanks again all.

 

[TrickyDicky]

It has made me think though, is it necessarily bad to have a lot of combinational logic in a design? SHould I be avoiding it, or should I just be more careful?!

It depends how long the logic chain is. Each LUT (as that what the logic is fitted to) will add a delay between the registers. The greater the delay, the slower the FMax. If you're hitting iteration limits, its usually a warning the logic is going to be pretty poor when it comes to timing analysis.

Moral of the story - increase the pipelining. Unless you need agressive clocking (like 300+MHz) you dont need to worry too much about how much logic there is between registers, as long as its no more than a simple function, like an add, or a few if's. Loops should really be avoided unless generating parrallel logic.

Check the build reports like KJ said. Look for warnings of latches or combinatorial loops. These should all be removed, as they are very bad things.