[AVR] Need to disable watchdog timer on ATmega328p

[Paul Crawford]

i have built a project to collect data from various sensors. The central computer is an Odroid N2L SBC by Hardkernel running Ubuntu 22.04.1 LTS It acts as database server, web server and data collection server. The data provided by five independent devices is collected via an RS485 network using the Modbus protocol that is connected to the N2L through an RS485 to RJ45 ethernet industrial serial server. Two of the devices are analog to digital DC voltmeters, the third is an analog to digital AC power meter, the fourth is a single phase 240 VAC to three phase variable frequency drive connected to a 370 watt electric motor and the fifth is an ATmega328p MCU.

The MCU mainly collects data from sensors but does have a control function as well that turns on a cut-off relay if a temperature sensor value exceeds a predetermined maximum. The MCU is connected to the RS485 network by a half duplex RS485 to TTL serial converter that is connected to the serial RX and TX ports on the MCU. The software running on the MCU is pure C to retrieve data from the sensors and includes code to receive commands on the RS485 network in pseudo Modbus fashion and transmit the data back to the SBC for processing, saving in the database and preparing presentation as a web page.

There is no problem with any of this system while only the data collection segments are running. The problems start when the SBC sends a command to the VFD to start the motor at a given frequency. When that occurs communication with the MCU is disrupted and resets occur in the MCU. If the MCU were only collecting slowly varying data, it would be tolerable to wait until the VFD is commanded to stop and communication is regained. With the MCU control function getting corrupted by the reset, however, the source of the interference must be determined and corrected or, in the alternative the control function must be reworked to ensure that it continues to operate correctly in spite of the interference.

I am not expecting direct assistance on sorting out the interference issue, but I would appreciate some assistance on exploring the MCU reset. My understanding is that there are only four causes of a reset on the MCU:
1. Power-on
2. External
3. Brown-out
4. Watchdog System

I have added code to the beginning of the MCU initialization to read the MCUSR. I have discovered that when the resets that occur that are coincident with the VFD operation the MCUSR value is always 0x08 indicating a watchdog reset. The SBC can send an external reset to the MCU, independent of the RS485 network, and when it is sent the value of the MCUSR is found to be 0x02 which is correct. The MCU code for this is:

#include <avr/wdt.h>
...
int main(void)
{
   cli();
   mcusr = MCUSR;
   MCUSR = 0;
   wdt_enable(WDTO_8S);
...

The value of mcusr is sent out to the RS485 network by the MCU with all of the sensor data on request from the SBC.

Since the interference seems to always trigger the Watchdog System Reset Flag, I wanted to try running the code without the WDT. If I comment out the wdt_enable line to get the default setting or explicitly stop the WDT with wdt_disable, there is no response from the MCU to commands sent from the SBC. Having run this code continuously for days without any resets, as long as the motor does not start, I am confident that there is no chance of runaway code or a hang so the watchdog may not be necessary. I realize that I should use it in the end anyway.

So my real question is: How can I prevent the WDT from running while still maintaining the command-response functionality at least as a test?

 

[KlausST]

Hi

.. a really long story ... but a lot of information is missing.

* complete code
* wiring information
* debugging methods
* schematics
* PCB layout.
... and so on.

*****
So to your question:

How can I prevent the WDT from running while still maintaining the command-response functionality at least as a test?

Just disable the watchdog.

****
We don't know wher/when in your code you do the watchdog_reset. Maybe move it to another place ...

***
We don't know what causes the "hang". (Noise on the RS485 does not necessarily need to cause a hang)
Is it the RS485 communication? Is it the noise on RS485 lines?
Simple tests: 1) start the motor (VFD) manually, without communication. 2) do VFD communication without starting the motor.

If it's the motor noise:
* how is the wiring of the motor cables and RS485 cables?
* is the RS485 circuit, including protection correct?
* proper 120 Ohm cables? Shielded?
* noise filter on motor signals?

...and maybe 100 more questions and possible reasons...

Klaus

 

[danadakk]

In Arduino :

Watchdog Timer in Arduino | Arduino

How to use Watchdog timer in Arduino to help the system recover from unexpected hang-ups

www.electronicwings.com

Using watchdog timer in your projects - Embedds

All AVR microcontrollers have an internal watchdog timer that can be successfully used in your projects. Atmega328 and other modern AVR microcontrollers have the so-called Enhanced Watchdog Timer (WDT). It has few beneficial features, including a separate 128kHz clock source, reset the...

embedds.com

Regards, Dana.

 

[Paul Crawford]

Klaus,

Yes it is a long story with lots still not included as you say.

As I noted I have tried to disable the wdt with wdt_disable(); from avr/wdt.h. Perhaps it does disable the WDT but as I also noted it seems to stop all communication with the MCU as well. So perhaps that idea is just a dead end in the troubleshooting.

The following is the full main function:

int main(void)
{
   cli();
   mcusr = MCUSR;
   MCUSR = 0;
   wdt_enable(WDTO_8S);
   Z1 = -RTD_A;
   Z2 = RTD_A * RTD_A - (4 * RTD_B);
   Z3 = (4 * RTD_B) / RTDNOMINAL;
   Z4 = 2 * RTD_B;
   uint8_t skiph = 0;
   uint8_t skipl = 0;
   spi_init_master ();
   temperature_init();
   serialInit();
   twi_init();
   timer_init();
   _delay_ms(5);
   acceleration_init();
   cnt = eeprom_read_byte((uint8_t*)32);
   maxabtemp = eeprom_read_byte((uint8_t*)33);
   maxtimer = eeprom_read_byte((uint8_t*)34);
   //ElapsedQSeconds = eeprom_read_byte((uint8_t*)35);
   temperaturertd = temperature(RREF);
   if(temperaturertd >= (float)maxabtemp)
   {
      PORTD |= (0x01 << PD7);
      skiph = 1;
      cnt++;
      eeprom_update_byte((uint8_t*)32,cnt);

   }
   if(temperaturertd < (float)maxabtemp) //&& temperaturertd >= MAXBT - BTDB)
   {
      PORTD &= ~(0x01 << PD7);
      skipl = 1;
   }
   for(;;)
   {
      temperaturertd = temperature(RREF);
      if(temperaturertd > (float)maxabtemp && !skiph)
      {
         PORTD |= (0x01 << PD7);  //set refrigerator cutoff relay (NC) on so not running
         skiph++;
         skipl--;
         cnt++;
         eeprom_update_byte((uint8_t*)32,cnt);
         resettimer();
      }
      if(((temperaturertd < (float)(maxabtemp - BTDB) && !skipl) || (!ElapsedQSeconds && !skipl)) && cnt <= MAXRESTARTS)
      {
         PORTD &= ~(0x01 << PD7);  //set refrigerator cutoff relay (NC) off so running
         skipl++;
         skiph--;
         resettimer();
      }
      wdt_reset();
      if(!tpcr)
      {
         tpcr = TPCRMAX;
      }
      wdt_reset();
      if(!tsht)
      {
         readbshttwi();  //temp removal to test
         tsht = TSHTMAX;
      }
      wdt_reset();

      if(!tbme)
      {
         readbmetwi();  //temp removal to test
         tbme = TBMEMAX;
      }
      acceleration_init();
      wdt_reset();
      readxyztwi();  //temp removal to test
      wdt_reset();
      serialHandleData();
      wdt_reset();
      if(ElapsedQ0MilliSeconds >= 8000)
      {
         if(ElapsedQSeconds > 0)
         {
            ElapsedQSeconds--;
            eeprom_update_byte((uint8_t*)35,ElapsedQSeconds);
            ElapsedQ0MilliSeconds = 0;
         }
      }
      wdt_reset();
      tpcr--;
      tbme--;
      tsht--;

   }
}

As you can see I have sprinkled wdt_reset(); calls through out the loop. I used to have only one call at the end of the loop, but adding the others makes no difference anyway. The loop runs in about 0.33 seconds so it is hard to imagine an 8 second timer would get tripped without a severe hang in the loop.

I have completed the simple tests. Running the motor manually from the VFD does definitely cause the communication errors. Disconnecting the motor leads from the VFD and sending the communication signal does not affect the communication with the MCU at all.

It would seem that main issue is the motor itself and not the VFD or the RS485. The VFD is located inside a steel electrical box and the motor leads go from the terminals on the VFD through steel electric metallic tube to a metal motor junction box. The motor itself is totally enclosed fan cooled. The RS485 network on the other four devices continues to work perfectly during the motor operation so it would appear that the motor noise is not affecting the network, only the MCU.

In my searches on this topic I have seen situations that were similar but there was no resolution documented:
https://forum.arduino.cc/t/atmega328-noise-by-3-phase-motor/442394

I am powering the MCU with a battery rather than a PSU to minimize the noise on the DC power. I also have a low pass filter which is not in the circuit yet but could be tried. There are capacitors on the Vcc pins at the MCU. Watching the DC voltage with a Fluke 177 DMM did not show any obvious loss of power but the DMM is not an oscilloscope.

Thank you so much for your time and response.

 

[Paul Crawford]

Should have been "Vdd pins at the MCU" not "Vcc pins at the MCU"

 

[KlausST]

Hi,

seems to stop all communication with the MCU

You need to understand that "disabling the watchdog" is not the problem. The problem is that the MCU hangs.
You need to find out why. The root cause.

You say the motor is the problem. But it is not. It's the noise sent out by the cable.
So you have two options:
* reduce the noise by using filters
* or make your software more fault proof
.. or both

...
I come back later.

Klaus

 

[Paul Crawford]

Would a AVR32DB28-I/SP be better choice than the ATmega328p I am presently using in terms of noise immunity ?

 

[Paul Crawford]

I do understand that disabling the watchdog is not the problem. I was only trying to see what would happen if it was. And apparently I did see that nothing works if it is disabled with the present software/

I also understand that the motor itself is not the problem. It is more likely to be EMI from the power cable to the motor. But I would have thought that putting the cable inside a metal tube would have reduced the amount of EMI radiated. Perhaps I am very wrong on that assumption.

I assume you mean reduce the noise on the input power to the MCU. I will wire in the filter that I have and see if that makes any difference.

Can you give me an example of how I might the software more fault proof?

 

[KlausST]

Hi,

You still miss to give informations about wiring. Which cables, how long, termination.....it's annoying for both of us that I need to repeat.
The same is true for schematic and PCB layout.
I guess this is not top secret information.

*******
I don't think it's an (AVR) power supply problem, and I don't think it's a hardware (AVR) problem.
Usually it's not the hardware (microcontroller) that get's upset with noise (unless it's a very heavy pulse causing latch up) ... and if, then I expect even the watchdog can't restore operation.
In all my decades of industrial design I never experienced (reoperable) true hardware fail caused by EMI. (And some of my AVRs work 24/7 close to a 3000A/2000V switched power supply without true shielding case in a silicon production site).

It's way more likely that EM noise causes an "erroneous bit" in the serial data stream ... and your software can't handle this wrong bit.
(But for sure my schematics and PCB layouts, as well as wiring was designed for high EMC. I still don't know about yours).

I need some time to check your code ..

Klaus

Added:
Just had a quick view on your code.
Am I dumb? I can't find the communication part in your code?
I clearly wrote "complete code" ...
Don't get me wrong ... but I don't get it. You write a big story about RS485 communication and problems ... but don't post the according code?

So I leave it to you to debug your code.
Do a search on loops that may become infinite on erroneous serial bitstream.. Like "while ....".
Also check for buffers (UART receive) that may overflow on erroneous messages .. and don't become cleared upon timeouts.
Also "expect" erroneous bits, erroneous message length, check on UART_errors and clear them - especially when they may stop further communication.
Expect that messages get out of sync, include some frame syncing in your code.

Btw: in my industrial function code there only rarely is a "busy wait". Neither "delay()", nor "while()" (besides the main loop), nor other loops.
This way I don't waste processing power and keep all (other) functions running.

Debugging hint:
Use spare pins as debug outputs. Set them to a dedicated state before entering a function (like UART receive, or message parsing) and clear them when they come back. Trigger the "hang" and read the pin's states.

Good luck

Klaus

 

[Paul Crawford]

Hi Klaus,

Thank you for all of your help. The complete code for the MCU is attached. The message that is sent from the SBC to the MCU is:

         msg[0] = 0x02;    // RS485 address
         msg[1] = 0x08;    // number of bytes in message
         msg[2] = 0x03;    // command byte for reading data
         msg[3] = 0x00;    // dummy to fill modbus command message size
         msg[4] = 0x00;    // dummy to fill modbus command message size
         msg[5] = 0x00;    // dummy to fill modbus command message size
         crc16(msg, MODBUSCMDSIZE - CRCSIZE);            // get crc appended

This is typical output from the MCU with and without the pump running:

0a-0:04 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:00 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:6d 23:14 
0a-0:05 1:0f 2:04 3:00 4:37 5:2a 6:08 7:00 8:3d 9:90 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:2f 23:f3 
0a-0:06 1:0e 2:fd 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:8b 23:3e 
0a-0:07 1:0f 2:01 3:00 4:37 5:2a 6:08 7:00 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:ff 23:4f 
0a-0:08 1:0f 2:01 3:00 4:37 5:2a 6:08 7:70 8:3e 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:33 23:28 
0a-0:09 1:0f 2:04 3:00 4:37 5:2a 6:08 7:20 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:7c 23:8e 
0a-0:0a 1:0f 2:04 3:00 4:37 5:2a 6:08 7:d0 8:3c 9:80 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:7e 23:be 
0a-0:0b 1:0f 2:07 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:b0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:9b 23:64 
0a-0:0c 1:0f 2:07 3:00 4:37 5:2a 6:08 7:00 8:3d 9:f0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:b3 23:0a 
0a-0:0d 1:0f 2:07 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:d0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:fc 23:ee 
0a-0:0e 1:0f 2:07 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:d0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:bf 23:ef 
0a-0:0f 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:b0 8:3e 9:c0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:22 23:55 
0a-0:10 1:0f 2:07 3:00 4:37 5:2a 6:08 7:10 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:40 23:ce 
0a-0:11 1:0f 2:07 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:40 23:5e 
0a-0:12 1:0f 2:04 3:00 4:37 5:2a 6:08 7:c0 8:3e 9:00 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:74 19:14 20:04 21:05 22:0a 23:1a 
0a-0:13 1:0f 2:07 3:00 4:37 5:2a 6:08 7:d0 8:3c 9:d0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:3d 23:e6 
0a-0:14 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:a0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:84 23:24 
0a-0:15 1:0f 2:07 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:aa 23:e9 
0a-0:16 1:0f 2:07 3:00 4:37 5:2a 6:08 7:20 8:3d 9:90 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:a6 23:9c 
0a-0:17 1:0f 2:04 3:00 4:37 5:2a 6:08 7:f0 8:3e 9:30 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:61 23:9d 
0a-0:18 1:0f 2:07 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:b0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:46 23:34 
0a-0:19 1:0f 2:07 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:80 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:f7 23:10 
0a-0:1a 1:0f 2:07 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:d0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:54 23:59 
0a-0:1b 1:0f 2:04 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:b0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:61 23:1e 
0a-0:1c 1:0f 2:04 3:00 4:37 5:2a 6:08 7:20 8:3d 9:b0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:b7 23:b0 
0a-0:1d 1:0f 2:07 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:03 23:57 
0a-0:1e 1:0f 2:0e 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:b0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:6c 23:8a 
0a-0:1f 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:10 8:3e 9:10 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:23 23:ed 
0a-0:20 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:00 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:fc 23:44 
0a-0:21 1:0f 2:07 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:e0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:fc 23:15 
0a-0:22 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:d0 8:3e 9:10 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:cc 23:33 
0a-0:23 1:0f 2:07 3:00 4:37 5:2a 6:08 7:b0 8:3d 9:d0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:6e 23:18 
0a-0:24 1:0f 2:0e 3:00 4:37 5:2a 6:08 7:e0 8:3e 9:00 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:ee 21:04 22:e8 23:c8 
0a-0:25 1:0f 2:04 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:f0 10:cb 11:7d 12:00 13:8e 14:4e 15:00 16:80 17:85 18:75 19:14 20:dc 21:04 22:d6 23:25 
0a-0:26 1:0f 2:07 3:00 4:37 5:2a 6:08 7:50 8:3e 9:e0 10:c7 11:7d 12:00 13:8a 14:4e 15:00 16:86 17:85 18:75 19:14 20:dc 21:04 22:6e 23:4a 
0a-0:27 1:0f 2:0e 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:c0 10:c7 11:7d 12:00 13:8a 14:4e 15:00 16:86 17:85 18:75 19:14 20:dc 21:04 22:be 23:05 
0a-0:28 1:0f 2:07 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:c0 10:c7 11:7d 12:00 13:8a 14:4e 15:00 16:86 17:85 18:75 19:14 20:dc 21:04 22:ed 23:c9 
0a-0:29 1:0f 2:07 3:00 4:37 5:2a 6:08 7:40 8:3d 9:f0 10:c7 11:7d 12:00 13:8a 14:4e 15:00 16:86 17:85 18:75 19:14 20:dc 21:04 22:fc 23:95 
run pump at 44.73 Hz for 9 s
0a-0:2a 1:0f 2:0e 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:d0 10:c7 11:7d 12:00 13:8a 14:4e 15:00 16:86 17:85 18:75 19:14 20:dc 21:04 22:9c 23:0c 
0a-0:00 1:0e 2:fd 3:00 4:37 5:2a 6:08 7:80 8:3d 9:30 10:c9 11:7d 12:00 13:8e 14:4e 15:00 16:83 17:85 18:72 19:14 20:ec 21:04 22:54 23:1d 
mcu reset at: 2a in cycle: 270
0a-0:01 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:e0 10:c9 11:7d 12:00 13:8e 14:4e 15:00 16:83 17:85 18:72 19:14 20:ec 21:04 22:6e 23:df 
stop pump for 9 s
run pump at 26.98 Hz for 117 s
0a-0:00 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:10 8:3d 9:a0 10:cf 11:7d 12:00 13:91 14:4e 15:00 16:73 17:85 18:6f 19:14 20:e2 21:04 22:22 23:53 
mcu reset at: 01 in cycle: 271
0a-0:00 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:a0 10:cb 11:7d 12:00 13:90 14:4e 15:00 16:7a 17:85 18:70 19:14 20:f6 21:04 22:16 23:11 
mcu reset at: 00 in cycle: 272
0a-0:01 1:0e 2:f0 3:00 4:37 5:2a 6:08 7:00 8:3e 9:00 10:cb 11:7d 12:00 13:90 14:4e 15:00 16:7a 17:85 18:70 19:14 20:f6 21:04 22:0d 23:88 
0a-0:02 1:0e 2:fd 3:00 4:37 5:2a 6:08 7:00 8:3d 9:d0 10:cb 11:7d 12:00 13:90 14:4e 15:00 16:7a 17:85 18:70 19:14 20:f6 21:04 22:6f 23:13 
0a-0:03 1:0f 2:01 3:00 4:37 5:2a 6:08 7:30 8:3d 9:d0 10:cb 11:7d 12:00 13:90 14:4e 15:00 16:7a 17:85 18:70 19:14 20:f6 21:04 22:db 23:32 
0a-0:04 1:0e 2:fd 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:90 10:cb 11:7d 12:00 13:90 14:4e 15:00 16:7a 17:85 18:70 19:14 20:f6 21:04 22:87 23:bd 
0a-0:00 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:c0 8:3e 9:e0 10:c9 11:7d 12:00 13:8f 14:4e 15:00 16:83 17:85 18:6f 19:14 20:e4 21:04 22:91 23:34 
mcu reset at: 04 in cycle: 273
0a-0:01 1:0e 2:f3 3:00 4:37 5:2a 6:08 7:c0 8:3e 9:20 10:c9 11:7d 12:00 13:8f 14:4e 15:00 16:83 17:85 18:6f 19:14 20:e4 21:04 22:9c 23:61 
0a-0:00 1:0e 2:f7 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:e0 10:cf 11:7d 12:00 13:93 14:4e 15:00 16:7d 17:85 18:6e 19:14 20:fd 21:04 22:30 23:51 
mcu reset at: 01 in cycle: 274
0a-0:00 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:31 8:3d 9:f0 10:c6 11:7d 12:00 13:90 14:4e 15:00 16:87 17:85 18:72 19:14 20:fb 21:04 22:0a 23:1f 
mcu reset at: 00 in cycle: 275
0a-0:01 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:d0 8:3e 9:00 10:c6 11:7d 12:00 13:90 14:4e 15:00 16:87 17:85 18:72 19:14 20:fb 21:04 22:b8 23:69 
0a-0:02 1:0e 2:f7 3:00 4:37 5:2a 6:08 7:c0 8:3e 9:90 10:c6 11:7d 12:00 13:90 14:4e 15:00 16:87 17:85 18:72 19:14 20:fb 21:04 22:37 23:3d 
0a-0:01 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:a0 8:3d 9:b0 10:c3 11:7d 12:00 13:8e 14:4e 15:00 16:8a 17:85 18:6f 19:14 20:f8 21:04 22:ff 23:52 
missed mega output: 00 in cycle: 275
0a-0:02 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:e0 8:3e 9:20 10:c3 11:7d 12:00 13:8e 14:4e 15:00 16:8a 17:85 18:6f 19:14 20:f8 21:04 22:0e 23:25 
stop pump for 13 s
0a-0:00 1:0e 2:f7 3:00 4:37 5:2a 6:08 7:c0 8:3c 9:f0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:90 23:58 
mcu reset at: 02 in cycle: 276
0a-0:01 1:0e 2:f7 3:00 4:37 5:2a 6:08 7:00 8:3d 9:90 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:01 23:a5 
0a-0:02 1:0f 2:01 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:c0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:9e 23:fa 
0a-0:03 1:0f 2:01 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:f0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:2f 23:de 
0a-0:04 1:0f 2:01 3:00 4:37 5:2a 6:08 7:d0 8:3e 9:10 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:74 23:56 
0a-0:05 1:0f 2:01 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:e0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:e7 23:68 
0a-0:06 1:0f 2:07 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:d0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:5d 23:eb 
0a-0:07 1:0f 2:07 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:f0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:d2 23:5f 
0a-0:08 1:0f 2:07 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:d0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:02 23:73 
0a-0:09 1:0f 2:07 3:00 4:37 5:2a 6:08 7:00 8:3d 9:f0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:4d 23:97 
0a-0:0a 1:0f 2:04 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:90 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:0b 23:55 
0a-0:0b 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:b0 8:3d 9:e0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:70 23:67 
0a-0:0c 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:d0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:47 23:41 
0a-0:0d 1:0f 2:07 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:f0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:d8 23:38 
0a-0:0e 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:d0 8:3d 9:e0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:85 23:70 
0a-0:0f 1:0f 2:07 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:a0 10:c6 11:7d 12:00 13:8e 14:4e 15:00 16:85 17:85 18:6f 19:14 20:f8 21:04 22:9b 23:a9 
run pump at 52.96 Hz for 8 s
stop pump for 10 s
run pump at 53.48 Hz for 93 s
0a-0:01 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:30 8:3d 9:a0 10:c5 11:7d 12:00 13:8f 14:4e 15:00 16:89 17:85 18:6c 19:14 20:e8 21:04 22:59 23:81 
missed mega output: 00 in cycle: 276
stop pump for 33 s
0a-0:00 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:b0 8:3e 9:00 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:ee 23:f8 
mcu reset at: 01 in cycle: 277
0a-0:01 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:10 8:3d 9:d0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:a2 23:de 
0a-0:02 1:0e 2:fd 3:00 4:37 5:2a 6:08 7:00 8:3e 9:00 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:48 23:e2 
0a-0:03 1:0e 2:fa 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:d0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:e1 23:4f 
0a-0:04 1:0f 2:01 3:00 4:37 5:2a 6:08 7:20 8:3d 9:d0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:e6 23:bb 
0a-0:05 1:0f 2:01 3:00 4:37 5:2a 6:08 7:e0 8:3d 9:90 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:08 23:e7 
0a-0:06 1:0f 2:04 3:00 4:37 5:2a 6:08 7:b0 8:3e 9:10 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:1b 23:59 
0a-0:07 1:0f 2:04 3:00 4:37 5:2a 6:08 7:80 8:3e 9:10 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:fb 23:81 
0a-0:08 1:0f 2:04 3:00 4:37 5:2a 6:08 7:c0 8:3d 9:c0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:38 23:63 
0a-0:09 1:0f 2:04 3:00 4:37 5:2a 6:08 7:b0 8:3d 9:f0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:d9 23:7b 
0a-0:0a 1:0f 2:04 3:00 4:37 5:2a 6:08 7:b0 8:3d 9:d0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:d5 23:0e 
0a-0:0b 1:0f 2:04 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:c0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:7b 23:62 
0a-0:0c 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:a0 8:3d 9:80 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:b7 23:86 
0a-0:0d 1:0f 2:0b 3:00 4:37 5:2a 6:08 7:f0 8:3d 9:f0 10:cb 11:7d 12:00 13:91 14:4e 15:00 16:81 17:85 18:6a 19:14 20:f8 21:04 22:88 23:46