Finding why a device resets in the field

A unit at a customer site restarts about once a day. The same firmware on your bench, with a debugger attached, runs for a week without a hiccup. Nothing was logged, because the code that would have logged it is the code that restarted.

This is one of the most common problems in shipped embedded products, and it is rarely solved by staring at source code. It is solved by making the device tell you what happened. That takes two pieces of instrumentation, both small, and both far easier to add before the product ships than after.

Read the reset cause, and read it correctly

Every microcontroller we work with records why it last came out of reset. On the Nordic nRF52 it is the RESETREAS register in the POWER peripheral. On Espressif parts, ESP-IDF wraps it as esp_reset_reason(), which separates power-on, brownout, panic, and the different watchdogs. On Zephyr, the portable call is hwinfo_get_reset_cause(). STM32 parts keep a set of reset flags in an RCC status register; the register name and the exact flags differ between families, so check the reference manual for the part on the board.

Reading the register is one line. Reading it correctly takes more care. Three mistakes come up again and again.

Not clearing it. Nordic’s documentation is direct: unless cleared, RESETREAS is cumulative, and a field is cleared by writing a 1 to it. Zephyr documents the same behavior for its API: successive calls return the same value until hwinfo_clear_reset_cause() is called. If the firmware never clears the flags, a watchdog reset from last Tuesday is still reported today, next to every reset since. Read once, early in boot, store the value, then clear.

Letting the bootloader eat it. If a bootloader runs first and reads and clears the flags, the application sees nothing useful. Decide which stage owns the register, and have it pass the value forward.

Assuming every cause has a flag. On the nRF52832, no flag set at all means the reset came from the on-chip reset generator, which Nordic describes as a power-on reset or a brownout reset. The register alone cannot tell those two apart. A design that needs to distinguish them has to get that information another way, such as a supply measurement taken at boot.

Reported cause	What it usually points to	Next check
Brownout or power-on	Supply dip, battery sag, connector	Scope the rail, triggered on reset
Watchdog	Hang, deadlock, a blocked loop	Which task stopped checking in
Software reset or panic	A fault handler or assert fired	The saved fault record
Reset pin	Noise or ESD on the reset line	Reset circuit against the datasheet
CPU lockup	A fault inside a fault handler	Stack and handler code

Store the cause with a boot counter in nonvolatile memory. A count of resets by cause, read out of a returned unit, often narrows the problem to either hardware or firmware before anyone opens a debugger.

Capture the fault before the reboot

The reset cause says what kind of event it was. It does not say where. For firmware crashes, the device needs to save a small record at the moment of the fault.

On Arm Cortex-M parts, which covers STM32, nRF52, and Microchip SAM, the core pushes eight registers onto the active stack when an exception is taken: r0 through r3, r12, the link register, the program counter, and the program status register. The fault status registers in the System Control Block record the class of fault. A fault handler that saves the stacked program counter, the link register, and the fault status registers has enough to name the function that crashed, given the matching build. One caveat matters here. Some bus faults are reported as imprecise, which means the stacked program counter is not the instruction that caused the fault. The status register flags this, so record it and interpret the address accordingly.

Where the record goes matters as much as what is in it. Writing to flash from inside a fault handler is risky. The system state cannot be trusted, and if the root cause is a failing supply, a flash write is the wrong thing to start. The usual pattern is to write the record to a region of RAM that the startup code does not initialize, mark it with a magic value and a checksum, and reset. On the next boot, healthy firmware validates the record and moves it to flash or sends it out. This works for warm resets where the supply stays up. Confirm on your specific part that RAM contents survive the reset types you care about.

The frameworks already provide much of this. ESP-IDF can save a core dump to a dedicated flash partition or send it over UART, and idf.py coredump-info decodes it into registers and call stacks. Zephyr has a core dump subsystem with logging and flash partition backends. One Zephyr default deserves attention: the stock fatal error handler halts the system. In a product with no watchdog running, that is a device that sits dead until someone cycles power. The handler is a weak symbol so that the application can set its own policy, and a shipping product should.

None of this helps without the matching build. A program counter value is only meaningful against the ELF file for that exact firmware. Put the firmware version in the fault record, and archive the ELF for every release that leaves the building.

Make the watchdog say something

A hardware watchdog reset leaves one bit behind. The firmware hung, somewhere.

The way to get more is to arrange for a warning before the hardware watchdog fires. A timer interrupt set to expire a little earlier can record which tasks had stopped checking in, and the interrupted program counter, and then let the reset happen. The Interrupt blog’s guide to watchdog timers walks through this pattern on an nRF52 and is worth reading in full.

Check the defaults as well. ESP-IDF runs two watchdogs. The interrupt watchdog invokes the panic handler on timeout. The task watchdog, by default, prints a warning and a backtrace and keeps running. On a deployed device with nobody reading the console, that warning goes nowhere. It can be configured to panic instead.

Where the watchdog is fed matters too. Feeding it from a timer interrupt proves only that interrupts still run. Feed it from a place that is reached only when every task that matters has reported in.

Telling a power problem from a firmware bug

Brownout resets tend to line up with events. A motor starts, a relay pulls in, a radio transmits, a battery gets cold or old. If the boot record includes uptime and, where an ADC channel is available, a supply reading, the pattern usually shows itself. On the bench, put a scope on the rail and trigger on the reset line, then reproduce the load event. This is also where the debugger misleads. A unit powered from a bench supply through a short cable is not the unit in the field on a long harness.

Reset pin causes point to the reset circuit. Compare it against the datasheet’s recommended circuit, and look at what else is routed near that trace.

Stack overflows are the firmware fault that hides best. The overflow corrupts a neighbor, and the crash shows up later in code that did nothing wrong. ESP-IDF addresses this by setting a watchpoint near the end of each task’s stack, as described in its fatal errors guide. On Cortex-M parts with a memory protection unit, a guard region does similar work. A set of fault records that point to unrelated places is a reason to measure stack usage first.

Finally, test the instrumentation on purpose. Force a fault, starve the watchdog, dip the supply, and confirm that the record names each one correctly. A fault handler that has never run is a guess.

When we take on a reset problem we start with the reset cause and a fault record, so send us the schematic, the firmware source or the exact build, and whatever you know about when the resets happen.