Utracer6 Crash and Burn

[Never Mind]

I have found a unpleasant failure mode with my utracer6 while using utracerJS.

While rechecking the uTeacer6's calibration I made some edits to the calibration file.

I did these edits manually with excel as the EEprom file was called a CSV file.

In hindsight it seems ill advised to manually edit the EEprom file with excel.

After my calibration file edit I found when I ran  I(Vg, Va) with Vs, Vh constant the utracerJS GUI became unresponsive. 

I then noticed the utracerJS WEB server that runs on the PC the utracer6 was connected to had stopped running.

I also stated to notice a "funny" smell. 

Touching T60, T80 , and L60, 61, 80 ,81 on the utracer6 I found that they were all hot.

I disconnected the power from the utracer6 and then reapplied power.

After restating the utracerJS web server on the PC and rerunning I(Vg, Va) with Vs, Vh again the GUI became unresponsive.

Again the utracer6 was becoming very warm.

Thinking the configuration file was corrupted and so causing the WEB server to crash and thus also crashing the utracer6 PIC processor I decided to remove my utracerJS install directory and to then reinstall the all the utracerJS files.

After doing this the utracer6 and utracerJS worked perfectly again. Bullet dodged.

So two take always.

1) Do NOT manually edit the utracerJS EEPROM. CSV file using excel.

2) There seems to be a nasty utracer6 PIC processor failure mode that can be induced by sending incorrect signals down the serial port.

In this failure mode on the utracer6 the PIC entered a invalid state that resulted in both T60 and T80 being driven permanently on. This results in L60,61, L80,81 saturating and the current then rising to a high level now only now limited by the low on resistance of T60,80 and the small DCR of L60,61 and L80,81.

So the current can reach many amperes in both T60 and T80. This results is a large power dissipation that will destroy these devices in a very short period of time.

The utracer6 hardware design has no hardware protection to prevent burning out T60 and T80 if the PIC enters a invalid state. This is not good.

In my case I believe my utracer6 was saved from permanent damage as it was powered by a 1.5 amp current limited lab bench power supply.

Had I used the recommenced laptop power supply that can normally supply much higher currents it seems clear the utracer6 would have suffered permanent damage when the PIC entered this invalid state.

For now I will make sure I only use a 1.5 amp current limited bench supply with my utracer6 with a current meter to allow me to monitor the current draw.

It seems clear the utracer6 as currently designed is far too easy to damage if a software error occurs in the GUI or the PIC being controlled that causes the PIC to crash.

The PIC code should have effective watchdog code that works with the PICs built in hardware watchdog timer and monitors the state of T60 and T80. If the watchdog code sees T60 or T80 freeze in a high state then the hardware watchdog timer is allowed to expire and so resetting the PIC. This type of watchdog code is pretty easily deployed as background code using spare real time in the PIC.

Unfortunately we do not have access to the PIC code so some other workaround seems required. I am going to look a creating a external watchdog that will monitor the state of T60 and T80 and apply a hardware reset to the PIC if they lock high.

Not sure how this will be done but a small 8 pin processor running a simple state machine comes to mind as one way. Perhaps a bit of CMOS logic and a few timing capacitors will be workable as well. Any other suggestions out there?

[Ihor]

What an experience! I am glad that in the end everything is up and running again. 

Indeed, it is probably not a good idea to edit that csv with excel, at the same time it si a simple text file so it can be edited with any text editor. Still, when it is done do, it is also good to check if the new values are properly picked up with the uTracerJS interface. It can be done by opening the Calibration tab in uTracerJS and checking if those numbers are reasonable and if there are no obvious number-parsing-from-file errors (words like NAN or undefined). 

With a wrong parsing in that case probably the worst possible combination was sent to the uTracer: high voltage measurement, with high current amplification settings and high grid voltage, probably the uTracer also could not stop charging or overcharged the caps... in the worst case the command could also specify that the measurements should be done in a "continuous" mode, which means that there is no measurement pulse going on but actually high voltage from the caps is constantly applied to the DUT. I do not have uTracer6 but only uTracer3 and I had once similar hangs (not necessary with uTracerJS) but also while messing up with the hardware communication and then I burned a couple of MOSFETs. I also saw some posts here as well, when the communication breaks and the HV LED continues to be on). In uTracer3 there is also a "mode" when for wrong calibration and low Va (less then 2V) the uTracer will hang in some unresponsive state because the algorithm that charges the caps cannot reach that level due to the limitation of the ADCs and so on, but at least there nothing burns. As far as I read on the uTracer6 webpage, the PIC's firmware has better watchdogs for charging the caps and introducing some timeout controls in several places which are not in the uTracer3 firmware, but probably it is not enough. Ronald wrote the whole firmware directly in assembler for PIC (versus a modern way of writing it in C and compiling for PIC) and typically it takes more effort to introduce all those watchdogs, "parallel" running routines and other nice debug and precaution features, so I am not sure how much time he can invest in that.