MightyBoard Failure Analysis

[Gary Crowell]

At least a start anyway.

It turns out that an associate here at work had a failed MightyBoard, and had received a replacement, but he had been too busy to swap it out.  I offered to put the new board in his Replicator, in exchange for getting my hands on the failed board.  Despite all my efforts, the new board in his Replicator appears to be working OK, and here's what I found.

The first thing I looked at was the X-stop cable:

• Inspected both connectors and the individual wires under microscope - nothing unusual.
• Did a discontinuity check on every pin combination, on each connector, and between connectors at each cable end - normal.
• Did a continuity check on each conductor from cable end to end - normal.
• Removed pins from connector shell on both ends, and inspected wire ends and crimps under microscope.  The crimps were very well done; there were no stray strands of wire.
• Removed shrink tube from both ends of cable; inspected - nothing unusual.
• Removed shrink tube from the 'bare' wire on both ends; inspected - nothing unusual.
• Removed insulation from the cable ~2" at a time; inspected - nothing unusual.

The X-stop cable did not cause this board failure.  But it is a weird cable construction, and is probably worthy of replacement anyway.  That bare wire removes the 'double insulation' between conductors that you would expect to find in a cable.

Then I pulled the BotSteps off of the failed MightyBoard.

• Inspected each under the microscope, both sides, looking for any visible signs of shorts or component failure - nothing like that was noted.
• Interesting things that I did see:
• The headers on the board are gold; headers on a new BotStep are tin.
• Headers have significant solder flux residue; no cleaning was done after the headers were soldered on.  New BotSteps are clean.
• One BotStep had some white, sticky residue on the driver chip.  Didn't look like flux or cleaning residue.  Maybe someone blew their nose or popped a zit.
• Nearly all of the board edges had been scored and removed from an assembly panel.  New BotSteps are routed on the long edges and scored only on the short edges, indicating a change in assembly panelization.  If not done correctly, depanelization can damage surface mount components on a board.  I didn't see anything that looked damaged (i.e., cracked capacitors) on any of the BotSteps.

Then I made resistance measurements of the BotStep's:

• VMot to Gnd measured 360K and 1.8M(leads reversed) on all boards.
• 5V to VMot measured 360K and 1.8M on all boards.
• 5V to Gnd on all boards measured between 1.2K and 20 Ohms.
• 5V to Gnd on a new BotStep measures 16.5K.

So it's pretty certain that all of the BotSteps were blown when the MightyBoard failed.  So don't expect to have extra drivers from those failed MightyBoards.

Comparing the failed MightyBoard to the new one:

• The old pcb was fabricated in 2011, workweek 52, the new in 2012 workweek 26.
• The old has a butt-ugly black soldermask, the new is normal green.
• PCB fabrication on the new board appears to be significantly higher quality.
• The old board appears to be assembled with lead-free solder (joints appear dull); the new appears to be assembled with leaded solder (bright, shiny joints).  (Cheers, no RoHS for us!)
• The old board had cleaning residue, the new board is perfectly clean.

I then inspected the failed board under the microscope, both sides, looking for any visible signs of shorts or component failure.  Other than the obviously blown 5V regulator, nothing was noted.

It has been thought that a 5V to 24V short might be responsible for the regulator failure, so I looked carefully at areas on the board where 5V and 24V are in proximity.  I found all such points on the CAD data and looked carefully at the corresponding points on the board.  There was no evidence of shorting at any of those points.  I was also looking for this type of short when I inspected the BotSteps.  One of the points that occurred to me later however, wasn't on the CAD data - it's the trace cut that was made on the board to switch the 3.3V regulator from a 24V to a 5V input.  This would be a very uncontrolled potential short area, if there were any slivers of copper left in the cut.  The trace cuts on both the old and new boards appeared very clean.  

I then began taking resistance measurements of the failed board, while sequentially removing components that are connected to the 5V rail.

• 24V to Gnd measured a normal capacitor response of the 24V bulk capacitor.
• 5V to 24V measured a normal capacitor response of the 24V bulk capacitor.
• 5V to Gnd measured 14 Ohms.
• Removed the 5V regulator from the board, also removing the wire jumper connection from 5V to the 3.3V regulator.  5V to Gnd measured 16 Ohms.
• Removed the LED driver (PCA9533D).  5V to Gnd measured 16 Ohms.
• Removed both thermocouple chips (MAX6675ISA).  5V to Gnd measured 19 Ohms.
• Removed X-axis digital pot (MCP4018).  5V to Gnd measured 20 Ohms.
• Removed the ATMEGA8U2-AU.  5V to Gnd measured 22 Ohms.
• Removed the ATMEGA1280.  5V to Gnd measured 428 Ohms.
• Removed Y and Z-axis digital pots.  5V to Gnd measured 2.7K.
• Removed Extruder A digital pot.  5V to Gnd measured 4.3K.
• Removed Extruder B digital pot.  5V to Gnd measured 4.7K.

What does this mean?  It has long been surmised that when the regulator fails, it applies 24V to the 5V rail, damaging many/most/all of the components connected to 5V.  These values might mean very little in finding the original point of failure.

From those resistance readings, we can work backwards with parallel resistance calculations to get an idea of the resistance of each chip

• Regulator: 120 Ohms
• LED driver: open
• Thermocouple IC's (both together): 100 Ohms
• X-axis pot: 370 Ohms
• ATMEGA8U2-AU : 170 Ohms
• ATMEGA1280 : 24 Ohms
• Y and Z axis pots together : 500 Ohms
• Extruder A pot : 7.3K
• Extruder B pot : 51K
• Rest of board : 4.7K
• Somebody might want to check me on this.

My best guess:  The wide range of apparent resistance values for the digital pots makes me suspicious.  If you look at the digital pots, they are surrounded on two sides by the walls of the BotStep header connectors; on one end by the motor connector, 50% blocked on the other end by a large tantalum capacitor, and covered by the BotStep.  The driver chip of the BotStep sits directly above it, radiating heat into this nice little oven compartment.  There is no airflow.  I'm thinking that a digital pot fails with a resulting high 5V current draw.  The linear regulator, which is burning the current difference between 5 and 24V as heat, immediately tries to radiate 5x the power that the pot is over-drawing, and blows.  The regulator short then toasts everything else.

This doesn't explain why boards fail shortly after turn-on.  Perhaps the pot has been damaged the last time the Replicator was used, and the pot current coupled with the turn-on surge blows the regulator.  Doesn't explain why brand new boards fail at first turn-on.

To do / questions / thoughts:

• Get a thermocouple into one of those little ovens on an operating Replicator.
• When a pot fails, what happens to its output?  If it failed so that the driver went to max current, the Replicator might appear to function normally.
• The MCP4018 digital pot does have a high upper temp range: 125C
• Get another dead board... and do what?
• A failed MightyBoard is pretty much irreparable. 
• Get a digital pot, put it on a breadboard, power it up, apply heat and see what happens?
• I have some of the components that I removed, I can try to measure their 5V to Gnd out-of-circuit resistance when I can get back to a microscope to see the tiny buggers.
• I notice that I missed at least one device on the 5V rail: 74HC3G14DP  I'll have to take a second look at the docs to see if I missed any others.

Gary

--
----------------------------------------------
Gary A. Crowell Sr., P.E., CID+

Linkedin  Elance  KE7FIZ

[Shawn]

Thanks Gary. This is the only reason I've been holding onto my failed
boards. Now I feel free to pass em on or toss/recycle them.

[drando...@gmail.com]

With all the work you've done Gary be sure to send an invoice to MBI for a consulting fee. :)

[Jetguy]

Thanks for the thorough analysis!!!!!

Just spitballing some ideas here:
I believe(and totally could be wrong here) that the failure comes
either from overload of current from some device on the 5 volt rail,
or it's the fact such a large differential between the 24 volts coming
in being reduced to 5 volts.
Another remote option is the power supply producing a voltage spike or
slow over voltage that changed over time that exceeds the regulators
specs.

Finally, and having a machine in my hands helped me see this, the
round poswer supply input connector can be jammed in inverted thus
supplying reverse 24 volts to the board and an obvious blowout the
instant that switch is flipped. I know the plug is polarized, BUT, it
appears it can make contact in reverse, especially if the switch is on
but not be plugged in past the "key".

I see a VERY likely scenario where the bot was moved, the switch was
on and the owner was plugging the cable into the back and had it
rotated. It makes contact reverse polarity and blows the board sky
high.

We need layers of protection here.
First, put a 12 volt regulator between the 24 volts and the 5 volts.
This adds another layer and some slighty higher input power
tollerance.
Next, the resetable circuit breakers rated just above monral current
levels but far belwo the max rating of the regulators.
Reverse input diode and a matching fuse. This way, if you plug it in
backwards, the diode clamps the current and blows the fuse.

In theory, if we did the above layered aproach, we might prevent some
failures.

One final failsafe would be a massive Zener diode on the 5 volt rail
fed by an appropriate sized fuse. This way, if all the above failed,
we still limit voltage to the 5 volt rail and block reverse current
there too. This idea comes from a fix over in the DIY Drones group who
kept blowing out their expensive boards for the same overvoltage and
reverse voltage reasons.

On Dec 16, 1:39 am, Gary Crowell <garyacrowel...@gmail.com> wrote:
> At least a start anyway.
>
> It turns out that an associate here at work had a failed MightyBoard, and
> had received a replacement, but he had been too busy to swap it out.  I
> offered to put the new board in his Replicator, in exchange for getting my
> hands on the failed board.  Despite all my efforts, the new board in his
> Replicator appears to be working OK, and here's what I found.
>
> The first thing I looked at was the X-stop cable:
>

>    - Inspected both connectors and the individual wires under microscope -
>    nothing unusual.
>    - Did a discontinuity check on every pin combination, on each connector,

>    and between connectors at each cable end - normal.

>    - Did a continuity check on each conductor from cable end to end -
>    normal.
>    - Removed pins from connector shell on both ends, and inspected wire

>    ends and crimps under microscope.  The crimps were very well done; there
>    were no stray strands of wire.

>    - Removed shrink tube from both ends of cable; inspected - nothing
>    unusual.
>    - Removed shrink tube from the 'bare' wire on both ends; inspected -
>    nothing unusual.
>    - Removed insulation from the cable ~2" at a time; inspected - nothing

>    unusual.
>
> The X-stop cable did not cause this board failure.  But it is a weird cable
> construction, and is probably worthy of replacement anyway.  That bare wire
> removes the 'double insulation' between conductors that you would expect to
> find in a cable.
>
> Then I pulled the BotSteps off of the failed MightyBoard.
>

>    - Inspected each under the microscope, both sides, looking for any

>    visible signs of shorts or component failure - nothing like that was noted.

>    - Interesting things that I did see:
>       - The headers on the board are gold; headers on a new BotStep are tin.
>       - Headers have significant solder flux residue; no cleaning was done

>       after the headers were soldered on.  New BotSteps are clean.

>       - One BotStep had some white, sticky residue on the driver chip.

>        Didn't look like flux or cleaning residue.  Maybe someone blew
> their nose
>       or popped a zit.

>       - Nearly all of the board edges had been scored and removed from an

>       assembly panel.  New BotSteps are routed on the long edges and
> scored only
>       on the short edges, indicating a change in assembly panelization.  If not
>       done correctly, depanelization can damage surface mount components on a
>       board.  I didn't see anything that looked damaged (i.e., cracked
>       capacitors) on any of the BotSteps.
>
> Then I made resistance measurements of the BotStep's:
>

>    - VMot to Gnd measured 360K and 1.8M(leads reversed) on all boards.
>    - 5V to VMot measured 360K and 1.8M on all boards.
>    - 5V to Gnd on all boards measured between 1.2K and 20 Ohms.
>    - 5V to Gnd on a new BotStep measures 16.5K.

>
> So it's pretty certain that all of the BotSteps were blown when the
> MightyBoard failed.  So don't expect to have extra drivers from those
> failed MightyBoards.
>
> Comparing the failed MightyBoard to the new one:
>

>    - The old pcb was fabricated in 2011, workweek 52, the new in 2012
>    workweek 26.
>    - The old has a butt-ugly black soldermask, the new is normal green.
>    - PCB fabrication on the new board appears to be significantly higher
>    quality.
>    - The old board appears to be assembled with lead-free solder (joints

>    appear dull); the new appears to be assembled with leaded solder
>    (bright, shiny joints).  (Cheers, no RoHS for us!)

>    - The old board had cleaning residue, the new board is perfectly clean.

>
> I then inspected the failed board under the microscope, both sides, looking
> for any visible signs of shorts or component failure.  Other than the
> obviously blown 5V regulator, nothing was noted.
>
> It has been thought that a 5V to 24V short might be responsible for the
> regulator failure, so I looked carefully at areas on the board where 5V and
> 24V are in proximity.  I found all such points on the CAD data and looked
> carefully at the corresponding points on the board.  There was no evidence
> of shorting at any of those points.  I was also looking for this type of
> short when I inspected the BotSteps.  One of the points that occurred to me
> later however, wasn't on the CAD data - it's the trace cut that was made on
> the board to switch the 3.3V regulator from a 24V to a 5V input.  This
> would be a very uncontrolled potential short area, if there were any
> slivers of copper left in the cut.  The trace cuts on both the old and new
> boards appeared very clean.
>
> I then began taking resistance measurements of the failed board, while
> sequentially removing components that are connected to the 5V rail.
>

>    - 24V to Gnd measured a normal capacitor response of the 24V bulk
>    capacitor.
>    - 5V to 24V measured a normal capacitor response of the 24V bulk
>    capacitor.
>    - 5V to Gnd measured 14 Ohms.
>    - Removed the 5V regulator from the board, also removing the wire jumper

>    connection from 5V to the 3.3V regulator.  5V to Gnd measured 16 Ohms.

>    - Removed the LED driver
> (PCA9533D<http://www.nxp.com/documents/data_sheet/PCA9533.pdf>).

>     5V to Gnd measured 16 Ohms.

>    - Removed both thermocouple chips
> (MAX6675ISA<http://datasheets.maximintegrated.com/en/ds/MAX6675.pdf>).

>     5V to Gnd measured 19 Ohms.

>    - Removed X-axis digital pot
> (MCP4018<http://ww1.microchip.com/downloads/en/DeviceDoc/22147a.pdf>).

>     5V to Gnd measured 20 Ohms.

>    - Removed the ATMEGA8U2-AU <http://www.atmel.com/Images/7799s.pdf>.  5V

>    to Gnd measured 22 Ohms.

>    - Removed the ATMEGA1280 <http://www.atmel.com/Images/2549s.pdf>.  5V to
>    Gnd measured 428 Ohms.
>    - Removed Y and Z-axis digital pots.  5V to Gnd measured 2.7K.
>    - Removed Extruder A digital pot.  5V to Gnd measured 4.3K.
>    - Removed Extruder B digital pot.  5V to Gnd measured 4.7K.

>
> What does this mean?  It has long been surmised that when the regulator
> fails, it applies 24V to the 5V rail, damaging many/most/all of the
> components connected to 5V.  These values might mean very little in finding
> the original point of failure.
>
> From those resistance readings, we can work backwards with parallel
> resistance calculations to get an idea of the resistance of each chip
>

>    - Regulator: 120 Ohms
>    - LED driver: open
>    - Thermocouple IC's (both together): 100 Ohms
>    - X-axis pot: 370 Ohms
>    - ATMEGA8U2-AU <http://www.atmel.com/Images/7799s.pdf> : 170 Ohms
>    - ATMEGA1280 <http://www.atmel.com/Images/2549s.pdf> : 24 Ohms
>    - Y and Z axis pots together : 500 Ohms
>    - Extruder A pot : 7.3K
>    - Extruder B pot : 51K
>    - Rest of board : 4.7K
>    - Somebody might want to check me on this.

>
> My best guess:  The wide range of apparent resistance values for the
> digital pots makes me suspicious.  If you look at the digital pots, they
> are surrounded on two sides by the walls of the BotStep header connectors;
> on one end by the motor connector, 50% blocked on the other end by a large
> tantalum capacitor, and covered by the BotStep.  The driver chip of the
> BotStep sits directly above it, radiating heat into this nice little oven
> compartment.  There is no airflow.  I'm thinking that a digital pot fails
> with a resulting high 5V current draw.  The linear regulator, which is
> burning the current difference between 5 and 24V as heat, immediately tries
> to radiate 5x the power that the pot is over-drawing, and blows.  The
> regulator short then toasts everything else.
>
> This doesn't explain why boards fail shortly after turn-on.  Perhaps the
> pot has been damaged the last time the Replicator was used, and the pot
> current coupled with the turn-on surge blows the regulator.  Doesn't
> explain why brand new boards fail at first turn-on.
>
> To do / questions / thoughts:
>

>    - Get a thermocouple into one of those little ovens on an operating
>    Replicator.
>    - When a pot fails, what happens to its output?  If it failed so that

>    the driver went to max current, the Replicator might appear to function
>    normally.

>    - The MCP4018<http://ww1.microchip.com/downloads/en/DeviceDoc/22147a.pdf>

> digital
>    pot does have a high upper temp range: 125C

>    - Get another dead board... and do what?
>    - A failed MightyBoard is pretty much irreparable.
>    - Get a digital pot, put it on a breadboard, power it up, apply heat and
>    see what happens?
>    - I have some of the components that I removed, I can try to measure

>    their 5V to Gnd out-of-circuit resistance when I can get back to a
>    microscope to see the tiny buggers.

>    - I notice that I missed at least one device on the 5V rail:
> 74HC3G14DP<http://www.nxp.com/documents/data_sheet/74HC_HCT3G14.pdf>

>    I'll have to take a second look at the docs to see if I missed any others.
>
> Gary
> --
> ----------------------------------------------
> Gary A. Crowell Sr., P.E., CID+

> Linkedin <http://www.linkedin.com/in/garyacrowellsr>
> Elance<http://www.linkedin.com/redirect?url=http%3A%2F%2Fgaryacrowellsr%2Eel...>
>   KE7FIZ <http://www.arrl.org>

[Jetguy]

Crap, not even 5 minutes into research DUH, here's the problem:
http://www.ti.com/lit/ds/symlink/lm1084.pdf

RIGHT IN THE DATSHEET max voltage is .... drumroll...
Absolute Maximum Ratings (1)
Maximum Input to Output Voltage Differential
LM1084-5.0 25V
(1) Absolute Maximum Ratings indicate limits beyond which damage to
the device may occur. Operating Ratings indicate conditions for which
the device is intended to be functional, but specific performance is
not guaranteed. For guaranteed specifications and the test conditions,
see the Electrical Characteristics.

Hmm, maybe you think operating a regulator 1 volt below it's absolute
MAX blow it up rating would be a good design choice? This is criminal
negligence if I ever saw it.

So yes, previous fixes would solve the problem. We need to cut the
trace feeding the 5 volt regulator from the 24 Volt input. We need to
put a properly rated device such as a 12 volt step down or other
switching module in between it (the LM1084 -5.0) and the 5 volt rails.

I'll have a mod up on Thingiverse soon today.

If you own one of these machines, you are rolling the dice daily!!!

Also, heat has little to do with this. This is a voltage thing likely
making the internal regulator section fail. The device could be cold,
but if your particular power brick supplies just over 25 volts or
there is just a minor max tollerance issue, your regulator would
blow.
Sorry, I didn't catch this sooner.

And, all another good reason for MakerBot to open the source for
Rep-2. Who's to say the same idiot who did this, didn't do it again on
that system. I think we all should have a look at the engineering here
on a $2,200 machine.

Does the Makerbot engineering staff EVEN read a datasheet? I've lost
count how many times they have made this EXACT type of error where a
browse of a data sheet would show they NEVER should have even sent the
board design out for fab.

> ...
>
> read more »- Hide quoted text -
>
> - Show quoted text -

[Jetguy]

Oops, caught myself there, it's differential voltage, not absolute.
Even I can make a mistake.

So it's 5 volt+ 25 volts =30 volts max?

What made me start looking was my proposed fix of using a 12 volt in
between. The 12 volt rating is:
LM1084-12 18V max differential, or (12+18=30V) the exact same value as
the 5 volt version and the 3.3 volt shows the same math.

This means it could be the power bricks pushing the limits. They are
rated at 24 volt, but who's to say they don't age or have stability
issues?

Still, I have a strong feeling that a (24-5=19 volt) differential, on
paper is not a good idea let alone what the reality could be with the
external supply.

Everything in my first post is still true. The proposed fixes would
still be valid, but I am running some end to end tests and caluations
for safety margins.

[Jetguy]

Sorry to spam but some other failure mode thoughts.

I cut the trace feeding the LM1084 -5.0 from the 24V, and then was
double checking the polarity of the power plug before I soldered my
reverse protection diode (an FR503-TP) directly to the bottom side of
the board on top the the power input connector pins (shunt the problem
at the source and lowest resistance point), when I noticed the botstep
driver's LEDs lit up.
So just as a thought, this makes me buy even more into the reverse
polarity theory in that it might even be the botsteps backfeed up the
5 volt rail, thus blowing everything out too. In other words, it might
not even be the regulator that fails first, the botsteps could
backfeed the 5 volt rail in some fault conditions as a backdoor that I
hadn't previously suspected.

In previous bots, we always had separate stepper drivers. If you blew
one, it was less likely to kill all your electronics as the stepepr
drivers had their own 5V supply from an entirely separate source.

It might be that we divide the 5 volt into a main logic rail, and then
a stepper driver logic rail.

And thus lies the problem. This is going to be hard to pinpoint the
exact failure. Since everything is driven by 1 single 5 volt source
with no isolation, any single fault could break the system. It is a
rather new interesting point that there are 2 possible (really 5+1)
places 24 V and 5V are present on the same chip right? The MOSFETs are
an unlikely candidate as they only interface a single output pin each
on the microcontroller and are on the ground side of the heater loops.
Thus, they might blow out an output of the Mega, but not cause the
type of destruction seen the the blown boards where everything blew.

One point we could ask is those who did blow up a board, what was the
exact sequence when it happened?
I realize people might not want to admit it, but we need the data to
fix the problem.

Did you move or unlug the brick to bot power cord before the event?

If yes, was the switch on or off when you plugged it in?

Did you get immediate smoke as soon as it was plugged in, or after the
switch was turned on?

Anyway, here is a quick and dirty mod that at least provides reverse
and overcurrent protection.
http://www.thingiverse.com/thing:38045

[Jetguy]

Also, sorry for my tone. I'm just frustrated with all the tiny fixes
and details. It seems every time a look at a part on the Replicator,
i think, "well that could be done a better way". So rather than just
gripe, I am at least trying to post some mods to Thingiverse.

BTW, I swapped out my buzzer on the mightyboard with a proper one from
a Make Elelectronics kit from Radioshack. Now I have nice tones
instead of crap. http://www.makershed.com/Make_Electronics_Components_Pack_2a_p/mecp2.htm
no spec but this ◦1 Piezoelectric Beeper

Not worthy of a full mod posting up on Thingiverse?

Just saying I had the soldering iron hot so a 5 minute fix at the same
time as the rest.

> and overcurrent protection.http://www.thingiverse.com/thing:38045

[Joseph Chiu]

Gary, I only have just started to look at the Mightboard schematic (found at http://thingiverse-production.s3.amazonaws.com/assets/6d/94/65/9b/90/MakerBot_MightyBoard_REVE_Schematic.pdf) , so these are wild guesses...

1) Do you know if your friend's bot was exposed to a damp environment?  Presence of flux on the board, description of "white stuff", and your mentions of some of the power rails being in close proximity makes me think of dendritic connections between voltage domains that could cause failures.    Do you have pics of the white sticky residue?   It sounds like it could be flux redeposited onto the board during a bad wash.  If the same wash tank was used for a long period of time, it's possible that you can actually deposit flux onto the board.  Then, if a final rinse wasn't done, that flux-saturated wash could have resulted in "water spots". 

2) Description of "white stuff" also makes me think of blown caps.  Based on your measurements, my initial impression is that the caps did not go bad, but it's certainly a possibility.

3) The digital pots are unlikely to be a problem.  The 125 C max temp for the part, plus how it is wired makes it unlikely to be the source of any problems.  It only supplies Vref to the botstep's, and has a 10K resistor in series with its internal R-ladder, so even in the pessimistic case that the chip shorts the A and B terminals, you're going to be limited to 5uA through there.

4) The ATmega1280 registering the equivalent of 24 ohms seems really low.  I don't know enough about the chip's behavior with over-voltage stresses, but perhaps the protection diodes have blown short due to over-voltage.  But to get there, you'd have to have over-voltage in the first place -- that is, it may not be the initial culprit, but may have contributed to the slugging of the board.

5) The MIC2120A 3.3v LDO does not appear to have the necessary input-side capacitor (0.1 uF) paired to it in the schematic -- though it's probably not needed because there are other caps on the input side. The output-side capacitor is definitely too small.   The datasheet warns that this can cause regulation errors.  I understand that at some point, they had switched the LDO input to the 5V rail -- any chance the failed board was still getting the input side from the 24V line?  The 3.3 output is mostly unused, but it does feed the SD card.

6) Speaking of SD cards -- the SD card talks to the ATmega1280 via SPI.  Looking at this, it's a little worrisome that the 5V MCU is talking directly to the SD card which operates at 3.3 volts.   SD Cards are not required to be 5V tolerant.    Maybe we've been "getting away with it"?  Mark Cohen has mentioned that he had a failure shortly after upgrading his firmware.  I wonder if he had his SD card still in the slot during the upgrade process -- and somehow that during the upgrade, the SPI lines were held at 5V for a prolonged period of time, stressing the SD card?     A possibility [pulled out of my posterior] is that prolonged 5V at the SD Card (during the update process) was causing the protection diodes to have a path to ground, and as it heated up enough, caused the SD card to latch-up, and then after that, the ATmega1280's SPI interface was drawing excess current and enters latch-up as well.  

--
 
 

[Dan Newman]

> 3) The digital pots are unlikely to be a problem. The 125 C max temp for
> the part, plus how it is wired makes it unlikely to be the source of any
> problems. It only supplies Vref to the botstep's, and has a 10K resistor
> in series with its internal R-ladder, so even in the pessimistic case that
> the chip shorts the A and B terminals, you're going to be limited to 5uA
> through there.

I was going to check this morning to see if there was a resistor
in series. If there wasn't and the digipot failed to go to its
midpoint on power up, then there could have been a problem as the
uProcessor doesn't initialize them until after the main code starts
running (after the bootloader waits a few seconds for a possible
firmware download).

> 6) Speaking of SD cards -- the SD card talks to the ATmega1280 via SPI.
> Looking at this, it's a little worrisome that the 5V MCU is talking
> directly to the SD card which operates at 3.3 volts. SD Cards are not
> required to be 5V tolerant. Maybe we've been "getting away with it"?
> Mark Cohen has mentioned that he had a failure shortly after upgrading his
> firmware. I wonder if he had his SD card still in the slot during the
> upgrade process -- and somehow that during the upgrade, the SPI lines were
> held at 5V for a prolonged period of time, stressing the SD card? A
> possibility [pulled out of my posterior] is that prolonged 5V at the SD
> Card (during the update process) was causing the protection diodes to have
> a path to ground, and as it heated up enough, caused the SD card to
> latch-up, and then after that, the ATmega1280's SPI interface was drawing
> excess current and enters latch-up as well.

FWIW, there have been reports of the regulator blowing during a firmware
download. Happened to Jetty and at least two other people have reported
on this list that their reg. blew during a firmware download. Of course,
it could just be coincidental that it blew then and may well have blown
regardless of what the bot was doing.

Dan

[Joseph Chiu]

Well, given the uncertainty, maybe it's better to be superstitious and suggest that the SD Card be removed during firmware updates?  From my quick read of the schematics, I didn't notice any other signal paths that screamed danger to me.  Of course, I'm biased because (as you might recall), I've now had two SD cards fail to work while in my Replicator -- I do think there's a design problem there.

When I have a little bit of time, I'll put together a 3.3V clamping circuit for the SD card interface and see if that helps.

Dan

--

[Joseph Chiu]

It occured to me that the schematic I was looking at might not be telling the whole story -- as the SD card slot is not directly on the Mightyboard, but is attached to it via a ribbon cable.  So perhaps there already is a 3.3V clamp on the SD card slot board?  I don't know where the schematic for that is, and I haven't taken my bot apart to see.  Does anyone else know?

[Joseph Chiu]

Spoke too soon. Should have Google'd that first.  http://thingiverse-production.s3.amazonaws.com/assets/e4/7d/f2/c3/c8/MakerBot_Replicator_Interface_REVB_Schematics_and_Fab_Files.pdf has the schematic for the interface, which also clearly shows a SN74AHC125D bus level shifter. 

But the SD card's DO (to MISO) is not protected.  It seems highly unlikely to be a source of problems during regular operation (as the 1280-MISO line is used as an input into the ATmega) -- but if the MISO pin is driven to 5V for any reason, that could still cause trouble for the SD Card.  I really am curious whether that MISO line is being driven during upgrades.  I could have sworn reading somewhere a while back that the SD Card could get corrupted during power-on/reset.   (Oh, yeah, here it is: http://support.makerbot.com/entries/21118211-incorrect-no-sd-card-found-message - what will I ever do if Google goes away?)  

[Jesse Donaldson]

I am not a hardware guy, but I thought I'd share a couple of things I've observed with our Replicator that struck me as Very Wrong.

1) I have occasionally been shocked by touching the outside of the USB cable as I was removing it from my laptop.  This only ever occurred rarely, but the shock continued for several seconds after the cable was removed (I could touch it, get shocked, remove my finger, touch it again, get shocked again, etc...).  Apart from that, the board worked fine.  MBI said they had not heard of this problem before, and they replaced it.  I haven't seen it with the new board.  The new board looked somewhat different, and seemed to have a lower revision letter even though I'd assumed it would be newer.

2) When levelling, etc., various instructions suggest that it's easier to just drag the extruders or build platform around when the steppers are disabled, rather than "driving" them where they need to be.  When I was moving the buildplatform the other day while the bot was off, I noticed that the front panel started to light up, blinking, etc.  Kind of freaked me out... I assumed this was the motor acting as a generator and powering components that were not supposed to be powered.

Jesse

On Sun, Dec 16, 2012 at 7:45 AM, Joseph Chiu <joe...@joechiu.com> wrote:

--
 
 

[RocketSled]

I deal with the mechanisms of failure in electrical and mechanical devices for a living.  I manage the Failure Analysis team for my company. It seems like there might be a few different failure scenarios being discussed here. But that's not the case.  All the failures involve the 5V regulator.  The differences are just what the machine happened to be doing at the time of failure and the collateral damage resulting from the failure.  While that might seem to matter, it doesn't have to.  In this case I'd argue that it does not.

The LM1084 has a pretty decent MTBF rating, according to TI.  1.2x10^9 hours.  The data is a little suspicious since they say they had zero failures in a 1000 hour test with 4165 samples (TI publishes this data).  You can't really calculate a statistical reliability rating if you don't get failures, and a "point estimate" would be 4165x1000 hours, not 1.2e9.  But the part should be pretty damn reliable even at the lower point estimate number.  About a 0.2% annual failure rate assuming 4M hour MTBF.

It's important to understand that it's not actually the voltage or current that kills a part.  It's the heat that's generated when that voltage/current flows through the part that kills it. This isn't necessarily the same as how hot the part gets on the outside.  It's the heat generated at the silicon chip that matters.  That chip is small, it heats up quickly.  If large enough, even short transients can do the necessary damage to cause instantaneous failures.  But higher operating temperatures cumulatively contribute to an increased probability of failure even if there are no transient events at all.  The reliability of the part goes down the higher the temperature it's been exposed to in operation.  It's described by something called the "Arrhenius Rule" and it actually says that as heat increases linearly, reliability goes down exponentially.  

The upshot is that when a part is operated at higher temperatures, it becomes less reliable.  The higher the temperature, the more quickly it becomes unreliable (the sooner it is likely to fail).  In an already-degraded state, it could fail prematurely in response to some specific stress event like a transient load change due to a reset after a FW download, or because of the higher inrush current at turn on (because until they're charged, all the caps on the PCBA look like dead-shorts to the power supply).  But it could also just fail for no apparent reason at all.  

This is IMO almost certainly what's happening with MB 5V regulators.  The regulator is rated for a maximum input voltage of 25V.  We're operating it at 24V.  96% of max.  Rule of thumb for regulator (or any power handling component) is 80% minimum derating.  To run at 24V, the part should be rated for 30V.  The maximum voltage for decent reliability of the TI part should be 20V (25*.8).  Interestingly, 20V is the highest operating voltage shown on TI's own data sheet for any of their stated test conditions. But even at 20V, the part should still have a heat sink. It's kind of crazy that MBI chose to operate the regulator at pretty much the max rated power dissipation without a heat sink (or at least, an oversized copper pad).  It's an almost certain formula for premature failure.

I think we are probably all susceptible to premature failure of this part.  There doesn't have to be any rhyme or reason to the failure conditions.  The baseline failure rate of the part is going to be very high simply because it's being operated so close to it's maximum rating.  

Taking TI's reliability data and using the point estimate of 4.165M hours (units*test_time), I get a FIT rate of about 240.  That equates to a 0.21% annual failure rate. If I then project reliability assuming operating at 96% of maximum temperature, the part runs at around 120C and the FIT rate goes up to about 47K, which increases the annual failure rate projection to about 33%. Assuming this math is right, perhaps as much as one out of every 3 of us could have a failure for every year of accumulated power-on time.  But even if it's off by a factor of 3x, that's still 1 in 10.

Scary!

There are no drop-in solutions I can find.  No replacement regulator that will fit the site and add operating margin with a higher maximum input voltage rating.  To fix this properly, it's going to need a dead-bug solution, a different type of component, connected to the regulator 5V output and 24V input pads with soldered-on leads.  This application really needs a Buck regulator, not a Linear.  Something like the TI LMZ23605 5A Switcher with a 36V maximum input rating would work quite well, but it's not a simple 3-lead component.

[RocketSled]

Great FA by Gary, BTW!

[Jetguy]

Actually, look at the power brick and read the label. You'll notice
the 24 Volt ground pins is labeled as being tied to the power outlet
ground. It's Winter, and takes nothing for you to create a static
charge with them goes through the USB out ground shield to the
motherboard USB and then to your wall. Or heaven forbid you have a
ground loop in your house where your PC is a different ground voltage
than the outlet the bot is plugged into. Normally, they either isolate
though the power brick or there is a very high value resistor (100
Meg) such that while it's grounded, it's not a death trap.

On Dec 16, 2:39 pm, Mark Cohen <markcoh...@gmail.com> wrote:
> 1-check your electrical outlet or plug it into a ups. Usb is only 5 volts
> so you would not likely feel it. You have some other problem.
> 2-thats normal and you are correct.
> On Dec 16, 2012 2:12 PM, "Jesse Donaldson" <je...@donaldsonworkshop.com>

> wrote:
>
>
>
> > I am not a hardware guy, but I thought I'd share a couple of things I've
> > observed with our Replicator that struck me as Very Wrong.
>
> > 1) I have occasionally been shocked by touching the outside of the USB
> > cable as I was removing it from my laptop.  This only ever occurred rarely,
> > but the shock continued for several seconds after the cable was removed (I
> > could touch it, get shocked, remove my finger, touch it again, get shocked
> > again, etc...).  Apart from that, the board worked fine.  MBI said they had
> > not heard of this problem before, and they replaced it.  I haven't seen it
> > with the new board.  The new board looked somewhat different, and seemed to
> > have a lower revision letter even though I'd assumed it would be newer.
>
> > 2) When levelling, etc., various instructions suggest that it's easier to
> > just drag the extruders or build platform around when the steppers are
> > disabled, rather than "driving" them where they need to be.  When I was
> > moving the buildplatform the other day while the bot was off, I noticed
> > that the front panel started to light up, blinking, etc.  Kind of freaked
> > me out... I assumed this was the motor acting as a generator and powering
> > components that were not supposed to be powered.
>
> > Jesse
>

> > On Sun, Dec 16, 2012 at 7:45 AM, Joseph Chiu <joec...@joechiu.com> wrote:
>
> >> Gary, I only have just started to look at the Mightboard schematic (found
> >> at

> >>http://thingiverse-production.s3.amazonaws.com/assets/6d/94/65/9b/90/...)

> >> On Sat, Dec 15, 2012 at 10:39 PM, Gary Crowell <garyacrowel...@gmail.com>wrote:
>
> >>> At least a start anyway.
>
> >>> It turns out that an associate here at work had a failed MightyBoard,
> >>> and had received a replacement, but he had been too busy to swap it out.  I
> >>> offered to put the new board in his Replicator, in exchange for getting my
> >>> hands on the failed board.  Despite all my efforts, the new board in his
> >>> Replicator appears to be working OK, and here's what I found.
>
> >>> The first thing I looked at was the X-stop cable:
>

> >>>    - Inspected both connectors and the individual wires under
> >>>    microscope - nothing unusual.
> >>>    - Did a discontinuity check on every pin combination, on each

> >>>    connector, and between connectors at each cable end - normal.

> >>>    - Did a continuity check on each conductor from cable end to end -
> >>>    normal.
> >>>    - Removed pins from connector shell on both ends, and inspected wire

> >>>    ends and crimps under microscope.  The crimps were very well done; there
> >>>    were no stray strands of wire.

> >>>    - Removed shrink tube from both ends of cable; inspected - nothing
> >>>    unusual.
> >>>    - Removed shrink tube from the 'bare' wire on both ends; inspected -
> >>>    nothing unusual.
> >>>    - Removed insulation from the cable ~2" at a time; inspected -

> >>>    nothing unusual.
>
> >>> The X-stop cable did not cause this board failure.  But it is a weird
> >>> cable construction, and is probably worthy of replacement anyway.  That
> >>> bare wire removes the 'double insulation' between conductors that you would
> >>> expect to find in a cable.
>
> >>> Then I pulled the BotSteps off of the failed MightyBoard.
>

> >>>    - Inspected each under the microscope, both sides, looking for any

> >>>    visible signs of shorts or component failure - nothing like that was noted.

> >>>    - Interesting things that I did see:
> >>>       - The headers on the board are gold; headers on a new BotStep are
> >>>       tin.
> >>>       - Headers have significant solder flux residue; no cleaning was

> >>>       done after the headers were soldered on.  New BotSteps are clean.

> >>>       - One BotStep had some white, sticky residue on the driver chip.

> >>>        Didn't look like flux or cleaning residue.  Maybe someone blew their nose
> >>>       or popped a zit.

> >>>       - Nearly all of the board edges had been scored and removed from

> >>>       an assembly panel.  New BotSteps are routed on the long edges and scored
> >>>       only on the short edges, indicating a change in assembly panelization.  If
> >>>       not done correctly, depanelization can damage surface mount components on a
> >>>       board.  I didn't see anything that looked damaged (i.e., cracked
> >>>       capacitors) on any of the BotSteps.
>
> >>> Then I made resistance measurements of the BotStep's:
>

> >>>    - VMot to Gnd measured 360K and 1.8M(leads reversed) on all boards.
> >>>    - 5V to VMot measured 360K and 1.8M on all boards.
> >>>    - 5V to Gnd on all boards measured between 1.2K and 20 Ohms.
> >>>    - 5V to Gnd on a new BotStep measures 16.5K.

>
> >>> So it's pretty certain that all of the BotSteps were blown when the
> >>> MightyBoard failed.  So don't expect to have extra drivers from those
> >>> failed MightyBoards.
>
> >>> Comparing the failed MightyBoard to the new one:
>

> >>>    - The old pcb was fabricated in 2011, workweek 52, the new in 2012
> >>>    workweek 26.
> >>>    - The old has a butt-ugly black soldermask, the new is normal green.
> >>>    - PCB fabrication on the new board appears to be significantly
> >>>    higher quality.
> >>>    - The old board appears to be assembled with lead-free solder

> >>>    (joints appear dull); the new appears to be assembled with leaded solder
> >>>    (bright, shiny joints).  (Cheers, no RoHS for us!)

> >>>    - The old board had cleaning residue, the new board is perfectly

> >>>    clean.
>
> >>> I then inspected the failed board under the microscope, both sides,
> >>> looking for any visible signs of shorts or component failure.  Other than
> >>> the obviously blown 5V regulator, nothing was noted.
>
> >>> It has been thought that a 5V to 24V short might be responsible for the
> >>> regulator failure, so I looked carefully at areas on the board where 5V and
> >>> 24V are in proximity.  I found all such points on the CAD data and looked
> >>> carefully at the corresponding points on the board.  There was no evidence
> >>> of shorting at any of those points.  I was also looking for this type of
> >>> short when I inspected the BotSteps.  One of the points that occurred to me
> >>> later however, wasn't on the CAD data - it's the trace cut that was made on
> >>> the board to switch the 3.3V regulator from a 24V to a 5V input.  This
> >>> would be a very
>

[Jetguy]

"Kind of freaked
me out... I assumed this was the motor acting as a generator and
powering
components that were not supposed to be powered. "

This is normal. All steppers do this even an old Cupcake.

On Dec 16, 2:39 pm, Mark Cohen <markcoh...@gmail.com> wrote:
> 1-check your electrical outlet or plug it into a ups. Usb is only 5 volts
> so you would not likely feel it. You have some other problem.
> 2-thats normal and you are correct.
> On Dec 16, 2012 2:12 PM, "Jesse Donaldson" <je...@donaldsonworkshop.com>
> wrote:
>
>
>

> > I am not a hardware guy, but I thought I'd share a couple of things I've
> > observed with our Replicator that struck me as Very Wrong.
>
> > 1) I have occasionally been shocked by touching the outside of the USB
> > cable as I was removing it from my laptop.  This only ever occurred rarely,
> > but the shock continued for several seconds after the cable was removed (I
> > could touch it, get shocked, remove my finger, touch it again, get shocked
> > again, etc...).  Apart from that, the board worked fine.  MBI said they had
> > not heard of this problem before, and they replaced it.  I haven't seen it
> > with the new board.  The new board looked somewhat different, and seemed to
> > have a lower revision letter even though I'd assumed it would be newer.
>
> > 2) When levelling, etc., various instructions suggest that it's easier to
> > just drag the extruders or build platform around when the steppers are
> > disabled, rather than "driving" them where they need to be.  When I was
> > moving the buildplatform the other day while the bot was off, I noticed
> > that the front panel started to light up, blinking, etc.  Kind of freaked
> > me out... I assumed this was the motor acting as a generator and powering
> > components that were not supposed to be powered.
>
> > Jesse
>

> > On Sun, Dec 16, 2012 at 7:45 AM, Joseph Chiu <joec...@joechiu.com> wrote:
>
> >> Gary, I only have just started to look at the Mightboard schematic (found
> >> at

> >>http://thingiverse-production.s3.amazonaws.com/assets/6d/94/65/9b/90/...)

> >> On Sat, Dec 15, 2012 at 10:39 PM, Gary Crowell <garyacrowel...@gmail.com>wrote:
>
> >>> At least a start anyway.
>
> >>> It turns out that an associate here at work had a failed MightyBoard,
> >>> and had received a replacement, but he had been too busy to swap it out.  I
> >>> offered to put the new board in his Replicator, in exchange for getting my
> >>> hands on the failed board.  Despite all my efforts, the new board in his
> >>> Replicator appears to be working OK, and here's what I found.
>
> >>> The first thing I looked at was the X-stop cable:
>

> >>>    - Inspected both connectors and the individual wires under
> >>>    microscope - nothing unusual.
> >>>    - Did a discontinuity check on every pin combination, on each

> >>>    connector, and between connectors at each cable end - normal.

> >>>    - Did a continuity check on each conductor from cable end to end -
> >>>    normal.
> >>>    - Removed pins from connector shell on both ends, and inspected wire

> >>>    ends and crimps under microscope.  The crimps were very well done; there
> >>>    were no stray strands of wire.

> >>>    - Removed shrink tube from both ends of cable; inspected - nothing
> >>>    unusual.
> >>>    - Removed shrink tube from the 'bare' wire on both ends; inspected -
> >>>    nothing unusual.
> >>>    - Removed insulation from the cable ~2" at a time; inspected -

> >>>    nothing unusual.
>
> >>> The X-stop cable did not cause this board failure.  But it is a weird
> >>> cable construction, and is probably worthy of replacement anyway.  That
> >>> bare wire removes the 'double insulation' between conductors that you would
> >>> expect to find in a cable.
>
> >>> Then I pulled the BotSteps off of the failed MightyBoard.
>

> >>>    - Inspected each under the microscope, both sides, looking for any

> >>>    visible signs of shorts or component failure - nothing like that was noted.

> >>>    - Interesting things that I did see:
> >>>       - The headers on the board are gold; headers on a new BotStep are
> >>>       tin.
> >>>       - Headers have significant solder flux residue; no cleaning was

> >>>       done after the headers were soldered on.  New BotSteps are clean.

> >>>       - One BotStep had some white, sticky residue on the driver chip.

> >>>        Didn't look like flux or cleaning residue.  Maybe someone blew their nose
> >>>       or popped a zit.

> >>>       - Nearly all of the board edges had been scored and removed from

> >>>       an assembly panel.  New BotSteps are routed on the long edges and scored
> >>>       only on the short edges, indicating a change in assembly panelization.  If
> >>>       not done correctly, depanelization can damage surface mount components on a
> >>>       board.  I didn't see anything that looked damaged (i.e., cracked
> >>>       capacitors) on any of the BotSteps.
>
> >>> Then I made resistance measurements of the BotStep's:
>

> >>>    - VMot to Gnd measured 360K and 1.8M(leads reversed) on all boards.
> >>>    - 5V to VMot measured 360K and 1.8M on all boards.
> >>>    - 5V to Gnd on all boards measured between 1.2K and 20 Ohms.
> >>>    - 5V to Gnd on a new BotStep measures 16.5K.

>
> >>> So it's pretty certain that all of the BotSteps were blown when the
> >>> MightyBoard failed.  So don't expect to have extra drivers from those
> >>> failed MightyBoards.
>
> >>> Comparing the failed MightyBoard to the new one:
>

> >>>    - The old pcb was fabricated in 2011, workweek 52, the new in 2012
> >>>    workweek 26.
> >>>    - The old has a butt-ugly black soldermask, the new is normal green.
> >>>    - PCB fabrication on the new board appears to be significantly
> >>>    higher quality.
> >>>    - The old board appears to be assembled with lead-free solder

> >>>    (joints appear dull); the new appears to be assembled with leaded solder
> >>>    (bright, shiny joints).  (Cheers, no RoHS for us!)

> >>>    - The old board had cleaning residue, the new board is perfectly

> >>>    clean.
>
> >>> I then inspected the failed board under the microscope, both sides,
> >>> looking for any visible signs of shorts or component failure.  Other than
> >>> the obviously blown 5V regulator, nothing was noted.
>
> >>> It has been thought that a 5V to 24V short might be responsible for the
> >>> regulator failure, so I looked carefully at areas on the board where 5V and
> >>> 24V are in proximity.  I found all such points on the CAD data and looked
> >>> carefully at the corresponding points on the board.  There was no evidence
> >>> of shorting at any of those points.  I was also looking for this type of
> >>> short when I inspected the BotSteps.  One of the points that occurred to me
> >>> later however, wasn't on the CAD data - it's the trace cut that was made on
> >>> the board to switch the 3.3V regulator from a 24V to a 5V input.  This
> >>> would be a very
>

[Shawn]

In my case, it was after the platform and/or nozzles had been heated and
were cooling off. I think I may have had the power switch in the on
position for one of the boards that blew when I plugged in the brick,
but know it was in the off position for the others. I have noticed that
if the switch is on, I'll hear little sparks when plugging in the cable.
Once I noticed that, I make sure my switch is off before plugging in
the brick.

So my sequence with the replaced boards was:
- plug in the brick
- turn on
- go through the initial power up script
- In one case I cancelled the levelling routine, in another I let it
complete but didn't bother with actually levelling (I have another
method for that).
- Test the heaters
- pre heat the HBP and make sure it was heating (in one test, I
skipped this step as I suspected the problem was related to the nozzles)
- pre heat the nozzles, one at a time, to make sure I had the cables
plugged in right
- cancelled the pre-heat once I was satisfied the right thing was
heating.

At that point, I would get the POP and smoke while letting things cool off.

[Gary Crowell]

This just posted on the Pop & Smoke thread:

"My 3rd MakerBot blew today.  As I was removing a print there was a small static discharge on the build platform.  I then placed my hand on the side of the Replicator near the SD card (as I always do when I navigate using the controls).  There was another small zap.  This time the screen flickered.  The system seemed like it was off, then it would click on, then instantly off again in a loop.  Like it knew there was a short somewhere (computers do this when they detect a short).  I turned it off asap.  When I turned it on, POP!  Smoke.  The voltage regulator literally exploded.  A big chunk was missing (right / top corner)."

That really sounds like ESD induced latch-up, probably the processor.  

I'm wondering if there are several sources, things that increase the 5V current beyond the normal.  The regulator is so close to margin that anything pops it.

Gary

--
----------------------------------------------
Gary A. Crowell Sr., P.E., CID+

Linkedin  Elance  KE7FIZ

[Jesse Donaldson]

This is normal. All steppers do this even an old Cupcake.

I wasn't surprised the steppers could act way, just that other sections of the mechanism would be allowed to be powered by it.  I guess I assumed that a robust electrical design would prevent that sort of thing.  That kind of behavior is generally accepted as harmless?

Jesse

[Gary Crowell]

Comments in line.

On Sun, Dec 16, 2012 at 8:45 AM, Joseph Chiu <joe...@joechiu.com> wrote:

Gary, I only have just started to look at the Mightboard schematic (found at http://thingiverse-production.s3.amazonaws.com/assets/6d/94/65/9b/90/MakerBot_MightyBoard_REVE_Schematic.pdf) , so these are wild guesses...

1) Do you know if your friend's bot was exposed to a damp environment?  Presence of flux on the board, description of "white stuff", and your mentions of some of the power rails being in close proximity makes me think of dendritic connections between voltage domains that could cause failures.    Do you have pics of the white sticky residue?   It sounds like it could be flux redeposited onto the board during a bad wash.  If the same wash tank was used for a long period of time, it's possible that you can actually deposit flux onto the board.  Then, if a final rinse wasn't done, that flux-saturated wash could have resulted in "water spots". 

2) Description of "white stuff" also makes me think of blown caps.  Based on your measurements, my initial impression is that the caps did not go bad, but it's certainly a possibility.

Pretty dry, high desert.  Humidity here usually 20-30%.

Pic of the white stuff is attached.  The raised up part of it is where I scraped it with a knife blade.  It was soft, tacky.  Although it looks like it in the picture, it doesn't really go down the side of the chip and contact the pins (as far as I can see).

All of the caps near it are ceramic, and the big solid tant on under it on the mightyboard.  Don't see anything else that could have vented.

3) The digital pots are unlikely to be a problem.  The 125 C max temp for the part, plus how it is wired makes it unlikely to be the source of any problems.  It only supplies Vref to the botstep's, and has a 10K resistor in series with its internal R-ladder, so even in the pessimistic case that the chip shorts the A and B terminals, you're going to be limited to 5uA through there.

4) The ATmega1280 registering the equivalent of 24 ohms seems really low.  I don't know enough about the chip's behavior with over-voltage stresses, but perhaps the protection diodes have blown short due to over-voltage.  But to get there, you'd have to have over-voltage in the first place -- that is, it may not be the initial culprit, but may have contributed to the slugging of the board.

Note the item just posted that looks like an ESD latch-up of the processor.

 

5) The MIC2120A 3.3v LDO does not appear to have the necessary input-side capacitor (0.1 uF) paired to it in the schematic -- though it's probably not needed because there are other caps on the input side. The output-side capacitor is definitely too small.   The datasheet warns that this can cause regulation errors.  I understand that at some point, they had switched the LDO input to the 5V rail -- any chance the failed board was still getting the input side from the 24V line?  The 3.3 output is mostly unused, but it does feed the SD card.

That was the trace cut that I checked pretty closely.  Looked like a really clean cut on both the old and new boards.

 

6) Speaking of SD cards -- the SD card talks to the ATmega1280 via SPI.  Looking at this, it's a little worrisome that the 5V MCU is talking directly to the SD card which operates at 3.3 volts.   SD Cards are not required to be 5V tolerant.    Maybe we've been "getting away with it"?  Mark Cohen has mentioned that he had a failure shortly after upgrading his firmware.  I wonder if he had his SD card still in the slot during the upgrade process -- and somehow that during the upgrade, the SPI lines were held at 5V for a prolonged period of time, stressing the SD card?     A possibility [pulled out of my posterior] is that prolonged 5V at the SD Card (during the update process) was causing the protection diodes to have a path to ground, and as it heated up enough, caused the SD card to latch-up, and then after that, the ATmega1280's SPI interface was drawing excess current and enters latch-up as well.  

Gary 

[Gary Crowell]

Well, that is another thought.  What you see on the display is from the motors to the 24V line and through the 3.3V regulator.  Presumably it would be applying some power via the 5V regulator to the rest of the board as well.  I wouldn't think it would cause a problem on 5 or 3.3V circuits as that power is effectively 'regulated', and not likely to bother the drivers as they expect that kind of treatment.  Wonder what the magnitude of that voltage is?  Could that be spiking the max on the 5V regulator?

A diode to isolate the motor 24V probably wouldn't be a bad idea.

Gary

Jesse

--
 
 

[Gary Crowell]

Comments in line:

On Sun, Dec 16, 2012 at 4:24 AM, Jetguy <barry...@hotmail.com> wrote:

Thanks for the thorough analysis!!!!!

Just spitballing some ideas here:
I believe(and totally could be wrong here) that the failure comes
either from overload of current from some device on the 5 volt rail,
or it's the fact such a large differential between the 24 volts coming
in being reduced to 5 volts.
Another remote option is the power supply producing a voltage spike or
slow over voltage that changed over time that exceeds the regulators
specs.

Finally, and having a machine in my hands helped me see this, the
round poswer supply input connector can be jammed in inverted thus
supplying reverse 24 volts to the board and an obvious blowout the
instant that switch is flipped. I know the plug is polarized, BUT, it
appears it can make contact in reverse, especially if the switch is on
but not be plugged in past the "key".

Aggravating this is that the connector is 'upside-down' from what people world normally expect; i.e., the flat side of the connector is usually 'up'.  Hadn't thought about it but that makes a good case for this thing.  Makes it impossible to start the connector wrong and adds good support to the connector.

 

I see a VERY likely scenario where the bot was moved, the switch was
on and the owner was plugging the cable into the back and had it
rotated. It makes contact reverse polarity and blows the board sky
high.

We need layers of protection  here.
First, put a 12 volt regulator between the 24 volts and the 5 volts.
This adds another layer and some slighty higher input power
tollerance.
Next, the resetable circuit breakers rated just above monral current
levels but far belwo the max rating of the regulators.
Reverse input diode and a matching fuse. This way, if you plug it in
backwards, the diode clamps the current and blows the fuse.

In theory, if we did the above layered aproach, we might prevent some
failures.

One final failsafe would be a massive Zener diode on the 5 volt rail
fed by an appropriate sized fuse. This way, if all the above failed,
we still limit voltage to the 5 volt rail and block reverse current
there too. This idea comes from a fix over in the DIY Drones group who
kept blowing out their expensive boards for the same overvoltage and 

reverse voltage reasons.

The Replicator 2 board uses a switcher for the 5V regulator, and there is a fuse - not sure what it's in line with though.

 

Gary

[Gary Crowell]

On Sun, Dec 16, 2012 at 7:15 AM, Jetguy <barry...@hotmail.com> wrote:

Also, sorry for my tone. I'm just frustrated with all the tiny fixes
and details.  It seems every time a look at a part on the Replicator,
i think, "well that could be done a better way". So rather than just
gripe, I am at least trying to post some mods to Thingiverse.

BTW, I swapped out my buzzer on the mightyboard with a proper one from
a Make Elelectronics kit from Radioshack. Now I have nice tones
instead of crap. http://www.makershed.com/Make_Electronics_Components_Pack_2a_p/mecp2.htm
no spec but this ◦1 Piezoelectric Beeper

Not worthy of a full mod posting up on Thingiverse?

Yes, it is.  With an explanation of why it's a neet thing to do.  I've got one in my cart for my next Digi-Key order.

Gary

[Gary Crowell]

On Sun, Dec 16, 2012 at 10:53 AM, Joseph Chiu <joe...@joechiu.com> wrote:

Spoke too soon. Should have Google'd that first.  http://thingiverse-production.s3.amazonaws.com/assets/e4/7d/f2/c3/c8/MakerBot_Replicator_Interface_REVB_Schematics_and_Fab_Files.pdf has the schematic for the interface, which also clearly shows a SN74AHC125D bus level shifter. 

But the SD card's DO (to MISO) is not protected.  It seems highly unlikely to be a source of problems during regular operation (as the 1280-MISO line is used as an input into the ATmega) -- but if the MISO pin is driven to 5V for any reason, that could still cause trouble for the SD Card.  I really am curious whether that MISO line is being driven during upgrades.  I could have sworn reading somewhere a while back that the SD Card could get corrupted during power-on/reset.   (Oh, yeah, here it is: http://support.makerbot.com/entries/21118211-incorrect-no-sd-card-found-message - what will I ever do if Google goes away?)  

Note the recent post of a blow-up that looks like an ESD touch on the SD card. 

[Gary Crowell]

On Sun, Dec 16, 2012 at 8:45 AM, Joseph Chiu <joe...@joechiu.com> wrote:

3) The digital pots are unlikely to be a problem.  The 125 C max temp for the part, plus how it is wired makes it unlikely to be the source of any problems.  It only supplies Vref to the botstep's, and has a 10K resistor in series with its internal R-ladder, so even in the pessimistic case that the chip shorts the A and B terminals, you're going to be limited to 5uA through there.

 

Yeah, I had this great theory, thinking that it was probably 85C.  Then I saw that 125.  Still want to get a thermocouple in there though.

Gary

 

[Gary Crowell]

Thanks to everyone; I thought that post would start some ideas cooking.  Was tied up today and couldn't get back to comment earlier, but I think I'm caught up now.

I didn't get back to work to check the resistances of the removed components, but I can do that in the morning.

Gary

--

[Gary Crowell]

You know, looking at the picture of the residue on the driver that I attached earlier, I see minimal solder and likely solder balls on the capacitor pad to the right of the driver, and excess solder on the capacitor to the left of the driver.  Not the best assembly.

[bagelturf]

A simple way to lower the dissipation in the regulator is to dump energy elsewhere. If the maximum 24V current draw into the regulator is known, then it's easy to calculate the value and power rating of a series resistor inserted between the 24V supply and the regulator to drop a specific voltage. For example, 2A max and a 10V drop needs 5 Ohms and will dissipate 20W max.

A resistor can be mounted some distance from the regulator, and should be relatively easy to wire in.

[Gary Crowell]

I like that idea...  need to get some current measurements.  I don't think there would be too great a variation in the 5V draw to come up with a good value.  The greatest variable is probably the processor.

Gary

--
 
 

[Gary Crowell]

Hey guys...

We need a Thing to mitigate the Replicator ESD danger.  I think the first/easiest thing to do is to strongly recommend this Keyboard Cover.  There are pins exposed on the back of this board that lead directly to the processor, and are very easy to touch.

Next, it appears that the recently described ESD event came from a discharge to the SD card/socket.  I checked, the socket shield is grounded to the board's ground plane, and the board ground connects to the MightyBoard via three conductors of the ribbon cable.  I suspect the inductance of that ground path is too much to prevent an ESD discharge that jumps to a signal pin.  What do you think about a square of adhesive metal foil, placed on the frame around the SD slot, edges of the slot cutout area folded into the slot, with a wire/braid/strip of ribbon cable soldered to it and running back to a ground point on the MightyBoard near the power input?  The idea being to catch a discharge there first.  That would just require some simple soldering; I didn't think most people would want to try to solder ESD protection devices to the board(s).

What think?  What else?

Gary

[Dan Newman]

On 17 Dec 2012 , at 10:13 AM, Gary Crowell wrote:

> Hey guys...
>
> We need a Thing to mitigate the Replicator ESD danger. I think the
> first/easiest thing to do is to strongly recommend this Keyboard

> Cover<http://www.thingiverse.com/thing:29858>.

> There are pins exposed on the back of this board that lead directly to the
> processor, and are very easy to touch.
>
> Next, it appears that the recently described ESD event came from a
> discharge to the SD card/socket. I checked, the socket shield is grounded
> to the board's ground plane, and the board ground connects to the
> MightyBoard via three conductors of the ribbon cable. I suspect the
> inductance of that ground path is too much to prevent an ESD discharge that
> jumps to a signal pin. What do you think about a square of adhesive metal
> foil, placed on the frame around the SD slot, edges of the slot cutout area
> folded into the slot, with a wire/braid/strip of ribbon cable soldered to
> it and running back to a ground point on the MightyBoard near the power
> input? The idea being to catch a discharge there first. That would just
> require some simple soldering; I didn't think most people would want to try
> to solder ESD protection devices to the board(s).

Probably not a bad idea although not something I have any expertise in myself.

> What think? What else?

Plenty of folks with ToMs and Cupcakes have run heavy ground wires from
various points of known static build up to the ATX PSU's safety ground. For
example, the metal clamping plate for the extruder as well as a the aluminum
build plate.

On a Rep 1 this is readily done with the Al build plate, but harder to do
with the extruders. The "easiest" way would be if the that long aluminum
block into which the extruder throats thread had a hole tapped on the side
to which a screw might be affixed with a ground wire.

Dan

[Gary Crowell]

And gosh, I forgot about the exposed signal pins on the limit switches.  Kapton tape wrap?

Gary

Dan

--

--

[Dan Newman]

On 17 Dec 2012 , at 10:35 AM, Gary Crowell wrote:

> And gosh, I forgot about the exposed signal pins on the limit switches.
> Kapton tape wrap?

Or a conformal coating with decent electrical insulation properties (e.g.,
some of the brush on acrylic coatings). Not as handy as Kapton tape
of course, but for those of us with the stuff around….

Dan

[BTHOON]

This wouldn't work?
http://www.newark.com/murata-power-solutions/78srh-5-2-c/ic-sync-switching-regulator-to/dp/40M6588
or this:
http://www.mouser.com/ProductDetail/Murata-Power-Solutions/7805SRH-C/?qs=sGAEpiMZZMukgiigmf73gLCRViTH0sTv

[Gary Crowell]

Yup, either of those would work, and look like good choices.  Only difficulty, not great, is that they aren't drop in.  I think the pinout is different, and the space is limited - they'd have to be deadbuged above the board somehow.

Gary

--
 
 

[Gary Crowell]

Like this: http://www.alliedelec.com/images/products/datasheets/bm/MG_CHEMICALS/70125821.pdf

Not too expensive, $10 for a 2oz bottle, here: http://www.alliedelec.com/search/productdetail.aspx?SKU=70125821

Gary

Dan

--

[BTHOON]

Alternately, and I know this is silly to some, the headers on the regulators could be desoldered and replaced with reasonably thick (but not so thick it wouldn't pass through the holes) wire, and after cutting down to roughly the same height, the wires could be swapped.  

All of this aside, I'm trying to think if this is a reasonable change to make before a failure happens.  Given the number of failure points on the board, I would just worry about MBI replacing the MB if something else went wrong if there were mods of this variety.

[Jetguy]

"I wasn't surprised the steppers could act way, just that other
sections of
the mechanism would be allowed to be powered by it. I guess I assumed
that
a robust electrical design would prevent that sort of thing. That
kind of
behavior is generally accepted as harmless?"

As I stated before all electronic versions from MBI have done this
since the beginning. Why? Because they all use the Allegro brand
stepper driver chips. The same chips used by everyone else. That
design shunts power from the motor as active braking. In a large CNC
machine, this is a desireable safety feature. This is because say the
machine is running and the power is either cut by accident, or by
human safety shutoff. Rather than letting the machine wind down
forever from inertia, they want active braking. So that is designed
into the chip. Well, the tiny chip alone cannot disipate the energy
so the system was designed to pass it up the power supply rails where
a larger system could disipate the energy.

On big CNCs, they use stovetop heater coils. Basically, there is a set
of diodes and FETs that direct backflow voltage heading towards the
power supply into those heaters.

So, yes, this is generally accepted and common practice in the CNC
community. It is generally fine in our application. Unless you re-
engineer the entire system and use different stepper driver chips,
then not easily eliminated. We could build the power supply circuits
to have a heater resistor system but a bit of overkill sense we
shouldn't normally have a great deal of inertia or motion to deal
with. It requires an entire re-design to implement and isolate the
stepper drivers from everything else.

Some solid related info : http://onehossshay.wordpress.com/2011/08/21/designing-a-diy-cnc-for-a-sherline-mill/

"In choosing a power supply, the supply voltage should be just under
the maximum voltage the drivers allow to allow for a little back EMF
buffer. (Sorry ATX PC power supplies will not work here!) The Pololu
stepper drivers have a maximum motor voltage of 32V and require
heatsinks for currents over 1A per coil, but you don’t need much of
one, if you drive the motors at the highest voltage possible. These
drivers recycle the energy already present in the motor coils and tend
to be more efficient with higher voltages. This is due to the high
voltages forcing the energy out of the motor windings faster and ends
up with less waste heat generated. Personally, I did see a huge
difference in temperature between 24V to 30V. At 24V, the heatsinked
drivers overheated within a minute at 1.5A per coil and tripped the
internal thermal protection, and when at 30V, the driver heatsinks
were hot, not searing, and didn’t require a fan."

http://www.cncdrive.com/downloads/BRKC_XXX_manual.pdf
4. Theory of operation
4.1. Operation detailed
All type of motors works with electricity, but they may be used in
reverse operation mode, when the
motor shaft is rotated with an external force the motor acts as a
generator and generating energy and
Voltage.
With using electric motor drives and when the motor is braking the
same thing happens and extra
Votlage is generated by the motor. This extra energy seen as a Voltage
and is charging back to the
power supply. Beyond a power level the power supply and it’s buffer
capacitors may be unable to
handle and buffer this extra Voltage and the Voltage in the power
supply may start to rise. The rising of
the Votlage could be infinate in theory and too much Voltage rising
may damage both the power supply
and the motor drive when Voltage rises above the Voltage rating of the
components inside these
devices.
The braking resistor is to discharge and dissipate this extra Voltage
and energy and to protect the motor
controllers and the power supply.
The working of the braking circuit is simple, it measures the Voltage
of the power supply and if it rises
above a set point the circuit triggers a Mosfet transistor with a
power resistor conected.
The resistor shunts the power supply rails and dissipates the extra
Voltage.
When the Voltage drops below the set point the Mosfet and of the
braking circuit closes removing the
power resistor from the power supply rails.

On Dec 16, 11:18 pm, Jesse Donaldson <je...@donaldsonworkshop.com>
wrote:

[Gary Crowell]

Current measurement.  Cutting the 24V trace into the regulator isn't feasible, lifting the pin would do.  Directly measuring the 5V would be more difficult, lifting the pin wouldn't suffice, as a second 5V trace comes off the tab area; it would require cutting a trace, lifting the pin or cutting another trace, and dealing with the 5V-3.3V jumper.  Then measuring at a couple of different places.  Not difficult, but more chopping than I wanted to do on a functional board.  Getting the 24Vin current should be sufficient anyway.

Ballpark some numbers here.  The 5V current draw couldn't be more than about 200ma, 'cause that would dissipate 3.8W in the regulator, and it couldn't take much more than that with just the limited board pad it has.  So say 200ma as an upper figure.  To drop 12V at that would take 60 Ohms, and the resistor would dissipate 2.4W; so you could probably use a 5W resistor if there was airflow around it.  The dissipation in the regulator would drop to 1.9W.

If the board only draws 100ma, that's 1.9W normally in the regulator.  Adding a 60 Ohm series resistor would drop 6V, dissipating 0.6W in the resistor, and dropping the regulator dissipation to 1.3W.  I suspect the current draw is somewhere between those values, so 60 Ohms might be a good starting point.  

Bit of an implementation problem though, just lifting the input leg of the regulator to insert the resistor, would isolate the input from the 24V input bulk capacitor.  The way the traces go, you'd probably need to add another capacitor deadbugged at the input leg.

Gary

[Jetguy]

Sorry, but a 20 watt heater is a dumb idea and an accident waiting to
happen. I mean the cartridge heater is 40 watts, a soldering iron is
typically 10-30 watts. How do you plan on safely disipating 20 watts?

> > all.  - Hide quoted text -

[Jetguy]

The greatest 5 volt draw is the LCD backlight , just like what I see
on Gen 4 with an LCD control panel.
I know this because for my bots, I'm not using a PC power supply, but
rather a 12 volt, 30 amp rated industrial power supply.
In order to power the gen 4 motherboard which requires 12, 5, and 3.3
volts, I use a perfboard from radioshack and a 7805 1 amp rated linear
regulator for the 5volt rail and a LD33V 3.3 volt regulator, both from
Sparkfun.
If I don't have the LCD plugged in, then the regulator doesn't even
need a heatsink or airflow from a fan at 12 volt. with the LCD panel
plugged in, it must have a heastink and either be in open air, or if
enclosed, some forced air. The 3.3 volt never ever even gets warm.

So what is different about gen4 VS Mightyboard.

The stepper drivers each has their own 7805 powering the logic side of
the Allegro stepper drivers. In my experience, those never get warm
even when driving the stepper driver at 24 volts (an experiment back
before the motor change on the T-O-M).

The extruder controller has a 7805 to power all it's logic too and
never gets hot on gen4. That means it powers the MAX 6675, the FTDI
chip, the 328P and whatever else.

So, on a mightyboard, we have 2 MAX 6675s, the Mega 1280, the LCD and
backlight, 5 each logic side of the stepper drivers, the digipots, the
32u2, and some other minor logic.
They use a single 5 amp rated regulator to power all that. In my
experience with gen 4, I can power the motherboard easily with a 7805
and not blow it sky high. The stepper drivers and everything else
don't seem to draw anything significant ever, but each has their own
7805 on gen4.

Further, as an interesting test to the mod I posted earlier, I took a
7812 and put it in series before the 5 volt regulator. It's only rated
at 1 amp MAX, and then at the 24 volts, the current rating goes down
due to the heat that must be disipated. I ran it for a few seconds
without any heatsink and let the board idle for a few minutes this
way. I never tried to print but the regulator didn't blow up instantly
either. So what this tells me is standard current draw is likely under
1 amp on the 5 volt circuit. I see no reason why printing would ever
increase this much. I removed the 12 volt regulator because it was
probably not a good fix to the problem and would require better
heatsinking. What this doesn't explain is how can we overload a 5 amp
rated regulator? The only real options are a major fault in another
component that dead shorts the bus. Items that are intial candidates
are the Botsteps, The MAX 6675, or the LCD. Other mechanical shorts
could happen with the endstops. Somehere I posted that there is NO
need for 5 volts at the endstops. It's a feature that powered the LED
on them to show when tripped to the user, but no other purpose. Simply
remove the white wire from the endstop socket at the motherboard.
People have measures the wires and said it's not bad cables. I say it
could easily short though at the endstop itself, either by the way
they are mounted with the square nuts on top opf the board, or the
exposed terminals.

Finally, one last though that is different between gen4 and
Mightyboard is the RGB LED controller. I haven't dug into it and it
has 24 volts present. I have a very bad suspicion that something
backfeeds the entire 5 volt bus with 24 volt current causing the
blowouts, most of the scenarios presented didn't seem like overcurrent
situations on the 5 volt rail.I actually think there is a fair chance
normal current on the 5 volt rail is under 1 amp making the 5 Amp
regulator over capacity and should never fail.

The static failures and the firmware update failures seem to point to
a device that bridges the the 24 and 5 volt rails. A spurios signal or
whatever seems to start the reaction but I'm having a hard time with
that 5 Amp regulator being damaged from overcurrent. Over voltage
seems to be the issue. Thus, again, I feel we either have a backfeed
situation by some component feeding the 5 volt bus with higher
voltage, or the fact I pointed out earlier, the 5 volt regulator is
see a high differential voltage and failing on the input side.

The million doller question is, a proper workaround that is cheap, and
easily done by a novice. I think a properly heatsinked 7805 can power
the 5 volt rail but would require good heatsinking and is near the max
limits at 24 volt input. Maybe though, it might be less static and
spike sensitive and not fail in the same way. Alternatively, 2 7805s
could load share by one powering and isolating the Botsteps, and the
other powers the rest of the 5 volt logic. If we still had failures,
at least isolation would determine where in the ciruit we have a
problem. Then comes the issue that we need mass testing of the mod.
Since we don't know the exact fault, we cannot reproduce it, and this
we need a large installed base to have the random accidents happen.

> Linkedin <http://www.linkedin.com/in/garyacrowellsr>
> Elance<http://www.linkedin.com/redirect?url=http%3A%2F%2Fgaryacrowellsr%2Eel...>
>   KE7FIZ <http://www.arrl.org>- Hide quoted text -

[Dan Newman]

On 17 Dec 2012 , at 11:12 AM, BTHOON wrote:

> Alternately, and I know this is silly to some, the headers on the
> regulators could be desoldered and replaced with reasonably thick (but not
> so thick it wouldn't pass through the holes) wire, and after cutting down
> to roughly the same height, the wires could be swapped.

You can just use point-to-point wiring and just cut the outer two legs of
the 5V reg (ground and Vin).

1. Install a DC/DC switching converter
1a. Wire ground to a ground pad for the PSU inlet to the board
1b. Wire the Vin (24V) to the switched 24V leg of the on/off switch
1c. Wire Vout (5V) to the jumper wire between the old 5V reg and the 3.3V reg
2. Mount the DC/DC switching converter to the inside of the chassis

Notes

A. No point in cutting the 5V leg on the old 5V reg as the tab is also
Vout on that reg. and it's soldered into the circuit as well.
B. The DC/DC switching converter requires a 100 uF cap across its Vout
and ground. The existing 100 uF cap which serves this same purpose
for the old 5V reg will serve: just leave it in the circuit.

Dan

[Jetguy]

Dan, While I agree we can replace the regulator, but the more we look
at this, what exactly does that gain us?

We have decent data showing the 5 volt load shouldn't be an Amp on a
bad day. The current regulator is rated at 5 Amps. A casual test shows
me thermal failure is a highly unlikely culprit. The surrounding board
isn't changing color over time, the regulator has never been even arm
when I touched it.

Either, we are chaning it because we believe it somehow doesn't meet
the specs, or, we assume another regulator can better handle the
fault.

So some more poking around at suspect areas, I say the LED driver
setup seems safe to count out, but a new prime candidate emerged and
that's the disabled in a very hacked way safety relay system. BTW it
also has 24 and 5 volt interfaces.
Hmm, some logic in front, a rigged solder patch job to disable,
instant failures during a firmware upgrade which might throw some
random bits around. This seems the most likely candidate and would
surely explain the failures during an update.

Further, just as a historical point. They sent these out originally
with the cutoffs enabled. They had some boards fail, so they came back
with the solder blob hack to disable. I just took close up pictures of
mine showing that the solder mask on extruder 1 (the one seeing the
most use, it damaged, a little copper is showing and the solder blob
for a bridge looks a little tarnished. Going back to the schematic,
they should be bridging all 3 pins on the relay output to fully high
current disable it but the stock fix only shorts 2 pins. Depending on
who did the QC, yours may look worse than mine. I know in the circuit
the SSRs are connected to the 5 volts through 4.7k, but the big
question is, if we put 24 volts (AKA 19 Volt differential, does that
in turn raise the voltage enough to blow the regulator and catastrophy
happens?

Maybe it's just me but if they are disabled, just remove the entire
chip, bridge all 3 pins on the output side and be done? No chance of
bleeding 24V into the 5 volt, one less possible failure point removed.

[Gary Crowell]

5V

• A4982  8ma x 5  ~ 40ma
• driver led's on 24V
• ATMEGA1280 ~ 50ma
• ATMEGA8U2 ~ 20ma
• MCP4018 0.08ma x 5 ~ 0.5ma
• PCA9533  <1ma
• 74HCT3G14  0.06ma
• fet led's on 24V
• digital pot divider ~0.1ma x 5 ~ 0.5ma
• 5V led  3ma
• status led's 3ma x 4  ~ 12ma
• tx/rx led's  3ma x 2  ~  6ma
• limit sw led's  3ma x 3  ~ 9ma
• keybd led's  3ma x 3  ~ 6ma
• NC7WZ02  0.01ma
• AVQ25  1ma x 2  ~ 2ma
• 6675  1.5ma x 2  ~ 3ma

<155ma  +assorted pull-ups, dividers, signal currents, and other stuff I missed.

• LCD   ??

3.3V

• HEF4090  0.15ma
• 75AHC125  0.04ma
• 3.3V led  3ma
• SD card

<~5ma

All of those values are pretty much maximums, and some guessing, leaning towards the high side.  Of course the 3.3V current is going to be added to the 5V.  I found data on the the LCD sometime in the past, but don't see it now.  Anyway, just picking a similar LCD on sparkfun, it runs 150ma with the backlight.

So just taking a SWAG at it I think the normal maximum could be as high as 300ma.  It might run as much as 20-25% less than that, I'd be very surprised if it was under 200ma.  

Measuring it would really be better.

OK, WOW.  I just did a quick calculation on the junction temperature of the 5V regulator.  Again, there's a lot of SWAG here, but at 200ma load, the first number I come up with is 187C.  The device limits in the spec?  150C.   This figure is dominated by the power dissipation, of course, but also by the heatsink.  In this case the heatsink is the copper areas on the top and bottom of the board that the regulator is soldered to.  I'm not looking at it at the moment, but I don't recall those areas being much larger than about an inch square.  So if its 25x25mm, top and bottom, that's a total of 1250mm sq.  On the handy chart included in the spec that's a thermal resistance of 40C/W for heatsink-to-ambient.  We can use zero for case-to-heatsink, 'cause it's soldered on.  And from the spec, the junction-to-case for that package is 2.7C/W.  So it's just 42.7C/W times the 3.8W dissipation, plus 25C for ambient = 187C.  But that chart is probably for convection cooling only (it doesn't say), and there is forced air over the regulator area, but not much really.  Need to find a better figure for the heatsink.

'course I think we already knew it was overstressed.

Gary

[Dan Newman]

On 17 Dec 2012 , at 3:39 PM, Jetguy wrote:

> Dan, While I agree we can replace the regulator, but the more we look
> at this, what exactly does that gain us?

Wasn't suggesting that it be replaced. Rather, I was commenting on
how to "dead bug" it to the person concerned about mounting.

I'm with you -- and have previously posted this -- that I think there's
an issue with 24V spilling onto the 5V rail. At least from my experience,
that's a sure fire way to blow those linear regulators with a loud pop.
Shorts to ground it has built in overcurrent protection for.

And, I'll be the first to admit, I have far, far less experience with
what happens when these are run overheated for too long.

Dan

[Joseph Chiu]

These devices are supposed to have thermal protection, but more for sudden upsets, not prolonged continuous high draw.  I agree with your estimate -- when I did a mental calculation, I came up with a SWAG around 200 ma's.  Still, with a regulator dropping 20 Volts at 200 mA, that's still about 4 W of heat dissipation.  Definitely better served by having a heatsink or active cooling.

Dan

--

[Jetguy]

I'm glad we are doing the math and thermal calcs. It's all good work
everyone is posting.

I just try to temper that with experience, and I've seen tons of
boards run hot with components that lasted for years overheated. The
point is, the board, solderjoints and surrounding parts always show
witness to this kind of abuse. The black solder mask makes it harder
as a green or red board would have a very easy to see color change if
that was the case. Hoever, it's glossy black and that would make it
easy to spot on the back side of the board. That's not happening on
any report I have heard, nor on the board in front of me.

Going over the schematic with a fine tooth comb, and the board and
bright light an lens in front of me, I must say, the most suspect area
of the board is the safety cutoff which has been disabled in a very
poor way. I just desoldered both SSRs and U5 for good measure. Which,
feel free to discuss, make me question it even more to be the SSR
problem. While in my previous post, I quoted the 4.7K pullups, but
actually, duh, the U5 NOR latch is DIRECTLY connected to the 5 volt
rail. IF the SSR fails internally then we feed 24 Volts into the U5,
blowing it up and sendign 24 volts up the 5 volt rail right near the
regulators anyway. Again, My logic here is they shorted the output of
the SSR, didn't even really do that right for current handling, and
then the SSR has a pretty direct path on failure to blow the board sky
high. Removing them is no big deal, but removes a huge area of risk.

Further, let's read the data sheet and we find out answer :
http://www.mouser.com/ProductDetail/Panasonic-Electric-Works/AQV252G/?qs=sGAEpiMZZMsuBfEaN9EhVe3oVbOumh5%252b
Rated current is 2.5 Amps
I measured a heater and it was 16.6 Ohms on my meter. I won't try to
act like that a calibrated measurement, but the math works out. It
would conduct about 1.4 Amps, and times 24 Volts we get 33.6 watts for
a 40 watt rated heater.

Actual measured current draw cold on mine was 1.56 Amps (again, not
claiming that's perfect), but a solid 37.5 watts cold.

So the question becomes why the solder mod in the first place? If they
removed it because of the thermal devices tripping prematurely
(happens on the MK7 due to the poor design based on the thermal
insulation resisting the heat), why not just put a jumper wire across
the screw terminals instead of the safety cutoff. Zero chance of a
screwup and no modifications to the board. But, ther didn't do that
easy fix and decided to short the output of the SSR?

The SSR data sheet says it's an LED and thus opto isolated to a very
high voltage. Thus one would be led to believe even in failure, it
shouldn't short to the LED side. But, I cannot ignore it's another
device connected to the 24 volt + rail. It just seems fishy all the
way around. I mean the ratings look right, but there were initial
failures, then the solder hacks by MBI, and then we still have
failures. I say if it's not in use, get it out of there, but I'm at a
loss here? This makes my head hurt.

[Joey]

Gary,
  Nice job on the thermal calcs. I double checked your numbers and the only change I have is the heat sink area. I measured the area from the brd file and got .742 in sq. (.371 x 2 sides) or about 480 sq mm.  That gives a thermal resistance of about 53 C/W. I measured the case temp with an IR thermometer and it was only about 100F. Confirmed this using my finger, just slightly warm.

I'm guessing the current is quite a bit lower than we think. Would be interesting to try this while it's printing, since the AVR is probably in idle mode most of the time unless it's printing. From what I here the AVR is pretty much maxed out while printing which would up the power draw significantly. Only problem is it's hard to run a print with the bot sitting on it's side so I can measure the regulator temp!

I have a harbor freight meter with a thermocouple probe on it. If I get a chance I'll try to take some readings with the bot printing.

Joey

[Gary Crowell]

Way cool info.  The 3.3V reg is still on the dead board I have, and it will be easy to test - if I had brought it to work this morning.  Your experience would explain the repetitive failures we've heard about.  A lot of people are apparently getting by without replacing the interface board, so I expect there are multiple ESD damage paths, even just from the SD card, i.e., direct to the processor.  Probably other failure modes too.

Thanks,

Gary

On Mon, Dec 17, 2012 at 7:16 PM, HelpingHands <concern...@me.com> wrote:

A friend of mine linked me to this thread while we were chatting about our debugging experiences with our Makerbots, maybe you folks would be interested.
I did some very basic discovery work I made when I first received (and then blew up, repeatedly) my Rep-1's mightyboard. When I first used the machine, it was very much a plug-and-play experience. The software (RepG something or other) was a pain to use, especially for the dual-color printing stuff, but I got used to it. I have some experience with other hobby 3D printers, and it was not hard to learn the Replicator.
Eventually though, I touched the SD card and shocked it with static electricity. I was in the process of unplugging the card, and when I put it in my laptop it was not readable. I returned it to the printer and the LCD immediately turned off. It blinked on again, showed two solid black bars, then went off. This is a symptom of the PSU shutting off to avoid an over-draw situation, and typically indicates a short (I replicated this later, with a screwdriver jammed in the PSU prongs while waiting for MakerBot support to send me a new board. Living on the edge, but I can afford to replace the PSU if I need to, and curiosity had the better of me). After a few blinks the mightyboard's regulator detonated, and the machine was effectively dead. The 24v rail was actually still fine, the mainboard fan was working, and the LEDs under the stepper boards were on. Everything on the 5v rail was totally dead. I chopped the traces from various components to the 5v rail to see just how dead, and found that pretty much everything was shorted one way or another. That seems like the work of a really dramatic over-voltage event, or perhaps just over-current on the same bus. In any case, it seems that one thing dying can kill everything else.
I checked the resistance of the SD card, and found that it was indeed shorted, as was the interface board! It seems that the 5v to 3.3v logic converter has failed, and dumped 5v into the SD card and had over-drawn the 5v regulator -- probably leading to it overheating rapidly and then exploding. I note that on the jumper-hacked boards, the 3.3v regulator is fed from the 5v regulator, rather than the 24v bus that it was originally designed to feed from. I wonder if they saw something like this happen in-house right before or right after they launched the machine. Makerbot, to their credit, happily replaced the mightyboard and SD card. They assured me it was not going to happen frequently, but had some general advice.
It happened immediately! I wanted to see if this was the case, so I plugged the new mightyboard in, leaving everything else as it was (but I threw away the SD card). Indeed, with the damaged interface, the board did the sad little rebooting loop, and then exploded. I should have taken a video of this, but alas, I was not sure what would happen, and it was late at night.
Makerbot again replaced the motherboard, as well as the interface. I now ground myself before touching anything on this machine.

There's an even happy ending to this though -- the Rep2 my company purchased (one of a few I pushed for purchasing) has a totally different interface board, with 3.3v supplied for the SD card right onboard. There is also some serious rework to the power distribution, a different brand PSU, a nice little switching regulator that can handle a short (I tested this by sticking a screwdriver in one of the exposed endstop cables, this was not scientific), no hacked SSRs, and the interface board is made of three separate PCBs, and try as I might I cannot do any damage to the machine by rubbing my feet on the carpet and touching it. It also turns off the LCD when uploading new firmware, and there is no stupid reset button timing. If someone here has access to a proper electrostatic discharge testing device they could try poking it and seeing what happens. I bet a Rep1 will explode, and a Rep2 will reboot safely. They must have hired some more electrical engineers!

TL;DR, shorting anything will cause the regulator to quickly explode (after overheating?), and the interface is a very very fragile circuit. I wish the new mightyboard's schematic was available to review, manually tracing it is a pain.

--
 
 

--

[Jetguy]

Well, I now have a dead board despite being "extra" carefull. It
happened during the classic firmware update with no SD card in the
slot.

The carnage =everything blown sky high.

BTW a reasonably safe way to test is to use some male to female jumper
from a FTDI cable powered by the typical USB port. On the standard
FTDI header cable, the black wire is ground, the red wire is 5 volts
(basically pin 1 and 3). USB is limited to 500ma and the PC will tell
you how much power is being drawn. And easy spot to inject with 5 volt
power is the endstop pins. The middle 2 pins are ground, the white
wire is the +5 volts.
When you do this provided the 5 volt is no longer dead shorted, all
three power LEDS will light even though there is no 24V being
supplied. If you remove that jumper feeding the 3.3 volt and just have
the 5 volt powered by the FTDI and jumper then only the 5 volts works
which is what we initally care about.
This low amount fo corretly regulated 5 volt power will make any chip
that is blown get hot, but not start smoking or anything. To my
surpise, the mega 1280 seems to have somewhat survived as with the LCD
not attached, I got the startup beeps. Feeling the chips, all 5
botsteps blown and getting hot, the suspected U5 NOR Latch is very hot
and may have been the source. The final kicker is the 8u2 seems dead
with no detection as a USB device when the 5 volt bus is powered.

Removing the blown components was easy, just remove all 5 botsteps and
the U5. At this point, the board 1280 seems happy, the botstep ports
measure correct voltages and signals meaning the micro sent the
digipots the correct data. For example, the ref pins on 4 of the 5
read exactly the same voltage ( I think 1.8 volts?) and then the Z
axis was like 0.6 V. This makes sense as the moons motors are roughly
a little more than double the current of the Z axis (0.86A VS ~0.4A)
When Idle, obviously since I have no working USB input and no LCD
control panel support, the other input pins of the botstep are pulled
high (as expected) because Allegro chips see low (AKA grounded) pins
as high or active. So for example, stepper enabled, the enable pin
goes low (no way to do this on this board).

I know for a fact from the DIY drones threads, the 8u2 will not
tollerate over voltage and they blow out very easily on the drone
board all the time when people use the wrong voltage or other wiring
errors. Supposedly as little as 5.5 can kill them.

Testing the LCD panel shows a bunch of low resistance meaning most
everything there was damaged in the event.

Hoefully, MBI support can ship me some boards ASAP.

Again, I think people could test with resonable safety of not causing
more damage from a limited power supply. Isolate the 3.3 volt. Mine at
least beeps, but would be hard if not impossible to repair as the 8u2
has to be either hotplate or reflow soldered. You could bypass it with
the an 8u2 breakout from Sparkfun, but not pretty. I believe the
Max6675s and the mega 1280 survived as they are tougher, but the known
weak points are seemingly the botsteps and U5.

Interestly, this points to the cause as being the U5 and disabled
safety SSRs (which shouldn't short but I consider highly suspect given
what I just saw and tested), or it's the botsteps failing. Both are
connected between 24 and 5 volt rails.

> > various components to the 5v rail to see just *how* dead, and found that

> > pretty much everything was shorted one way or another. That seems like the
> > work of a really dramatic over-voltage event, or perhaps just over-current
> > on the same bus. In any case, it seems that one thing dying can kill
> > everything else.
> > I checked the resistance of the SD card, and found that it was indeed

> > shorted, as was the *interface board!* It seems that the 5v to 3.3v logic

[Gary Crowell]

On Thu, Dec 20, 2012 at 8:10 AM, Jetguy <barry...@hotmail.com> wrote:

Well, I now have a dead board despite being "extra" carefull. It
happened during the classic firmware update with no SD card in the
slot.

I was wondering if erasing/writing the 1280 flash drew additional current, but I didn't notice anything in a skim of the datasheet.

Gary 

--
----------------------------------------------
Gary A. Crowell Sr., P.E., CID+

Linkedin  Elance  KE7FIZ

[Jetguy]

> I was wondering if erasing/writing the 1280 flash drew additional current,
> but I didn't notice anything in a skim of the datasheet.

Not enough to even matter. I mean I can program the bootloader via a
USBtiny programmer which bus powered the microcontroller via the USB
bus limited to 500ma. We have 5 Amp rated 5 volt regulator (I
understand subject to thermal contstraints but we don't hit them). The
regulator should be bunring up board traces if the devices are
shorting or excessive load. As I stated, I did a test with a 7812
linear regulator feeding the 5 volt regulator from 24 volts and you
could keep for finger on it for 30 seconds with no heatsink. We are
FAR below the threshhold. There are no witness marks, the regulator is
not getting remotely hot under normal conditions conditions.

Seeing one blow up and then disecting this, the Mega seems to survive,
the 8u2 is likely to be killed every single time.

Now that I have a failed one in front of me, I can provide reall
meaningful analysis.
That said, I am more sure than ever it's either the Botsteps failing
and the Z axis is my prime suspect (I'll explain later but just say I
have some info), or it's the stupid SSRs, failing and backfeeding the
5 volt bus.

Armed with that info, I think a couple of trace cuts could greatly
reduce the chance of blowouts. MBI seems to think the X cable is the
main culprit and it might be a cause in some cases. I already came up
with the easy for for that, pull the white wires out of the endstop
connectors as we do not need 5 volts at the endstops. Why tempt fate?
Anybody with 5 minutes and a paper clip can do that fix.

If it's the botsteps, I already have the idea and could work out a way
to replace them with bog standard Pololu 4983s or 88s. It's not even
that hard. A couple of the header pins just not soldered in and maybe
some resistors in place to control the MS jumpers.

The mods to cut traces around the safety cutoff system are safe as
there are no nearby traces that would kill something important.

My goal is a series of fixes that can be done in the field, low or no
cost, and do not require any special tools. I don't think we need to
go out and replace the regulator, I don't think we need to go wild
adding stuff, but worst case is replacing the Botsteps.

I want to do these as an alternative to the day MBI stops supporting
Mightyboard 1. They have great support now, and have helped many
people out including me, but at some point with Replicator2 on the
market, the supply of parts will dry up as being not viable for MBI to
support. As a community effort, I want an alternate solution in place
for folks for when that day comes.

On Dec 20, 11:01 am, Gary Crowell <garyacrowel...@gmail.com> wrote:

> On Thu, Dec 20, 2012 at 8:10 AM, Jetguy <barrych...@hotmail.com> wrote:
> > Well, I now have a dead board despite being "extra" carefull. It
> > happened during the classic firmware update with no SD card in the
> > slot.
>
> I was wondering if erasing/writing the 1280 flash drew additional current,
> but I didn't notice anything in a skim of the datasheet.
>
> Gary
>
> --
> ----------------------------------------------
> Gary A. Crowell Sr., P.E., CID+

[Gary Crowell]

On Sat, Dec 15, 2012 at 11:39 PM, Gary Crowell <garyacr...@gmail.com> wrote:

At least a start anyway.

 

From those resistance readings, we can work backwards with parallel resistance calculations to get an idea of the resistance of each chip

• Regulator: 120 Ohms
• LED driver: open
• Thermocouple IC's (both together): 100 Ohms
• X-axis pot: 370 Ohms
• ATMEGA8U2-AU : 170 Ohms
• ATMEGA1280 : 24 Ohms
• Y and Z axis pots together : 500 Ohms
• Extruder A pot : 7.3K
• Extruder B pot : 51K
• Rest of board : 4.7K
• Somebody might want to check me on this.

Finally got around to measuring the resistances of the chips I took off the board.  These are the only ones that I still had that I could get a good measurement on:

• ATMega1280   25 Ohms
• MAX6675  (1) 800K    (2)  200 Ohms

... Pretty close to what I got on the board.

Gary

--
----------------------------------------------
Gary A. Crowell Sr., P.E., CID+

Linkedin  Elance  KE7FIZ

[RocketSled]

On Monday, December 17, 2012 2:06:05 PM UTC-5, Gary wrote:

Yup, either of those would work, and look like good choices.  Only difficulty, not great, is that they aren't drop in.  I think the pinout is different, and the space is limited - they'd have to be deadbuged above the board somehow.

Gary

Alas, I don't think so even if the pinout is the same.  The Murata doesn't look small enough, and neither provides enough current.  At least as I understand it, we want 5A.  The Murata is 2, and the Mouser is only 0.5.

 

[RocketSled]

I make SSDs.  Erasing Flash usually draws the *highest* current.  Max, maybe 2 or 3x higer than Read current.  Programming is in between.

[Jetguy]

We don't need 5A. Basic math 101 says 5 Amps @ 5 Volts is 25 Watts.
25 Watts takes a heasink on every chip to disipate that much energy.

Let's fix the problem. It's either the botsteps failing and shorting
24 Volts into the 5 Volt Logic supply (VDD), or I'm seeing faults that
should never occur in the safety SSR function that does the exact same
thing. Both conditions would blow up any regulator.

I'm not saying you guys shouldn't investigate alternate 5 volt
regulators, but it's not the root cause or a solution to the problem
at hand.Even a shunt heavy duty Zener regulator which would be an
attempt at effective overvoltage protection can't deal with a 24 volt
short into the 5 volt rail.

It's funny, but the children's game, "Which on of these is not like
the other", is what gave me the clue as to where to look for the
failure. Of the 5 botsteps, 4 see the same motor but the Z axis is the
one that has only 0.40 Amp rating and high inductance. Those two
factors mean that stepper driver and specfically, the reverse clamping
diodes to block the ringback into the FETs from the inductance are
taking a serious beating.

I honestly believe the failure sequence goes like this:
Either via the firmware update or normal usage, the Z axis botstep
blows out when the inductance and the high voltage ring that occurs
blows out the shunting diodes protecting the FETs in the H-bridge. I
believe the firmware update could either send very fast pulses or
manipulate the digipot, causing a condition which stresses the
botstep.
The botstep blows and shorts the 24 Volt VMotor to the 5 volt logic
VDD
At that point, we are seeing 24 volts on the 5 volt rail which takes
out the 8U2 (USB interface), blows the regulator sky high, and causes
U5 (the Nor logic for the unused safety cutoff.

The other supposed failure mode MakerBot is suggesting in the X cable
which does flex a lot, it shorting 5 volt to ground, causing an
overload of the 5 volt regulator. While this is possible, the fix is
stupidly easy, remove the white wire (5 volt) at each of the 3 endstop
cables into the motherboard. This is a good, safe, easy fix everyone
should do right now today. Sure, the stupid LED no longer lights up on
the endstop when switched. Not the end of the world, and far better
than the chance of blowing the thing up.

The harder problem is this massive blowout we are seeing that
typically can happen with a firmware update.
My causal advise that is easy to implement as a first precaution is
remove all 5 botsteps before doing a firmware update. This is nothing
more than an action or user training. No mods are taking place.
The next more invasive step is to remove U5. It's surface mount, it's
tiny, but can be done with a soldering iron rather easily.
Other fixes include isolating the VDD pins from the rest of the 5 volt
bus and provide a second voltage regulator just for them. We could put
a 1 amp silicon diode in the path from the current regulator but that
means a standard 0.7 volt drop thus 4.3 volts at the VDD pin of the
botsteps. Not sure if that is a good plan. If we use only 1 diode,
then if one botstep fails, they all fail. We could use 5 diodes, one
directly in place of each VDD pin.

Other alternatives inlcude swapping the motor body and coils from a
standard Moon motor from the MBI store on the Z axis to make it a low
inductance motor and thus make it stronger and less prone to blowing
out the botsteps. This is so easy, you don't even have to take the bot
apart. just remove the 4 screws on the back, remove the cover and the
body, do the same on the new motor, swap body and cover over the rotor
and replace the 4 screws.
I think in onboard preferences you can see the botstep current
settings and adjust them to match the other motors? Or is it from the
LCD menu in Sailfish?

The basic principals are, let's segregate the logic 5 volt VDD into
sections for the botsteps and "the rest of the logic", investigate
solutions to prevent the suspected botstep from blowing up in the
first place, and possibly get rid of the already disabled SSR section
at the bottom of the boards. I feel these easy to do steps are more
than enough to "fix" the problem.

[Andrew Plumb]

Hey Mark,

W.r.t the order of things, I always turn on the 'bot first, then plug in the USB.  Powering off, I first unplug the USB, then power off the 'bot.

I've gotten in the habit of always doing a power-off reboot after firmware upgrade, after noticing that the board would soft-reset into a state where all the heaters would default to "75C".  That stopped happening a while back, but I still make a habit of doing power-off restarts, just in case something else strange happens.

Andrew.

On 2012-12-21, at 8:11 AM, Mark Cohen wrote:

When I said my board blew after the firmware update and I did sd print
it was probably in the process of moving the Z axis.
Also would it not just be a good idea to turn off the machine, unplug
the usb and pull the power plug after a firmware update so that
everything goes back to normal?

understand it, we want 5A.  The Murata is 2, and the Mouser is only 0.5.- Hide quoted text -

- Show quoted text -

--

--

"The future is already here.  It's just not very evenly distributed" -- William Gibson

Me: http://clothbot.com/wiki/

[Dan Newman]

On 21 Dec 2012 , at 6:31 AM, Andrew Plumb wrote:

> Hey Mark,
>
> W.r.t the order of things, I always turn on the 'bot first, then plug in the USB. Powering off, I first unplug the USB, then power off the 'bot.
>
> I've gotten in the habit of always doing a power-off reboot after firmware upgrade, after noticing that the board would soft-reset into a state where all the heaters would default to "75C".

MBI added that to RepG 39 for all bots. In RepG 39 - Sailfish, we disabled it for
Cupcakes & ToMs and raised it as a concern with MBI. In RepG 40 I seem
to recall disabling it for Sailfish on a Rep 1 as well and MBI picked up the
change.

Dan

[RocketSled]

But I am saying it's the root cause.  The design lacks margin, that's the problem.  A dead short needs a fuse, and that would be a worthy addition too.  But the fundamental issue is that the regulator is operating at very near it's design limit with no mitigation for the stresses that operation induces.  The part is fundamentally more prone to failure because of this.  It's a law of physics, higher temperatures accelerate probability of failure.  And that means the regulator can fail for any reason at any time.  Sure, there are things we could do to aggravate it.  When you're operating close to the breaking point already, it only takes one extra straw to push the system in to catastrophic failure.  But even operating with no transient stress, at the dissipation level the regulator is operating at, it is going to fail at a much higher rate.  A regulator with more operating margin would still be susceptible to a dead short failure, but it would be more reliable under all other operating conditions...

[Jetguy]

Update to my latest protective measures.

I took the time to desolder the 5 volt pin on each of the botsteps. I
did this because I read the data sheet for the A4982 stepper driver
which states worst case current draw on the 5 volts should be around
8-10ma. In place of the pin and the plastic strips that is part of the
header row, I used a sginal diode from radioshack rated at 75ma cold
and 10ma at 150C. I'm obviously hoping not to be at the 150C point but
more than adequate. The data sheet show VDD at the botstep cango down
to just above 3 volts. Typical calc says that a silicon diode drops
0.7 volts. Thus, with a 5 volt source, I'm now feeding each of the
botsteps VDD with 4.3 volts measured, well within spec. The point of
this diode is that when/if the botstep blows up, because the Vmotor is
24 Volts, it's now much better protected against backfeeding the 5
volt bus.

Part 2 of the mod is to just use a 7812 voltage regulator also from
Radioshack to reduce the 24 volts to 12 volts and feed the 5 volt and
3.3 volt regulators. I used a standard heatsink from radioshack, some
#18 wire, heatshrink, and 2 each 100uf 50 volt rated caps on the input
and output, to mount the regulator remotelu from the board right in
front of the cooling fan (actually on the intake side. Even before I
mounted it near the fan, the temps were completely reasonable after
printing for hours.

All that said, and alternative would be a 12 volt zener properly rated
to do the same thing, see the last example
http://www.evilmadscientist.com/2012/basics-introduction-to-zener-diodes/

I did all of the above, plus the previous steps:
Large diode for reverse plug input protection
Removed the 3 white wires from the endstop cable shells at the
motherboard and used heatshrink to insulate them.

I am pretty confident I have a bullet proof mightyboard. You can't
kill it by plug reversal, if a botstep blows, it's only that botstep
failing, the endstop cables cannot short out the 5 volt rail, the SSR
and U5 logic chip have been removed and can no longer pose a threat
from 24 volts backfeeding the 5 volt rail. The 5 volt regulator which
is more than adequate, is now only fed 12 volts. The 7812 seems well
with spec, and worse case, failing feeds the system just as before in
the pre-mod state.



On Dec 16, 10:22 pm, Shawn <sgro...@open2space.com> wrote:/
> In my case, it was after the platform and/or nozzles had been heated and
> were cooling off.  I think I may have had the power switch in the on
> position for one of the boards that blew when I plugged in the brick,
> but know it was in the off position for the others.  I have noticed that
> if the switch is on, I'll hear little sparks when plugging in the cable.
>   Once I noticed that, I make sure my switch is off before plugging in
> the brick.
>
> So my sequence with the replaced boards was:
> - plug in the brick
> - turn on
> - go through the initial power up script
> - In one case I cancelled the levelling routine, in another I let it
> complete but didn't bother with actually levelling (I have another
> method for that).
> - Test the heaters
>    - pre heat the HBP and make sure it was heating  (in one test, I
> skipped this step as I suspected the problem was related to the nozzles)
>    - pre heat the nozzles, one at a time, to make sure I had the cables
> plugged in right
>    - cancelled the pre-heat once I was satisfied the right thing was
> heating.
>
> At that point, I would get the POP and smoke while letting things cool off.
>
> On 12-12-16 07:05 AM, Jetguy wrote:
>
>
>
> > Sorry to spam but some other failure mode thoughts.
>
> > I cut the trace feeding the LM1084 -5.0 from the 24V, and then was
> > double checking the polarity of the power plug before I soldered my
> > reverse protection diode (an FR503-TP) directly to the bottom side of
> > the board on top the the power input connector pins (shunt the problem
> > at the source and lowest resistance point), when I noticed the botstep
> > driver's LEDs lit up.
> > So just as a thought, this makes me buy even more into the reverse
> > polarity theory in that it might even be the botsteps backfeed up the
> > 5 volt rail, thus blowing everything out too. In other words, it might
> > not even be the regulator that fails first, the botsteps could
> > backfeed the 5 volt rail in some fault conditions as a backdoor that I
> > hadn't previously suspected.
>
> > In previous bots, we always had separate stepper drivers. If you blew
> > one, it was less likely to kill all your electronics as the stepepr
> > drivers had their own 5V supply from an entirely separate source.
>
> > It might be that we divide the 5 volt into a main logic rail, and then
> > a stepper driver logic rail.
>
> > And thus lies the problem. This is going to be hard to pinpoint the
> > exact failure. Since everything is driven by 1 single 5 volt source
> > with no isolation, any single fault could break the system. It is a
> > rather new interesting point that there are 2 possible (really 5+1)
> > places 24 V and 5V are present on the same chip right? The MOSFETs are
> > an unlikely candidate as they only interface a single output pin each
> > on the microcontroller and are on the ground side of the heater loops.
> > Thus, they might blow out an output of the Mega, but not cause the
> > type of destruction seen the the blown boards where everything blew.
>
> > One point we could ask is those who did blow up a board, what was the
> > exact sequence when it happened?
> > I realize people might not want to admit it, but we need the data to
> > fix the problem.
>
> > Did you move or unlug the brick to bot power cord before the event?
>
> > If yes, was the switch on or off when you plugged it in?
>
> > Did you get immediate smoke as soon as it was plugged in, or after the
> > switch was turned on?
>
> > Anyway, here is a quick and dirty mod that at least provides reverse
> > and overcurrent protection.
> >http://www.thingiverse.com/thing:38045
>
> > On Dec 16, 7:19 am, Jetguy <barrych...@hotmail.com> wrote:
> >> Oops, caught myself there, it's differential voltage, not absolute.
> >> Even I can make a mistake.
>
> >> So it's 5 volt+ 25 volts =30 volts max?
>
> >> What made me start looking was my proposed fix of using a 12 volt in
> >> between. The 12 volt rating is:
> >> LM1084-12 18V max differential, or (12+18=30V) the exact same value as
> >> the 5 volt version and the 3.3 volt shows the same math.
>
> >> This means it could be the power bricks pushing the limits. They are
> >> rated at 24 volt, but who's to say they don't age or have stability
> >> issues?
>
> >> Still, I have a strong feeling that a (24-5=19 volt) differential, on
> >> paper is not a good idea let alone what the reality could be with the
> >> external supply.
>
> >> Everything in my first post is still true. The proposed fixes would
> >> still be valid, but I am running some end to end tests and caluations
> >> for safety margins.
>
> >> On Dec 16, 7:03 am, Jetguy <barrych...@hotmail.com> wrote:
>
> >>> Crap, not even 5 minutes into research DUH, here's the problem:http://www.ti.com/lit/ds/symlink/lm1084.pdf
>
> >>> RIGHT IN THE DATSHEET max voltage is .... drumroll...
> >>> Absolute Maximum Ratings (1)
> >>> Maximum Input to Output Voltage Differential
> >>> LM1084-5.0 25V
> >>> (1) Absolute Maximum Ratings indicate limits beyond which damage to
> >>> the device may occur. Operating Ratings indicate conditions for which
> >>> the device is intended to be functional, but specific performance is
> >>> not guaranteed. For guaranteed specifications and the test conditions,
> >>> see the Electrical Characteristics.
>
> >>> Hmm, maybe you think operating a regulator 1 volt below it's absolute
> >>> MAX blow it up rating would be a good design choice? This is criminal
> >>> negligence if I ever saw it.
>
> >>> So yes, previous fixes would solve the problem. We need to cut the
> >>> trace feeding the 5 volt regulator from the 24 Volt input. We need to
> >>> put a properly rated device such as a 12 volt step down or other
> >>> switching module in between it (the LM1084 -5.0) and the 5 volt rails.
>
> >>> I'll have a mod up on Thingiverse soon today.
>
> >>> If you own one of these machines, you are rolling the dice daily!!!
>
> >>> Also, heat has little to do with this. This is a voltage thing likely
> >>> making the internal regulator section fail. The device could be cold,
> >>> but if your particular power brick supplies just over 25 volts or
> >>> there is just a minor max tollerance issue, your regulator would
> >>> blow.
> >>> Sorry, I didn't catch this sooner.
>
> >>> And, all another good reason for MakerBot to open the source for
> >>> Rep-2. Who's to say the same idiot who did this, didn't do it again on
> >>> that system. I think we all should have a look at the engineering here
> >>> on a $2,200 machine.
>
> >>> Does the Makerbot engineering staff EVEN read a datasheet? I've lost
> >>> count how many times they have made this EXACT type of error where a
> >>> browse of a data sheet would show they NEVER should have even sent the
> >>> board design out for fab.
>
> >>> On Dec 16, 6:24 am, Jetguy <barrych...@hotmail.com> wrote:
>
> >>>> Thanks for the thorough analysis!!!!!
>
> >>>> Just spitballing some ideas here:
> >>>> I believe(and totally could be wrong here) that the failure comes
> >>>> either from overload of current from some device on the 5 volt rail,
> >>>> or it's the fact such a large differential between the 24 volts coming
> >>>> in being reduced to 5 volts.
> >>>> Another remote option is the power supply producing a voltage spike or
> >>>> slow over voltage that changed over time that exceeds the regulators
> >>>> specs.
>
> >>>> Finally, and having a machine in my hands helped me see this, the
> >>>> round poswer supply input connector can be jammed in inverted thus
> >>>> supplying reverse 24 volts to the board and an obvious blowout the
> >>>> instant that switch is flipped. I know the plug is polarized, BUT, it
> >>>> appears it can make contact in reverse, especially if the switch is on
> >>>> but not be plugged in past the "key".
>
> >>>> I see a VERY likely scenario where the bot was moved, the switch was
> >>>> on and the owner was plugging the cable into the back and had it
> >>>> rotated. It makes contact reverse polarity and blows the board sky
> >>>> high.
>
> >>>> We need layers of protection  here.
> >>>> First, put a 12 volt regulator between the 24 volts and the 5 volts.
> >>>> This adds another layer and some slighty higher input power
> >>>> tollerance.
> >>>> Next, the resetable circuit breakers rated just above monral current
> >>>> levels but far belwo the max rating of the regulators.
> >>>> Reverse input diode and a matching fuse. This way, if you plug it in
> >>>> backwards, the diode clamps the current and blows the fuse.
>
> >>>> In theory, if we did the above layered aproach, we might prevent some
> >>>> failures.
>
> >>>> One final failsafe would be a massive Zener diode on the 5 volt rail
> >>>> fed by an appropriate sized fuse. This way, if all the above failed,
> >>>> we still limit voltage to the 5 volt rail and block reverse current
> >>>> there too. This idea comes from a fix over in the DIY Drones group who
> >>>> kept blowing out their expensive boards for the same overvoltage and
> >>>> reverse voltage reasons.

>
> >>>> On Dec 16, 1:39 am, Gary Crowell <garyacrowel...@gmail.com> wrote:
>
> >>>>> At least a start anyway.
>

> >>>>> It turns out that an associate here at work had a failedMightyBoard, and
> >>>>> had received a replacement, but he had been too busy to swap it out.  I
> >>>>> offered to put the new board in his Replicator, in exchange for getting my
> >>>>> hands on the failed board.  Despite all my efforts, the new board in his
> >>>>> Replicator appears to be working OK, and here's what I found.
>
> >>>>> The first thing I looked at was the X-stop cable:
>
> >>>>>     - Inspected both connectors and the individual wires under microscope -
> >>>>>     nothing unusual.
> >>>>>     - Did a discontinuity check on every pin combination, on each connector,
> >>>>>     and between connectors at each cable end - normal.
> >>>>>     - Did a continuity check on each conductor from cable end to end -
> >>>>>     normal.
> >>>>>     - Removed pins from connector shell on both ends, and inspected wire
> >>>>>     ends and crimps
>
> ...
>
> read more »

[Jetguy]

I know the question was asked earlier so to help others get a good
answer on how much current we require, here is the answer from a brand
new replcement board.
The idle (doing nothing) current for the 5 volt rail (powered by
feeding exactly 5 volts into the board via the endstop connectors
since they present the direct +5V and GND pins) is between 108-115mA.
This is with the LCD panel connected.
Running the motors only makes it get into around 120-128mA
The highest inrush I've seen is about 138mA at first power on.
Keep in mind, this is the backside of the 5 volt regulator which in
theory, should represent the true load. Also, it's powering the 3.3
volt regulator and I did test with the SD card in.

Even flashing the firmware, the most I ever saw was 125mA draw.

I'm doubting running current is the problem nor is programming.

This goes back to my theory. One of the failed boards in front of me.
The Z axis botstep blew after the firmware update. The only way I can
say the 2 are related is that likely, there was a change in the
digipot value and due to the Z axis having a high inductance motor, it
seems likely that contrary to the fact it is the lowest current
stepper, it means that botstep has to work harder to PWM at 24 Volts
and the inductive kick off that motor could be very high.

I've never liked the idea of SSRs being a root cause (they are
supposed to be opto isolated even in failure) and the fact the
regulator never seems stressed has troubled me too to explain the
failures. The MakerBot theory that the shorted X enstop cable is
killing these doesn't explain the relationship to the reports of it
happening at firmware update.
What seems most logical is the Z axis botstep failing.

So this goes back to the fix. http://www.radioshack.com/product/index.jsp?productId=2062587
By replacing the 5 volt pin on each botstep with these, it acts like a
tiny fuse in the case of overload, but also should protect against
voltage feeding back into the 5 volt rail by a failing botstep. The
procedure is simple. Use a desoldering iron to remove the pin, use
neednose pliers to remove the plastic around the pin by breaking it
away. Fold over the non-striped end of the diode the same length as
the other pins from the body od the diode to double up so it fits the
socket better. Insert it into the socket, then insert the botstep and
solder the diode in place of the 5 volt pin.

Sorry for the dark pics http://www.flickr.com/photos/90025904@N04/8305729950/
http://www.flickr.com/photos/90025904@N04/8304682521/

I know they aren't great but the directions are simple. The stripe on
the diode must go towards the Botstep and the non-striped pin to the
motherboard. If done incorrectly, the botstep doesn't get VDD and thus
doesn't work. Easy enough to measure the voltage at the pin in circuit
and it should be around just over 4 volts.

On Dec 23, 11:43 pm, Jetguy <barrych...@hotmail.com> wrote:
> Update to my latest protective measures.
>
> I took the time to desolder the 5 volt pin on each of the botsteps. I
> did this because I read the data sheet for the A4982 stepper driver
> which states worst case current draw on the 5 volts should be around
> 8-10ma. In place of the pin and the plastic strips that is part of the
> header row, I used a sginal diode from radioshack rated at 75ma cold
> and 10ma at 150C. I'm obviously hoping not to be at the 150C point but
> more than adequate. The data sheet show VDD at the botstep cango down
> to just above 3 volts. Typical calc says that a silicon diode drops
> 0.7 volts. Thus, with a 5 volt source, I'm now feeding each of the
> botsteps VDD with 4.3 volts measured, well within spec. The point of
> this diode is that when/if the botstep blows up, because the Vmotor is
> 24 Volts, it's now much better protected against backfeeding the 5
> volt bus.
>
> Part 2 of the mod is to just use a 7812 voltage regulator also from
> Radioshack to reduce the 24 volts to 12 volts and feed the 5 volt and
> 3.3 volt regulators. I used a standard heatsink from radioshack, some
> #18 wire, heatshrink, and 2 each 100uf 50 volt rated caps on the input
> and output, to mount the regulator remotelu from the board right in
> front of the cooling fan (actually on the intake side. Even before I
> mounted it near the fan, the temps were completely reasonable after
> printing for hours.
>
> All that said, and alternative would be a 12 volt zener properly rated

> to do the same thing, see the last examplehttp://www.evilmadscientist.com/2012/basics-introduction-to-zener-dio...

>
> I did all of the above, plus the previous steps:
> Large diode for reverse plug input protection
> Removed the 3 white wires from the endstop cable shells at the
> motherboard and used heatshrink to insulate them.
>

> I am pretty confident I have a bullet proofmightyboard. You can't

> ...
>
> read more »- Hide quoted text -

[RocketSled]

I love data.  But I'm really surprised at how low the current draw is...  I was under the impression it was much higher.  Closer to the limit of the regulator.  I still think running the regulator at 96% of max without a heatsink is bad.  But even at 50% higher current than you measured, it's not that bad.  I concur with your conclusion now that I understand it better.  The 5V regulator failures are likely the result of some other failure someplace else.  Thanks for taking the time to explain.

[Jetguy]

More info on the latest 12volt reduction mod before the failing 5 volt
regulator. http://www.weisd.com/store2/nte1936.pdf

The key thing is read that data sheet and the note about how a diode
should be placed from output to input to prevent power from back
feeding the regulator and thus blowing it sky high. I'm thinking the
same protection should be in place on the 5 volt regulator in addition
to all the other mods.

On Dec 23, 11:43 pm, Jetguy <barrych...@hotmail.com> wrote:

> Update to my latest protective measures.
>
> I took the time to desolder the 5 volt pin on each of the botsteps. I
> did this because I read the data sheet for the A4982 stepper driver
> which states worst case current draw on the 5 volts should be around
> 8-10ma. In place of the pin and the plastic strips that is part of the
> header row, I used a sginal diode from radioshack rated at 75ma cold
> and 10ma at 150C. I'm obviously hoping not to be at the 150C point but
> more than adequate. The data sheet show VDD at the botstep cango down
> to just above 3 volts. Typical calc says that a silicon diode drops
> 0.7 volts. Thus, with a 5 volt source, I'm now feeding each of the
> botsteps VDD with 4.3 volts measured, well within spec. The point of
> this diode is that when/if the botstep blows up, because the Vmotor is
> 24 Volts, it's now much better protected against backfeeding the 5
> volt bus.
>
> Part 2 of the mod is to just use a 7812 voltage regulator also from
> Radioshack to reduce the 24 volts to 12 volts and feed the 5 volt and
> 3.3 volt regulators. I used a standard heatsink from radioshack, some
> #18 wire, heatshrink, and 2 each 100uf 50 volt rated caps on the input
> and output, to mount the regulator remotelu from the board right in
> front of the cooling fan (actually on the intake side. Even before I
> mounted it near the fan, the temps were completely reasonable after
> printing for hours.
>
> All that said, and alternative would be a 12 volt zener properly rated

> to do the same thing, see the last examplehttp://www.evilmadscientist.com/2012/basics-introduction-to-zener-dio...

>
> I did all of the above, plus the previous steps:
> Large diode for reverse plug input protection
> Removed the 3 white wires from the endstop cable shells at the
> motherboard and used heatshrink to insulate them.
>

> I am pretty confident I have a bullet proofmightyboard. You can't

> ...
>
> read more »