Toggeling LDO3 causes crash/shutdown of the AXP209. Some verification help needed.

[Olliver Schinagl]

Hey all,

I've seen some discussion in the past about the state of LDO3 on several
boards. Some people mentioned hangs on enabeling them from the kernel etc.

Anyway, I don't think anybody really looked into what was happening. Due
to an issue we had I spend some time on digging herein and found the
following.

What happend was, when we issued a reboot with our u-boot and kernel
configuration, 75% of the time, the SPL would start and print the first
line, and hang some where between displaying the first few letters of DRAM:.

After some poking and prodding, we first thought the PMIC for some
reason or a nother disabled the ext-en pin on the lime2 we where using,
which causes 3.3 V power to drop and thus the board not getting what it
needs.

Digging deeper however, it was discovered that it was not just ext-en,
but the entire PMIC shutting down.

It turns out, turning on LDO3, after it was turned off before, causes
the PMIC to shut down. Even enabeling all interrupts and checking the
interrupt pin, nothing could be found as to why however. It is thus my
believe that there is a design flaw in the AXP209, that causes it to
shutdown if LDO3 is enabled. This does however happen 'randomly' at 90%
of the time. Sometime it does indeed work properly.

I want to now know, before sending patches, if this happens on other
boards next to the Olinuxino Lime2. A simple test can be performed to
find out.

Start the board and interrupt u-boot to get a u-boot console. Then
copy/paste (e.g. run) the following code from the u-boot console.

setenv pwrctrl 'i2c md 0x34 0x12 1'
setenv ldo34_on 'i2c mw 0x34 0x12 0x5f'
setenv ldo3_off 'i2c mw 0x34 0x12 0x1f'
setenv ldo4_off 'i2c mw 0x34 0x12 0x57'

while true; do run ldo34_on; run pwrctrl; sleep 2; run ldo3_off; run
pwrctrl; sleep 2; done



what this code does is enable and disable ldo3 every 4 seconds. For me,
on a small number of boards, I saw failure after 1 - 5 toggles of the power.

Thank you for your results and help.

Olliver

[Marcus Weseloh]

Hi Oliver,

2017-02-24 23:13 GMT+01:00 Olliver Schinagl <olive...@schinagl.nl>:

It turns out, turning on LDO3, after it was turned off before, causes the PMIC to shut down. Even enabeling all interrupts and checking the interrupt pin, nothing could be found as to why however. It is thus my believe that there is a design flaw in the AXP209, that causes it to shutdown if LDO3 is enabled. This does however happen 'randomly' at 90% of the time. Sometime it does indeed work properly.

I want to now know, before sending patches, if this happens on other boards next to the Olinuxino Lime2. A simple test can be performed to find out.

I tested this on an Olimex A20-SOM-EVB board. It boots up with LDO3 and LDO4 disabled:

=> i2c md 0x34 0x12 1

0012: 17    .

After enabling LDO3 manually, it crashes instantly (i've tried it a few times, with the same result):

=> i2c mw 0x34 0x12 0x57

uboot unresponsive afterwards

Toggling LDO4 works fine, though.

LDO3 seems to be connected as power supply for CSI0 and is set to 2.8V:

=> i2c md 0x34 0x29 1

0029: 54    T

Hope this helps,

Marcus

[Priit Laes]

On Fri, 2017-02-24 at 23:13 +0100, Olliver Schinagl wrote:
> Hey all,
>
> I've seen some discussion in the past about the state of LDO3 on
> several 
> boards. Some people mentioned hangs on enabeling them from the kernel
> etc.
>
>

[..]

Tried this on my sunxi zoo, AXP version/revision is same on all of
them:
=> i2c md 0x34 0x03 1
0003: 41    A

I also included AXP209 markings.

1) Cubietruck (A20), AXP209 DC096CB 61E1 OK
2) CHIP (R8), AXP209 E3012CB 68V1  OK
3) CHIP (R8), AXP209 E3012CB 68V1 OK
4) CHIP (R8), AXP209 E3012CB 68V1 OK
5) CHIP Alpha (CS13??), AXP209 D5060CB 67S1 OK

Could it be a layout/design issue with Olimex boards?

Päikest,
Priit :)

[Marcus Weseloh]

2017-02-26 11:31 GMT+01:00 Priit Laes <pl...@plaes.org>:

Could it be a layout/design issue with Olimex boards?

It may well be! Looking at the schematic of the A20-SOM-EVB (the base board for the A20-SOM), it seems like the CSI interface is powered via the +5V supply and converter to 2.8V. But it is *also* connected to the LDO3_2.8V rail, which is connected to LDO3 on the AXP and the F19 Pin of the A20 (which powers port E).

So when we switch on LDO3 and the AXP powers that rail, there are now two sources driving that line. Might be that the AXP notices this, maybe even sees current flowing into it's output, and shuts down immediately.

I'll test this by removing the SOM from the EVB and report back.

Cheers,

Marcus

[Marcus Weseloh]

2017-02-26 13:19 GMT+01:00 Marcus Weseloh <mwese...@gmail.com>:

So when we switch on LDO3 and the AXP powers that rail, there are now two sources driving that line. Might be that the AXP notices this, maybe even sees current flowing into it's output, and shuts down immediately.

I'll test this by removing the SOM from the EVB and report back.

Same behaviour if the SOM is removed from the EVB. Enabling LDO3 instantly causes the AXP to shut down. So it's got nothing to do with the dual supply of the CSI interface. Sorry for the noise...

For completeness sake:

=> i2c md 0x34 0x03 1

0003: 41    A

Markings on chip:

AXP209 E1031CB 32AI

Cheers,

   Marcus

[Marcus Weseloh]

Hi again,

more experiments with LDO3:

If I set the voltage of LDO3 to the minimum value (0.7V):

=> i2c mw 0x34 0x29 0x0

then I can toggle LDO3 on and off without the AXP shutting down.

Even increasing LDO3 voltage after it's been enabled works without problems:

=> i2c mw 0x34 0x12 0x57

=> i2c mw 0x34 0x29 0x54

So it might be a capacitance issue? Maybe the cap on LDO3 and the added input capacitance of the Port E power supply draws so much power that the AXP over-current protection kicks in.

Maybe a viable solution would be to set LDO3 to it's minimum value before switching it back on, then set it to the correct value afterwards. Or even use a slightly higher staring voltage... 1,75V also seems to work reliably my SOM, slightly higher values work sometimes, anything in the region of 2V always shuts down the AXP.

Cheers,

   Marcus

--

Marcus Weseloh
Elektronische Musikinstrumente
Fuhlendorfweg 27a
22589 Hamburg
USt-ID: DE295961146

mwese...@gmail.com
+49 (0)176 48816340
+49 (0)40 67301933 (nur Abends / only in the evening)

[oliver]

Hey all,

top reply, sorry :)

So after the test results I have received so far, for which I am
grateful indeed, we did some more experiments here.

One is to remove C185 (10 microF) and the problem went away. Looking at
other designs, cubieboard 1 for example, uses the same layout, but only
4.7 microF. So it seems that charging of all the capacitance (input
C106) the board itself, the output (C185) may be too much for the
AXP209.

Digging through the datasheet I also found the LDO3 Dynamic voltage
scaling parameter settings, but it did not make a difference.

So next, I will start doing current measurements to put some numbers
there.

Next will be to add a software voltage ramping of LDO3 (maybe behind a
quirk)

[Marcus Weseloh]

Hi,

2017-02-27 13:11 GMT+01:00 oliver <oli...@schinagl.nl>:

One is to remove C185 (10 microF) and the problem went away. Looking at other designs, cubieboard 1 for example, uses the same layout, but only 4.7 microF. So it seems that charging of all the capacitance (input C106) the board itself, the output (C185) may be too much for the AXP209.

Great, that confirms my suspicion that the capacitance is the main problem on the A20-SOM. The reference design for the A20 from Allwinner also suggests 4.7uF on LDO3, Olimex probably used 10uF there to keep the BOM smaller?

At least on the A20-SOM-EVB we have another problem though (as explained in an earlier mail): When then SOM is attached to the EVB, the LDO3_2.8V rail on the SOM is powered from the EVB via +5V and a DC converter. Even with the LDO3 voltage set to it's minimum, turning on LDO3 shuts down the AXP immediately. Probably because the AXP sees an external voltage applied to it's input, it might even see reverse current flowing into it's ouput. I'd say that this is a design flaw on the EVB. So at least on the A20-SOM-EVB combination, LDO3 should never be be allowed to be switched on.

Cheers,

   Marcus

[Lyubcho Haralanov]

Hi,

VR1 is not assembled (NA). It is not placed. Only pads are provided in case you don't want to power the camera from the AXP209 but externally, in which case you should place VR1 and disconnect the SMT jumper LDO3_2.8V_E.

By original design the camera is powered only by the AXP209.

Best regards,

Lub

[Marcus Weseloh]

Hi Lyubcho,

2017-02-27 15:36 GMT+01:00 Lyubcho Haralanov <lub...@gmail.com>:

VR1 is not assembled (NA). It is not placed. Only pads are provided in case you don't want to power the camera from the AXP209 but externally, in which case you should place VR1 and disconnect the SMT jumper LDO3_2.8V_E.

By original design the camera is powered only by the AXP209.

Ah, thanks for that! I based my assumtion on the EVB schematic and didn't notice the "NA" in there (and didn't have a multimeter handy to check). So maybe the cause for the immediate AXP shutdown is the extra capacitance of C28 (22uF)?

Cheers,

   Marcus

[oliver]

Hey Lub

On , Lyubcho Haralanov wrote:
> Hi,

>
> VR1 is not assembled (NA). It is not placed. Only pads are provided in
> case you don't want to power the camera from the AXP209 but
> externally, in which case you should place VR1 and disconnect the SMT
> jumper LDO3_2.8V_E.
>
> By original design the camera is powered only by the AXP209.
>

> Best regards,
> Lub

Thanks for the explanation of the EVB :)

Can you also elaborate on the 10 uF for all designs however?

Some extra information, I ran some current tests on the LDO3
power-up/shutdown and while 200 mA max sourcing current is never
reached, the chip does shut down before the max voltage has been
reached. For example setting ldo3 to max ( i2c mw 0x34 0x29 0x7f ) we
get screenshot 3v6.png where we can see the max voltage is only 2.32.

Repeating the same test however with 2.3 V ( i2c mw 0x34 0x29 0x40 )
yields 2V3.png, which also causes an AXP209 failure. This supprised me a
little, as I expected the 2.3V to actually work, as that is the voltage
we failed before. Also failure happens sooner and max current is lower.

After some trial and error, we find that 1.95 V, in this case, still
works, but that I suppose really depends on the capacitance, but by no
means reliably!

Due to the small timebase (50 uS) we cannot clearly see the power rail
dropping, but be assured, for the 2V3 and 3V6, power slowly drops as
capacitors discharge.


While this can and should be solved in hardware (smaller capacitor)
there's hundreds of thousands of boards in the wild with this
configuration and thus a software solution is needed.

The AXP209 does have slew rate control, however this does not apply when
toggling the LDO3 output switch. What I thus propose, is a quirk-flag,
for buggy boards, where we set the minimal voltage, enable power and
than set the desired voltage, letting the internal slew rate control
slowly ramp up voltage.



What I still would love to hear from you guys, is why this could be
happening. The spec-sheet does say 200 mA of sourcing capability. But it
seems this is not exactly true? At least not when toggling LDO3 via
reg12 for sure.

Olliver

[Lyubcho Haralanov]

It seems like what Marcus suggested (turning off LDO3, setting voltage to minimum 0.7V, then turning on LDO3 and then increasing the voltage to 2.8V) works fine with the LIME2:

i2c mw 0x34 0x12 0x1f

i2c mw 0x34 0x29 0x00

i2c mw 0x34 0x12 0x5f

i2c mw 0x34 0x29 0x58

I'm yet to test the other designs. Nothing is final at the moment since we are still testing, but probably the future hardware revisions would have 4.7 uF on LDO3 and LDO4 instead of 10 uF (LDO1 is always on, LDO2 is connected to the memory and turning it off naturally kills the board).

[Olliver Schinagl]

Hey Lub,

On 28-02-17 07:43, Lyubcho Haralanov wrote:
> It seems like what Marcus suggested (turning off LDO3, setting voltage
> to minimum 0.7V, then turning on LDO3 and then increasing the voltage to
> 2.8V) works fine with the LIME2:
>
> i2c mw 0x34 0x12 0x1f
> i2c mw 0x34 0x29 0x00
> i2c mw 0x34 0x12 0x5f
> i2c mw 0x34 0x29 0x58

Yeah, I ran the same tests and even 0x7f (3.6V) 'works fine'. The only
problem herein is, that this solution can't be simply rolled out to all
boards I think. (All meaning the CHIP, cubieboard etc). For one, it
would change current behavior of boards that are not broken.

But more importantly, how does electronics behave that get this (fast)
ramp up, rather than instant on power? Are there people using LDO3 with
the assumption power is immediately available?

So I think this alternate behavior should live behind a toggle, which is
enabled for all boards that need it. (Currently all Olimex boards?)

Olliver

[Lyubcho Haralanov]

The question that really bugs me is: why toggling LDO4 doesn't affect the board but toggling LDO3 kills it...

[Marcus Weseloh]

2017-02-28 13:19 GMT+01:00 Lyubcho Haralanov <lub...@gmail.com>:

The question that really bugs me is: why toggling LDO4 doesn't affect the board but toggling LDO3 kills it...

Can you measure the time it takes LDO3 and LDO4 to ramp to up the their final value? LDO4 seems to have a different architecture than LDO3, the datasheet calls it "low noise LDO". Maybe the different architecture means it has a much slower (or even just a working) slew rate than LDO3?

Olivers measurements also seem to suggest that the documented slew rate of LDO3 (VRC, register 25H) of 1.6mV/us is actually not what's happening when re-enabling LDO3? If my maths is correct, then it should take LDO3 about 1.75ms to go from 0V to 2.8V. But if the attached 10uF leads to current in excess of 200mA, then it seems like the voltage rises much faster than that (less than 100us)?

Cheers,

    Marcus

[o.sch...@ultimaker.com]

Hey Lub,

On Tue, 2017-02-28 at 04:19 -0800, Lyubcho Haralanov wrote:
> The question that really bugs me is: why toggling LDO4 doesn't affect
> the board but toggling LDO3 kills it...

I was wondering the same thing, but it could be very well because it
has a different circuitry behind it inside thee AXP209. LDO3 is
separate, LDO4 is shared with LDO2.

From my measurements it does seem that the AXP209 may have a small
glitch however. I do not see 200 mA inrush currents, 190-ish, which is
close yes, but I did not expect shutdown. Also the interrupt never
fires. Not that there's anything we can do about that however...

[o.sch...@ultimaker.com]

Hey Marcus,

On Tue, 2017-02-28 at 13:58 +0100, Marcus Weseloh wrote:
> 2017-02-28 13:19 GMT+01:00 Lyubcho Haralanov <lub...@gmail.com>:
> > The question that really bugs me is: why toggling LDO4 doesn't
> > affect the board but toggling LDO3 kills it...
> >
>
> Can you measure the time it takes LDO3 and LDO4 to ramp to up the
> their final value? LDO4 seems to have a different architecture than
> LDO3, the datasheet calls it "low noise LDO". Maybe the different
> architecture means it has a much slower (or even just a working) slew
> rate than LDO3?

Yeah, different hardware inside the AXP I think ...

>
> Olivers measurements also seem to suggest that the documented slew
> rate of LDO3 (VRC, register 25H) of 1.6mV/us is actually not what's
> happening when re-enabling LDO3? If my maths is correct, then it
> should take LDO3 about 1.75ms to go from 0V to 2.8V. But if the
> attached 10uF leads to current in excess of 200mA, then it seems like
> the voltage rises much faster than that (less than 100us)?

Actually, VRC should work, but it only works when changing an already
enabled LDO3. E.g. the slew rate control only applies when going from
0.7 -> 3.6 V. Further more, it is disabled by default and not enabled
in u-boot at all. The default slew rate of ldo3 (e.g. the instant on
case) is something like 0.016 V/uS.

I've finished my patch and am testing it right now, but running into an
issue still before sending it off. I added an option to enable the VRC
on boards that need it.

>
> Cheers,
>
>     Marcus