Contactor issue

[Andre Eisenbach]

Today marked the first prolonged charging session with my new OpenEVSE v3.

It charged for about 2h continuously at 27A. Temperature showed 45C.

When unplugging the charging cable after that, my firmware (NoSpark) crashed and the station auto-reset. When it came back up, the relay check failed. I tried removing power to the unit and re-starting it a few times, but not a single click from the contactor.

After letting the unit sit for 5-10mins (with no power) and coming back to it, it started up correctly on first try.

Any idea what might be going on there?

I'm guessing heat build-up caused the contactor to be stuck somehow.

Defective contactor maybe?

[Nick Sayer]

I doubt the contactor was at fault. 45 °C is a temperature that is even tolerable for humans.

It's possible that the SSR failed. I don't believe the stock design of OpenEVSE v3 includes any provisions for handling the coil collapse voltage from turning off the contactor. You can add your own MOV across the contactor coils to at least protect against damaging the SSR. To also add some noise suppression, consider adding a .01 µF cap and 39 ohm resistor in series across the coil as well. The resistor should probably be 1/3 or 1/2 watt rated and the cap should be 1 kV minimum.

I went through a bunch of designs of my contactor adapter board before I got the snubber right. It was the second most painful design iteration process I've done so far (the first being a rechargeable USB power brick that just never worked properly for reasons I never did figure out). I wound up using 1/3 watt SMD resistors and 1 kV rated ceramic caps and even then I decided to add a MOV as extra insurance.

[Craig Kirkpatrick]

Andre, I would say that the OpenEVSE V3 is pretty well proven to be solid hardware with so many units shipped and other than the I2C pull-up resistor problem appearing on some build lots of the V3 board. The V3 with contactor has proven to be very reliable.

I do admire Nick's design too.

I like the idea of adding an MOV like Nick and Chris have referenced. I'll just add it at the screw terminals for the AC RELAY since it is simple to add there.

45C is just warm = 113F. You are not witnessing any sort of thermal runaway just normal warming in my opinion. Maybe this weekend I'll change a flag in my code to always display temperatures on my 50A kit and let you know what I see as normal at 28A for an hour of charging with the same hardware you have.

I'm puzzled as you are at the symptoms you witnessed. But I'd say not to sweat 45C.

[Andre Eisenbach]

Well, I'm only pointing out the temperature because the after letting it sit for a moment the contactor started working again.

Any other ideas what could cause a stuck relay?

And no, unfortunately I don't know if it was stuck open or closed.

[Craig Kirkpatrick]

The correct J1772 protocol has the EVSE relay only opening or closing when no current is flowing. The one exception is a Ground Fault which is an emergency and the relay should open asap. A high current contactor opening with 27A still flowing to the car will cause a furry spark that will erode the contactor's internal contacts and begin a spiral of problems like possibly welded contacts or higher resistance leading to heating.

I'm hoping you have the NoSpark code's J1772 protocol correct and avoiding arcing the contactor.

[chris1howell .]

The most common cause of a stuck closed relay is an abrupt opening while under load. Stuck open could be bad crimps or poor connections.

Like Craig said v3 contactors have been very reliable in the field. If the issues persist I would suggest running a OpenEVSE HEX build for a few days.

--
You received this message because you are subscribed to the Google Groups "OpenEVSE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openevse+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Danny ter Haar]

On Thursday, April 23, 2015 at 3:30:21 AM UTC-7, Andre Eisenbach wrote:

When unplugging the charging cable after that, my firmware (NoSpark) crashed and the station auto-reset. When it came back up, the relay check failed. I tried removing power to the unit and re-starting it a few times, but not a single click from the contactor.

Could it be the EV was still connected ?

I've just blown your 0.03 code in my v3 and with the EV attached power cycled a couple of times and during the POST the relay does not click. With the EV disconnected, it does.

I think this is a safety issue (and rightly so!) 

Could that have been the case ? 

[Andre Eisenbach]

Yupp, it checks whether the EV is attached at startup and does NOT cycle the relay when it is.

[Andre Eisenbach]

What should the disconnect procedure look like?

I read the OpenEVSE source code again and I don't see anything that would show any compensation on disconnect. When the state changes to "A", the relays are clicked off. 

Based on what you guys said in this thread here, it seems like when state changes C->A, there should be a delay on disconnecting the relay?

That seems dangerous to me as it would leave the cable powered even after it's been disconnected. And again, I don't see anything like it in the firmware.

What am I missing?

[Nick Sayer]

On Apr 23, 2015, at 8:35 PM, Andre Eisenbach <int...@gmail.com> wrote:

What should the disconnect procedure look like?

I read the OpenEVSE source code again and I don't see anything that would show any compensation on disconnect. When the state changes to "A", the relays are clicked off. 

Based on what you guys said in this thread here, it seems like when state changes C->A, there should be a delay on disconnecting the relay?

That seems dangerous to me as it would leave the cable powered even after it's been disconnected. And again, I don’t see anything like it in the firmware.

No. It’s the vehicle’s job to only move C->B when there’s no (or little) load. As for C->A, there would only happen after the car got a proximity transition, and would have lowered the load in response to that.

An EVSE must turn the power off without delay when transitions to state B or A occur. Otherwise, the prospect of exposing the hot pins while energized to weather might raise itself, and that would likely be a bad idea.

What am I missing?

On Thursday, April 23, 2015 at 5:41:59 PM UTC-7, Chris wrote:

The most common cause of a stuck closed relay is an abrupt opening while under load. Stuck open could be bad crimps or poor connections.

Like Craig said v3 contactors have been very reliable in the field. If the issues persist I would suggest running a OpenEVSE HEX build for a few days.

On Apr 23, 2015 5:31 PM, "Andre Eisenbach" <int...@gmail.com> wrote:

Well, I'm only pointing out the temperature because the after letting it sit for a moment the contactor started working again.

Any other ideas what could cause a stuck relay?

And no, unfortunately I don't know if it was stuck open or closed.

On Thursday, April 23, 2015 at 5:25:14 PM UTC-7, Craig Kirkpatrick wrote:

Andre, I would say that the OpenEVSE V3 is pretty well proven to be solid hardware with so many units shipped and other than the I2C pull-up resistor problem appearing on some build lots of the V3 board.   The V3 with contactor has proven to be very reliable.

I do admire Nick's design too.

I like the idea of adding an MOV like Nick and Chris have referenced. I'll just add it at the screw terminals for the AC RELAY since it is simple to add there.

45C is just warm = 113F. You are not witnessing any sort of thermal runaway just normal warming in my opinion. Maybe this weekend I'll change a flag in my code to always display temperatures on my 50A kit and let you know what I see as normal at 28A for an hour of charging with the same hardware you have.

I'm puzzled as you are at the symptoms you witnessed. But I'd say not to sweat 45C.

--
You received this message because you are subscribed to the Google Groups "OpenEVSE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openevse+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "OpenEVSE" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openevse/hVj54uIdy5o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openevse+u...@googlegroups.com.

[Andre Eisenbach]

The "normal" cable removal sequence would be C->A correct? Or is there a brief transition through B as the button is pressed?

I found a 250ms debounce in the firmware when transitioning states. So I assume that those 250ms are enough to let the cable be pulled out and the power to be removed before the relay clicks?

I'll add that to NoSpark.

[Nick Sayer]

On Apr 23, 2015, at 8:46 PM, Andre Eisenbach <int...@gmail.com> wrote:

The "normal" cable removal sequence would be C->A correct? Or is there a brief transition through B as the button is pressed?

It’s C->A. The proximity transition event isn’t seen by the EVSE and removes load prior to that.

I found a 250ms debounce in the firmware when transitioning states. So I assume that those 250ms are enough to let the cable be pulled out and the power to be removed before the relay clicks?

I wouldn’t add such a delay.

[Andre Eisenbach]

On Thursday, April 23, 2015 at 9:23:14 PM UTC-7, Nick Sayer wrote:

I wouldn’t add such a delay

I don't understand.

1. The delay is in the OpenEVSE firmware (25ms for the transition to STATE_A, 250ms for all others)

2. How else can one possibly prevent opening the relay under load if the FW has no control over the cable removal?

Note the comment below from the OpenEVSE firmware:

// must transition to state A from contacts closed in < 100ms according to spec
// but Leaf sometimes bounces from 3->1 so we will debounce it a little anyway

[Lee Howard]

On 04/23/2015 08:46 PM, Andre Eisenbach wrote:
> The "normal" cable removal sequence would be C->A correct? Or is there
> a brief transition through B as the button is pressed?
>
> I found a 250ms debounce in the firmware when transitioning states. So
> I assume that those 250ms are enough to let the cable be pulled out
> and the power to be removed before the relay clicks?
>
> I'll add that to NoSpark.

The specification requires the EVSE to open the contactors within 100ms
for any transition from C-->A or D-->A. That would include any fast
transition through B, as well. So, basically, any time you are
transitioning to A you must do so within 100ms. Since the pilot pin is
shorter than the charging pins in the Type 1 connector the 100ms is
apparently less time than the time in connector removal between when the
pilot disconnects and the charging pins disconnect.

The proximity button on the Type 1 connector tells the EV to quickly
draw-down or reduce current draw. On the Chevy Volt pressing the
proximity button causes the EV to transition to State B. So, if you're
charging in State C and remove the cable the button must be depressed,
and so there will be a brief moment in State B between States C and A.

So, a C-to-A transition really only happens in an emergency situation
where the EV becomes disconnected in an unexpected fashion.

The last time I looked at OpenEVSE code there was no hysteresis on the
pilot voltage read. So, that 250 ms "debounce" is there because as you
switch between resistors you do tend to get some noise. The transition
is not always smooth, and you don't want to act on the noise. However,
if hysteresis were implemented on the pilot reading without thinking
about this consequence then you would still have a problem, because a
C-to-A transition must not "slide" through B on it's way down (which is
what it would do). D-to-A must not "slide" through C and B, either.

With the OpenEVSE code the way that it was when I last looked at it I
could put a potentiometer or variable resistor on the pilot terminal and
slowly get to a resistance where the module would "flutter" between two
states - A and B or, worse, B and C. In the wild I've only ever found
one EV + EVSE combination where the pilot resistance was problematic
like that, however, and even in that instance hysteresis did not fully
prevent the "flutter".

Thanks,

Lee.

[Lee Howard]

On 04/23/2015 09:28 PM, Andre Eisenbach wrote:
> On Thursday, April 23, 2015 at 9:23:14 PM UTC-7, Nick Sayer wrote:
>
> I wouldn’t add such a delay
>
>
> I don't understand.
> 1. The delay is in the OpenEVSE firmware (25ms for the transition to
> STATE_A, 250ms for all others)
> 2. How else can one possibly prevent opening the relay under load if
> the FW has no control over the cable removal?

Barring some malfunction, in a C-to-A transition you're either going to
open the contactors under load or the cable will be disconnecting from
the EV under load. Take your pick of where you want the arcing to happen.

If it's any consolation, the EV has the same constraint. The EV has 100
ms to stop drawing current whenever it sees the proximity switch opened
or PWM cease.

So, I guess if we're considering a hypothetical situation where the
proximity switch was staying closed but where the cable was otherwise
being disconnected normally, then *both* the EVSE and the EV would be
under a concurrent and simultaneous 100 ms constraint to drop the load.
So, in this case the EVSE may want to be at the tail-end of the 100 ms
window (hopefully slower than the EV) in order to avoid arcing in the
contactors needlessly. But when you consider the reaction time of the
controller and the reaction time of the contactors, there just isn't a
lot of wriggle-room for much of an intentional delay.

Thanks,

Lee.

[Andre Eisenbach]

Good info, thanks!

Since the relay is only open in state C, wouldn't it be as simple as always doing a ~75ms delay in before toggling the relay ON or OFF? That would mean any transition TO C or OUT OF C would be preceded with a delay.

Sounds like that would be standards compliant and safe, no?

Just wondering if the 75ms are enough for the EV to really turn the current off...

Stupid question, in STATE_B, should the relay be ON  or OFF?

[Lee Howard]

On 04/23/2015 09:47 PM, Andre Eisenbach wrote:
> Since the relay is only open in state C, wouldn't it be as simple as
> always doing a ~75ms delay in before toggling the relay ON or OFF?
> That would mean any transition TO C or OUT OF C would be preceded with
> a delay.
>
> Sounds like that would be standards compliant and safe, no?
>
> Just wondering if the 75ms are enough for the EV to really turn the
> current off...
>
> Stupid question, in STATE_B, should the relay be ON or OFF?

As I mentioned in my last message, you could delay 75ms in hopes that
the EV stopped drawing current before then. However, you're then giving
yourself only a 25 ms window for the contactor to react and open in
order to be standards-compliant. Furthermore, you're also then
deliberately introducing a 75 ms window for arcing to occur at the
disconnection point.

Remember that C-to-A transitions are an emergency situation unless there
is some damage to the proximity switch or something. Think of the
charging cable being cut or torn-out or something where the thinner
pilot wire is severed first. The EV should shut-off the current draw,
yes, and we do want to avoid arcing in the contactors, but it's quite
possible that situation necessitates it or risks arcing out where the
cable is being cut or otherwise disconnected from the EV.

In State B the contactors should be opened. There should be no power on
the charging pins on the connector.

Keep in mind that the whole purpose of the EVSE over a dumb electrical
socket is as a safety device.

Thanks,

Lee.

[Andre Eisenbach]

Getting more confused :D

The core of the problem I encountered with my EVSE it seems to be opening the contactor under load. This makes sense and would be corroborated by the fact that currently the NoSpark firmware will open the contactor as soon as we transition out of STATE_C. So, the only way for me to prevent that from the EVSE side would be to introduce a delay on the transition out of state C to allow the EV to power off first.

But now you're telling me a delay is BAD(tm)...

So, what is one supposed to do here then?

[chris1howell .]

You should open the contactor when going out of C.  It will not be under load because pushing the button will signal to the EV to knock it off via proximity.

Ill post more on transitions and states tomorrow.

--

[Andre Eisenbach]

Ok. So then that's not my problem, I guess ...

[Lee Howard]

On 04/23/2015 10:14 PM, Andre Eisenbach wrote:
> The core of the problem I encountered with my EVSE it seems to be
> opening the contactor under load.

If that is true then your EVSE is opening the contactors *during* State
C. The C-->B transition is made by the EV.

> This makes sense and would be corroborated by the fact that currently
> the NoSpark firmware will open the contactor as soon as we transition
> out of STATE_C.

I don't think so. The EV isn't going to switch away from State C while
it is drawing current. So, when the proximity button is pressed the EV
will stop drawing current and switch to State B, in that order. It has
100 ms to stop drawing current.

> So, the only way for me to prevent that from the EVSE side would be to
> introduce a delay on the transition out of state C to allow the EV to
> power off first.

To be clear, on the EVSE-side "transitioning out of State C" means
"opening the contactors" when going from C-->B and additionally
"stopping PWM" when going from C-->A. If the EV tells the EVSE to
change to State B before it stops drawing current, then the EV has a
serious problem. And, if that's the case, I don't see how a 75 ms delay
(or any other delay) is going to prevent it.

Thanks,

Lee.

[Nick Sayer]

> On Apr 24, 2015, at 6:06 AM, Lee Howard <fax...@howardsilvan.com> wrote:
>
> On 04/23/2015 10:14 PM, Andre Eisenbach wrote:
>> The core of the problem I encountered with my EVSE it seems to be opening the contactor under load.
>
> If that is true then your EVSE is opening the contactors *during* State C. The C-->B transition is made by the EV.
>
>> This makes sense and would be corroborated by the fact that currently the NoSpark firmware will open the contactor as soon as we transition out of STATE_C.
>
> I don't think so. The EV isn't going to switch away from State C while it is drawing current. So, when the proximity button is pressed the EV will stop drawing current and switch to State B, in that order. It has 100 ms to stop drawing current.

The EV won’t necessarily switch to state B on a proximity transition. The spec says it only has to stop drawing significant power. There’s no requirement for a state change. It is perfectly ok to transition to state A only when the pilot pin breaks. What is not ok is for that to happen when “significant” current is being drawn. But that can’t happen unless you pull the plug without a proximity transition - like if the button or latch were broken. I’ve seen J1772 plugs with the latch broken off. It’s important if you ever find yours in that situation that you push the button anyway. :)

>
>> So, the only way for me to prevent that from the EVSE side would be to introduce a delay on the transition out of state C to allow the EV to power off first.
>
> To be clear, on the EVSE-side "transitioning out of State C" means "opening the contactors" when going from C-->B and additionally "stopping PWM" when going from C-->A. If the EV tells the EVSE to change to State B before it stops drawing current, then the EV has a serious problem. And, if that's the case, I don't see how a 75 ms delay (or any other delay) is going to prevent it.
>
> Thanks,
>
> Lee.
>

[Craig Kirkpatrick]

Andre, A few days ago I tossed out the idea that possibly the NoSpark firmware was leading to arcing of the contactor.  It simply was one possible explanation of the symptoms you witnessed.  Then a really nice discussion of J1772 state protocol followed.  I think that was useful for everyone to review.  Still it was just a guess that I tossed out there and maybe I ignited a lot of assumptions about the root cause of the trouble you witnessed.

Chris also mentioned that it could be explained by more simple things like a loose crimp on the wires to the contactor coil - I am paraphrasing.  Still the problem may have some very basic root cause.

I think the simplest thing to do would be to use an EV simulator and I think you have one.  Compare state transition behavior between the NoSpark and OpenEVSE firmware.  That may reveal something that is harder to analyze by only studying the source code of both firmwares.

I can offer that I have time and interest to look into this for you today.  I have an EVSE on the bench.  I can use my simulator and do the firmware comparison if you want me to look at it for you.  

Probably the easiest thing is if you can provide a hex file since I do not think I have the same development environment as you do in order to compile NoSpark.  Or let me know what I would need in order to help.

I'm very impressed with the progress you have made writing NoSpark from the ground up.

-Craig

[lincomatic]

Turning on serial debugging will echo all of the state transitions out the serial port. Just call EnableSerDbg(1)

Sent from my iCrap

On Apr 25, 2015, at 9:32 AM, Craig Kirkpatrick <crai...@comcast.net> wrote:

Andre, A few days ago I tossed out the idea that possibly the NoSpark firmware was leading to arcing of the contactor.  It simply was one possible explanation of the symptoms you witnessed.  Then a really nice discussion of J1772 state protocol followed.  I think that was useful for everyone to review.  Still it was just a guess that I tossed out there and maybe I ignited a lot of assumptions about the root cause of the trouble you witnessed.

Chris also mentioned that it could be explained my more simple things like a loose crimp on the wires to the contactor coil - I am paraphrasing.  Still the problem may have some very basic root cause.

I think the simplest thing to do would be to use an EV simulator and I think you have one.  Compare state transition behavior between the NoSpark and OpenEVSE firmware.  That may reveal something that is harder to analyze by only studying the source code of both firmwares.

I can offer that I have time and interest to look into this for you today.  I have an EVSE on the bench.  I can use my simulator and do the firmware comparison if you want me to look at it for you.  

Probably the easiest thing is if you can provide a hex file since I do not think I have the same development environment as you do in order to compile NoSpark.  Or let me know what I would need in order to help.

I'm very impressed with the progress you have made writing NoSpark from the ground up.

-Craig

On Thursday, April 23, 2015 at 5:31:26 PM UTC-7, Andre Eisenbach wrote:

Well, I'm only pointing out the temperature because the after letting it sit for a moment the contactor started working again.

Any other ideas what could cause a stuck relay?

And no, unfortunately I don't know if it was stuck open or closed.

On Thursday, April 23, 2015 at 5:25:14 PM UTC-7, Craig Kirkpatrick wrote:

Andre, I would say that the OpenEVSE V3 is pretty well proven to be solid hardware with so many units shipped and other than the I2C pull-up resistor problem appearing on some build lots of the V3 board.   The V3 with contactor has proven to be very reliable.

I do admire Nick's design too.

I like the idea of adding an MOV like Nick and Chris have referenced. I'll just add it at the screw terminals for the AC RELAY since it is simple to add there.

45C is just warm = 113F. You are not witnessing any sort of thermal runaway just normal warming in my opinion. Maybe this weekend I'll change a flag in my code to always display temperatures on my 50A kit and let you know what I see as normal at 28A for an hour of charging with the same hardware you have.

I'm puzzled as you are at the symptoms you witnessed. But I'd say not to sweat 45C.

--

You received this message because you are subscribed to the Google Groups "OpenEVSE" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openevse+u...@googlegroups.com.

[Andre Eisenbach]

Craig,

Yes, very nice summary you posted there, I fully agree!

Today I fixed the biggest problem that has been troubling me which was the watchdog reset that happened sometimes when unplugging the device. This was a software problem that is now fixed and it's working great!

I don't think that this will fix the contactor issue I've seen. I'm charging right now for a long charge again and will see if this is still happening at the end of a 15+ kWh charge. At least now that the software is fixed, I should be able to debug the hardware issue better.

I'll post the latest hex over in the NoSpark announce thread if you want to try it out (remember, v3 only right now).

lincomatic,

Or "debug 1" in NoSpark :D

[Craig Kirkpatrick]

Andre, I'm crossing my fingers that actually you have no contactor issue at all.

I'm happy to read you located something in the code that wasn't right.

I'll stand by to help as much as I can.

-Craig

[Craig Kirkpatrick]

Andre, I do realize now that I'm not 100% setup to test NoSpark on my bench.  I have an OpenEVSE v3 controller but with the typical 30A kit built with relays instead of a contactor right now on my bench.   I do have a Packard C240C contactor on hand to retrofit in place of the relays and fuse block but I just have not done that hardware change yet.  Once I replace the relays with the AC contactor then I will be in a better position to play software QA engineer for you.

Because I have the relays and not the contactor in place I was not able to check the state transitions.  But it seems you found the trouble anyway.

I did try the hex file you made.  I like your user interface design.  On the bench with zero current actually flowing (still with the CT attached which picks up some noise) I saw the Amperage flip from zero to 73A about 50% duty cycle which accumulated a nice kWh even though no current was flowing.  Understand I'm not criticizing anything, I'm just playing software QA and helping to test things for you.  

Keep plugging away at NoSpark.  Everything you are doing is very impressive.

-Craig

On Saturday, April 25, 2015 at 3:11:27 PM UTC-7, Andre Eisenbach wrote:

Craig,