Lepton common issue, but no solution

[b...@labyrinthlabs.ca]

Hi All,

I'm experience an issue that I've also read about from a few other people in this group.

I'm using a RPi to read images through VoSPI from a LeptonV3 with some great success.

However, after 5-10 minutes of continuous operation, the Lepton stops transmitting data. This is evidently the problem, seen with an oscilloscope. The clock and CS pins from the RPi are generated correctly, but the Lepton just refuses to give data.

A number of people (using V2 or V3) have tried to disconnect and reconnect the breakout board's ground pin. This helps for my system, too, but this is not a real solution. For my application it's important that we have minimal downtime.

Anybody have some input on this? How can I get the Lepton to work continuously?

Thanks. Ben.

[JeffW]

I have the same or a similar issue.  For me - after a while the module still gives data - but SPI bus seems to be out of sync between master and slave.  The lepton datasheet says to hold chip select and clock high for > 185 msec, but that only resyncs most of the time, not all of the time.  I can run the device for a couple of days but eventually it will go totally off the rails and the only reset is to pull the device.

 

[b...@labyrinthlabs.ca]

Hi Jeff.

You're having successful operation for a straight couple of days? Is that with the CS/CLK high for >185ms? How often does your system actually perform this resync?

Thanks for posting!

[JeffW]

It resyncs quite a bit.  Make sure you are using  SPI mode 3 (so clock idles high) and when the frames fail to report after X attempts, hold for ~200 msec and stuff should sync back up.  You may have to re-initialize the Pi side of things too (depending on how your code is organized).

However, even with this approach - eventually the thing goes off the rails and wont come back without a hard reset.

Let me know if you find any other tricks.

[b...@labyrinthlabs.ca]

Thanks for the info. Also using SPI mode 3 here. I do get some decent frames while it's working...just haven't been able to successfully run it continuously.

If I figure it out, I'll let you know.

[cn...@uncc.edu]

Hi Ben,

Something that helped me out a great deal was to use a Pi Proto board that connects directly to the GPIO pins of the RPi.  I too am using a Pi V3 and what I did was solder female socket pins to my board to connect the camera and then solder wires directly from the 3V3, SPI, and I2C pins directly to the socket pins.  I'm using the default 'raspberry_pi video' code and am able to stay in sync for long periods of time (days).  I saw a post about shortening the wires a while back and this really seems to help a lot.  Just try shortening your wires and you should see improved results!

NOTE: I was using 6" jumpers to a breadboard before this and had sync issues.

Hope this helps,

Cameron

[b...@labyrinthlabs.ca]

Cameron, thanks for the reply.

I too am using jumpers (4 inch female-female) directly from the RPi pins to the Lepton's breakout board pins - no breadboard in between.
I don't want to solder them, but it seems multiple sources suggest it helps a lot, so I will.

Thx

[b...@labyrinthlabs.ca]

Well.. I've soldered wires of 2 inches length directly from the breakout board to the RPi pins. It shows improvement, but still isn't perfect.

It still has the occasional (maybe an hour) halt and never begins sending frames again until the Lepton is powered down and then up (ie: remove ground wire or remove sensor from the breakout board).

I was reading the datasheet (page 17) and thought maybe it has to do with overtemp. I have NOT noticed the Lepton warming up at all. This is the only written reason that I see that could cause the Lepton to enter and remain in Standby mode.

The datasheet is here: http://media.wix.com/ugd/53cdb6_5191be73d1c943d78d2e1a095cb7f3b8.pdf

Thoughts?

[b...@labyrinthlabs.ca]

It also shows on the same datasheet (page 18-19) the sequence required to get out of standby. But I'm not sure how this is implemented.

This Software interface description document doesn't contain any of this.

Software IDD can be found here: https://lepton.flir.com/wp-content/uploads/2015/06/flir-lepton-idd.pdf

[Ben Kluwe]

Hi Ben,

If you are using the lepton sdk libraries, you can read out the FPA temperature using LEP_GetSysFpaTemperatureCelcius (or LEP_GetSysFpaTemperatureKelvin, both found here: https://github.com/groupgets/LeptonModule/blob/890e25d7fc2025ceff792a072a1b9152075733bc/software/beagleboneblack_video/leptonSDKEmb32PUB/LEPTON_SYS.c) to check for overheating (>80°C is highly unlikely).

I myself have tried to get around this problem multiple times with tweaks to the hardware (shorter wires, longer wires, pull-down resistor on clk, etc.) and software (sleeping, master clock speed, etc.), but it simply doesn't go away. The maximum I have got it running for is about two to three hours. These days it runs for about 10-20 minutes, but maybe 2 hours, its varies greatly.

What you see on pages 18-19 of the lepton datasheet can't be done if you use the Lepton Breakout Board from PureEngineering (as seen here: http://www.pureengineering.com/projects/lepton) because the pins PWR_DOWN_L and RESET_L aren't accessible from the breakout board.

Cheers,
(coincidentally also) Ben

[b...@labyrinthlabs.ca]

Thanks Ben. I don't expect that it reaches even close to 80 deg C either.

Also, if I understand correctly, you're suggesting that the only way to do such is by a hardware sequence on the Lepton pins themselves (not the 8 breakout board pins).

However, there's a quote from the datasheet (page 18): 
    It is also possible to transition between standby and on states via software commands, as further defined in the software IDD.

I'd love to figure this out.

[Ben Kluwe]

You're right! The software IDD has been updated and I've not got the latest version running. Have a look at section 4.7, specifically 4.7.2 (OEM Power on) and 4.7.3 (OEM Power off), this should allow you to power the lepton off and back on again.

Reading on there is also a dedicated Camera Reboot function (4.7.11). In theory, these should be present in the lepton module software (same project as linked in my previous post) as LEP_OEM.{c,h}, but it seems that they aren't implemented yet. It seems that neither the OEM nor the RAD module are implemented.

Ben

[Ben Kluwe]

Hi again,

I've implemented the OEM Module as far as I could for the raspberry pi libs. You can find it here: https://github.com/BenKluwe/LeptonModule. So far, the power down function works but I can't get the power to turn back on, yet when running i2cdetect -y 1 the Lepton (0x2a) address still responds.

Feel free to have a go at it. The first function in the LEPTON_OEM.c file is where the power is supposed to be turned on again. Unfortunately, the Software IDD is quite unclear about where the Camera I2C Device ID is, or I am just blind.

Cheers,
Ben

[b...@labyrinthlabs.ca]

Wow... I was looking at the old IDD too where Section 4 ended at 4.6...

For reference, there is an updated IDD (Rev. 100; Dec 2016) I found here: http://www.flir.com/uploadedFiles/OEM/Products/LWIR-Cameras/Lepton/FLIR-Lepton-Software-Interface-Description-Document.pdf

Thanks for looking into this! I'll give it a go over the next few days.

[b...@labyrinthlabs.ca]

By the way, there is a note for the PowerOn command (4.7.2) that might relate to why you had trouble with the command.

    Note that this command is not fully supported. It works the first time, after that, a power cycle is required.

[Ben Kluwe]

Yup, I found out the solution. Use the LEP_RunOemReboot function. It works as expected, just be sure to run the FFC command after the reboot or you might get odd results. This might only be applicable if it is set to manual mode.

Later that day I was told by Kurt Kiefer that the two 'new' modules OEM and RAD had already been implemented in the purethermal1 middleware github repository...

I've added both files to my repo so if you download it you will have the original LeptonModule repository with the OEM and RAD modules in there as well.

It's intresting to note that the way the PowerOn function is implemented in the purethermal1 also does not work, they were essentially doing the same thing as I was, writing a 0x0 to the 0x0 register (Power state).

I've since experimented with writing 0xFFFF, 0x0001 etc. to it, but nothing changes the fact that that function doesn't work.

Ben

[b...@labyrinthlabs.ca]

Morning Ben Kluwe. Thanks for finding all of this.

I have yet to give it a try. But with respect to our original issue...the stopped flow of frames after 1-2 hours (in some cases more than a day):
Do you see LEP_RunOemReboot() as an effective way to implement the needed reset?

Thx
Ben

[Ben Kluwe]

Hi Ben, I think it is, but it depends on what you want to do with it really. I've been testing it today (its 4:30 PM here) and its been consistently on for around 7-8 hours. The reboot does take a few seconds, but it's better than nothing! I've set it to send the reboot command when the program doesn't get a new image for three seconds. This way it takes around 5-6 seconds for it to come back up when it 'fails'. For our purposes it's not ideal, but we can live with it.

Ben

[Alexis Meneses]

Hi Ben, 
I have a doubt. How did you use the function? 
You need to define type:

typedef struct  LEP_CAMERA_PORT_DESC_T_TAG

    {

        LEP_UINT16  portID;

        LEP_CAMERA_PORT_E   portType;

        LEP_UINT16  portBaudRate;

        LEP_UINT8 deviceAddress;

    }LEP_CAMERA_PORT_DESC_T, *LEP_CAMERA_PORT_DESC_T_PTR;

But, how Do I identify the portID, and the portType. The portBaudRate I guess it is how you open the port. The deviceAdress I asume that it is the address of the device. So, now I am a bit lost. 
How Should I defined?

[Ben Kluwe]

Hi Alexis, check LEPTON_SYS.c:

LEP_RESULT LEP_OpenPort(LEP_UINT16 portID,

LEP_CAMERA_PORT_E portType,

LEP_UINT16 portBaudRate,

   LEP_CAMERA_PORT_DESC_T_PTR portDescPtr)

you never populate the values directly. If you search for LEP_OpenPort in the code on github, you can find multiple examples.

most (if not all) are of the type:

LEP_CAMERA_PORT_DESC_T port;
LEP_OpenPort(1, LEP_CCI_TWI, 400, &port);

you can then use the port variable as a gateway to the API.

[Alexis Meneses]

This works perfect. 

Also, the method of LEP_RunOemReboot().

For people who dont know, you need to do this:

LEP_CAMERA_PORT_DESC_T port;
LEP_OpenPort(1, LEP_CCI_TWI, 400, &port);

LEP_RunOemReboot(&port)

to restart the device. Basically, works fine, but you need to wait maybe 4 to 5 seconds to re synchronized. 

[b...@labyrinthlabs.ca]

Hi Alexis,

If you look for example in Ben Kluwe's code, the port handling is done in the Lepton_I2C files. Lepton SDK functions that are used are written into other functions in there.

I believe it's doing the same as you are suggesting for every LEP_* type of function.

Thx

[Robert Dawes]

Can I also add my thanks to Ben - this was going to be the next thing I tried after I encountered the same problem and it was a great relief to come across this thread and discover someone else has had the same trouble and that it worked as a solution.

One thing to add from my tests - when I call LEP_RunOemReboot() it sometimes returns -26 and sometimes 0. When It returns -26 everything works fine. When it returns 0 the camera image is normally very degraded and the system will usually fail again within minutes. I'm going to add a condition to make the system reboot again unless it receives -26.

Robert

[Austin McElroy]

Hey Ben,

Thanks for posting this, it has helped with some of the disconnect issue. After a few hours to tens of hours, the several of our Radiometery Leptons will still enter an unrecoverable state where the only solution is to reseat the Lepton or reboot the micro controller. I just opened an issue with FLIR support, do you have any other guidance on this?

Thanks,

Austin McElroy

[Ben Kluwe]

Hi Austin,

by radiometric leptons do you mean the lepton version 2.5? I have tested my code on the lepton 2 and 3. I didn't test any longer than a day yet but that might be something to start doing now that you brought up tens of hours. Unfortunately, thats as far as my knowledge goes. When you say unrecoverable state, does it mean that the lepton doesn't even respond to I²C commands? If you have an interface (GUI/terminal application) to send commands manually to the lepton then you could try and test out which functions return what values with which parameters after it goes into the state. I know that this is a painstacking process because it may very well be that it is still responding on the I²C and quirks such as an unrecoverable state if you start the spi too soon may pop up, but its probably the only way to get clarity.

Ben

[Austin McElroy]

Hey Ben,

Thanks for responding so fast. These are the Radiometery Leptons with 80x60 resolution, so I guess 2 or 2.5 as the nomenclature goes. Unrecoverable means that the SPI doesn't return any valid packets, but seems to respond to I2C, though I am almost immediately rebooting the lepton to try and nip the issue as soon as it occurs. Unfortunately, I am running this headless as a server side application and have tons of logging, and can almost always attach the debugger to the remote code and see what is going on since the issue never resolves.

I've also tried wiring up a solid state relay to the power line of the breakout board and trying to reset the board that way, but that doesn't seem to work either. Only physically removing and reseating the lepton in the socket or cutting power to the R Pi seems to do the trick. It seems like a bus issue or charge buildup or something like that, but I am 100% positive this issue will happen as time approaches tens of hours.

This issue where the Lepton stops responding seems to be prevalent and not suitable for industrial applications.

[Austin McElroy]

Hey Ben,

Another update. I just went into the unrecoverable state, so AMA and I can try and probe around. The Status Register is reading 63751 coming out of several reboot attempts, so that means it is reading busy, ready, and booted with an error code of LEP_UNDEFINED_FUNCTION_ERROR, or -7. Perhaps the busy bit is held in this high state and that is causing the stability issue?

The server is written using Java, so there is limited code injection that can be done if there is anything y'all would like to test.

Austin

[Ben Kluwe]

That sounds interesting... I realised that I don't actually have a Lepton handy at the moment so I won't be able to test this behaviour for a while. Did you have any luck probing the functions other than the status register? Maybe setting various SPI parameters such as the header at the beginning/end/off? That specific setting caused me some trouble as well at some point...

Ben

[Austin McElroy]

No, I’ve not had any luck. Issuing Shutdown also gives back a -7 error in the status reg. All other I2C commands are returning the proper status but the Reboot and now the Shutdown. Everything seems as per the manual when issuing the commands, set the OEM Protection Bit & OEM Bank & the run command.

I am using Pi4J, can you mention a bit more about the SPI parameters issue? 

[Ben Kluwe]

Sorry, I used the wrong terms. I meant enabling/disabling the telemetry and putting it either before or after the data.

Ben

[Austin McElroy]

Ok, I am not using Telemetry. 

In your code that had issues like SPI disconnects or other odd communication issues, did you happen to do anything odd with the FFC? Personally, I had it disabled for this project. Last night, I turned it back based on another thread you commented in about reset triggering an FFC (my reset still doesn't work). With some code to handle frames when the FFC was on, things have been stable for 24 hours and running. Fingers crossed... 

If this works I'm going to try and minimize communication to the Lepton during FFC and only poll the status register, and the FFC State (SYS command 0x4C) and just leave it alone.

Correlation does not equal causation, but I am grasping at straws on this issue of long term stability!

[manuel esperanza]

i have a problem with that, it says "undefined reference to `LEP_OpenPort' ". I think its the port definition, any idea?

[Ben Kluwe]

This is a problem that occurs when you haven't added '-l <library>' to the command line. This means instead of doing: gcc <my.c> you need to do gcc <my.c> -l leptonSDK.

This will most likely result in another error in which it can't find the leptonSDK unless you have installed it to a location where the system searches by default.

To remedy this you need to add -L <directory_with_leptonSDK>. So the command should now look like this: gcc -L/path/to/leptonsdk/build <my.c> -l leptonSDK.

In 90% of cases when you see the error 'undefined reference to xxx', it is this exact problem.

Ben