LeptonV3 with breakoutboard SPI problems.

[한규동]

Hi, everyone

 

fortunately, I could buy Lepton V3 about three weeks ago.

 

I was trying to edit source code base on https://github.com/groupgets/LeptonModule  /software/raspberrypi_video.

 

I had to succeed to display four segments like below jpg file.

 

But Four segments sync was not correct.

 

I think that cause of the problem is SPI speed.

 

When comparing the leptonV2 and leptonV3, leptonV3 SPI speed is four times as fast than leptonV2.

 

So I changed code like spi_speed 10MHz -> 40Mhz in SPI.cpp file. But Problem is same.

 

Does 'spi_bitsPerWord = 8' also need to change?

 

Please give the help.  

[Andrei]

Hey,

Did you get the chance to fix this issue ? I have a similar problem while connecting Lepton V3 with raspberry pi.

Best,

Andrei

[Kurt Kiefer]

Your bottleneck is probably not in the SPI clock rate, but rather the time it takes for your program to call into the driver and request data from the peripheral. Ultimately you should probably try to get in sync and then read as much data as you can at once. So take a 4k max buffer size for the rpi's SPI driver and divide by the size of a VOSPI message, and that's 24 lines you can read with one call. Note that you could make this buffer bigger with /sys/module/spidev/parameters/bufsiz if you want to read an entire frame with one call.

[Andrei]

Hello Kurt,

Thanks for your fast reply. I understand your point.

So, I tried to wait for the beginning of the segment (which should be package 0), then read 20 packages at once, 3 times.

Also, the function that calls the SPI read method is not doing any processing, and the frame buffer is fully pre-allocated.

But it is not working :(  I always get the first segment, or a strange black\white version of the 1 or 2 segment. I will try to capture that and post it.

At this point, I'm not even interested in reading a full frame (4 segments). I just want to see that I'm able to continuously read complete segments.

Is this the wrong approach ? And I should always look for reading the 4 segments ? (I just tried to isolate my problem)

For the SPI configuration I'm using Mode3, 8 bit, and 16MHz (which will be rounded to 15.6 MHz, clock div 16) .

Best,

Andrei

[Ben Kluwe]

If you look at the lepton specifications, it says that the SPI speed should be 2MHz-20MHz... so 40MHz is overkill and might be causing this. You also dont need to increase the speed because the read timeout is per-packet (1 / 27 sec).

This means increasing the speed will increase the cycle wait for each packet. If this is not clear, have a look at point 9.2 (page 28) in the specifications: https://cdn.sparkfun.com/datasheets/Sensors/Infrared/FLIR_Lepton_Data_Brief.pdf

in conclusion, set the clock to around 4 MHz and you will be fine. The readout will be the same for the lepton v3, only that you need to stitch four 80x60 images together instead of just getting one 80x60 image.

Ben

[Andrei]

Hey Ben,

Thanks for you comment. You are right about the 40MHz speed. That is too much. (Also, that was the initial post of someone else, I just tried to see if he/she solved the problem, since I have a similar one)

Anyway, the lepton v3 specification states that you have 4 times more data to read, since each frame is now 4 segments, where each segment is the equivalent of a lepton v2 frame.

So the readout speed becomes >8MHz. And I'm using exactly what you said 4MHz per segment * 4 segments => 16MHz

Also, there is this open source project of Max: https://github.com/maxritter/DIY-Thermocam/blob/master/Hardware/Lepton.h

That uses the lepton v3 with an arduino. The SPI is configured to 30MHz and it works for him. so, I'm little bit confused right now. 

Best,

Andrei

[Andr Sam]

Hello.

Can you tell where you could find Lepton V3 for purchasing? Or you just disassembled Flir One?

Thanks.

пʼятниця, 7 жовтня 2016 р. 05:53:08 UTC+3 користувач 한규동 написав:

[Andrei]

Your guess is correct. I have it from a Flir One for Andorid.

Best,

Andrei

[Andr Sam]

Lepton 3 datasheet says this:

"This section is provided for customers already familiar with the Lepton 2 VoSPI protocol. It concisely summarizes the difference between Lepton 2 and Lepton 3. Much of the protocol is identical, including the following:

1. The physical layer is identical, including the SPI mode and timing.
2. The minimum VoSPI transaction is a packet, consisting of 164 bytes of data when in Raw14 video mode or 244 bytes of data when in RGB888 mode. The packet protocol, including the packet header and payload, are unchanged. However, it is worth noting a single packet represented a single 80-pixel video line for Lepton 2 whereas it represents half of a 160-pixel video line in Lepton 3.
3. The synchronization requirements are identical with one exception. To maintain synchronization, Lepton 2 requires each video frame to be read out prior to the next available frame. In contrast, Lepton 3 requires each segment to be read out prior to the next available segment, where a segment represents one-quarter of a video frame.
4. For both Lepton 2 and Lepton 3, each unique video frame is duplicated twice. The duplicate frames must be read out to maintain synchronization.

The four most significant differences between the Lepton 2 VoSPI interface and that for Lepton 3 are:

1. On Lepton 2, reconstructing a video frame from the individual packets requires the host to decode the packet number from each packet header. On Lepton 3, the host must decode both the packet number and the segment number.
2. There is 4X more data to be read per frame on Lepton 3 compared to Lepton 2. Therefore, the minimum SPI clock rate to read a frame of data is 4X higher.
3. If the sync pulse is enabled (see section 9.2.3), its frequency is 4X higher on Lepton 3 than on Lepton 2. For Lepton 3, the sync pulse represents when the next available segment is available whereas for Lepton 2 it indicates when the next available frame is available.
4. When telemetry is enabled in Lepton 2, it results in three extra video lines (63 total packets per frame). When telemetry is enabled in Lepton 3, it results in 1 additional packet per segment for a total of 2 extra video lines.

Let's do simple math:

Each packet is 164 bytes long. It represents half of 160-pixel line, so we need 2*164 bytes for line and 120*2*164 for frame. Also, like in Lepton V2, each frame is duplicated twice, so overall we need to read 3*120*2*164 bytes in each video segment, which repeats every 106 ms. So, minimal frequency of SPI must be at least (1/0.106)*8*3*120*2*164 ~= 8.9MHz. So, as you mentioned above, 16MHz must be fairly enough.

2Andrei теска)

середа, 19 жовтня 2016 р. 18:31:09 UTC+3 користувач Andrei написав:

[Ben Kluwe]

yeah sorry, I realised too late that I was replying to the OP. I even need to apologise because didn't actually fully understand the version 3 protocol! However, only the minimum speed requirement changes. You are required to read out a packet in 1 / 106 seconds, 8.5 MHz - 20MHz as oppose to 2MHz - 20MHz. So you should be good.

If you go by the DIY-Thermocam code, then you should be using SPI mode 1 anyway ;)

If it isn't the clock speed then you might want to check the settings that the camera is running on (Raw14 or RGB888, with or without telemetry, maybe check the CRC values?). Also remember that frames will be duplicated twice and you are required to read them out... Effectively, unique frames occur every 3rd received frame.

Hope it helps,
Ben

[Andrei]

Not a problem Ben. Thanks for your help.

Some extra details about the behavior of my configuration:

- I'm able to receive the first segment

- Sometimes, the second segment also comes through

- but, most of the time Lepton "resets" the communication, and I'm reading over and over the first segment.

Wrt the configuration, I'm pretty sure it is Raw14, since this is the default mode, and I'm also able to correctly read a segment.

Also, I know that there are some latency issues with the SPI protocol that may create some issues at high frequencies.

I read few post on this topic and it seems that the problem should be addressed. But, I'm using raspbian 3.18 (build from Jan 2015), and one of the posts mentioned that some serious fixes were added in kernel version 4.1.

Anyone have a better understanding of this issue ? (In parallel I will try to install the latest raspbian on a different sdcard to see if this fixes the problem)  

Best,

Andrei

[Andrei]

Hey guys,

A quick update here. I managed to receive the frames from Lepton V3, hooray !!!!!!!

Tips (that worked for me):

- use latest Raspbian (or any up-to-date unix distribution) -> it seems that the SPI drivers are better, especially starting with kernel 4.1

- do not ground the MOSI !!!! even though the documentation states that this pin should be grounded, if you use raspberry pi you should not do that (unclear for me why? it may be a limitation of SPI drivers on the raspberry pi). So, connect the MOSI pin to the Lepton V3.

- use 16 MHz SPI (which is actually 15.6MHz, due to the integer nature of the clock divisor)

- read more packets at a time. (please see Kurt's post above for more details)

Anyway, I still have a big issue :(

After few seconds, the synchronization with Lepton V3 is lost, and I'm not able to recover from that state.

So, the program is an "infinite loop" of trying to read a complete frame..... I also have a classic reset counter, that will force the SPI reset (close-wait-open) after 500 attempts of reading a complete frame. 

Anyone experienced same issue before with Lepton V3 or V2 ? 

Best,

Andrei

[Ben Kluwe]

I think I might have actually had the same problem! I couldn't set the SPI speed. The raspberry pi would simply insist on using a specific frequency (4 MHz I think). I don't have my pi infront of me right now so I can't tell you which kernel its running on, but will do tomorrow.

Ben

[Andrei]

Hmmm... it is working now. The problem was that I reset the entire frame every time a discard package came, but instead you should reset just the segment (not the frame).

I was confused by the documentation which states that a discard package is sent until a new frame becomes available.

Now, the last problem that I have to fix is the speed :) For some reason I receive only 6~7 unique frames per second, instead of ~9fps.

It may be due to too many resets per frame... So, the sensors drops frames.

Any suggestions guys ?

[Federico Polacov]

Hi, Andrei. I'm trying to do exactly the same thing you've done but in python. I followed all the tips you and Kurt explain here and I have the lepton v3 recording on the raspberry for about a day. After that looks like the lepton enters in a bad mode and starts sending a lot of malformed packages and it never returns a good frame after that. I tryed with 3 different leptonV3 with 2 different raspberries (one 3 and one 2) so I thing is a problem in the driver and I can't undestand also why sudo reboot is not enough for fixing it (I need to unplug the raspberry from AC and plug it again for getting it working). Any idea on where to find this issue will be apreciate.

Thanks in advance

[Andrei]

Hey Federico,

I have a similar problem: after a while my LeptonV3 freezes. I can't get any packet from it. 

Anyway, in my case, it is enough to disconnect the GND pin of Lepton from raspberry pi, then connected back. I don't need to reboot my pi.

I known that it may sound stupid, but you should try this too. If it works, and you need to do this automatically, you can create a small "reset" circuit for this pin. 

I did not get that far, since for me was not that big of a problem (I'm more research oriented and not product focused).

Btw, are you able with python to get all 9(unique) frames from a second ? In my case, I'm around 6-7 fps only.... I think there are still too many reset happening on the SPI protocol and I may lose some frames.

Also, Lepton freeze is happening more frequently that in your case. My Lepton will freeze in less than 1 hour. 

Best,

Andrei

[Federico Polacov]

Hi Andrei. Thanks for the advice. I guess i will make the reset circuit but it has to be a problem with spi drivers that after a while start oscilating the frequency. I'm based on the fact that when i increase the frequency of spi clock to 31.2Mhz somehow takes more time to freeze the Lepton. I'm not sure if i'm getting less frames than expected, i'm processing each frame before requesting an other one, so i'm loosing some time there. anyway i will measure that later and i will let u know. If u have any clue or idea for avoiding the reset circuit, let me know. It sounds like a very unnecessary patch.

Regards,
Federico

[Andrei]

Hey guys,

My camera still stops working after 20-30 min. Never got it to work for 1h continuously.

I'm pretty sure that it is an issue with the SPI protocol between raspberry Pi and Lepton... I may still miss something.

Did anyone have a similar configuration with no freez issues ?

Best,

Andrei

[Kurt Kiefer]

It could just be something periodic happening in your system. That's Linux for you. Try to make the program less `nice`, and if that's not enough, spawn a thread and use SCHED_FIFO or SCHED_RR. I don't see that suggestion on here a lot, but this is something I always have to do on Linux to e.g. keep video timestamps from jittering when doing such a thing in userspace.

[Bart]

Hi all,

For my university graduation project I have been experimenting with the FLIR Lepton v3 sensor. I have the Lepton up and running but my feed is still pretty unstable. 

So far I have read most of the posts in this thread / forum to try to implement some of the suggested fixes. I was able to change the SPI clock speed succesfully and it has indeed improved feed stability.

Can someone give me more detailed pointers on where in the code I have to look and what to adjust to be able to read more packets at a time / increase the buffer size?

Also, I do not fully understand how to implement resetting just the segment and not the entire frame. 

I am using the pylepton v3 code from:  https://github.com/groupgets/pylepton/tree/lepton3

However, my knowledge of protocol programming is unfortunately rather limited which makes it hard for me to comprehend what is happening in the code.

Any help is much appreciated. Maybe it would be nice to make these suggested improvements available on github in an updated library to help along less experienced people like myself?

Thank you for the great suggestions posted in this group so far!

Kind regards,

Bart

[Tomas K]

After some unsuccessful experiments with PureThermal USB board, we decided to use Lepton 3 with PureEngeneering SPI board and we came across exactly the same problem. SPI transfers are highly unstable and it goes out of sync very soon and when it happens it is almost impossible to resync back. Rebooting camera via I2C does not help, but disconnecting voltage pin (#1) for a while seems to be reliable - is it possible to disconnect the voltage by software?

We figure out that the problem is probably connected to higher HW/CPU/bus load, because simple SPI streaming and writing one pixel value into the terminal runs for an hour, but any increase in HW load leads immediately to loss of synchronization. CPU core isolation and changing reading thread's CPU affinity makes no change. Playing with real-time priority makes no change.

The most stable transfers we get by reading only one packet at time (= 164 Bytes) and NOT more packets as mentioned earlier in this topic. When reading more packets, increased HW load leads to the total unrecoverable loss of sync. When reading one packet only, increased HW load leads "only" to random errors in individual packets which looks like bad pixels in the image.

But we would like to find any better solution. We want to display Lepton thermal image and Pi camera image at the some time and it is almost impossible now (starting the PiCam stream leads immediately to errors in SPI transfers).