Is PureThermal 2 compatible with Chrome browser?

[Fernando Paz]

Hi everyone,

TL;DR: is it possible to get thermal image from PureThermal 2 on Chrome without installing drivers?

I've recently acquired a Lepton PureThermal 2 camera, with the goal of capturing the temperature and showing colourised video on my current solution: a web app running on a Chrome OS.

My app works fine with standard webcams, but I get no image at all (blank screen) from the thermal camera, even though it is being recognised by the Chrome browser. 

I then decided to test it on Windows 10 (also on Chrome), but I get a low resolution, very blurry black and white image instead. However the default camera app on Windows shows the thermal image I was expecting. 

In both cases I didn't deliberately install any additional driver, it was all plug and play.

With that being said, my question is: is PureThermal 2 compatible with Chrome browser? I know I will probably need to write my own driver to be able to capture the temperature, but I wasn't expecting to to hit this roadblock with the video stream.

Best regards,

Fernando

[Andrew Jones]

So the FAQs [https://groupgets.com/manufacturers/getlab/products/purethermal-2-flir-lepton-smart-i-o-module]
seem to suggest that this kind of use-case (for using it as a webcam!)
should appear to work:

```
Q. Which apps work with PT2 on various platforms?
A. While not comprehensive, the short list below should get you
started on your favorite platform:

Linux: VLC Media Player, gstreamer, Guvcview
macOS: VLC Media Player, Facetime, iChat
Windows: yawcam (recommended), VLC Media Player, OpenBroadcastSoftware
Android: UVCCamera (your mileage may vary)
```

Have you tried any real webcam apps (i.e., not Chrome) on Windows? How
does it work?

Your graining image: what was that captured using? Have you tried
using GetThermal [https://github.com/groupgets/GetThermal] on Mac or
Linux to make sure your Lepton + PT2 are working okay?

> Attention: "This communication is confidential and may be legally privileged. If you are not the intended recipient, (i) please do not read or disclose to others, (ii) please notify the sender by reply mail, and (iii) please delete this communication from your system. Failure to follow this process may be unlawful. Thank you for your cooperation."
>
> --
> You received this message because you are subscribed to the Google Groups "Flir Lepton" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to flir-lepton...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/flir-lepton/08fe5cec-2457-4c80-aae3-3da8209fe613n%40googlegroups.com.

[Fernando Paz]

Thanks for the reply, guys.

@Andrew Jones I tested it with a bunch of apps, got some interesting results:

• The camera isn't working well on Chrome, Opera and even on the new Edge... they're all showing the same image below (tested on multiple websites)
• 
• Tweaking the level and contrast on Photoshop gives me the image below, so there IS something being captured, it's just not clear enough:

• 

• It works well on both Firefox and old Microsoft Edge - both are NOT Chromium-based browsers:
• 
  
• Here's a comparison table from https://webcamtests.com:
• 
  
• It works on VLC, but the image's on a different colour palette (not a big deal though)
• 

My conclusion so far is that this is a problem with how Chromium-based browsers handle video input. 

I tried adding different flags to the Chrome launcher, such as the ones below, but to no avail:

• enable-media-foundation-video-capture
  
• run-video-capture-service-in-browser
  
• --enable-features=RunVideoCaptureServiceInBrowserProcess
  
• --disable-web-security

I'll keep looking, but any suggestion would be greatly appreciated.

@Basavaraj Amogi Thanks for the suggestion. The camera doesn't seem to be the problem though (see my notes above). 

I haven't got to the point of reading the temperature - I'm afraid that may take a while, as I couldn't find any sample code in JavaScript to use as a reference.

If you're aware of any code snippet that could help, that would be awesome.

Best regards,

F.

--

Regards,

Fernando Paz Perfecting Clinic Workflow PHONE: +64 22 135 7193 WEB: www.florence.digital EMAIL: fern...@florence.digital ADDRESS: 31 Napier Road, Havelock North, New Zealand 4130

   

Attention: "This communication is confidential and may be legally privileged. If you are not the intended recipient, (i) please do not read or disclose to others, (ii) please notify the sender by reply mail, and (iii) please delete this communication from your system. Failure to follow this process may be unlawful. Thank you for your cooperation."

   

[Kurt Kiefer]

So, it looks like in the Chromium browser, it is actually grabbing the raw 16-bit data WITH the two lines of telemetry (which is why the contrast isn't stretched properly). It might look weird on the screen, but if you're planning on doing a custom app (and you have access to that data, this I'm not sure about, and would be pretty key), this is actually really cool and unexpected, and would allow you to do your own colorization and take radiometric measurements. Now, I'm not sure how you could convince it to grab a different format. If you use another application to set the format, and then open it in Chrome maybe?

[Fernando Paz]

@Kurt Kiefer interesting right? I tried using the camera with both VLC and Chrome, but regardless of which one I open first, the second app always fails with an error message like "device is already in use".

What you said makes sense, but something tells me that Chromium is not doing this deliberately, specially because the integration looks quite buggy (notice how there's a trail from the previous tab at the bottom of the video when I switch tabs).

I was really hoping that some flags on Chrome would be the key to sort this out, so I'll keep looking at that.

Also I haven't read all the docs yet so I'm not completely sure what you meant regarding the telemetry data, hopefully I'll figure it out soon.

@Basavaraj Amogi, thanks again, definitely that will be my next stop after I fix the thermal image. 

Best regards,

F.

You received this message because you are subscribed to a topic in the Google Groups "Flir Lepton" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flir-lepton/88PrZI1-A8I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flir-lepton...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flir-lepton/a30f4f49-4137-4ae6-a50f-a7076cd0d000n%40googlegroups.com.

--

Regards,

Fernando Paz Perfecting Clinic Workflow PHONE: +64 22 135 7193 WEB: www.florence.digital EMAIL: fern...@florence.digital ADDRESS: 31 Napier Road, Havelock North, New Zealand 4130

   

Attention: "This communication is confidential and may be legally privileged. If you are not the intended recipient, (i) please do not read or disclose to others, (ii) please notify the sender by reply mail, and (iii) please delete this communication from your system. Failure to follow this process may be unlawful. Thank you for your cooperation."

[Fernando Paz]

Hi everyone,

Just an update about this issue:

I found this tool from WebRTC Hacks that tests your webcams at different resolutions, and the PureThermal actually worked with QQVGA (160x120):

So I updated my code to force that resolution and now Chrome (on Windows) is showing the right thermal image.

However, my Chrome OS - which is running Chrome 84.0.4147.127 - is still showing just that grey-ish image.

In the end I asked this question on StackOverflow with the hope someone could come up with an idea to workaround this issue. It includes a sample code to reproduce it on a web page.

[Kurt Kiefer]

Does the browser think it is capturing a depth format buffer? You may be able to set the constraints to disable depth, or you might have a look at this, for an example for how to deal with a buffer with arbitrary data and render it yourself:

https://01.org/chromium/blogs/astojilj/2017/depth-camera-capture-html5

On Aug 30, 2020, at 9:19 PM, Fernando Paz <fern...@florence.digital> wrote:

Hi everyone,

Just an update about this issue:

I found this tool from WebRTC Hacks that tests your webcams at different resolutions, and the PureThermal actually worked with QQVGA (160x120):

<image.png>

So I updated my code to force that resolution and now Chrome (on Windows) is showing the right thermal image.

However, my Chrome OS - which is running Chrome 84.0.4147.127 - is still showing just that grey-ish image.

In the end I asked this question on StackOverflow with the hope someone could come up with an idea to workaround this issue. It includes a sample code to reproduce it on a web page.

On Fri, 28 Aug 2020 at 12:47, Fernando Paz <fern...@florence.digital> wrote:

@Kurt Kiefer interesting right? I tried using the camera with both VLC and Chrome, but regardless of which one I open first, the second app always fails with an error message like "device is already in use".

What you said makes sense, but something tells me that Chromium is not doing this deliberately, specially because the integration looks quite buggy (notice how there's a trail from the previous tab at the bottom of the video when I switch tabs).

To view this discussion on the web visit https://groups.google.com/d/msgid/flir-lepton/CADfRHny5siqZ%3D0XpyCrQQ8agGan_W49SE1gr7m8L1vJ2CufRpg%40mail.gmail.com.

[Fernando Paz]

Thanks Kurt. I couldn't find any constraint related to depth so I'm afraid I'll need to handle the buffer manually as you're suggesting.

Wasn't expecting to do that, but after checking out the PureThermal UVC Capture examples it seems I need to do low level operations anyway to capture the temperature in JavaScript.

To view this discussion on the web visit https://groups.google.com/d/msgid/flir-lepton/9791B145-A833-45BF-96FF-5BAA998CC153%40gmail.com.

[Fernando Paz]

Lepton's FAQ mentions that the grey image is probably the thermal image pre AGC. Why that's not being done properly only on Chrome OS remains a mystery.

I've decided to focus on temperature capturing, but I'm struggling to understand how it can be done.

• This datasheet says I could collect "Optional radiometry for temperature values of every pixel". Does "optional" here mean I can enable/disabled it? If so, how can I ensure it's enabled?
• Later on the same document describes the Output Format as "User-selectable 14-bit, 8-bit (AGC applied), or 24-bit RGB (AGC and colorization applied)". How can I select that? And how can I tell I'm using one option over the others?
• This other datasheet explains a little bit about the structure of the telemetry data, but how do I select the Telemetry mode (e.g.: header, footer)? And how to read the temperature from it? 
• I had a look at the Python and C++ samples on GitHub, but it's not clear to me how the temperature is being retrieved from the video frames. I think I'll need to read one specific channel of each pixel in each frame, and then somehow calculate the temperature in Kelvin. Is my assumption correct?

Sorry for all the questions, I'm just not used to programming at this low-level, so any help would be greatly appreciated.

[Fernando Paz]

@Kurt Kiefer you mentioned in an earlier post the following:

(...) it looks like in the Chromium browser, it is actually grabbing the raw 16-bit data WITH the two lines of telemetry (which is why the contrast isn't stretched properly) (...)

Would you care to explain what these 2 lines of telemetry are? I can't make sense of the datasheets I found (see my previous post), perhaps I'm missing some other documentation? 

I thought the temperature could be captured by getting each pixel of the image, reading the red channel (represented in 2 bytes), and finally normalising it to get the temperature em Kelvin. If that's correct, why would I need these 2 lines of telemetry and where are they coming from?

[Fernando Paz]

I had more progress with the web integration over the weekend, but so far it all points to an issue with Chrome OS.

Following Kurt's suggestion, I came up with this code to read the 16-bit pixel value from the red channel before drawing it onto an HTML canvas (which supports 8-bit only). 

The logs show the value for the first pixel in each frame, and as expected it varies from 0.0 to 1.0 instead of 0 to 255. It works on Windows, but on Chrome I only get values around 0.45 (thus the grey image) which means the image is already being streamed wrong.

I ended up raising the issues below but don't have much hope on getting answers any soon:

• Issue on Chromium bug tracker: https://bugs.chromium.org/p/chromium/issues/detail?id=1125329
• Question on Chromebook Help Centre: https://support.google.com/chromebook/thread/68940451?hl=en
• Question on StackOverflow: https://stackoverflow.com/questions/63663370/wrong-image-colour-when-streaming-from-16-bit-depth-webcam-on-chrome-os

With that being said, my question from the previous posts remains: is this the only way to read the temperature on this camera? If not, what else can be done?

[Kurt Kiefer]

I looked through this really quickly and I didn't see any normalization done in there. When you're dealing with thermal data, you have to contrast-extend the images if you want to see something. On a Lepton 2.5/3.5 your 16-bit values represent temperatures in centikelvin (kelvin * 100), so for example, a pixel that is at 29.5C would be a value of 30265, which is right around 0.45 * 65535, the values you're seeing. The whole scene is going to be around this temperature, so you have to scale these values up so that you have contrast in the range you're interested in seeing. You can do this based on max/min in the scene, or you can set a fixed max and min, and make those values be 0.0 and 1.0, then offset and scale the data into that range.

[Fernando Paz]

Thanks for the response Kurt. 

That code example isn't covering the normalization part, I just wanted to confirm that pixel channels can be read in 16-bit in HTML with the help of WebGL.

If I got what you're saying, the only way to capture the temperature is indeed reading the red channel in a 0-65535 range, which is not possible at the moment on Chrome OS because for some reason it's streaming an image in a much shorter range.

I'm waiting for my contacts at Google to come back to me to confirm whether or not this is a problem on their end, will keep you guys posted. 

To view this discussion on the web visit https://groups.google.com/d/msgid/flir-lepton/005f0010-e343-4cbe-95eb-148a3c448c19n%40googlegroups.com.

--

Regards,

Fernando Paz Perfecting Clinic Workflow PHONE: +64 22 135 7193 WEB: www.florence.digital EMAIL: fern...@florence.digital ADDRESS: 31 Napier Road, Havelock North, New Zealand 4130

   

Attention: "This communication is confidential and may be legally privileged. If you are not the intended recipient, (i) please do not read or disclose to others, (ii) please notify the sender by reply mail, and (iii) please delete this communication from your system. Failure to follow this process may be unlawful. Thank you for your cooperation."

[Fernando Paz]

@Kurt After reading your post again, I decided to do another experiment. Even though the value range on Chrome OS is (apparently) shorter than on Windows, it's still long enough when I scale up as you suggested. 

If I set 0.44 to be the new min and 0.47 the new max, I can still get a good range and parse it to 0-255 to display the colours correctly, which is what I did here and it worked:

Why Chrome OS is capturing a different range is still unknown to me, but at least this "calibration" is something I can deal with on my web app.

Thanks so much for your help