Problem with Webviewer: "header fields are too long for the server to interpret"

[Frank Bröker]

Hello,

I'm trying to see a webstream with the webviewer (inculded in Appinventor).

The problem is:

When I try the applikation with the connection "all Companion" and scan the code with my mobile, erverything is fine: I can see the webstream. perfect.

But, after I created a apk-file and install the apk ob my phone, the web Viewer says: "Header fileds are too long for the server to interpret"

I have tried a lot of things, but nothing helps.

any idea?

thanks

Frank

[TimAI2]

Please show your relevant blocks, so we can see what you are trying to do

Right Click in Blocks Editor and select "Download Blocks as Image"

(use image editor to crop etc. if required)

In the forum, click on the image button to upload your blocks image

[Frank Bröker]

HI,

that was vwery fast.

attached You can see what I'm doing.

The basic Idea is, to view a webstream from a ESP32Cam Module.

best regards

Frank

[BodyMindPower]

This is not a problem that could be solved with AI2 (via the webviewer). It seems that every browser is affected by this, which try to access the ESP32 server.

https://github.com/espressif/arduino-esp32/issues/2983

[Ghica]

Look also in this thread, there it seems to be working: https://groups.google.com/forum/#!searchin/mitappinventortest/esp32$20cam%7Csort:date/mitappinventortest/9Tih_KHmQ9Q/7vC3FxumAgAJ

Cheers, Ghica.

[Frank Bröker]

the strange thing is, that I can see the stream when I'm connected to he phone (with "all companion") see attached picture, but not, when I create a apk (with the same programm)

See attached pic.

That is very strange.

[Chris Ward]

Perhaps the error message is one of a few in a list and is actually not an accurate account of the issue. 

I think we need to see all of your blocks Frank. There are of course differences between the Companion and an APK. In particular, file paths and permissions.

[Frank Bröker]

Hi Chris,

thank you.

Attached You can see all my blocks:

PS: The LED Control (switch LED on and off) is working (in both cases)

[Frank Bröker]

Hi to all,

I can't belive it, but it is working now.

What was the solution:

I'm using a Samsung Smartphone with the Samsung Internet Browser. That doesn't work!!!

Now I activated (installed) the Chrome Browser and see: It is working.

So, the problem was the browser App on the phone.

I didn't change anything on the AppInventor Programm.

Absolutely perfect. Now the webstream is on my phone.

thanks to all.

Best regards

Frank

[massimiliano comunalemassimiliano@live.com]

Is there any way to make the background webwiever transparent?

[Chris Ward]

Hi Frank

That's good news, but equally weird - should have no effect at all, given that the companion worked and your App is not using the phone's Browser anyway. Except perhaps installing Chrome has changed or added a security/privacy permission........

It will be interesting to learn what Evan thinks. 

[Chris Ward]

Hi massimiliano

You should open your own topic, not hijack this one which is a different subject.

[Frank Bröker]

bad News!

I had to update the chrome Browser (because the version, which I activated, was to old) and now:

I have the problem again.

Message in the Webviewer: "Header Fileds are too long for server to interpret"

What a bad idea to update the Chrome Browser. It must be really a security problem, in combination with the AppInventor App and the Browser.

So, any further ideas?

Regards

Frank

[Chris Ward]

Hi Frank

I suspect it is a permissions issue. I know the ESP error message does not suggest that but very often these things have not been written by an English speaker and mistakes are easily made, or the error was unknown and the developer did not make provision for that scenario.

Let's wait for Evan's input. There is a good chance that he can shine a light on the problem. If it works with the Companion, we should be able to make it work as an APK.

[Frank Bröker]

Hi Chris,

thanks for your help!!

It is really reproducible. I deinstalled the chrom Browser. ( I can't deinstall it completely, because it's part of the Android OS) but with the "old" version, it is working. With the new update not.

That is really weired.

I'm really glad to hear from "Evan". Hopefully he can help.

What I did in the last minutes: I made a wire Shark record, to see what's going on. And I think the request (in the non working case) didn't leave the phone. I saw also in the ESP that there is no incomming http request.

maybe that helps.

regards

Frank

[Chris Ward]

Hi Frank

Something to try now: Switch on 'Location'. 

[Chris Ward]

... on my Samsung it's in Settings, Biometrics and security, Privacy, Location.

[Frank Bröker]

Hi Chris,

I tried it. I updated Chrome again and activated the location, but it doesn't work. Same error message in the Webviewer.

Regards

Frank

[BodyMindPower]

See also here, heading: "Problems with HTTP header fields":

https://www.ionos.com/digitalguide/hosting/technical-matters/http-400-bad-request-finding-the-causes/

[Evan Patton]

Hi Frank,

I poked around a bit and tried to mimic this as best I could given that I don't have a ESP32 Cam. Here are the results of my analysis.

Test apparatus:

A python server running on port 8000 with a single file stream. The IP address of the machine had the same length as the one given in your pictures.

This means that the HTTP request line and the Host headers are exactly the same length as in your scenario.

I used Wireshark on the server to capture incoming packets to the python server. Here's an example of the packet sent from my phone. It may differ slightly from your phone, but I think the point should generally be the same:

GET /stream HTTP/1.1

Host: xxx.xxx.x.xxx:8000

Connection: keep-alive

Upgrade-Insecure-Requests: 1

User-Agent: Mozilla/5.0 (Linux; Android 8.1.0; Pixel Build/OPM1.171019.012; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome 78.0.3904.96 Mobile Safari/537.36

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3

X-Requested-With: edu.mit.appinventor.aicompanion3

Accept-Encoding: gzip, deflate

Accept-Language: en-US,en;q=0.9

Based on what I've found about the ESP32 HTTP server processing of headers, the request line doesn't count against the 512 bytes of headers, so we can start counting from the start of the Host field. The values, including the \r\n characters are as follows:

26

24

30

173

128

52  << this is the X-Requested-With header

32

33

which totals up to 498, 14 bytes shy of the 512 byte limit. If the host page has set cookies, then you'll probably be above 512 and you'll get the error. My guess is that your older version of Chrome that you downgrade to is not sending as many headers, and therefore gives you more flexibility in what works.

So how does this fit into compiled apps? As you can see on the X-Requested-With header we have the package name of the app. Your app's package name will be appinventor.ai_<email address before @>.<project name>, which is likely bound to be longer than edu.mit.appinventor.aicompanion3. For the example app I built, the package name was appinventor.ai_ewpatton.WebviewHeaderTest. The resulting packet was 507 bytes, just 5 bytes short of the limit. If I had made my project name much longer, or had a longer email address, then I would have tripped the issue.

If you send me the version of Chrome you downgrade to in order to get the project to work, I might be able to give you some more insights into what exactly is going on when you upgrade Chrome, but the best course of action here is probably to reload the firmware on your ESP with a bigger allocation for header processing.

Regards,

Evan

[Frank Bröker]

Hi Evan,

thank you very much for your investigation.

And you are absolutely right. I tested it on my system and the Header (in my case) is 528 bytes long. Very long is the field "User-Agent"

The ESP (in the example program) is using the HTTP-Server library.

Unfortunately It was not possible for me to change the max Header length. It is fix set to 512 byte (only:-) )

But:

I found a solution. In this solution the "standard" Webserver-library is used. And here the max header length is adjustable. The default value is 1024byte.

And the result is: it is working. With the "all companion" and also as "*.apk" file.

Attached you can see a phone screenshot, with the working APK-file.

So, AppInventor (with the WebViewer component) is working proberly!

For all, who are running in the same issue, here the link to the ESP32 library (using the ESP WebServer):

https://github.com/yoursunny/esp32cam

Thanks to all and:

AppInventor is awesome. I have build a lot of Apps in combination with Arduino,ESP8266 and ESP32.

best regards

Frank

[Evan Patton]

Hi Frank,

I'm glad that the analysis helped you out. Happy inventing!

Cheers,

Evan

[_ REC]

Hello I'm a university student living in Korea I'm a university project I want to see the live streaming through the web viewer using esp32 cam through the web viewer. Can I see the standard" Webserver-library?

Please send me your answer here.(fla...@naver.com)

[Chris Ward]

_REC

In Frank's post above:

[_ REC]

Hi   Chris Ward

Can it be modified in the Arduino integrated development environment?

Or should I fix it in esp32 idf? 

(I am using Arduino IDE)