Telemetry record_wpr still fails with error of 404 Not Found when record https pages.

[yaojie yan]

Hi

I am newbie of telemetry, I try to record some https pages with record_wpr, but it fails with error of “404, Not Found”, Could anyone help with me if you can successfully record https website?

My recording environment

1.      Chromium android repo,

commit b162d02e0a80252119ff75bf00c4d2d848d55cde

Author: blink-deps-roller <blink-de...@chromium.org>

Date:   Fri Jan 16 00:27:04 2015 -0800

2.      Host

       Ubuntu 13.04 + python 2.7.4

3.      Device: Mi Pad(non-rooted) + android(the spec: http://www.mi.com/mipad/)

 Steps:

1.      Set the https and http proxy by "export https_proxy=http://xxx.com:xxx and http_proxy=http://xxx.com:xxx" in host

2.      Run “tools/perf/record_wpr -v --browser=android-jb-system-chrome polymer_page_set” to record the page.

For http page set, it run correctly, but later, it fails with 404 Not Found, when to load "https://www.polymer-project.org"

I attached the adb logcat(adb_log_mi.log) and the output of "record_wpr -v" log of host(host_https_404_wpr_err.log), could you help me with this.

Thanks
Yaojie

[yaojie yan]

+1

I am looking forward to your guidance.:)

在 2015年1月26日星期一 UTC+8下午4:55:24，yaojie yan写道：

Hi

I am newbie of telemetry, I try to record some https pages with record_wpr, but it fails with error of “404, Not Found”, Could anyone help with me if you can successfully record https website?

My recording environment

1.      Chromium android repo,

commit b162d02e0a80252119ff75bf00c4d2d848d55cde

Author: blink-deps-roller <blink-deps-roller@chromium.org>

[Tony Gentilcore]

Do you have the ability to test whether it works without the proxy? It's possible that WPR gets confused by the proxy and I doubt that code path is well used.

BTW, if you haven't seen it, there is some documentation here:

http://www.chromium.org/developers/telemetry/record_a_page_set

-Tony

[Akshay Aurora]

Facing the same issue. Getting SSLError in logs. 

Traceback - http://gist.github.com/0b47271de2d2d717df6b

On Thursday, February 5, 2015 at 6:31:44 AM UTC+5:30, Tony Gentilcore wrote:

Do you have the ability to test whether it works without the proxy? It's possible that WPR gets confused by the proxy and I doubt that code path is well used.

BTW, if you haven't seen it, there is some documentation here:

http://www.chromium.org/developers/telemetry/record_a_page_set

-Tony

On Wed Feb 04 2015 at 12:42:20 AM yaojie yan <yaoji...@gmail.com> wrote:

+1

I am looking forward to your guidance.:)

在 2015年1月26日星期一 UTC+8下午4:55:24，yaojie yan写道：

Hi

I am newbie of telemetry, I try to record some https pages with record_wpr, but it fails with error of “404, Not Found”, Could anyone help with me if you can successfully record https website?

My recording environment

1.      Chromium android repo,

commit b162d02e0a80252119ff75bf00c4d2d848d55cde

Author: blink-deps-roller <blink-de...@chromium.org>

[Stephen Lamm]

As Tony mentioned earlier, the Chromium team does not use WPR with a proxy, so we do not have experience with that.

On thing I notice in the "Steps" listed earlier is that the https_proxy env variable is set to an "http:" URL. That seems problematic.

-slamm

[Akshay Aurora]

Hi Stephen,

How can I record web pages using WPR without proxy? 

On OSX, I changed system proxy settings and on Ubuntu I changed proxy via command line parameter. So, don't think the https_proxy env variable should be problematic.

--

- Akshay Aurora

[Stephen Lamm]

Do you have a network administrator that you can ask? I am unsure why you need a proxy on your network.

What is a proxy server? https://kb.iu.edu/d/ahoo

I added the proxy code based on a patch that a user submitted. I do not plan to work on this as it is not needed for my current projects. However, if you debug it and figure out a patch, I can work on integrating the fix.

Best,
-slamm

[Akshay Aurora]

I don't have proxy on my network, I'm just following the instructions in web-page-replay docs[0]

[0] - https://github.com/chromium/web-page-replay/blob/master/documentation/GettingStarted.md

--

- Akshay Aurora

[Akshay Aurora]

So, I tried recording pagesets using telemetry[0] with this command.

➜  telemetry  ./record_wpr --browser=system top25_smooth_page_set

The first request made is - https://www.google.com/#hl=en&q=barack+obama

When I opened the browser, its showed 404 not found error. Screenshot attached.

I checked the web page replay logs, and its shows SSL Handshake error. 

Traceback - https://gist.github.com/iakshay/9b5b83143242d2f15024

I tried opening other https websites and got the same error, while http websites worked fine. Is there anything else I need to configure to record https websites?

[0] - http://www.chromium.org/developers/telemetry/run_locally

--

- Akshay Aurora

[Akshay Aurora]

Would be great, if you guys could help. I'm just following instructions in telemetry documentation for recording pages.

--

- Akshay Aurora

[Stephen Lamm]

A Google engineer ran into what seems to be the same problem. 

The 404 error is actually generated by WPR (shows that the browser is connecting to WPR fine). However, when WPR goes to fetch the original resource, it is unable to do so.

We are still investigating why that is the case.

-slamm

[Akshay Aurora]

Thanks for the update Steve. Let me know when you guys fix the issue, been stuck on this for a while.

I got 404 not found in browser and SSL handshake error from server when I tried recording using telemetry. 

When using WPR alone - I got no data sent in browser and SSL Proxy Error from server.

--

- Akshay Aurora

[Akshay Aurora]

Is there a bug I can follow, for updates on this?

--
- Akshay Aurora

[Akshay Aurora]

Is anybody working on this issue?

--

- Akshay Aurora

[jamshed...@gmail.com]

Wanted to bump this issue and see if it has been resolved?

[Sumeet Agrawal]

Hi ,
I am facing the same issue from telemetry in windows . I followed the same documentation for wpr which worked for me in linux. If you have solved the issue in windows kindly help ...

[Ulan Degenbaev]

I can reproduce this issue on macbook with python 2.7.10. Recording the same page set works fine on linux with python 2.7.6:

The command line: tools/perf/record_wpr --browser-executable="$CHROME" --extra-browser-args="--no-sandbox" v8_infinite_scroll

I think the issue is caused by certificate checks enabled by default in python ssl library in 2.7.9: https://www.python.org/dev/peps/pep-0476/

If I patch the replay.py with the workaround from pep-0476, then recording starts working on macbook with 2.7.10 for all pages except twitter.

The twitter page is complaining about mismatch between hostnames provided by SNI and HTTP. Could anyone help with this error?

[Sumeet Agrawal]

Hi Ulan,
Is your patch working for windows??
If yes , can you please mail me the patch replay.py , coz i want to access youtube.com from telemetry.

Thanks

[Sarath Singapati]

I am facing the same issue on Android Chromium / content-shell, when recording https:// pages from Ubuntu machine.

https://groups.google.com/a/chromium.org/forum/#!topic/telemetry/iPRV-I0yVCU

Also attached the net-log.

Hi Ulan, Could you please share what workaround you have done in replay.py? I am using Python 2.7.11+

Thank you.

[Ned]

Hmhh, I think this should be fixed by https://github.com/chromium/web-page-replay/commit/7e572527906e28e819c643c97d13c1725536a719

Can you pull the latest WPR?

--
You received this message because you are subscribed to the Google Groups "telemetry" group.
To unsubscribe from this group and stop receiving emails from it, send an email to telemetry+...@chromium.org.

[Sarath Singapati]

Works. Great! Thank you very much!