News-Crawler: Is there an option to use HTTP/1.0 or 1.1 for the Warc File

[Yuxin Zhu]

Hi there,

I tried to use the news-crawler, and the produced Warc files have HTTP/2 in the status line. I'm wondering if there's an option in the config files to use HTTP/1.0 or 1.1? Thanks in advance.

[Sebastian Nagel]

The easiest way (without code modifications) is to use Java 8 because there is no support for HTTP/2 and okhttp will fall back to HTTP/1.1

Otherwise you need to modify the code in the method
  com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol#configure(Config conf)
and define which protocols [1] are allowed. See Nutch's protocol-okhttp as example [2].

Please open an issue on [3] to make the selection of the protocol configurable via the crawler-conf.yaml.

Of course, you could also fix the HTTP status line written to WARC files. This would happen at [4].

Best,

Sebastian

[1] https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/-builder/protocols/

[2] https://github.com/apache/nutch/blob/680df6ba1dc68ad5ede5fca743304593d4d5b0a3/src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttp.java#L113

[3] https://github.com/DigitalPebble/storm-crawler/issues/

[4] https://github.com/DigitalPebble/storm-crawler/blob/54bb5934918ff0f8919d58372405cfb14d63ad93/external/warc/src/main/java/com/digitalpebble/stormcrawler/warc/WARCRecordFormat.java#L175

[Yuxin Zhu]

Hi Sebastian,

Thanks so much for the detailed reply. I have another question(might be a little dumb)...I am using Newscrawler instead of StormCrawler. In the News-crawler's github page, in the prerequisites section, you mentioned "Clone and compile StormCrawler". Where should I clone the StormCrawler repo to? Should it be placed as a subdirectory under the NewsCrawler repo? Thanks.

Yuxin

[DigitalPebble]

Hi Yuxin

I don't think this is necessary anymore as the news crawler relies on a released version of SC and pulls it like any other dependency.

BTW you'd get a wider audience if you ask questions on StackOverflow

Julien

--
You received this message because you are subscribed to the Google Groups "DigitalPebble" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalpebbl...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/digitalpebble/5be5b503-b018-4b11-8284-3df38736201en%40googlegroups.com.

--

 
Open Source Solutions for Text Engineering
 
http://www.digitalpebble.com

http://digitalpebble.blogspot.com

@digitalpebble

[Sebastian Nagel]

Yes, the master branch now - since there have been complaints about unstable versions -  always relies on a released SC package which is pulled from a Maven repository. If you want to use a development version of SC you'd need to compile SC and "install" it to your local Maven repository (~/.m2/repository/ on Linux). I'll update the news-crawl README and mark this option as "expert".

[Sebastian Nagel]

Update and correction:

- recent Java 8 JDK packages may support HTTP/2 if they include the ALPN backport, see

   https://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-January/011042.html

   https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation

- I'll update the WARC bolt soon to allow to restrict the protocol version used

- but also to make the WARC record writer "fake" the recorded HTTP header

Sebastian