Golang tool/package for Common Crawl data extraction

[Rustem Kamalov]

Hi guys!

I've developed a solution that can help you extract data from Common Crawl :)
You can use it as a separate tool, or import it into your Go project.
Github: https://github.com/karust/gogetcrawl

I hope the package will be useful to someone!

[LUCKY OKEWORO]

This is amazing, great work

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/cd219df8-36db-4e4b-a986-0b72e7ead05cn%40googlegroups.com.

[Sebastian Nagel]

Hi Rustem,

thanks! I've added the tool to the list of code examples, tools and
libraries on the Common Crawl website:
https://commoncrawl.org/the-data/examples/

Best,
Sebastian

[LUCKY OKEWORO]

This is wonderful

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/d1e35ce3-9b45-c0f2-3806-47c4eb63e111%40commoncrawl.org.

[tauseef jamal]

Great work 

To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/CAFHVMcY0kjOLRqVq8P_4OUuPbHogzq9WLbrqaFgPtoJ7nGpZJw%40mail.gmail.com.

[Wesam Al-Nabki]

Many thanks for this great tool. 

I understood from the --ext we can download files or HTML pages, can we use it to download the WET (the clean text) files of a website? Also, can we set a date range "from" "to" date-stamps?

Regards,

Wesam 

[Rustem Kamalov]

Hi Wesam,

You are right, we can set the file extension to search/download via `--ext` (you can also do it via `--filter` and MIME type).
I updated the solution, now you can set a date range using `--from` and `--to` arguments.

We have "length" and "offset" params in JSON response of the Index server for WARC archive, which allow us to download a certain page.
Currently, I'm not sure how I can do it for WET, because WET archive should have different offsets...

Regards,
Rustem

воскресенье, 4 июня 2023 г. в 14:30:23 UTC+3, wesam....@gmail.com: