Cloufront bandwidth

61 views
Skip to first unread message

AlexGahe

unread,
Aug 30, 2022, 9:39:46 AMAug 30
to Common Crawl
Hi Sebastian, 
Will connection issues with Cloudfront (max bandwidth suddenly down to 1MB/s) be fixed or is it now permanent ?

Regards
AG

Greg Lindahl

unread,
Aug 30, 2022, 3:13:47 PMAug 30
to common...@googlegroups.com
Have you authenticated yourself to cloudfront? There are 2 classes
of downloads with different limits: anonymous and authenticated.
Sebastian's advice has always been to use authentication.

-- greg
> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/common-crawl/d61da213-a9ed-4fd1-887f-d73b75927ce6n%40googlegroups.com.

AlexGahe

unread,
Sep 1, 2022, 5:03:49 AMSep 1
to Common Crawl
Hi, thanks for your help.
Nevertheless, I thought using HTTPS/Cloudfront was intended to be used as the path to access data without authentication. I used this method without any trouble from april to mid august, downloading WARC files in few seconds with wget. Now it takes 18mn per file.
Did something change ?
Is there now something like a limit in terms of volume per month ?

BR

Florian Hantke

unread,
Sep 1, 2022, 6:30:18 AMSep 1
to Common Crawl
Hi all,
I also experienced some download limitations the previous days, even though I never had problems before. It seems to work fine again, at least from my side.
Best, Florian

Lawrence Stewart

unread,
Sep 19, 2022, 10:35:55 PM (6 days ago) Sep 19
to Common Crawl
I've noticed this as well. 

Lawrence Stewart

unread,
Sep 20, 2022, 12:12:34 PM (5 days ago) Sep 20
to Common Crawl
switch to downloading from s3 is a drastic improvement. 
Reply all
Reply to author
Forward
0 new messages