Curl check before download and Multithreading Question

76 views
Skip to first unread message

curlppnoob

unread,
May 26, 2012, 2:43:06 PM5/26/12
to cur...@googlegroups.com
Hello Curl Community,

i'm building a webcrawler, now i need to tune, my crawler, the crawler was build in php, but i have some problems.

Is this in Cpp possible:

e.g. I make a request to http://domain.com/4/dir/somepdf.pdf

When my crawler makes a request to that url, and the header returns, a for example application/pdf ,  than I want, that curl does not download the file (in one request). (I just want only text/html to download). In php i could do this in two requests like just receive the header, check, if ok than make a new request. I want todo this in one request check, stop or go.
The same applies to 404, 403, 503 HTTP errors i dont want fetch the body if the header returns a these errors.

Real Multi-threading, on PHP you dont have a real multi-threading, curl is so fast as your slowest request. Is this in Cpp possible to make a real multi threaded application?

As you can see im coming from PHP, does the cpp client has some other feat, which the php not has? which are worth to mention?

Thanks for help








Reply all
Reply to author
Forward
0 new messages