setting curl options / non-blocking Hyrdra.run

[Adam]

Where do you set curl options like CURL_USERPWD?

I'm trying to retrieve urls from servers that need authentication....

Also, is there a way to do a something more akin to a select instead
of a run? Since Hyrdra.run will block, it would be nice to re-queue
connections that have completed rather than having to wait for all the
queued requests to complete or timeout.

Thanks,
Adam

[Billy Doughty]

Where do you set curl options like CURL_USERPWD?

I'm trying to retrieve urls from servers that need authentication....

If basic authentication is all you require, then you can do that by setting the request headers.

    login = "somelogin"
    password = "somepassword"
    b64_encoded = Base64.b64encode("#{login}:#{password}")
    headers = {"Authorization" => "Basic #{b64_encoded}"}
    request = Typhoeus::Request.new("http://twitter.com/statuses/user_timeline.json?screen_name=username&page=1&count=200",
          :headers => headers
    )


 

Also, is there a way to do a something more akin to a select instead
of a run?  Since Hyrdra.run will block, it would be nice to re-queue
connections that have completed rather than having to wait for all the
queued requests to complete or timeout.

I am dealing with a similar problem.  The solution I chose was to have each request oncomplete block check a "work" queue before returning.  Items in the work queue are turned into new requests and queued on the hydra before the current request exits.  The hydra.run only needs to be called once as long as each request generates another request.  If there are no more items in the work queue, hydra.run will exit once all existing requests complete.


--Bill

[Adam]

Thanks. The auth part worked like a charm.

I'm having some trouble with the on_complete. I create a req_hash
with some things related to the current request, like next run time.
It seems the on_complete method only has access to the most recently
updated req_hash. This is strange because I was expecting each
on_complete to have it's own copy of the req_hash based on Ruby
closure rules, which I may not understand correctly...

Here's the pseudo code:

req = Typhoeus::Request.new(c.snapshot_url)

req_hash = { # stuff I want to keep related to this request ...}

req.on_complete do |response|
# process the response
# updated req_hash with next run time
# process wait queue
end

Any help would be appreciated.

Regards,
Adam

On Nov 25, 7:07 am, Billy Doughty <billyd....@gmail.com> wrote:
> > Where do you set curl options like CURL_USERPWD?
>
> > I'm trying to retrieve urls from servers that need authentication....
>
> If basic authentication is all you require, then you can do that by setting
> the request headers.
>
>     login = "somelogin"
>     password = "somepassword"
>     b64_encoded = Base64.b64encode("#{login}:#{password}")
>     headers = {"Authorization" => "Basic #{b64_encoded}"}

>     request = Typhoeus::Request.new("http://twitter.com/statuses/user_timeline.json?screen_name=username&p...

[Adam]

I turned off memoization and it seemed to fix things. I'm not sure
why though. I have different requests going to different servers....