Scrapy with NewRelic

[Anderson Caco]

Has anyone here used Scrapy with NewRelic? Is the twisted.internet.reactor fully supported by the NewRelic agent?

Thanks!

--

Anderson Ferraz

[Pablo Hoffman]

They say the have experimental support for Twisted.WEB:

https://newrelic.com/docs/python/status-of-python-agent

Which may work for Scrapy too, or be made more generic to support any Twisted async app.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Anderson Caco]

I almost forgot! I actually submited a request ticket to NewRelic, which they responded (requires account):

https://support.newrelic.com/requests/41771

Basically, they gave me some configs I could try, but it seems that they would not work with Scrapy. "(...) Twisted support is experimental (...) these config options are designed to work in the context of a web application using twisted.web framework. If you're directly hooking into the twisted.internet.reactor engine you might not see any data from the agent."

2013/6/5 Pablo Hoffman <pa...@scrapinghub.com>

-- 
Anderson Ferraz

[Pablo Hoffman]

Are you planning to use it for logging errors or also performance metrics? (I've never used NewRelic btw). If it's for errors, I would recommend Sentry, whose client has twisted async builtin support.

[Jordi Llonch]

Feel free to use https://pypi.python.org/pypi/scrapy-sentry

2013/6/6 Pablo Hoffman <pa...@scrapinghub.com>

[Anderson Caco]

Great to see that there is already a nice Scrapy/Sentry integration! We are definetely going to use it. But we also want a performance dashboard, and NewRelic is amazing for this.

Many thanks, Jordi and Pablo!

2013/6/5 Jordi Llonch <llo...@gmail.com>

--

Anderson Ferraz

[Jordi Llonch]

NewRelic is a impressive product. Please also consider https://github.com/noplay/scrapy-graphite and graphite to track crawl stats.

2013/6/6 Anderson Caco <ca...@jusbrasil.com.br>

[Anderson Caco]

Priceless recommendations! Shouldn't something like "related projects" be present in the docs? I can create such document and this aggregates value to Scrapy.

2013/6/5 Jordi Llonch <llo...@gmail.com>

--

Anderson Ferraz

[Jordi Llonch]

You're right! Maybe a section in scrapy.org will be useful.

2013/6/6 Anderson Caco <ca...@jusbrasil.com.br>