Error: is not JSON serializable

[Anne]

Hi all,

I'm rather new to scrapy, but managed to get a spider up and running
without too much effort: great work, thanks!

I would love to (at least) monitor my scraper using the webservice (as
described on http://doc.scrapy.org/topics/webservice.html). So I've
enabled it in my settings.py, by setting WEBSERVICE_ENABLED = True.

Now, if I point my browser at http://localhost:6080/crawler (as
mentioned in the documentation) I get a traceback with essentially
this error:
<type 'exceptions.TypeError'>: <scrapy.crawler.CrawlerProcess object
at 0x23d9a50> is not JSON serializable

The same goes for http://localhost:6080/stats. But http://localhost:6080/enginestatus,
on the other hand, seems fine.

What am I doing wrong?

Best,
Anne

[Takahiro Fujiwara]

Hi, Anne.

You are misunderstanding about scrapy webservice.

It means web API service like Amazon eCommerce or Twitter API etc.

You can access to information though scrapy webservice using JSON-RPC 2.0 protocol.

You need to create a monitoring web application if you get it from your web browser.

I found this app created another developer on this forum but forgot it.

If you search it, you can find it maybe.

-- 
Takahiro Fujiwara
Sent with Sparrow

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

[Anne Schuth]

Hi Takahiro,

Thank you for answering me!  I'm not sure I misunderstood how the webservice worked. I was expecting to receive JSON instead of an error when I pointed my browser to http://localhost:6080/crawler. And in fact I did receive JSON when browsing to http://localhost:6080/enginestatus. I understand I will have to write (or indeed find) a client that talks the JSON-RPC 2.0 protocol. But shouldn't  scrapy produce JSON instead of the error I mentioned?

Best,

Anne

[Takahiro Fujiwara]

Oh, sorry Anne.

I understand that you wanna say and I was misunderstanding you said.

I have not use enginestatus API yet.

Sorry, I can't help you.

-- 
Takahiro Fujiwara
Sent with Sparrow

[Amir F]

Did you figure out what was wrong? I have the same problem.

To unsubscribe from this group, send email to scrapy-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To post to this group, send email to scrapy...@googlegroups.com.

To unsubscribe from this group, send email to scrapy-users+unsubscribe@googlegroups.com.

[Steven Almeroth]

It seems there is a bug in the way Twisted is processing the scrapy.crawler.CrawlerProcess object.  In the method twisted.web.resource.getChildForRequest() the resource.isLeaf is not True and thus the request.postpath value of ['crawler'] is being rendered, instead of the scrapy.webservice.RootResource instance, which, when rendered, gives the JSON object {"resources": ["enginestatus", "stats", "crawler"]}.

I will inform the developers of this issue, and thanks for reporting.

--steven

[Pablo Hoffman]

There was an issue in the webservice code which I fixed here (also fixed in stable branch for next release):
https://github.com/scrapy/scrapy/commit/9c3b9f29687c78b56825abcc78f4faa326a90e0a

Still, you shouldn't be able to access http://localhost:6080/crawler directly, as the object itself is not JSON serializable. Instead you can do a JSON-RPC call to invoke a method of it. See the extras/scrapy-ws.py script for some examples.

Pablo.

--

You received this message because you are subscribed to the Google Groups "scrapy-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/JFv-2bTCEoIJ.

To post to this group, send email to scrapy...@googlegroups.com.

To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.

[Παναγιώτης Αγγελόπουλος]

Hi, I still having this issue.

I'm using python 2.7.8 (Linux), scrapy 0.24.4 and i'm getting the following error through localhost:6080/crawler

exceptions.TypeError: <scrapy.crawler.Crawler object at 0x7f23ccf44e90> is not JSON serializable

/usr/local/lib/python2.7/dist-packages/twisted/web/server.py:189 in process
 
 188                    self._encoder = encoder
 189            self.render(resrc)
 190        except:

Also, scrapy command tool shows the following piece of code :

 [TelnetTransport,0,127.0.0.1] Unhandled Error
    Traceback (most recent call last):
    Failure: twisted.internet.error.ConnectionDone: Connection was closed cleanly.

[...]

File "/usr/lib/python2.7/json/encoder.py", line 184, in default
        raise TypeError(repr(o) + " is not JSON serializable")
    exceptions.TypeError: <scrapy.crawler.Crawler object at 0x7f23ccf44e90> is not JSON serializable

Any ideas?