Error: is not JSON serializable

1133 views
Skip to first unread message

Anne

unread,
May 13, 2011, 6:08:48 AM5/13/11
to scrapy-users
Hi all,

I'm rather new to scrapy, but managed to get a spider up and running
without too much effort: great work, thanks!

I would love to (at least) monitor my scraper using the webservice (as
described on http://doc.scrapy.org/topics/webservice.html). So I've
enabled it in my settings.py, by setting WEBSERVICE_ENABLED = True.

Now, if I point my browser at http://localhost:6080/crawler (as
mentioned in the documentation) I get a traceback with essentially
this error:
<type 'exceptions.TypeError'>: <scrapy.crawler.CrawlerProcess object
at 0x23d9a50> is not JSON serializable

The same goes for http://localhost:6080/stats. But http://localhost:6080/enginestatus,
on the other hand, seems fine.

What am I doing wrong?

Best,
Anne

Takahiro Fujiwara

unread,
May 13, 2011, 12:13:52 PM5/13/11
to scrapy...@googlegroups.com
Hi, Anne.

You are misunderstanding about scrapy webservice.
It means web API service like Amazon eCommerce or Twitter API etc.
You can access to information though scrapy webservice using JSON-RPC 2.0 protocol.

You need to create a monitoring web application if you get it from your web browser.
I found this app created another developer on this forum but forgot it.
If you search it, you can find it maybe.

-- 
Takahiro Fujiwara
Sent with Sparrow
--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

Anne Schuth

unread,
May 14, 2011, 4:22:15 AM5/14/11
to scrapy...@googlegroups.com
Hi Takahiro,

Thank you for answering me!  I'm not sure I misunderstood how the webservice worked. I was expecting to receive JSON instead of an error when I pointed my browser to http://localhost:6080/crawler. And in fact I did receive JSON when browsing to http://localhost:6080/enginestatus. I understand I will have to write (or indeed find) a client that talks the JSON-RPC 2.0 protocol. But shouldn't  scrapy produce JSON instead of the error I mentioned?

Best,
Anne

Takahiro Fujiwara

unread,
May 14, 2011, 7:23:45 AM5/14/11
to scrapy...@googlegroups.com
Oh, sorry Anne.

I understand that you wanna say and I was misunderstanding you said.
I have not use enginestatus API yet.
Sorry, I can't help you.

-- 
Takahiro Fujiwara
Sent with Sparrow

Amir F

unread,
Apr 28, 2012, 12:10:50 AM4/28/12
to scrapy...@googlegroups.com
Did you figure out what was wrong? I have the same problem.
To unsubscribe from this group, send email to scrapy-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users+unsubscribe@googlegroups.com.

Steven Almeroth

unread,
Apr 29, 2012, 2:54:56 PM4/29/12
to scrapy...@googlegroups.com
It seems there is a bug in the way Twisted is processing the scrapy.crawler.CrawlerProcess object.  In the method twisted.web.resource.getChildForRequest() the resource.isLeaf is not True and thus the request.postpath value of ['crawler'] is being rendered, instead of the scrapy.webservice.RootResource instance, which, when rendered, gives the JSON object {"resources": ["enginestatus", "stats", "crawler"]}.

I will inform the developers of this issue, and thanks for reporting.

--steven

Pablo Hoffman

unread,
May 3, 2012, 11:10:57 AM5/3/12
to scrapy...@googlegroups.com, Steven Almeroth
There was an issue in the webservice code which I fixed here (also fixed in stable branch for next release):
https://github.com/scrapy/scrapy/commit/9c3b9f29687c78b56825abcc78f4faa326a90e0a

Still, you shouldn't be able to access http://localhost:6080/crawler directly, as the object itself is not JSON serializable. Instead you can do a JSON-RPC call to invoke a method of it. See the extras/scrapy-ws.py script for some examples.

Pablo.
--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/JFv-2bTCEoIJ.

To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.

Παναγιώτης Αγγελόπουλος

unread,
Dec 20, 2014, 12:23:39 PM12/20/14
to scrapy...@googlegroups.com, sro...@gmail.com

Hi, I still having this issue.

I'm using python 2.7.8 (Linux), scrapy 0.24.4 and i'm getting the following error through localhost:6080/crawler


exceptions.TypeError: <scrapy.crawler.Crawler object at 0x7f23ccf44e90> is not JSON serializable
/usr/local/lib/python2.7/dist-packages/twisted/web/server.py:189 in process
 
 
188                    self._encoder = encoder
 
189            self.render(resrc)
 
190        except:


Also, scrapy command tool shows the following piece of code :


 [TelnetTransport,0,127.0.0.1] Unhandled Error
   
Traceback (most recent call last):
   
Failure: twisted.internet.error.ConnectionDone: Connection was closed cleanly.

[...]
File "/usr/lib/python2.7/json/encoder.py", line 184, in default
       
raise TypeError(repr(o) + " is not JSON serializable")
    exceptions
.TypeError: <scrapy.crawler.Crawler object at 0x7f23ccf44e90> is not JSON serializable



Any ideas?
Reply all
Reply to author
Forward
0 new messages