Scrapy on scrapinghub strange signal handler error but the scraper seems to work fine
Luca Fiaschi
I have a scraper which uses the Item pipeline to save items in a database.
When running my scraper on scrapinghub I get that all the requests are registered as failed and the following errors are reported.
Traceback (most recent call last):
File "/app/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 149, in maybeDeferred
result = f(*args, **kw)
File "/usr/local/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "/usr/local/lib/python2.7/site-packages/sh_scrapy/extension.py", line 39, in item_scraped
item = self.exporter.export_item(item)
File "/app/python/lib/python2.7/site-packages/scrapy/exporters.py", line 304, in export_item
result = dict(self._get_serialized_fields(item))
File "/app/python/lib/python2.7/site-packages/scrapy/exporters.py", line 65, in _get_serialized_fields
field_iter = six.iterkeys(item)
File "/usr/local/lib/python2.7/site-packages/six.py", line 593, in iterkeys
return d.iterkeys(**kw)
AttributeError: 'NoneType' object has no attribute 'iterkeys'
From the scraping hub dashboard I can see that all items appears as failed (0 processed), however the scraper seems to work fine and serielizes the correct data in the database.
Anybody can explain this?
Does it maybe depend on the fact that I am not retorning items as jsons, but only serializing to database?
Rolando Espinoza
CONFIDENTIALITY NOTICE: This message (including any attachments) is confidential and may be privileged. It may be read, copied and used only by the intended recipient. If you have received it in error please contact the sender (by return e-mail) immediately and delete this message. Any unauthorized use or dissemination of this message in whole or in parts is strictly prohibited.
DISCLAIMER: The information contained in this email, attachment, document and/or presentation have been researched by us with due care and all and any evaluations or assessments stated herein represent our personal opinions. We advise you that some of the available information has been independently verified and may be based on statements by third persons, but no representation or warranty, expressed or implied, is made as to, and no reliance should be placed on, the fairness, accuracy, completeness or correctness of this information or opinions contained herein. Certain statements contained herein may be statements of future expectations and other forward-looking statements that are based on our current personal views and assumptions and involve known and unknown risks and uncertainties that cause actual results, performance or events to differ materially from those expressed or implied in such statements. None of our team shall assume any liability whatsoever (in negligence or otherwise) for any loss howsoever arising from any use of this email, attachment and/or presentation or the statements contained herein as to unverified third person statements, any statements of future expectations and other forward-looking statements, or the fairness, accuracy, completeness or correctness of statements contained herein, or otherwise arising in connection with this email, attachment, document and/or presentation.
This email, attachment, document and/or presentation does not constitute an offer or invitation to purchase or subscribe for any investment and neither it nor any part of it shall form the basis of or be relied upon in connection with any commitment or contract whatsoever, and is solely intended to communicate a favourable impression of our team members and their business track-records. No one should base their investment decisions on this email, attachment, document and/or presentation, which principally constitutes an advertising message, as their main source of information or reliance but should form their personal opinions solely on the basis of further available information and evaluations and assumptions. The legal documentation held ready for any prospective investor shall be authorised as the sole source of relevant information and neither it nor any part of it shall constitute any representation, warranty or other undertaking, expressed or implied, as to the economic success of the offered investments. --
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscribe@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
Luca Fiaschi
Chairman of the Supervisory Board: Prof. Dr. Marcus Englert
Registration Court: Amtsgericht Charlottenburg, Registration No.: HRB 165662 B
You received this message because you are subscribed to a topic in the Google Groups "scrapy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scrapy-users/NNqSw0hBmu8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.