Where is the item returned by the spider?

[Zeynel]

When I run the spider in the command prompt with

python scrapy-ctl.py crawl whitecase.com

I see the FirmItem in the return items:

[whitecase.com] INFO: Passed FirmItem(title=[u'White & Case LLP -
Lawyers - Rachel B. Wagner '])

but where is this item? How do I get it and put it into a file?

Thank you for the help.

[Pablo Hoffman]

By writing an item pipeline to do that:
http://doc.scrapy.org/topics/item-pipeline.html

You can use the builtin File Export Pipeline for that:
http://doc.scrapy.org/topics/item-pipeline.html#module-scrapy.contrib.pipeline.fileexport

Which uses the Item Exporters:
http://doc.scrapy.org/topics/exporters.html

Or you can implement your own item pipeline, either using the Item Exporters or
from scratch.

Pablo.

[Zeynel]

Thanks! I just started to study those. But what is wrong with this
pipeline?

http://dpaste.org/HibR/

Do I need something more than that? If not what do I need to do make
this one work?

On Nov 20, 12:52 pm, Pablo Hoffman <pablohoff...@gmail.com> wrote:
> On Fri, Nov 20, 2009 at 08:39:04AM -0800, Zeynel wrote:
> > When I run the spider in the command prompt with
>
> > python scrapy-ctl.py crawl whitecase.com
>
> > I see the FirmItem in the return items:
>
> > [whitecase.com] INFO: Passed FirmItem(title=[u'White &amp; Case LLP -
> > Lawyers - Rachel B. Wagner '])
>
> > but where is this item? How do I get it and put it into a file?
>
> By writing an item pipeline to do that:http://doc.scrapy.org/topics/item-pipeline.html
>

> You can use the builtin File Export Pipeline for that:http://doc.scrapy.org/topics/item-pipeline.html#module-scrapy.contrib...

[Pablo Hoffman]

On Fri, Nov 20, 2009 at 10:08:51AM -0800, Zeynel wrote:
> Thanks! I just started to study those. But what is wrong with this
> pipeline?
>
> http://dpaste.org/HibR/
>
> Do I need something more than that? If not what do I need to do make
> this one work?

Add it to your project's Item pipelines, by adding this to your settings.py:

ITEM_PIPELINES [
'myproject.pipelines.CsvWritePipeline'
]

[Zeynel]

Yes, thanks. That line is there. I still don't understand how I get
the scraped data into items.csv.

I run the pipelines.py in IDLE but nothing happens. There is a file
items.csv but there is nothing in it :)

[Zeynel]

Hi,

Thanks for your help with this. The first sentence here
http://doc.scrapy.org/topics/item-pipeline.html says

"After an item has been scraped by a spider it is sent to the Item
Pipeline which process it through several components that are executed
sequentially."

So, I didn't have to do anything :) The Pipeline opens the items.cvs
and pastes the scraped data in there. Next, I'll use the File Export
Pipeline with csv.

On Nov 20, 12:52 pm, Pablo Hoffman <pablohoff...@gmail.com> wrote:
> On Fri, Nov 20, 2009 at 08:39:04AM -0800, Zeynel wrote:
> > When I run the spider in the command prompt with
>
> > python scrapy-ctl.py crawl whitecase.com
>
> > I see the FirmItem in the return items:
>
> > [whitecase.com] INFO: Passed FirmItem(title=[u'White &amp; Case LLP -
> > Lawyers - Rachel B. Wagner '])
>
> > but where is this item? How do I get it and put it into a file?
>
> By writing an item pipeline to do that:http://doc.scrapy.org/topics/item-pipeline.html
>

> You can use the builtin File Export Pipeline for that:http://doc.scrapy.org/topics/item-pipeline.html#module-scrapy.contrib...