Input/Output Processors with multiple arguments?

246 views
Skip to first unread message

Edward

unread,
Oct 17, 2011, 11:00:07 AM10/17/11
to scrapy-users
Hi,

I wondered if it was possible to define input/output processors with
multiple arguments?

I have this:

def function(x, y):
body = x.split("|")
for q in body:
if something specific to value y:
return x[some_index:]


Then, in my Item class I define a field as follows:

title = Field(input_processor=MapCompose(function, y="some value",
output_processor=Identity())

However, I get a TypeError when I try and run this, and I don't
understand how to declare the processor I want to use to the
MapCompose processor along with the second parameter.

Cheers!

Rolando Espinoza La Fuente

unread,
Oct 18, 2011, 10:18:14 AM10/18/11
to scrapy...@googlegroups.com
I think you can use partial or better a class:

class MyProc(object):

def __init__(self, y):
self.y = y

def __call__(self, value):
# do something to value


proc = MapCompose(MyProc("foo"), MyProc("bar"))

~Rolando

> --
> You received this message because you are subscribed to the Google Groups "scrapy-users" group.
> To post to this group, send email to scrapy...@googlegroups.com.
> To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.
>
>

Reply all
Reply to author
Forward
0 new messages