def myfun(s,i=[0]):
print(s)
i[0]+=1 # mutable variable get evaluated ONCE
return i[0]
pyspider is designed to be distributed, your request might be processed by different handler instance.
--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/CAH-droyyw%2BncnqS2VqsSbAMMAKNW7yGU0vS1AjBBeGDAXCVvyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
E.g. you have two box running pyspider, the first URL is handled by box A, you increase the counter to 1. The URL B is handled by box B, it will still find the counter is 0.
On Wed, 7 Jun 2017, 15:40 Nick Gilmour, <nicke...@gmail.com> wrote:Thanks for the reply but I don't really understand what you mean.Could you be more specific?
On Thu, Jun 8, 2017 at 12:36 AM, Roy Binux <r...@binux.me> wrote:
pyspider is designed to be distributed, your request might be processed by different handler instance.
On Wed, 7 Jun 2017, 14:41 Nick Gilmour, <nicke...@gmail.com> wrote:
Hi all,I have a website, where I have to pass some parameters to the URL. I have defined the params dict and is working fine for the first page. In the dict there is a parameter which controls where to start, like an index offset. No matter what I do, I cannot increment this value in order to get the next results, I'm always getting only the first ten results. How can I solve this problem?Is there some kind of a trigger I can use?Even this example in which a counter is incremented every time the function is called:def myfun(s,i=[0]): print(s) i[0]+=1 # mutable variable get evaluated ONCE return i[0]
from here:is not working within pyspyder. I don't understand why.
--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
That's only an example, even with one box, pyspider will recreate the script context at any time. You shouldn't do that.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
pyspider will recreate the script context at any time
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
A script is actually a string basically. It need to be executed before handle any requests. When it gets created, it's a brand new context. All variables is rested to its default value.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
Generate all links once in single callback.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
I don't know what you trying to do. Can you provide some sample code?
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
links_offset never changed
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
links_offset never changed
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
Can you put the loop in index page or make it as a function and call or with response.text
links_offset never changed
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/CAH-droyyw%2BncnqS2VqsSbAMMAKNW7yGU0vS1AjBBeGDAXCVvyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-user...@googlegroups.com.
To post to this group, send email to pyspide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/CAH-drowzpVU4pMaf5SYT%3DMzwb%2BJ5YsjSpYWKJ3d0J03J__cxXQ%40mail.gmail.com.
links_offset never changed
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyspider-users/CAH-droyyw%2BncnqS2VqsSbAMMAKNW7yGU0vS1AjBBeGDAXCVvyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyspider-users+unsubscribe@googlegroups.com.
To post to this group, send email to pyspider-users@googlegroups.com.
I'm still not sure you are trying to do. Self.crawl is to submit a URL to fetch. You can use loop to submit multiple URL.
While debug, you can select any tasks you submitted to go deeper.
Use break to break a Loop
links_offset never changed