Can't get a script to run until the end

[Rodrigo Burgarelli]

Hi guys,

I'm pretty new on this and I am having some trouble on running a python script to scrape some data from a brazilian government website. I'm having 3 specific problems:

1) I can't run it from the Scraper Overview screen - everytime I press run, it shows a message saying that it will run as soon as possible. But I waited for over a day and it didn't go off.

2) If I enter the script editing page, I can run it, but it'll never pass the first 100 url access (and it's supposed to run for all the 5700 brazilian cities). It just freezes at a certain time and nothing else happens. If I try just a small state like Roraima, with 13 cities, it finishes allright, but even if I put 6 our 7 small states to run all together it doesn't finish running for all the cities.

3) In other ocasions, the script just doesn't start running even in the script editing page. I get a "disconnect error" (sorry, didn't save the whole message and I can't reproduce it now) and I'm unable even to start the scrapping.

Am I doing anything wrong?

Link to the script: https://scraperwiki.com/scrapers/siope/

Thanks a lot.

[Rodrigo Burgarelli]

Ok, just got the error message when #2 happens: CPUTimeExceededError: ScraperWiki CPU time exceeded

[Zarino Zappia]

Hi Rodrigo,

1) The "run now" button on the scraper overview page never really worked as anyone expected (here's why). I've removed it.

2) Sounds like you've now noticed you get a CPU timeout error. I'd suggest writing your scraper so that it saves its position in the list of 5700 cities, and picks up where it left off.

3) The disconnect error is a known bug – the piece of software that connects your web browser to our server sometimes unexpectedly fails. Since we're now focussing our development time on the new ScraperWiki, this is unlikely to be fixed.

Hope that helps,

Zarino Zappia

VP of Product, ScraperWiki

@zarino

--
You received this message because you are subscribed to the Google Groups "ScraperWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scraperwiki...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Rodrigo Burgarelli]

Thanks a lot for the answers, Zarino! I'll try rewriting it then.