Server down?

[Emil]

i get this error from all my scrapers

*** internalwarning: <class 'twisted.internet.error.ConnectionLost'>


what can i do?


https://scraperwiki.com/scrapers/craigs2/

[Zarino Zappia]

Seems to be working for me now. When did the error show up?

Zarino Zappia

VP of Product, ScraperWiki

@zarino

--
You received this message because you are subscribed to the Google Groups "ScraperWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scraperwiki...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Emil]

Hey yeah its working for me now as well, and thanks fot the quick response.

The error showed up at around 16.00 today

[Emil]

I have another problem now with

https://scraperwiki.com/scrapers/craigs-whirlpool/edit/

i get the:

error: [Errno 32] Broken pipe


what does this mean?

[Zarino Zappia]

The errors are related, and to do with communication being lost between your browser and the server that runs your code.

Broken pipe is usually intermittent – is it happening constantly for you?

Z

[Emil]

I was declaring a variable from a value from a value that didnt exist in the database.
 
last_page = scraperwiki.sqlite.get_var("last_page")

Once i removed that line the brokenpipe error went away.


Im now running into "CPUTimeExceededError: ScraperWiki CPU time exceeded". Im currently trying to rebuild the scrapers to save their progress so they can pick up failed runs.
But im Wondering what does this error mean? Im only saving very light weight craigslist posts.

https://scraperwiki.com/scrapers/craigs-bosch/

Thanks / Emil

[Zarino Zappia]

Here's an FAQ about the CPUTimeout error:

https://scraperwiki.com/docs/python/faq/#cpu_limit

I've taken a quick glance at your scraper, and I can't see any quick wins for making it more efficient. Anybody else got any ideas?

Z

[Svavar Kjarrval]

What I'd do would be to separate the functionality, at least in your case. Let one script worry about collecting the ad urls and another to collect the ad contents. That way, even if the former script fails, the other will still be processing ads. And besides, if the urls are uniform, you can also avoid requesting the same ad more than once.

- Svavar Kjarrval