missing data in local watchdog instance

[Matt Price]

hi ,

hope this list is appropriate for this question.

in an effort to understand the watchdog framework a little better i'm
trying to run a local copy on my laptop. i'm following the README
instructions but the make process fails because i'm unable to download
some of the data directories -- "make sync" which runs a simple rsync
command, produces some errors, e.g.:

> receiving file list ... rsync: opendir "/crawl/opensecrets/cansum" (in data) failed: Permission denied (13)
> rsync: opendir "/crawl/opensecrets/pfd" (in data) failed: Permission denied (13)
> rsync: opendir "/crawl/almanac/nationaljournal.com" (in data) failed: Permission denied (13)
> rsync: opendir "/crawl/usps" (in data) failed: Permission denied (13)
> done

this leads to problems in later make commands, e.g. this, in 'make' when
exectuted from the 'import' directory.

> make
> python load/almanac.py > ../data/load/districts/almanac.json.tmp
.....

> Traceback (most recent call last):
> File "load/almanac.py", line 69, in <module>
> print simplejson.dumps(main(), indent=2, sort_keys=True)
> File "load/almanac.py", line 26, in main
> assert os.path.exists(ALMANAC_DIR), ALMANAC_DIR
> AssertionError: ../data/crawl/almanac/nationaljournal.com/pubs/almanac/2008/
>

anyway, reading code is slow for me, but wondering if there's a way to
either create the database w/out the almanac data or failing that, a
way to get access to the currently restricted directories.

thanks,
m

[Aaron Swartz]

> hope this list is appropriate for this question.

Yes, of course. If you want to work with the almanac data we can give
you an account; otherwise we should try and publish some extracted
version of it that can be used. I'll look into that.

[Matt Price]

i don't especially need that data, just trying to get watchdog up and
running so i have a better idea of the plumbing & the database
architecture. a summary would be fine, but if it's simpler to give me
an account that would be great too. either way, this shouldn't be a
high priority.

thanks as always,
matt

> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "Watchdog Volunteers" group.
> To post to this group, send email to watchdog-...@googlegroups.com
> To unsubscribe from this group, send email to watchdog-volunt...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/watchdog-volunteers?hl=en
> -~----------~----~----~----~------~----~------~--~---
>
--
Matt Price
matt....@utoronto.ca

[christopher bdnk]

On Sat, Sep 20, 2008 at 3:01 PM, Matt Price <matt....@utoronto.ca> wrote:

On Sat, 2008-09-20 at 07:51 -0400, Aaron Swartz wrote:

i don't especially need that data, just trying to get watchdog up and
running so i have a better idea of the plumbing & the database
architecture.  a summary would be fine, but if it's simpler to give me
an account that would be great too.  either way, this shouldn't be a
high priority.

If you just run the touch command on districts/almanac.json I think you should be set. I had the same problem before grabbing the almanac data and that resolved it for me. Of course, that won't give you access to that data.

[Aaron Swartz]

> i don't especially need that data, just trying to get watchdog up and
> running so i have a better idea of the plumbing & the database
> architecture. a summary would be fine, but if it's simpler to give me
> an account that would be great too. either way, this shouldn't be a
> high priority.

You should be able to get the extracted data at:

http://watchdog.net/data/parse/almanac.njs

It's not really set up to import that right now, but it should give you a sense.