end game

[Jim Garrison]

Hi all,

I spent quite a bit of time this weekend thinking about end game for the Wikiotics project.  At our board meeting last year, we concluded that it would be valuable to have a static version of the site before shutting it down.  But I am finding that migrating the content to a static site is a nontrivial undertaking, something that cannot be done in a few hours or even a single day, at least with my current toolbox.  Even if we were to have a static version of the site, there are still outstanding issues with curation/categorization of content (e.g. English lessons marked target-language:es and vice versa) -- things which have always been a problem, and I'm not sure why now would be the time to fix them.  I really can't put much more time into trying to setup up a static site.  If somebody wants to crawl the site, feel free.  The internet archive has archives of many lessons, but it seems it does not pick up the photos (see e.g. here).  I am hoping that a change like this will make it more likely the photos are picked up, but it took me an hour to dig through the code long enough to make that change, and I can't remember enough about the server environment to run django-compressor again, so it is currently undeployed.  (One could try disabling django-compressor as a next step, but doing so would take some time to figure out as well.)  I have little interest in working on this further at this point.

[Laurent Savaëte]

Hi Jim and everyone,

Thanks for bringing this up. While I don't think I'm ready to pick up wikiotics and rebuild it from scratch, I happen to have a bit of spare time on my hands so I'd be happy to see if I can crawl the site to make a static version of it. I picked up scrapy recently for something else, and I think it might do the trick. I don't think I'll spend more than a day on it, but hopefully it may be enough.

Cheers,

Laurent

On 17/09/18 03:13, Jim Garrison wrote:

Hi all,

I spent quite a bit of time this weekend thinking about end game for the Wikiotics project.  At our board meeting last year, we concluded that it would be valuable to have a static version of the site before shutting it down.  But I am finding that migrating the content to a static site is a nontrivial undertaking, something that cannot be done in a few hours or even a single day, at least with my current toolbox.  Even if we were to have a static version of the site, there are still outstanding issues with curation/categorization of content (e.g. English lessons marked target-language:es and vice versa) -- things which have always been a problem, and I'm not sure why now would be the time to fix them.  I really can't put much more time into trying to setup up a static site.  If somebody wants to crawl the site, feel free.  The internet archive has archives of many lessons, but it seems it does not pick up the photos (see e.g. here).  I am hoping that a change like this will make it more likely the photos are picked up, but it took me an hour to dig through the code long enough to make that change, and I can't remember enough about the server environment to run django-compressor again, so it is currently undeployed.  (One could try disabling django-compressor as a next step, but doing so would take some time to figure out as well.)  I have little interest in working on this further at this point.

--
You received this message because you are subscribed to the Google Groups "wikiotics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wikiotics+...@googlegroups.com.
To post to this group, send email to wiki...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wikiotics/783f74b0-668e-9dbd-fddd-13e8ba4193c3%40wikiotics.org.
For more options, visit https://groups.google.com/d/optout.

[Jim Garrison]

Hi Laurent,

Thanks for the email.  That would be most wonderful.  There is a recent LWN article, too, on scraping and archiving web sites (https://lwn.net/Articles/766374/).  It does not mention scrapy, but the alternatives listed may prove useful if scrapy does not work easily/reliably.

Hope you are well.

Cheers,

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/wikiotics/32244466-4624-3217-5c07-d41118419500%40wikiotics.org.

[garrison]

Hi everyone,

I have made significant progress on migrating Wikiotics to a static site generated by Hugo.  The site https://wikiotics.org is now generated from a repository hosted on github:

https://github.com/wikiotics/wikiotics.org

https://github.com/wikiotics/wikiotics.org/pull/4

I have only a few remaining steps until I merge the above pull request and consider this complete.

The lessons remain editable through the github repository, so there is no significant barrier to cleaning up the content at some point in the future.

Cheers,

Jim

On Thursday, October 11, 2018 at 9:35:50 PM UTC-4, Jim Garrison wrote:

Hi Laurent,

Thanks for the email.  That would be most wonderful.  There is a recent LWN article, too, on scraping and archiving web sites (https://lwn.net/Articles/766374/).  It does not mention scrapy, but the alternatives listed may prove useful if scrapy does not work easily/reliably.

Hope you are well.

Cheers,

Jim

On 10/09/2018 06:43 AM, Laurent Savaëte wrote:

Hi Jim and everyone,

Thanks for bringing this up. While I don't think I'm ready to pick up wikiotics and rebuild it from scratch, I happen to have a bit of spare time on my hands so I'd be happy to see if I can crawl the site to make a static version of it. I picked up scrapy recently for something else, and I think it might do the trick. I don't think I'll spend more than a day on it, but hopefully it may be enough.

Cheers,

Laurent

On 17/09/18 03:13, Jim Garrison wrote:

Hi all,

I spent quite a bit of time this weekend thinking about end game for the Wikiotics project.  At our board meeting last year, we concluded that it would be valuable to have a static version of the site before shutting it down.  But I am finding that migrating the content to a static site is a nontrivial undertaking, something that cannot be done in a few hours or even a single day, at least with my current toolbox.  Even if we were to have a static version of the site, there are still outstanding issues with curation/categorization of content (e.g. English lessons marked target-language:es and vice versa) -- things which have always been a problem, and I'm not sure why now would be the time to fix them.  I really can't put much more time into trying to setup up a static site.  If somebody wants to crawl the site, feel free.  The internet archive has archives of many lessons, but it seems it does not pick up the photos (see e.g. here).  I am hoping that a change like this will make it more likely the photos are picked up, but it took me an hour to dig through the code long enough to make that change, and I can't remember enough about the server environment to run django-compressor again, so it is currently undeployed.  (One could try disabling django-compressor as a next step, but doing so would take some time to figure out as well.)  I have little interest in working on this further at this point.

--
You received this message because you are subscribed to the Google Groups "wikiotics" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wiki...@googlegroups.com.

To post to this group, send email to wiki...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wikiotics/783f74b0-668e-9dbd-fddd-13e8ba4193c3%40wikiotics.org.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "wikiotics" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wiki...@googlegroups.com.

[Jim Garrison]

I believe the scrape is now complete. There is a link on the upper
corner of each page at https://wikiotics.org/ to compare with its
version on the old site. Please let me know if you can spot any
regressions. I am most interested now in issues having to do with the
automated scrape, but also interested in general feedback.

> send an email to wikiotics+...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wikiotics/582ace40-bf74-41fb-b782-c75b424961c1o%40googlegroups.com
> .

[Ian Sullivan]

I think it looks fantastic! Do we want to add a link from the homepage
to this wiki page: https://wikiotics.org/en/Wikiotics_Foundation/

Once the migration is complete, I'll go through and do a pass updating
that, and the various liked "donate" and "contribute" pages to more
accurately reflect the current model. Created issue #5 [^1] to capture
the pages I could find that need updating.

Ian

[^1]: https://github.com/wikiotics/wikiotics.org/issues/5

[Jim Garrison]

Great! I followed up to the question on the github link you posted.

Also, I am trying to see if we can achieve 50 stars on github. Please
star the repository if you have a github account:

https://github.com/wikiotics/wikiotics.org/issues/3