Update on the India Flood Alert Portal

[Ajay Kumar]

Hi Guys,
Some of us have been working on the Flood Alert portal and we need some help now :)
The details are here:
http://ictd.asia/wiki/CWC_Flood_Forecast_-_India

We primarily need help with the "presentation" of the information/data we are sourcing from the CWC which posts daily updates on River Water Level, Rainfall & Flood Alert Situation(if any) for each of the flood forecasting sites in India that are publically listed.

So if you have any of the listed skills and have time to help, please respond and feel free to ask questions :)

Our next task is:
1) Publishing Feeds
2) the UI and Mapping the information and displaying it on our portal

Apart from the skills, your comments, suggestions and advice are welcome as well!~

More soon!

--
Best Regards,

Ajay Kumar

http://www.ajuonline.net

[konark modi]

Hi

Good to hear after so long. I am familiar with working in beautiful soup,XML,RSS feeds, mapping so would love to contribute in that. 

Let me know how and when to start.

Regards

Konark.K.Modi

+91-9888211816

[Ravi chandra padmala]

On Wed, Aug 25, 2010 at 10:00 PM, konark modi <modi....@gmail.com> wrote:

Hi

Good to hear after so long. I am familiar with working in beautiful soup,XML,RSS feeds, mapping so would love to contribute in that. 

Let me know how and when to start.

Hi

This is a snapshot of what I'm doing.
http://ictd.asia/code/ictdasia-cwc.tar.gz

I'm sure you'll find a lot of bugs. So, go right ahead.

Regards

Konark.K.Modi

+91-9888211816

On Wed, Aug 25, 2010 at 7:33 PM, Ajay Kumar <ajuo...@gmail.com> wrote:

Hi Guys,
Some of us have been working on the Flood Alert portal and we need some help now :)
The details are here:
http://ictd.asia/wiki/CWC_Flood_Forecast_-_India

We primarily need help with the "presentation" of the information/data we are sourcing from the CWC which posts daily updates on River Water Level, Rainfall & Flood Alert Situation(if any) for each of the flood forecasting sites in India that are publically listed.

So if you have any of the listed skills and have time to help, please respond and feel free to ask questions :)

Our next task is:
1) Publishing Feeds
2) the UI and Mapping the information and displaying it on our portal

Apart from the skills, your comments, suggestions and advice are welcome as well!~

More soon!

--
Best Regards,

Ajay Kumar

http://www.ajuonline.net

--
Ravi Chandra

[S Anand]

Looks like the source is missing the .py with the definition of
FloodData (in alertscraper.py)?

iwscraper.py worked fine for me, though.

[Ajay Kumar]

Ravi,

On 25 August 2010 22:37, Ravi chandra padmala <neenao...@gmail.com> wrote:

This is a snapshot of what I'm doing.
http://ictd.asia/code/ictdasia-cwc.tar.gz

I'm sure you'll find a lot of bugs. So, go right ahead

I looked at the data being saved in the DB.
Table: site_profile , field: site_type. Values: "level forecasting site" or "inflow forecasting site"

I think there is no need to save the complete text. The flood forecast site types are: Inflow/Level.

And in the DB only site IDs from 1-79 and 148 are saved in site_profile ?

[Ravi chandra padmala]

On Thu, Aug 26, 2010 at 12:56 PM, Ajay Kumar <ajuo...@gmail.com> wrote:

Ravi,

On 25 August 2010 22:37, Ravi chandra padmala <neenao...@gmail.com> wrote:

This is a snapshot of what I'm doing.
http://ictd.asia/code/ictdasia-cwc.tar.gz

I'm sure you'll find a lot of bugs. So, go right ahead

I looked at the data being saved in the DB.
Table: site_profile , field: site_type. Values: "level forecasting site" or "inflow forecasting site"

I think there is no need to save the complete text. The flood forecast site types are: Inflow/Level.

 

Ok, will fix that.

And in the DB only site IDs from 1-79 and 148 are saved in site_profile ?

 

Fixed that yesterday. Will upload a snapshot tonight.

--
Best Regards,

Ajay Kumar

http://www.ajuonline.net

--
Ravi Chandra

[Ravi chandra padmala]

On Thu, Aug 26, 2010 at 12:36 PM, S Anand <root...@gmail.com> wrote:

Looks like the source is missing the .py with the definition of
FloodData (in alertscraper.py)?

We're not using that anymore.
cwc-script.py should get everything we need and store it in the database specified in settings.py

 

--
Ravi Chandra

[Ajay Kumar]

Hi Ravi.

On 27 August 2010 17:59, Ravi chandra padmala <neenao...@gmail.com> wrote:

cwc-script.py should get everything we need and store it in the database specified in settings.py

I flushed the DB and ran the script again.
1) I noticed that some Sites dont have any PWL value, and hence the 3 entries for pwl in the table "site_updates" is NULL.
They usually have a text on the site, "Present Water Level is Less than Warning Level."
Source: http://www.india-water.com/ffs/static_info.asp?Id=21

2) The timestamps (with timezone) are in UTC. Can we just directly save them in IST since this is an India specific project? Will save us from converting timezones twice, during insert to db/post to portal.

Converting Information/Data to Action?

Also I once we have this setup, before we can actually map it or publish feeds, I am thinking of getting a list of agencies/NGOs who could use this data and use it for their response efforts just like one of our office is doing.
Maybe I will put up a shared Google Spreadsheet for collecting this information and link it up on the wiki.

[Ajay Kumar]

Hi!
Last night me and Ravi were testing the code and cross checkin data collected.
Although the code has some problems running on our ICTD server, it runs fine locally, so we have started crawling and archiving the data from today morning.

While I work on the server side issue of identifying the problem and solving it, I have updated the wiki page with more tasks and next steps.

http://ictd.asia/wiki/CWC_Flood_Forecast_-_India

Comments and suggestions for publishing the data and feeds is welcome!

[Ajay Kumar]

On 29 August 2010 13:20, Ajay Kumar <ajuo...@gmail.com> wrote:

While I work on the server side issue of identifying the problem and solving it, I have updated the wiki page with more tasks and next steps.

Ok. the code seems to be crashing after some iterations on the server. Possible memory issue. Although works fine on the local PC, which of course has more resources than the shared nature of our server.

[Ajay Kumar]

On 1 September 2010 16:29, Ajay Kumar <ajuo...@gmail.com> wrote:

Ok. the code seems to be crashing after some iterations on the server. Possible memory issue.

After a little code optimisation. This seems to work on another VPS (thanks to Ravi)
Cron is setup and data is being fetched 4 times a day :)
I think we'd need a dedicated server for so much of data crawling with the ODI? :P

We should start publishing soon for NGOs to recieve updates via email.
More on this soon!

[Ravi chandra padmala]

Hi

I just exported my mercurial repository to bazaar and pushed to launchpad.
The code is here https://code.launchpad.net/~india-ictdasia/floodportalindia .

File/fix bugs there, thanks.

--
Cheers,
Ravi Chandra

[Thejesh GN]

I dont have capable servers to run the scripts so I wrote a scraper

Check

http://scraperwiki.com/scrapers/show/india_flood_report/data/

Thej
--
Thejesh GN
http://thejeshgn.com

[Ajay Kumar]

Lovely!

On 10 September 2010 00:32, Thejesh GN <i...@thejeshgn.com> wrote:

I dont have capable servers to run the scripts so I wrote a scraper

Check

http://scraperwiki.com/scrapers/show/india_flood_report/data/

I just noticed this as well that you put up: http://media.thejeshgn.com/visualizations/india-flood-alert.html

While we are already scraping data, we are still stuck at ways to "present" it and put it online. I will have some time in a couple of days to look back at this, but meanwhile if you and Ravi could see what can be done it would be great :)

[Ajay Kumar]

On 10 September 2010 11:01, Ajay Kumar <ajuo...@gmail.com> wrote:

http://scraperwiki.com/scrapers/show/india_flood_report/data/

I just noticed this as well that you put up: http://media.thejeshgn.com/visualizations/india-flood-alert.html

Missed to post the link to your blog post: http://thejeshgn.com/2010/09/10/parsing-and-visualizing-indian-flood-forecast/

[Ravi chandra padmala]

On Fri, Sep 10, 2010 at 12:32 AM, Thejesh GN <i...@thejeshgn.com> wrote:

I dont have capable servers to run the scripts so I wrote a scraper

Check

http://scraperwiki.com/scrapers/show/india_flood_report/data/

 

Nice,

If you're interested in helping develop the UI, I could give you access to the server on which we're running the scraper. Though, a standard desktop should be capable enough to run the scripts.

I could use some help with the UI development.
 

Thej
--
Thejesh GN
http://thejeshgn.com

--
Cheers,
Ravi Chandra

[Thejesh GN]

Thej
--
Thejesh GN
http://thejeshgn.com

On Fri, Sep 10, 2010 at 12:58 PM, Ravi chandra padmala <neenao...@gmail.com> wrote:

On Fri, Sep 10, 2010 at 12:32 AM, Thejesh GN <i...@thejeshgn.com> wrote:

I dont have capable servers to run the scripts so I wrote a scraper

Check

http://scraperwiki.com/scrapers/show/india_flood_report/data/

 

Nice,

If you're interested in helping develop the UI, I could give you access to the server on which we're running the scraper. Though, a standard desktop should be capable enough to run the scripts.

Sure. Can I access the data in JSON format?

Which gets automatically updated everyday.

 

I could use some help with the UI development.

I will work onit.

[Ajay Kumar]

On 11 September 2010 14:43, Thejesh GN <i...@thejeshgn.com> wrote:

Sure. Can I access the data in JSON format?

Which gets automatically updated everyday.

For now we just run the scraper daily 4 times since they update data at multiple instances on a day. The data is stored in pgsql. If you want I can send SSH access details to the server?

We are sharing resources right now, i.e. running the scraper on another VPS while we will publish it on the vps that ictd.asia is hosted on, so a little load sharing happening there :)

For now the priorities are:
1) To enable anyone to sign up to recieve these updates for their region(or a set of sites)  over Email at those intervals in a day. I.e. we run the script 4 times a day, an email sent 4 times a day instantly.
2) Publish it online as a nice info display.
3) We could link the RSS feed to third party provider like identi.ca/twitter for people to subscribe? As I am not sure how much extra load will opening up the API/RSS etc add to the server.

Comments?

[Thejesh GN]

On Sat, Sep 11, 2010 at 2:52 PM, Ajay Kumar <ajuo...@gmail.com> wrote:

On 11 September 2010 14:43, Thejesh GN <i...@thejeshgn.com> wrote:

Sure. Can I access the data in JSON format?

Which gets automatically updated everyday.

For now we just run the scraper daily 4 times since they update data at multiple instances on a day. The data is stored in pgsql. If you want I can send SSH access details to the server?

We are sharing resources right now, i.e. running the scraper on another VPS while we will publish it on the vps that ictd.asia is hosted on, so a little load sharing happening there :)

Sure. How about Google App Engine to pull that data from your sever and serving the world through AppEngine. That way we will reduce load on your server and we can give access to others.

 

For now the priorities are:
1) To enable anyone to sign up to recieve these updates for their region(or a set of sites)  over Email at those intervals in a day. I.e. we run the script 4 times a day, an email sent 4 times a day instantly.

This is difficult to do with limited resources.

 

2) Publish it online as a nice info display.

We could do this. I can work on this.

3) We could link the RSS feed to third party provider like identi.ca/twitter for people to subscribe? As I am not sure how much extra load will opening up the API/RSS etc add to the server.

This is great. We could publish to twitter with tagging. And for RSS, lets publish it through feedburner. With PubSubHubBub, its realtime and reduces burden on our server.

 

[Ajay Kumar]

On 11 September 2010 15:03, Thejesh GN <i...@thejeshgn.com> wrote:

Sure. How about Google App Engine to pull that data from your sever and serving the world through AppEngine. That way we will reduce load on your server and we can give access to others.

No experience with GAE. Not sure if Ravi has worked on it either (PS: He is offline until Monday) I think the load issues crops up when we want to expose the API and publish Feeds etc. Just for data visualisation I think the server can handle that. 

 

For now the priorities are:
1) To enable anyone to sign up to recieve these updates for their region(or a set of sites)  over Email at those intervals in a day. I.e. we run the script 4 times a day, an email sent 4 times a day instantly.

This is difficult to do with limited resources.

 

Ah, ok. For now I was thiknig to have a simple form on the site that has a list (checkboxes) of Forecasting sites for people to choose from and enter their email address. We could then add it to our list, and write a seperate script that sends them email alerts.
Sending emails using Gmail? If we send alerts from say, flood...@ictd.asia using the Gmail Hosted solution, it has some daily limits, like 500-1000. Good to start with then we can figure out some other solution once we hit that limit.

Also once we have converted data into JSON/RSS formats it becomes easier after that isnt it?

 

2) Publish it online as a nice info display.

We could do this. I can work on this.

 
OK. will send across SSH details and you can work on this.
Also check: http://ictd.asia/wiki/CWC_Flood_Forecast_-_India#Proposed_Design and add your suggestions?

On a side note: Are you on Launchpad? Could add you to the team so that you have commit access and can push the code there as well for everyone?
If you are on irc, #ictd-asia I am there if you need to discuss anything.

[Ajay Kumar]

On 11 September 2010 14:43, Thejesh GN <i...@thejeshgn.com> wrote:

Sure. Can I access the data in JSON format?

Which gets automatically updated everyday.

Emailed you the server access details :) Let me know if you need anything.