Warwickshire Schools: An OpenData Application on Yahoo Pipes

[SundayStrummer]

Hi all. An "also ran" in the bacon sarnie stakes, I suspect, but I am
slightly proud of it given my starting point of ZERO skills in this
field.

To run it: http://pipes.yahoo.com/sundaystrummer/warwickshireschools

And to be amazed by the KML output on Google Maps, try this:
http://pipes.yahoo.com/pipes/pipe.run?_id=76c0d7cb35668ed87e9f4f8a6f2a6a89&_render=kml&phase=Secondary

I've documented it and hope to post this on the wiki. I just have to
work out how to do that now!

Graham

[James]

Nice one SundayStrummer (Graham). I am having a play with Yahoo Pipes
myself and hope to post something here too.
If you need any help with posting on the Wiki (I am sure you won't)
just let us know, happy to help.

I used this site for viewing your KML file.
http://display-kml.appspot.com/

James

On 13 May, 18:07, SundayStrummer <sundaystrum...@yahoo.co.uk> wrote:
> Hi all.  An "also ran" in the bacon sarnie stakes, I suspect, but I am
> slightly proud of it given my starting point of ZERO skills in this
> field.
>
> To run it:http://pipes.yahoo.com/sundaystrummer/warwickshireschools
>

> And to be amazed by the KML output on Google Maps, try this:http://pipes.yahoo.com/pipes/pipe.run?_id=76c0d7cb35668ed87e9f4f8a6f2...

[SundayStrummer]

Thanks, James. Alas I feel the wiki editor has its own learning curve
and I'm "learnt out" this week. Maybe next week. In the meantime,
here's a link to the documentation I've written on the work I've done
so far, which I've also posted on a wiki page:
http://homepage.ntlworld.com/hydeaway/wcc/AnAdventureWithYahooPipes.pdf

I viewed the KML feed directly in Google Maps, but it was interesting
to view the KML file and post it on the KML viewer.

By the way, I've bought the PDF versions of Tony Loton's books on
Yahoo Pipes: http://www.lotontech.com/computing so let me know if
you'd like to have a look at them.

Graham

[Owen Stephens]

If you are interested in Yahoo Pipes I really recommend having a look at Tony Hirst's (many) blog posts about it

http://blog.ouseful.info/?s=pipework

The amount of information on here may be a bit overwhelming to start with, but as a starting point http://blog.ouseful.info/2009/07/09/mashlib-pipes-tutorial-2d-journal-search/ gives a very good intro/tutorial to Pipes

Owen

Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com
Email: ow...@ostephens.com
Telephone: 0121 288 6936

[SundayStrummer]

Hi Owen. Thanks for the link, and also thanks very much for your blog
article http://www.meanboyfriend.com/overdue_ideas/2010/04/putting-warwickshire-libraries-on-the-map/
that helped get me started. I've duly credited you for that in my
write up of the project (see link to pdf file above). And I gather
this is now being considered for publication on Yahoo's Developer
Blog. http://developer.yahoo.net/blog/ Yikes!

Thanks again.

Graham

On 17 May, 09:55, Owen Stephens <o...@ostephens.com> wrote:
> If you are interested in Yahoo Pipes I really recommend having a look at Tony Hirst's (many) blog posts about it
>
> http://blog.ouseful.info/?s=pipework
>

> The amount of information on here may be a bit overwhelming to start with, but as a starting pointhttp://blog.ouseful.info/2009/07/09/mashlib-pipes-tutorial-2d-journal...gives a very good intro/tutorial to Pipes

>
> Owen
>
> Owen Stephens
> Owen Stephens Consulting
> Web:http://www.ostephens.com

> Email: o...@ostephens.com

> Telephone: 0121 288 6936
>
> On 13 May 2010, at 23:01, SundayStrummer wrote:
>
> > Thanks, James.  Alas I feel the wiki editor has its own learning curve
> > and I'm "learnt out" this week.  Maybe next week.  In the meantime,
> > here's a link to the documentation I've written on the work I've done
> > so far, which I've also posted on a wiki page:
> >http://homepage.ntlworld.com/hydeaway/wcc/AnAdventureWithYahooPipes.pdf
>
> > I viewed the KML feed directly in Google Maps, but it was interesting
> > to view the KML file and post it on the KML viewer.
>
> > By the way, I've bought the PDF versions of Tony Loton's books on

> > Yahoo Pipes:http://www.lotontech.com/computingso let me know if

[Owen Stephens]

Hi Graham,

Glad you found the blog post useful. 

I've had a look at your project and if it is of interest I've done an alternative version which uses a mixture of Yahoo Pipes and YQL to achieve the same result, but without doing the pre-merge of data you had to do. As you noted, there isn't really a good way of doing a 'join' in Pipes. My approach uses a combination of a Pipes loop and a YQL query which extracts only the data you really want from the second dataset. I haven't got time for more detailed documentation at the moment I'm afraid, but feel free to have a look at the pipe source if you are interested and I'll answer questions if I can.

http://pipes.yahoo.com/ostephens/78f5296ca27bb3340320cc8e45b27861

Best wishes,

Owen

Owen Stephens

Owen Stephens Consulting

Web: http://www.ostephens.com
Email: ow...@ostephens.com
Telephone: 0121 288 6936

[SundayStrummer]

Ah, yes. I like it, Owen. Neat, given the contraints of the tool.

I guess the only question I have is how you decided what to assign the
results of the loop to. You used item.loop:yql. What does the colon
mean? It seems to imply there's a hierarchy in the data, which I
guess is right. And what would have happened if more than one row had
been returned from the YQL?

Anyway, thanks again.

On 18 May, 15:06, Owen Stephens <o...@ostephens.com> wrote:
> Hi Graham,
>
> Glad you found the blog post useful.
>
> I've had a look at your project and if it is of interest I've done an alternative version which uses a mixture of Yahoo Pipes and YQL to achieve the same result, but without doing the pre-merge of data you had to do. As you noted, there isn't really a good way of doing a 'join' in Pipes. My approach uses a combination of a Pipes loop and a YQL query which extracts only the data you really want from the second dataset. I haven't got time for more detailed documentation at the moment I'm afraid, but feel free to have a look at the pipe source if you are interested and I'll answer questions if I can.
>
> http://pipes.yahoo.com/ostephens/78f5296ca27bb3340320cc8e45b27861
>
> Best wishes,
>
> Owen
>
> Owen Stephens
> Owen Stephens Consulting
> Web:http://www.ostephens.com

> Email: o...@ostephens.com
> Telephone: 0121 288 6936
>

> On 18 May 2010, at 12:13, SundayStrummer wrote:
>
> > Hi Owen.  Thanks for the link, and also thanks very much for your blog

> > articlehttp://www.meanboyfriend.com/overdue_ideas/2010/04/putting-warwickshi...

> > that helped get me started.  I've duly credited you for that in my
> > write up of the project (see link to pdf file above).  And I gather
> > this is now being considered for publication on Yahoo's Developer
> > Blog.  http://developer.yahoo.net/blog/ Yikes!
>
> > Thanks again.
>
> > Graham
>
> > On 17 May, 09:55, Owen Stephens <o...@ostephens.com> wrote:
> >> If you are interested in Yahoo Pipes I really recommend having a look at Tony Hirst's (many) blog posts about it
>
> >>http://blog.ouseful.info/?s=pipework
>

> >> The amount of information on here may be a bit overwhelming to start with, but as a starting pointhttp://blog.ouseful.info/2009/07/09/mashlib-pipes-tutorial-2d-journal...a very good intro/tutorial to Pipes

>
> >> Owen
>
> >> Owen Stephens
> >> Owen Stephens Consulting
> >> Web:http://www.ostephens.com
> >> Email: o...@ostephens.com
> >> Telephone: 0121 288 6936
>
> >> On 13 May 2010, at 23:01, SundayStrummer wrote:
>
> >>> Thanks, James.  Alas I feel the wiki editor has its own learning curve
> >>> and I'm "learnt out" this week.  Maybe next week.  In the meantime,
> >>> here's a link to the documentation I've written on the work I've done
> >>> so far, which I've also posted on a wiki page:
> >>>http://homepage.ntlworld.com/hydeaway/wcc/AnAdventureWithYahooPipes.pdf
>
> >>> I viewed the KML feed directly in Google Maps, but it was interesting
> >>> to view the KML file and post it on the KML viewer.
>
> >>> By the way, I've bought the PDF versions of Tony Loton's books on

> >>> Yahoo Pipes:http://www.lotontech.com/computingsolet me know if

[Owen Stephens]

The results of the loop essentially populates the named element with a list of results from the loop - so yes, it does create a sort of hierarchy. However, I've just used the default name suggested by pipes when I created the loop - but as far as I'm aware the colon is simply convention here - it doesn't need to be there, and if you choose another name it doesn't make any difference.

Since I've just chosen to take the first result (as I know it is a safe assumption in this data each school only has one lat/lon) these two elements just get put into the results. If I chose to include all the results - whether there is one or more - I get a list like:

item.loop:yql
-> 0
->lat1
->lon1
->1
->lat2
->lon2

etc.

the loop just numbers up for as many results as there are.

Hope that makes sense

Owen

Owen Stephens
Owen Stephens Consulting
Web: http://www.ostephens.com

Email: ow...@ostephens.com
Telephone: 0121 288 6936

[SundayStrummer]

Makes perfect sense, Owen. Thanks for the explanation.

Just out of interest the article I wrote detailing my experiences with
Yahoo Pipes is now on the Yahoo Developer Network blog:
http://developer.yahoo.net/blog/archives/2010/05/short_adventure_with_yahoo_pipes.html

On 19 May, 20:51, Owen Stephens <o...@ostephens.com> wrote:
> The results of the loop essentially populates the named element with a list of results from the loop - so yes, it does create a sort of hierarchy. However, I've just used the default name suggested by pipes when I created the loop - but as far as I'm aware the colon is simply convention here - it doesn't need to be there, and if you choose another name it doesn't make any difference.
>
> Since I've just chosen to take the first result (as I know it is a safe assumption in this data each school only has one lat/lon) these two elements just get put into the results. If I chose to include all the results - whether there is one or more - I get a list like:
>
> item.loop:yql
>         -> 0
>                 ->lat1
>                 ->lon1
>         ->1
>                 ->lat2
>                 ->lon2
>
> etc.
>
> the loop just numbers up for as many results as there are.
>
> Hope that makes sense
>
> Owen
>
> Owen Stephens
> Owen Stephens Consulting
> Web:http://www.ostephens.com

> >>>> The amount of information on here may be a bit overwhelming to start with, but as a starting pointhttp://blog.ouseful.info/2009/07/09/mashlib-pipes-tutorial-2d-journal...very good intro/tutorial to Pipes

>
> >>>> Owen
>
> >>>> Owen Stephens
> >>>> Owen Stephens Consulting
> >>>> Web:http://www.ostephens.com
> >>>> Email: o...@ostephens.com
> >>>> Telephone: 0121 288 6936
>
> >>>> On 13 May 2010, at 23:01, SundayStrummer wrote:
>
> >>>>> Thanks, James.  Alas I feel the wiki editor has its own learning curve
> >>>>> and I'm "learnt out" this week.  Maybe next week.  In the meantime,
> >>>>> here's a link to the documentation I've written on the work I've done
> >>>>> so far, which I've also posted on a wiki page:
> >>>>>http://homepage.ntlworld.com/hydeaway/wcc/AnAdventureWithYahooPipes.pdf
>
> >>>>> I viewed the KML feed directly in Google Maps, but it was interesting
> >>>>> to view the KML file and post it on the KML viewer.
>
> >>>>> By the way, I've bought the PDF versions of Tony Loton's books on

> >>>>> Yahoo Pipes:http://www.lotontech.com/computingsoletme know if

> >>>>> you'd like to have a look at them.
>

> >>>>> Graham- Hide quoted text -
>
> - Show quoted text -