Open Source Pipes

[Bram Pitoyo]

Yesterday, we briefly talked about making Pipes not only open source,
but also distributable, in order to reduce load and minimize feed
delivery time.

After the meetup, six of us had dinner whilst continuing to talk about
this. I thought I’d start a new post on the group to keep the
discussion going.

To get you started, here was the framework that provoked me:

http://javascript.neyric.com/wireit/


-Bram

[David Frey]

My two cents on the matter is that whatever form the GUI side of an
open source version of Yahoo Pipes has, it should also have a
programming interface as well. One of my biggest frustrations with
Pipes is that relatively simple routines are made horribly complex by
requiring multiple nested pipes to accomplish something that could
have simply been written in one line of code in most scripting
languages.

I think having this option would allow more advanced programmers to
write modules/plugins to everyone's benefit.

[brian walsh]

+1 very, very interested in open source ypipes
-brian
(503) 407-2768
br...@bwalsh.com
http://www.bwalsh.com

[Amber Case]

agreed - i am willing to invest some time every week on this -- in
person, somewhere.

[tagalus]

Bram,

Could you summarize the main points that were discussed during dinner
for those of us that missed it?

I am also agreed with Amber - definitely willing to spend some time on
this project.

Any discussion yet on language preference for server-side code?

- John Nastos

On Jan 14, 10:26 am, Bram Pitoyo <brampit...@gmail.com> wrote:

[Dawn Foster]

Jerry has also been doing a bunch of work on specs for an open source pipes-like app. I'm also going to add him to the agenda for the 27th after Marshall's presentation to talk about his ideas.

Dawn

[Jerry Hilts]

I've hacked together a proposal for an open pipes framework. I ran a
brief outline by Dawn and Beer and Blog. Was hoping to tag you too,
Bram but you had something else going on.

Been hacking away some more this weekend.

I've got an initial, high-level architecture for the whole system
(pipes, pipe segments, pipe server, and pipe development tools).
I've also worked up a basic DSL for building a pipe. I should have
enough of the core runtime classes pulled together to demo to anyone
who wants to stay late at the next PDPUG meeting.

j.
----

p.s. "Open Pipes" is already taken by anther open source effort, so
I'm calling the project "Tuttle" for now. (Brownie points and maybe
a beer if you know why.) When I get the code to a cohesive point,
I'll think about posting it to github.

[Dawn Foster]

@unclenate just mentioned that http://apatar.com is an open source
project that has some pipes-like functionality. I don't have time to
evaluate it. Anyone want to take a look at it and post back to the
group?

Dawn

[Bram Pitoyo]

John,

On the last Data Plumbers meeting, I talked about WireIt, a JavaScript
framework that can handle the presentation of Yahoo!Pipes like systems.

http://javascript.neyric.com/wireit/

Besides being used for Tarpipe to wire many web apps to each other,
WireIt could possibly be used as the presentation layer of our open
source Yahoo!Pipes project (codename Tuttle, I think.)

Another idea thrown was about some sort of a scripting or markup
language that could be translated into Pipes-style stucture, but could
also be coded from scratch.

Currently, Leif Werner, Eric Dreschel and Amber Case are sitting
beside me, scheming the next move for this technology.

-Bram

[Bram Pitoyo]

Jerry,

I would very much like to hear your proposal! It sounds like you
already have something going in the high level.

-Bram

[brian walsh]

Took a quick look. Don't see a specific pipes like functionality.


http://apatar.com/capabilities.html

Apatar brings you a set of unmatched capabilities in an open source package:

Connectivity to Oracle, MS SQL, MySQL, Sybase, DB2, MS Access, PostgreSQL, XML, InstantDB, Paradox, BorlandJDataStore, Csv, MS Excel, Qed, HSQL, Compiere ERP, SalesForce.Com, SugarCRM, Goldmine, any JDBC data sources and more.

Single interface to manage all your integration projects

Flexible deployment options

Bi-directional integration

Platform-independent, runs from Windows, Linux, Mac; 100% Java- based

Easy customization, Java source code included

No coding! Visual job designer and mapping enable non-developers to design and perform transformations

[brian walsh]

+1 for API ( both client & server side )

On Thu, Jan 15, 2009 at 1:01 PM, David Frey <intui...@gmail.com> wrote:

[Nate DiNiro]

Apatar does exactly what pipes does, albeit Apatar support more native
DB formats and lacks some of the "Web" formats like RSS, JSON, etc.
Yahoo! Pipes is simply a web-centric, SaaS version of data
transformation tools, but really more focused on the web. TIBCO is
probably the most well-known example of tools that fall into this
class.

On Jan 19, 5:18 pm, "brian walsh" <br...@bwalsh.com> wrote:
> Took a quick look. Don't see a specific pipes like functionality.
>
> http://apatar.com/capabilities.html
>
> Apatar brings you a set of unmatched capabilities in an open source package:
> Connectivity to Oracle, MS SQL, MySQL, Sybase, DB2, MS Access, PostgreSQL,
> XML, InstantDB, Paradox, BorlandJDataStore, Csv, MS Excel, Qed, HSQL,
> Compiere ERP, SalesForce.Com, SugarCRM, Goldmine, any JDBC data sources and
> more.
> Single interface to manage all your integration projects
> Flexible deployment options
> Bi-directional integration
> Platform-independent, runs from Windows, Linux, Mac; 100% Java- based
> Easy customization, Java source code included
> No coding! Visual job designer and mapping enable non-developers to design
> and perform transformations
>

> On Mon, Jan 19, 2009 at 4:52 PM, Dawn Foster <GeekyGirlD...@gmail.com>wrote:
>
>
>
> > @unclenate just mentioned thathttp://apatar.comis an open source

[Jerry]

I'm haven't worked with Apatar, but spent a lot of time with TIBCO.   The transformation piece is similar, but TIBCO and other service bus solutions have a lot of features that are a bit beyond a simple pipes app:  queuing, guaranteed msg delivery, pub-sub, etc.    The same is true of ETL packages:  many have graphical transformation tools, but that's just a part of what they do.

j.

[Leif Warner]

I was thinking XSLT does a lot of pipe-like functionality.
At the meeting where this was brought up, someone asked about wether
Yahoo Pipes offered the functionality of being fed in a list of feeds,
e.g. an OPML file?
So I made a trivial template that transforms a list of URLs into table
on a web page, with the first few entries on each feed on each row.
http://xslt.portlanddatasystems.com/proc.php?_xml=feeds.xml
It can also be run on the client, in browser (with no server-side
processing other than the proxy to get around the Firefox same-domain
restriction)
http://xslt.portlanddatasystems.com/client_feeds.xml

That's just something pulled together from all the feeds each time the
page is requested, so it would't exactly be scalable for loading up
your opml of 900 feeds -- generally I was thinking you'd want to cache
normalized feeds into a database something through a REST interface
prefferably. I had done this with the typefaces OPML -- fleshed out
the outline, and put it into an XML database, but I don't have a web-
accessible Java runtime at the moment.

As a more direct fit and general-purpose app, there's a rather new
open standard XML Pipeline langauge "XProc" with a couple
implementations out.
XProc has an advantage of mapping directly to the pipes structure
(connecting boxes together with tubes). Also, existing pipelines can
be boxed up and used as boxes within other pipelines (wired-together
boxes as a box in another pipeline).
It does seem more general in scope than pipes, besides aggregating,
sorting, mashups, it can also be used as a general build sysytem,
interacting with databases, generating PDFs...

Common to both languages, is that they're written in XML, which would
make it simple to programaticlly parse / generate to and from the
visual layout. And they both can load additional modules from URLs,
so one could use portions of a friend's pipe anywhere on the net.

I'd like to get together with some folks and work on making pipe-
things!

[Bram Pitoyo]

Leif,

Funny, I was looking into XML & XSLT this morning, and found out that
the way it’s written actually separates and splices data into
manageable chunk. I agree with your point: XSLT can do many things
that pipe can do—if we can find out how to properly query and index it.

-Bram

[Eric Hayes]

Yep Bram, that's the thing with XSLT... it is AMAZING, you just need
to remove your brain, and re-install it backwards to figure it out.

-eric

[M. Edward (Ed) Borasky]

On Tue, Feb 24, 2009 at 10:40 AM, Eric Hayes <ejh...@gmail.com> wrote:
>
> Yep Bram, that's the thing with XSLT...  it is AMAZING, you just need
> to remove your brain, and re-install it backwards to figure it out.
>
> -eric

Unfortunately, my brain is already installed inside out and mirror
imaged. As a result, I read FTW as WTF. :)
--
M. Edward (Ed) Borasky
http://www.linkedin.com/in/edborasky

I've never met a happy clam. In fact, most of them were pretty steamed.