Proposal: Using pandas as backend

[Stefan Urbanek]

Hi,

Here is a proposal of using Pandas data analysis library as one of potential backends for Brewery:

http://blog.databrewery.org/post/28088920149

Note that having dependency on Pandas will impose dependency on numpy and couple other packages. I do not think that more dependencies it good idea, therefore it should be only an alternative. It will require different approach to stream construction and different Node interface.

What do you think?

Stefan

Twitter: @Stiivi

Home: http://stiivi.com

Brewery: http://databrewery.org

Github: https://github.com/Stiivi

[Adrian Klaver]

On 07/27/2012 01:58 AM, Stefan Urbanek wrote:
> Hi,
>
> Here is a proposal of using Pandas data analysis library as one of
> potential backends for Brewery:
>
> http://blog.databrewery.org/post/28088920149

The blog page seems to be down. I cannot access the above post.

>
> Note that having dependency on Pandas will impose dependency on numpy
> and couple other packages. I do not think that more dependencies it good
> idea, therefore it should be only an alternative. It will require
> different approach to stream construction and different Node interface.
>
> What do you think?

As far as I gotten with Pandas is bookmarking the site for further
review. It looks interesting so I could see integrating it. I will try
to delve in deeper this weekend.

>
> Stefan

Thanks,

--
Adrian Klaver
adrian...@gmail.com

[Stefan Urbanek]

On 28.7.2012, at 16:23, Adrian Klaver <adrian...@gmail.com> wrote:

> On 07/27/2012 01:58 AM, Stefan Urbanek wrote:
>> Hi,
>>
>> Here is a proposal of using Pandas data analysis library as one of
>> potential backends for Brewery:
>>
>> http://blog.databrewery.org/post/28088920149
>
> The blog page seems to be down. I cannot access the above post.
>

It is tumblr hosted, sometimes refreshing multiple times helps (or accessing just blog.databrewery.org and then article).

>>
>> Note that having dependency on Pandas will impose dependency on numpy
>> and couple other packages. I do not think that more dependencies it good
>> idea, therefore it should be only an alternative. It will require
>> different approach to stream construction and different Node interface.
>>
>> What do you think?
>
> As far as I gotten with Pandas is bookmarking the site for further review. It looks interesting so I could see integrating it. I will try to delve in deeper this weekend.
>

That would be nice. Here is the backend idea explained:

http://yfrog.com/z/kl45czp

Basically Brewery streams is metadata-based description of workflow/thought flow alternatives. The backend serves as computational engine.

I am going to be offline (for most of the time) for next week, be back on 5th. We can discuss it afterwards.

[Naveen Michaud-Agrawal]

Hi Stephan,

I think this is a great idea! Has any progress been made on using pandas as a backend?

Naveen

[Stefan Urbanek]

Hi Naveen,

I didn't had too much time to implement that yet, unfortunately. However, I found another alternative: carray

https://github.com/francescAlted/carray/

Nice small framework which provides fast column-based storage layer with persistence. Implementation would be simpler than with Pandas, while still providing structures for algorithms that work with numpy arrays:

http://cl.ly/image/271k0d1k0815

What do you think?

s.

Stefan Urbanek

data analyst and data brewmaster

[Naveen Michaud-Agrawal]

Carray looks like a good alternative for persistence. I was thinking that pandas would help in implementing some of the higher level backend interface, since it has very fast grouping and filtering.

Naveen