Account Options

  1. Sign in
Google Groups Home
« Groups Home
help with aggregation & group by
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  11 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Chuck Remes  
View profile  
 More options Nov 5 2007, 10:43 pm
From: Chuck Remes <cremes.devl...@mac.com>
Date: Mon, 5 Nov 2007 21:43:23 -0600
Local: Mon, Nov 5 2007 10:43 pm
Subject: help with aggregation & group by
I have a table where I'm trying to get the average time delta for each  
minute of the past X minutes. I have the SQL written and working but  
I'm unsure how to translate it into Sequel ruby; it may not even be  
possible but I thought I'd ask the list.

Here's my table definition and the SQL I am running against it:

class Order < Sequel::Model(:orders)
set_schema do
   primary_key :id, :auto_increment => true
   varchar :market, :size => 12
   decimal :created_at, :elements => [15,5]
   decimal :ack, :elements => [15,5]
end
end

select market, minute(from_unixtime(ack)) AS minute, round(avg((ack -  
created_at) * 1000), 2) AS ack_average from orders  where 'ack' >  
(UNIX_TIMESTAMP(now()) - (15 * 60) and 'market' = 'ICE' group by  
minute(from_unixtime(ack));

As you may infer, the ack and created_at columns are large floating  
point numbers (seconds plus microseconds) representing time relative  
to the UNIX epoch. Is it possible to write this in Sequel and have it  
generate the correct SQL? Here's an attempt...

market = 'ICE'
Order.filter{:market => market && :ack >  
15.minutes.ago}.avg(:no_idea_what_to_do_from_here).group(:still_lost)

I need to do math (subtraction) on two columns in the result set and  
average that value. I also need access to mysql's +minute+ function  
and +unix_timestamp+ function. I looked through the code but got lost  
pretty quickly when trying to figure out how to monkey patch this in.

Am I best off just executing the SQL above and saving it into a  
dataset for ruby manipulation?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sharon Rosner  
View profile  
 More options Nov 6 2007, 12:21 am
From: Sharon Rosner <cico...@gmail.com>
Date: Mon, 05 Nov 2007 21:21:31 -0800
Local: Tues, Nov 6 2007 12:21 am
Subject: Re: help with aggregation & group by

> Am I best off just executing the SQL above and saving it into a
> dataset for ruby manipulation?

Right now yes. You can do that easily:

  sql = "select market, minute(from_unixtime(ack)) AS minute,
round(avg((ack -
  created_at) * 1000), 2) AS ack_average from orders  where 'ack' >
  (UNIX_TIMESTAMP(now()) - (15 * 60) and 'market' = 'ICE' group by
  minute(from_unixtime(ack));"

  dataset = DB[sql]
  dataset.each {|r| p r}

I've played with it a bit and right now, field quoting in the mysql
adapter is broken, so you can't use any functions or formulas in
#select, #group etc. Ultimately our goal should look like this:

  market = 'ICE'
  ack_stamp = 15.minutes.ago
  Order.query do
    select do
      :market,
      :minute[:from_unixtime[:ack]].AS :minute
      :round[:avg[(:ack - :created_at) * 1000], 2].AS :ack_average
    end
    where do
      :ack > ack_stamp
      :market == market
    end
    group_by :minute[:from_unixtime[:ack]]
  end

With the help of ParseTree I think it's possible to do that, and also
to fix field quoting. Can you please open an issue for this?

best
Sharon


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Van De Vyver  
View profile  
 More options Nov 6 2007, 2:18 am
From: "Mark Van De Vyver" <mvy...@gmail.com>
Date: Tue, 6 Nov 2007 18:18:38 +1100
Local: Tues, Nov 6 2007 2:18 am
Subject: Re: help with aggregation & group by
Hi,
A note on this...

On Nov 6, 2007 4:21 PM, Sharon Rosner <cico...@gmail.com> wrote:

... the following presentation (discussed on the Nitro mail list) and
the code beneath it illustrate a neat implementation of just this sort
of thing.  It seems very elegant and it would simplify the above
syntax, so may be worth looking into?

 http://www.xaop.com/articles/2007/10/07/metaprogramming

HTH
Mark


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sharon Rosner  
View profile  
 More options Nov 6 2007, 3:50 am
From: Sharon Rosner <cico...@gmail.com>
Date: Tue, 06 Nov 2007 00:50:02 -0800
Local: Tues, Nov 6 2007 3:50 am
Subject: Re: help with aggregation & group by

> ... the following presentation (discussed on the Nitro mail list) and
> the code beneath it illustrate a neat implementation of just this sort
> of thing.  It seems very elegant and it would simplify the above
> syntax, so may be worth looking into?

Sequel is already much more advanced than that. Have a look at the way
you can specify filters in Sequel. My plan is to take the the great
filtering capability that we have and extend it so you can specify
Ruby expressions for any SQL clause and have it automagically
translated to SQL.

sharon


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Van De Vyver  
View profile  
 More options Nov 6 2007, 4:24 am
From: "Mark Van De Vyver" <mvy...@gmail.com>
Date: Tue, 6 Nov 2007 20:24:37 +1100
Local: Tues, Nov 6 2007 4:24 am
Subject: Re: help with aggregation & group by
On Nov 6, 2007 7:50 PM, Sharon Rosner <cico...@gmail.com> wrote:

> > ... the following presentation (discussed on the Nitro mail list) and
> > the code beneath it illustrate a neat implementation of just this sort
> > of thing.  It seems very elegant and it would simplify the above
> > syntax, so may be worth looking into?

> Sequel is already much more advanced than that. Have a look at the way
> you can specify filters in Sequel.

Apologies, I'm not yet familiar with the filters.  I was just
superficially comparing the code inside the query block with Peter's,
in particular I thought:

from   :c => Customer, :b => Bank, :a => Account
  select b.name
  where  ((c.firstname == first1) |
          (c.firstname == first2)), etc.

appeared simpler than having blocks and .AS, but no doubt there is a
limit to his approach I'm unaware of.

> My plan is to take the the great
> filtering capability that we have and extend it so you can specify
> Ruby expressions for any SQL clause and have it automagically
> translated to SQL.

Sounds great!
Thanks for all the great work.

Regards
Mark


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sharon Rosner  
View profile  
 More options Nov 6 2007, 4:40 am
From: Sharon Rosner <cico...@gmail.com>
Date: Tue, 06 Nov 2007 01:40:08 -0800
Local: Tues, Nov 6 2007 4:40 am
Subject: Re: help with aggregation & group by

> from   :c => Customer, :b => Bank, :a => Account
>   select b.name
>   where  ((c.firstname == first1) |
>           (c.firstname == first2)), etc.

> appeared simpler than having blocks and .AS, but no doubt there is a
> limit to his approach I'm unaware of.

Actually for that kind of thing you can specify your filter as a hash:

  where(:firstname => [first1, first2])

Notice how you can provide an array and Sequel takes care of
constructing the filter. Also gives you nicer-looking SQL:

  ... WHERE firstname IN ('john', 'george')

If you need to qualify the field name you can use the special symbol
notation:

  where(:c__firstname => [first1, first2])

Sequel also lets you use any comparison operator, not just ==:

  where {:price > 100}

You can also specify mathemtic expressions and compare against other
fields:

  where {:price < :avg[:price] - 100}

Have a look at the cheat sheet:

  http://code.google.com/p/ruby-sequel/wiki/CheatSheet

sharon


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Van De Vyver  
View profile  
 More options Nov 6 2007, 5:23 am
From: "Mark Van De Vyver" <mvy...@gmail.com>
Date: Tue, 6 Nov 2007 21:23:47 +1100
Local: Tues, Nov 6 2007 5:23 am
Subject: Re: help with aggregation & group by
I'm sold, I've just seen the :table__field___alias
 syntax too :)

On Nov 6, 2007 8:40 PM, Sharon Rosner <cico...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Chuck Remes  
View profile  
 More options Nov 6 2007, 7:05 am
From: Chuck Remes <cremes.devl...@mac.com>
Date: Tue, 6 Nov 2007 06:05:54 -0600
Local: Tues, Nov 6 2007 7:05 am
Subject: Re: help with aggregation & group by

On Nov 5, 2007, at 11:21 PM, Sharon Rosner wrote:

>> Am I best off just executing the SQL above and saving it into a
>> dataset for ruby manipulation?

> Right now yes. You can do that easily:

>  sql = "select market, minute(from_unixtime(ack)) AS minute,
> round(avg((ack -
>  created_at) * 1000), 2) AS ack_average from orders  where 'ack' >
>  (UNIX_TIMESTAMP(now()) - (15 * 60) and 'market' = 'ICE' group by
>  minute(from_unixtime(ack));"

>  dataset = DB[sql]
>  dataset.each {|r| p r}

This is the method I'm using right now. It's somewhat ugly but still  
very functional!

Ooh, that's pretty!

> With the help of ParseTree I think it's possible to do that, and also
> to fix field quoting. Can you please open an issue for this?

Just opened it. Listed as Defect #88.

I'll take a stab at providing some of this functionality but it means  
learning ParseTree and probably ruby2ruby which might be a bit beyond  
my ruby capabilities right now. Don't hold your breath waiting for my  
patch... ;-)

cr


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Sharon Rosner  
View profile  
 More options Nov 6 2007, 1:06 pm
From: Sharon Rosner <cico...@gmail.com>
Date: Tue, 06 Nov 2007 10:06:19 -0800
Local: Tues, Nov 6 2007 1:06 pm
Subject: Re: help with aggregation & group by

> Ooh, that's pretty!

Column quoting is fixed in the trunk. The only thing that's missing to
make the example work is for #select to accept blocks. We'll get
there.

sharon


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Lee  
View profile  
 More options Jan 3 2008, 5:03 pm
From: David Lee <davidomu...@gmail.com>
Date: Thu, 3 Jan 2008 14:03:43 -0800 (PST)
Local: Thurs, Jan 3 2008 5:03 pm
Subject: Re: help with aggregation & group by
You guys might already be aware of this, but SQLDSL by Jay Fields
already supports this type of direct ruby-to-sql.

http://sqldsl.rubyforge.org/

It could be helpful to collaborate SQLDSL.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
David Lee  
View profile  
 More options Jan 3 2008, 5:11 pm
From: David Lee <davidomu...@gmail.com>
Date: Thu, 3 Jan 2008 14:11:17 -0800 (PST)
Local: Thurs, Jan 3 2008 5:11 pm
Subject: Re: help with aggregation & group by
Another food for thought would be to have the Order.query cache the
generated SQL query so the second time the block is run, it doesn't
have to generate the SQL code all over again.

A potential problem is the fixation of dynamically generated values
such as the "market" or "ack_stamp" variables in the example code.

I'm not sure how far you can go with ParseTree, but you could search
for the presence of dynamic elements (variables, methods...) in the
block passed to Order.query and turn on caching only if it only
contains static elements (constants, integers, symbols...).


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »