Dynamic eager load options

218 views
Skip to first unread message

John Firebaugh

unread,
Feb 28, 2011, 9:54:17 PM2/28/11
to seque...@googlegroups.com
I have a problem I've been banging my head against for awhile and I can't seem to figure out a good approach. Imagine an application that collects some historical data over time for a related model:

class Albums < Sequel::Model
  one_to_many :daily_sales_figures, :order => :date
end

class DailySalesFigure < Sequel::Model
  # Includes date column and numerous statistics such as units sold, average price, etc.
  many_to_one :album
end

The user wants to see a list of albums, each with a sparkline showing a daily sales figure for that album. For the sparkline, they want to be able to choose both the time period (last quarter, YTD, etc.) and the statistic (units sold, average price, etc.).

The obvious way to implement this is as a query over albums, eager loading :daily_sales_figures. Here's the nut of the problem: suppose sales figures go back for decades, and include hundreds of statistics, so that for performance reasons, it would be highly advantageous to be able to filter the eager loaded dataset to just the chosen time period, and select only the single column needed for the sparkline. There doesn't seem to be an easy way to do this with Sequel -- #eager uses the association options (e.g. :conditions, :select) that were fixed at association definition time, whereas the user's choices change from request to request.

Right now I'm doing some crazy stuff involving dynamically creating new association reflections at query time, but it's really really ugly. I've also considered constructing the join and updating association caches manually -- this doesn't seem ideal either, as it essentially means a bespoke reimplementation of #eager at every place this needs to be done (and I have multiple analogous models). Is there some other way to get some dynamism with eager loading?

John Firebaugh

unread,
Feb 28, 2011, 11:49:23 PM2/28/11
to seque...@googlegroups.com
I wonder if it would be feasible to support datasets as the values in the hash passed to #eager:

   Albums.eager(:daily_sales_figures => DailySalesFigures.filter(...).select(...))

This could provide a way to customize the associated dataset, and still nest eager loading:

   Albums.eager(:daily_sales_figures => DailySalesFigures.filter(...).eager(:foo => :bar))

Jeremy Evans

unread,
Mar 1, 2011, 1:52:51 AM3/1/11
to sequel-talk
If you think about what your asking for, then you really are creating
a new association per query, since you are changing the parameters
under which the association operates. However, in your case, hard
coding each possible combination probably isn't acceptable.

If you really want to do this, it's been possible without much
trickery since the eager loader started accepting a single hash of
options. You'll need to have a custom eager loader, but you can
reference the :self entry in the hash to get the parent dataset. Then
you can set eager loader parameters like:

class Album < Sequel::Model
one_to_many
daily_sales_figures, :order=>:date, :eager_loader=>(proc do |h|
opts = h[:self].opts[:daily_sales_figures_opts]
DailySalesFigure.filter(opts[:filter]).select(*opts[:select])...
end)

def_dataset_method(:with_daily_sales_figures) do |v|
clone(:daily_sales_figures_opts => v).eager(daily_sales_figures)
end
end

Album.with_daily_sales_figures(:filter=>foo, :select=>[:col1, :col2])

I suppose it would be possible to use datasets as hash values, as you
suggest, even though hash values are currently used for recursive/
chained eager loading. That's obviously an easier way to get dynamic
eager loading compared to my code above. It would require changing
the default eager loaders, and putting the dataset to use in the hash
passed to the eager loaders. You'd also need to deal with possible
recursion when also specifying a dataset argument. Those shouldn't be
too difficult cases to handle, though. After 3.21.0 is released
tomorrow, give a shot a modifying the eager_load method to handle that
case using your proposed API, and see how it works in your app. If
you find it works well, send in a patch with specs.

Thanks,
Jeremy

Jeremy Evans

unread,
Mar 1, 2011, 3:39:04 PM3/1/11
to sequel-talk
On Feb 28, 10:52 pm, Jeremy Evans <jeremyeva...@gmail.com> wrote:
> I suppose it would be possible to use datasets as hash values, as you
> suggest, even though hash values are currently used for recursive/
> chained eager loading.  That's obviously an easier way to get dynamic
> eager loading compared to my code above.  It would require changing
> the default eager loaders, and putting the dataset to use in the hash
> passed to the eager loaders.  You'd also need to deal with possible
> recursion when also specifying a dataset argument.  Those shouldn't be
> too difficult cases to handle, though.  After 3.21.0 is released
> tomorrow, give a shot a modifying the eager_load method to handle that
> case using your proposed API, and see how it works in your app.  If
> you find it works well, send in a patch with specs.

I gave this some more thought and think that instead of providing a
dataset, you can provide a proc that is called with default dataset
that would be used, and should return a dataset to be used:

Albums.eager(:daily_sales_figures =>
proc{|ds| ds.filter(...).select(...)})

Using callbacks in this way is much more flexible, and probably more
concise in most cases. What do you think?

Jeremy

Michael Lang

unread,
Mar 2, 2011, 10:17:15 AM3/2/11
to seque...@googlegroups.com, Jeremy Evans
> I gave this some more thought and think that instead of providing a
> dataset, you can provide a proc that is called with default dataset
> that would be used, and should return a dataset to be used:
>
>  Albums.eager(:daily_sales_figures =>
>    proc{|ds| ds.filter(...).select(...)})
>
> Using callbacks in this way is much more flexible, and probably more
> concise in most cases.  What do you think?
>

I wish I understood using procs better. :-)

Are there any blog postings or tutorials out there that cover
employing procs in a more readable/use-case scenario? I've seen a few
that talk about procs and give examples, but not in the context of how
I can actually put one to use. The examples are usually quite exotic,
at least to me. The above short example is actually the few *useful*
scenarios I've seen for how one might employ procs.

Michael
--
http://codeconnoisseur.org

Scott LaBounty

unread,
Mar 2, 2011, 10:35:03 AM3/2/11
to seque...@googlegroups.com, Michael Lang, Jeremy Evans
I'd take a look at Ruby Metaprogramming (http://www.amazon.com/Metaprogramming-Ruby-Program-Like-Pros/dp/1934356476/ref=sr_1_1?ie=UTF8&qid=1299079977&sr=8-1) by Paolo Perrotta for a good explanation of proc's (and lambda's etc.).


On Wed, Mar 2, 2011 at 7:17 AM, Michael Lang <mwl...@cybrains.net> wrote:

I wish I understood using procs better.  :-)

Are there any blog postings or tutorials out there that cover
employing procs in a more readable/use-case scenario?  I've seen a few
that talk about procs and give examples, but not in the context of how
I can actually put one to use.  The examples are usually quite exotic,
at least to me. The above short example is actually the few *useful*
scenarios I've seen for how one might employ procs.

Michael
--
http://codeconnoisseur.org

--

--
Scott
http://steamcode.blogspot.com/
Reply all
Reply to author
Forward
0 new messages