Possible bug with chained where methods on 3.0.4

182 views
Skip to first unread message

Colin Law

unread,
Feb 16, 2011, 4:52:21 AM2/16/11
to rubyonra...@googlegroups.com
HI

I notice on Rails 3.0.4 that the following two statements generate
different results.

State.where(:abbreviation => 'TX').where(:abbreviation => 'NE')
SQL: SELECT `states`.* FROM `states` WHERE (`states`.`abbreviation` =
'TX') OR (`states`.`abbreviation` = 'NE')

State.where(:abbreviation => 'TX').where("abbreviation = 'NE'")
SQL: SELECT `states`.* FROM `states` WHERE (`states`.`abbreviation` =
'TX') AND (`states`.`abbreviation` = 'NE')

Using Rails 3.0.3 they both generated AND clauses which seems to me to
be correct.

Is this a bug or by design please?

Colin

Ernie Miller

unread,
Feb 16, 2011, 6:21:54 AM2/16/11
to rubyonra...@googlegroups.com

The OR functionality is new, but intentional. See
Relation::QueryMethods#collapse_wheres. Since the idea of having a
column being equal to two different values doesn't make sense,
equalities with the same left-hand side get combined with OR, which is
more likely the desired result if two scopes are merged and both
include equality conditions on the same column.

Since your example above used a string in the second where call, it
didn't result in an Arel::Nodes::Equality, so it wasn't ORed.

Robert Pankowecki

unread,
Feb 16, 2011, 6:35:57 AM2/16/11
to rubyonra...@googlegroups.com, Ernie Miller
On Wed, Feb 16, 2011 at 12:21 PM, Ernie Miller <er...@metautonomo.us> wrote:
> which is
> more likely the desired result if two scopes are merged and both
> include equality conditions on the same column.

I doubt that. There are a lot of cases (especially when using
search/filter functionality of application) where you want it to
generate an empty set because there are no matching records. Everybody
everywhere in current code assumes that the condition will be added
with 'AND' operator and now we are breaking this assumption ?

If I want to have 'OR' then I would build the query this way:

State.where(:abbreviation => ['TX', 'NE'])
or using longer but perfectly fine and valid arel version...

An example:

def where_something(scope, value)
scope.where(:column => value)
end

This code is ambiguous now. I do not know what sql it is going to
generate. Will it be
"AND column = value"
or
"(... OR column = value)" ?

Please revert it to the original behavior and provide a different way
of achieving this like:

State.possible(:abbreviation => 'TX').possible(:abbreviation => 'NE') =>
sql: "states.abbrevation = 'TX' OR states.abbrevation = 'NE"

or whatever different method name that reveals the intentions clearly

Robert Pankowecki

Will Bryant

unread,
Feb 16, 2011, 6:37:34 AM2/16/11
to rubyonra...@googlegroups.com
On Thu, Feb 17, 2011 at 12:21 AM, Ernie Miller <er...@metautonomo.us> wrote:
> Since your example above used a string in the second where call, it
> didn't result in an Arel::Nodes::Equality, so it wasn't ORed.

Even on just that point alone, this is a really bad API idea - it is
completely inconsistent, and this will cause subtle and confusing bugs
when people change between two otherwise-identical forms of where()
argument.

And to do it at all breaks the relational algebra idea badly. If I
pass a scope to something, and it does several different queries on it
using its own where scopes, each of those scopes should further
restrict the original scope, not broaden it.

With this commit, now neither callers nor callees have any idea what
where(:hash => arguments) will result in - whether the returned
results will match the conditions hash - unless they have constructed
the entire scope from scratch.

I also struggle to understand how it was decided that a minor point
patch release - for a security issue no less - should include an API
behavior change that will cause preexisting code to start getting
completely different query results.

Colin Law

unread,
Feb 16, 2011, 6:39:00 AM2/16/11
to rubyonra...@googlegroups.com

OK, thanks for that. If that is the way it is supposed to be then
that is the way it is supposed to be. It seems odd to me that
chaining a second where *adds* to the dataset of the first where. It
would seem particularly unexpected in the case
items = Model.where( :x => y)
...
...
myitems = items.where(:x => z)
resulting in myitems having more records than items, and in particular
records that do not satisfy :x => z. Note that at this point the code
would not necessarily 'know' the details of the first scope.

Colin

Ernie Miller

unread,
Feb 16, 2011, 7:24:16 AM2/16/11
to rubyonra...@googlegroups.com

Sound arguments. I didn't implement collapse_wheres, just in case a
witch hunt starts. I was just the first person awake this morning to
pose the answer. :)

The change probably shouldn't have made it into a point release, but I
do find a certain convenience and logic to it, and I long for a day
when we aren't hacking about trying to make a String and a Hash act
like they're the same thing. Hashes aren't strings, and (IMHO, and I
know there are people on the core team who disagree) it would be
better if we gradually eliminated hand coded SQL string conditions and
relied more heavily on hashes and other data structures, converting
them to SQL when needed via ARel. They're already a last resort in my
own code and I expect them to be relatively brittle when I use them.

Ernie Miller

unread,
Feb 16, 2011, 7:40:51 AM2/16/11
to Ruby on Rails: Core


On Feb 16, 7:24 am, Ernie Miller <er...@metautonomo.us> wrote:
>
> Sound arguments. I didn't implement collapse_wheres, just in case a
> witch hunt starts. I was just the first person awake this morning to
> pose the answer. :)
>
> The change probably shouldn't have made it into a point release, but I
> do find a certain convenience and logic to it, and I long for a day
> when we aren't hacking about trying to make a String and a Hash act
> like they're the same thing. Hashes aren't strings, and (IMHO, and I
> know there are people on the core team who disagree) it would be
> better if we gradually eliminated hand coded SQL string conditions and
> relied more heavily on hashes and other data structures, converting
> them to SQL when needed via ARel. They're already a last resort in my
> own code and I expect them to be relatively brittle when I use them.

I should also probably note that I went off on a sort of anti-String
tangent there, unrelated to the topic at hand. So please read:

"...a certain convenience and logic to it, and I long for..."

as:

"...a certain convenience and logic to it. On another note, I long
for..."


Thanks. :)

Brian Morearty

unread,
Feb 16, 2011, 8:35:43 AM2/16/11
to Ruby on Rails: Core
> Sound arguments. I didn't implement collapse_wheres, just in case a
> witch hunt starts. I was just the first person awake this morning to
> pose the answer. :)

Understood. :-)

> With this commit, now neither callers nor callees have any idea what
> where(:hash => arguments) will result in - whether the returned
> results will match the conditions hash - unless they have constructed
> the entire scope from scratch.

I agree with this reasoning, too. Under the 3.0.4 logic, we won't be
able to chain independent scopes together on the fly with any
confidence. E.g., a loop that iterates over various input conditions
and adds scopes one by one based on the input.

+1 in favor of reverting to the original behavior: chaining 'where'
clauses should consistently use 'AND'.

Brian



On Feb 16, 4:24 am, Ernie Miller <er...@metautonomo.us> wrote:

Chris Cruft

unread,
Feb 16, 2011, 8:40:32 AM2/16/11
to Ruby on Rails: Core
+1 for the the original behavior where chained clauses are ANDed
(POLS).

Aaron Patterson

unread,
Feb 16, 2011, 11:59:38 AM2/16/11
to rubyonra...@googlegroups.com
On Wed, Feb 16, 2011 at 06:21:54AM -0500, Ernie Miller wrote:
> On Feb 16, 2011, at 4:53 AM, Colin Law <cla...@googlemail.com> wrote:
>
> > HI
> >
> > I notice on Rails 3.0.4 that the following two statements generate
> > different results.
> >
> > State.where(:abbreviation => 'TX').where(:abbreviation => 'NE')
> > SQL: SELECT `states`.* FROM `states` WHERE (`states`.`abbreviation` =
> > 'TX') OR (`states`.`abbreviation` = 'NE')
> >
> > State.where(:abbreviation => 'TX').where("abbreviation = 'NE'")
> > SQL: SELECT `states`.* FROM `states` WHERE (`states`.`abbreviation` =
> > 'TX') AND (`states`.`abbreviation` = 'NE')
> >
> > Using Rails 3.0.3 they both generated AND clauses which seems to me to
> > be correct.
> >
> > Is this a bug or by design please?
> >
> > Colin
>
> The OR functionality is new, but intentional. See
> Relation::QueryMethods#collapse_wheres. Since the idea of having a
> column being equal to two different values doesn't make sense,
> equalities with the same left-hand side get combined with OR, which is
> more likely the desired result if two scopes are merged and both
> include equality conditions on the same column.

The intentional change fixes this bug:

https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/4598-default_scope-treats-hashes-and-relations-inconsistently-when-overwriting

If there is a different way to fix that ticket, I'm happy to apply
patches.

--
Aaron Patterson
http://tenderlovemaking.com/

Robert Pankowecki

unread,
Feb 16, 2011, 12:10:25 PM2/16/11
to rubyonra...@googlegroups.com, Aaron Patterson
On Wed, Feb 16, 2011 at 5:59 PM, Aaron Patterson
<aa...@tenderlovemaking.com> wrote:
> The intentional change fixes this bug:
>
>  https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/4598-default_scope-treats-hashes-and-relations-inconsistently-when-overwriting
>
> If there is a different way to fix that ticket, I'm happy to apply
> patches.

The different way is to make both cases return an empty array. David
(who created the ticket) expects something different but I am not sure
that this is what most people expect.

Could we provide a different method for such behavior as expected by David ?

Robert Pankowecki

James B. Byrne

unread,
Feb 16, 2011, 12:52:30 PM2/16/11
to rubyonra...@googlegroups.com

On Wed, February 16, 2011 12:10, Robert Pankowecki wrote:
>
>
> The different way is to make both cases return an empty array. David
> (who created the ticket) expects something different but I am not
> sure that this is what most people expect.
>
> Could we provide a different method for such behavior as expected by
> David ?
>
> Robert Pankowecki

Perhaps:

#where_in == OR

#where == AND


???

--
*** E-Mail is NOT a SECURE channel ***
James B. Byrne mailto:Byr...@Harte-Lyne.ca
Harte & Lyne Limited http://www.harte-lyne.ca
9 Brockley Drive vox: +1 905 561 1241
Hamilton, Ontario fax: +1 905 561 0757
Canada L8E 3C3

Jason King

unread,
Feb 16, 2011, 1:22:54 PM2/16/11
to rubyonra...@googlegroups.com
Just want to get this onto the table, "what most people expect" is not the only measure of a good API choice.

To specifically answer Aaron's question, for #4598, I think the fix for that would be for subsequent `default_scope`s to just overwrite previous ones entirely.  Not to merge with them.

For the broader question which seems to be what this thread is about, ie. what to do with: User.where(:foo => 1).where(:foo => 2) how about adding alias `and` as an alias for `where`, and adding `or` as an alias for "where that does ORs".  So the syntax could be:

User.where(:foo => 1).and(:foo => 2)

And:

User.where(:foo => 1).or(:foo => 2)

I'm sure I don't need to state what SQL each of those turn into (the real test for a good API :)


--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To post to this group, send email to rubyonra...@googlegroups.com.
To unsubscribe from this group, send email to rubyonrails-co...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.


Steve Schwartz

unread,
Feb 16, 2011, 12:23:31 PM2/16/11
to rubyonra...@googlegroups.com
Agreed, the OR behavior seems inconsistent and unintuitive. In fact, the original solution suggested was to AND them together to make case 1 consistent with case 2 (rather than ORing them to make case 2 consistent with case 1).


That seems more sane and less radical than having a single special case where chained methods get ORd. Especially when Arel itself already has its very own #or method to accomplish this within the relation chaining.

It seems odd to me that chaining a second where *adds* to the dataset of the first where.

Exactly.

-Steve
@jangosteve

Jason King

unread,
Feb 16, 2011, 1:27:26 PM2/16/11
to rubyonra...@googlegroups.com
Btw, that could also resolve the "how to combine `default_scope`s issue, if that's something that is still desired.

default_scope where( :foo => 1 )

And then:

default_scope where( :foo => 2 ) # effectively negates previous default scope
default_scope and( :foo => 2 ) # same as above
default_scope or( :foo => 2 ) # merges with original

All nice and explicit.

AND:

User.or( :foo => 2 )

...would also combine with any existing default_scope, whereas:

User.where( :foo => 2 )

Would AND it and therefore negate the default scope.

Ernie Miller

unread,
Feb 16, 2011, 1:29:53 PM2/16/11
to rubyonra...@googlegroups.com
On Feb 16, 2011, at 1:22 PM, Jason King wrote:

Just want to get this onto the table, "what most people expect" is not the only measure of a good API choice.

To specifically answer Aaron's question, for #4598, I think the fix for that would be for subsequent `default_scope`s to just overwrite previous ones entirely.  Not to merge with them.

Pretty sure the reason default scopes are cumulative is to support STI classes refining their parent classes default scope, but I could be wrong. Outside of that context, I can't think of a use case for calling it multiple times, off the top of my head. Happy to be corrected, though.

Matt Jones

unread,
Feb 16, 2011, 1:59:11 PM2/16/11
to rubyonra...@googlegroups.com

On Feb 16, 2011, at 1:22 PM, Jason King wrote:

> Just want to get this onto the table, "what most people expect" is not the only measure of a good API choice.
>
> To specifically answer Aaron's question, for #4598, I think the fix for that would be for subsequent `default_scope`s to just overwrite previous ones entirely. Not to merge with them.
>
> For the broader question which seems to be what this thread is about, ie. what to do with: User.where(:foo => 1).where(:foo => 2) how about adding alias `and` as an alias for `where`, and adding `or` as an alias for "where that does ORs". So the syntax could be:
>
> User.where(:foo => 1).and(:foo => 2)
>
> And:
>
> User.where(:foo => 1).or(:foo => 2)
>
> I'm sure I don't need to state what SQL each of those turn into (the real test for a good API :)

I suspect the 'and' behavior is already pretty well understood as the result of chaining 'where's. What about an explicit one for the 'or' case:

User.where(:foo => 1).or_where(:foo => 2)

The only issue I could see with this syntax is that it's somewhat unclear what should happen here:

User.where(:foo => 1).where(:bar => 2).or_where(:bar => 3)

as it could be interpreted as either

Version A: (foo = 1 AND bar = 2) OR (bar = 2)

or

Version B: (foo = 1) AND (bar = 1 OR bar = 2)

The issue is especially tricky once you're constructing relations in more than one place. For instance, this should do something reasonable:

r = User.where(:foo => 1)
def foo(some_relation)
some_relation.where(:bar => 1).or_where(:bar => 2) # probably expects version B SQL
end
foo(r) # probably expects version B SQL

but so should this:
r2 = User.where(:foo => 1).where(:bar => 1)
def other_foo(some_relation)
some_relation.or_where(:bar => 2)
end
other_foo(r2) # probably expects version A SQL

the difficulty is that the sequence of method calls on the relation are identical, they're just in different scopes.

On the other hand, isn't this sort of issue exactly why Arel has operators to combine scopes? The chained notation is alright for some things, but it's never going to be sufficient to create arbitrary SQL without a lot of fussing.

--Matt Jones

Jason King

unread,
Feb 16, 2011, 3:27:52 PM2/16/11
to rubyonra...@googlegroups.com
Perhaps or() and where/and (I mainly add `and` because it's symmetrical with `or` and adds a lot of readability IMHO) could take other where() snippets?  So your examples become:

r = User.where( :foo => 1 )
r = r.and( r.where( :bar => 1 ).or( :bar => 2 ))

...and...

r = User.where( :foo => 1 ).where( :bar => 1 )
r = r.or( :bar => 2 )

...respectively.


--

Jason King

unread,
Feb 16, 2011, 3:50:26 PM2/16/11
to rubyonra...@googlegroups.com
I get the feeling that those in the know are just hanging back waiting for us to tie ourselves in syntactic knots.

I don't want to derail the original thread here, which is dealing with the much more common and simple case of providing a consistent API for when to add an AND and when to combine same-name parameters.

If we postpone any thoughts of creating complex nested WHERE clauses, and in effect (probably using the IN function) limit our parenthesizing to same-name columns, then I think there's still a lot to be gained by making the OR explicit.

Then for non-same-name columns, we just leave it up the programmer and the call order (ie. no other parenthesis).

Jeremy Evans

unread,
Feb 16, 2011, 4:16:01 PM2/16/11
to rubyonra...@googlegroups.com

Having where behave differently depending on whether the column was already used elsewhere in the relation is pure madness.  If you do:

  relation.where(:column => 1).all

All records returned by it should have 1 as the value of column, regardless of the contents of relation.  Using where should filter the relation, it should always return a subset of the receiver.

FWIW, in Sequel, #where always operates as a filter (uses AND), and you can call #or if you want to use OR.

Jeremy

Akira Matsuda

unread,
Feb 17, 2011, 1:39:39 AM2/17/11
to Ruby on Rails: Core
+1 for reverting this ASAP.
The reason was already very clearly explained in the 4th post of this
thread.
I believe, this is the most important point.

On Wed, Feb 16, 2011 at 8:37 PM, Will Bryant <will....@gmail.com>
wrote:
> And to do it at all breaks the relational algebra idea badly.

Now, let me show you my real world tragedy.

# model
class User
scope :admin, where(:id => [1, 2, 3, 4])
end

# controller
def admin?
User.admin.where(:id => session[:user_id]).exists?
end

Imagine what happened to my app!


As for the "OR" API, I think squirrel had the best syntax against
ambiguity.

Book.written_by('david').published #=> AND

Book.or {
written_by('david')
published
} #=> OR

--
Akira Matsuda<ron...@dio.jp>

Akira Matsuda

unread,
Feb 17, 2011, 1:10:36 AM2/17/11
to rubyonra...@googlegroups.com

Robert Pankowecki

unread,
Feb 23, 2011, 6:21:03 PM2/23/11
to rubyonra...@googlegroups.com
This is fixed in
https://github.com/rails/rails/commit/41c35abed778a7a03e7a9c14009321cd6bc11c26
and released in v3.0.5.rc1 right ? Shall we update the ticket
https://rails.lighthouseapp.com/projects/8994/tickets/6188-model-name-for-default-submit-button-value-should-be-titleized
and make sure it works the same way in master ?

Robert Pankowecki

Aaron Patterson

unread,
Feb 23, 2011, 6:40:02 PM2/23/11
to rubyonra...@googlegroups.com
On Thu, Feb 24, 2011 at 12:21:03AM +0100, Robert Pankowecki wrote:
> This is fixed in
> https://github.com/rails/rails/commit/41c35abed778a7a03e7a9c14009321cd6bc11c26
> and released in v3.0.5.rc1 right ? Shall we update the ticket
> and make sure it works the same way in master ?

It's fixed in 3.0.5.rc1 and in master. :-)

Reply all
Reply to author
Forward
0 new messages