ActiveRecord interface to Arel is awkward, requires find_by_sql(query.to_sql)

[Bruce Perens]

Hi,

I have a complex query that, as far as I can tell, either requires
Arel or SQL. Performing Arel queries with ActiveRecord is awkward.
Getting the query into ActiveRecord requires
project(node_table.columns)
and then
find_by_sql(query.to_sql)
There should be a method of ActiveRecord that accepts an Arel query,
does the appropriate projection, and executes the query. Converting
Arel queries to SQL defeats the purpose of Arel. Arel should not be
dependent on SQL and should be able to work on no-SQL databases.

My entire query looks like this

space_table = [1, 2, 3]
path = 'Layouts/Default'

node_table = self.arel_table
dependency_table = Perens::Cms::Dependency.arel_table

query = node_table \
.project(node_table.columns) \
.where(node_table[:space_id].in(space_list))
.join(dependency_table) \
.on(dependency_table[:node_id].eq(node_table[:id])) \
.where(dependency_table[:path].eq(path))

dependents = self.find_by_sql(query.to_sql)

Core to this issue is the ActiveRelation vs. Arel dichotomy.
ActiveRelation is Arel's poor cousin that can't measure up to it. It's
an easier interface as long as your main query is where(field: value),
but doesn't provide all of the functionality of Arel.

Thanks

Bruce

[Ryan Bigg]

Hi Bruce,

On Sunday, 3 June 2012 at 5:05 AM, Bruce Perens wrote:

Hi,

I have a complex query that, as far as I can tell, either requires

Arel or SQL. Performing Arel queries with ActiveRecord is awkward.

Getting the query into ActiveRecord requires

project(node_table.columns)

and then

find_by_sql(query.to_sql)

There should be a method of ActiveRecord that accepts an Arel query,

does the appropriate projection, and executes the query. Converting

Arel queries to SQL defeats the purpose of Arel. Arel should not be

dependent on SQL and should be able to work on no-SQL databases.

ARel is a SQL builder language designed for relational databases. It is used to construct queries which are then executed on a database by using something such as find_by_sql. It will not work on no-SQL [sic] databases because it is designed for *relational* databases, which is not what a no-SQL [sic] database is!

The syntax provided by ActiveRecord::Relation is designed to provide shortcuts to most of the "cool" methods inside of ARel, and I acknowledge that it's a bit lacking at times.

However, I think your query *can* be done in that syntax. I am not sure what relation node has to dependency, so you may need to change :dependencies in the `joins` here to `:dependency`

Node.where(:space_id => space_table).joins(:dependencies).where("dependencies.path" => path) 

The code above will return an ActiveRecord::Relation object and when you iterate over it or call to_a or inspect on it, it will return the resultant objects.

Core to this issue is the ActiveRelation vs. Arel dichotomy.

ActiveRelation is Arel's poor cousin that can't measure up to it. It's

an easier interface as long as your main query is where(field: value),

but doesn't provide all of the functionality of Arel.

If you can provide exact use cases in how to improve ActiveRelation rather than this childish "OMG ACTIVERELATION SUCKS!!!", that would be most appreciated.

Thank you.

[Xavier Noria]

Hi Bruce,

Arel does not really belong to the public interface of Active Record. It happens to be used internally, but in theory Active Record could switch to something else tomorrow.

Some few methods like arel_table are published in the API, but I believe there is consensus in core that they are not public interface either and should be nodoc'ed. (Please someone correct that otherwise.)

[Bruce Perens]

Hi Ryan and Xavier,

It's true that the available "no-SQL" databases are also non-relational, and that's not what I'm after. It doesn't seem to me that Arel must always emit SQL, though. The syntax is sufficient to control a relational database directly.

Arel was one of the most promoted features of rails 3. Perhaps this wasn't your intent, but it's so. In contrast, these are the only methods of ActiveRecord::QueryMethods with a word of documentation on api.rubyonrails.org: extending, reorder, select, uniq. Even where has no documentation. joins() is similarly undocumented. And when one figures out ActiveRecord::QueryMethods, it's only to find that many of the methods require SQL fragments as arguments and aren't particularly portable across databases. If you look around the net for help on doing complex queries with Rails 3, the instructions are to drop into Arel and then feed it back to ActiveRecord.

With all due respect, you seem to have a disconnect with your users.

[Xavier Noria]

On Sun, Jun 3, 2012 at 7:46 AM, Bruce Perens <br...@perens.com> wrote:

Arel was one of the most promoted features of rails 3.

Not really, what was promoted from Rails was that Active Record had a new interface thanks to its new integration with Arel (Pratik Naik did that work). It is true, however, that in the early days some people misunderstood the role of Arel for end-users, due to blog posts and stuff that is outside the control of the project.

But I think generally speaking that is clear nowadays. In particular the current edition of AWDwR, Rails 3 in Action, and The Rails 3 Way do not even mention Arel except in some glossary or in passing. They cover the Active Record interface.

Perhaps this wasn't your intent, but it's so. In contrast, these are the only methods of ActiveRecord::QueryMethods with a word of documentation on api.rubyonrails.org: extending, reorder, select, uniq. Even where has no documentation. joins() is similarly undocumented. And when one figures out ActiveRecord::QueryMethods, it's only to find that many of the methods require SQL fragments as arguments and aren't particularly portable across databases. If you look around the net for help on doing complex queries with Rails 3, the instructions are to drop into Arel and then feed it back to ActiveRecord.

Yeah, this guide is certainly better

    http://guides.rubyonrails.org/active_record_querying.html

though the API should not only have more content than the guide, but should be the comprehensive reference. That should be fixed.

With all due respect, you seem to have a disconnect with your users.

Naahhh, appreciate your hypothesis, but I am pretty sure that is not the case generally speaking.

[Andrew White]

On 3 Jun 2012, at 06:46, Bruce Perens wrote:

> It's true that the available "no-SQL" databases are also non-relational, and that's not what I'm after. It doesn't seem to me that Arel must always emit SQL, though. The syntax is sufficient to control a relational database directly.

Perhaps some history may provide a context for this discussion. Originally Arel was developed by Nick Kallen as a relational algebra engine for Ruby. The original implementation had an ActiveRecord/SQL engine but I believe that some alternative engines were developed by other people (LDAP springs to mind for some reason). Nick lost interest/moved on to other things (I believe he was involved in improving Twitter's messaging platform) and Bryan Helmkamp developed it further.

At some point (I don't know when) it was decided that rather than constructing SQL out of string fragments, Rails 3 would use Arel to build SQL but with a separate query interface that maintained backwards compatibility but added functionality (ActiveRecord::Relation). However because Arel was already quite well known by that point some people got the wrong end of the stick, especially with the protracted development of Rails 3.

Once Rails 3 was out there were reports of slow query performance which turned out to partly down to Arel. To solve the problem Aaron Patterson re-wrote Arel for version 2.0 to focus on generating SQL as fast as possible - a lot of the abstract relational algebra features were removed because Rails didn't need them and they slowed things down. At this point Arel is really just a component of Rails - it's development is pretty much directed by what Rails needs. The original implementation by Nick Kallen is still available on GitHub at https://github.com/nkallen/arel if you're interested.

> Arel was one of the most promoted features of rails 3. Perhaps this wasn't your intent, but it's so. In contrast, these are the only methods of ActiveRecord::QueryMethods with a word of documentation on api.rubyonrails.org: extending, reorder, select, uniq. Even where has no documentation. joins() is similarly undocumented. And when one figures out ActiveRecord::QueryMethods, it's only to find that many of the methods require SQL fragments as arguments and aren't particularly portable across databases. If you look around the net for help on doing complex queries with Rails 3, the instructions are to drop into Arel and then feed it back to ActiveRecord.

ActiveRecord::Relation is designed with the Pareto principle in mind - the majority of web apps have relatively simple querying requirements most of the time and when you do need to do something complex then you can drop down to SQL. As for the issue about database independence, well how many web apps have to be database independent to the degree where you can just flip a switch. If you're writing an app that can be deployed on top of different database then you can special case each query - it's likely to be only a few queries anyway.

TL;DR - Rails is a framework for building real web applications and not a computer science research project. Sometimes we do things which aren't the 'right' way but it's usually with the best of intentions.

> With all due respect, you seem to have a disconnect with your users.

Actually, I'd say the opposite - there was a widespread concern amongst users that Active Record in Rails 3 was slow and it was preventing them from upgrading. We addressed that concern by developing a new version that focussed on what it was being used for and not some theoretical use.


Andrew White

[Maurizio Casimirri]

Il giorno 03/giu/2012, alle ore 13.35, Andrew White ha scritto:
>
> At some point (I don't know when) it was decided that rather than constructing SQL out of string fragments, Rails 3 would use Arel to build SQL but with a separate query interface that maintained backwards compatibility but added functionality (ActiveRecord::Relation). However because Arel was already quite well known by that point some people got the wrong end of the stick, especially with the protracted development of Rails 3.

Could it be better if ActiveRecord users were completely unaware of Arel? i understand it was an unwanted side effect of the adoption of a well known component, but there are even some simple situations in which i usually resort to Arel, or at least I'm tempted to do so.


Just a little example:

term = params[:term]

@tags = if term.size >= 3
condition = ActsAsTaggableOn::Tag.arel_table[:name].matches("%#{term}%")
ActsAsTaggableOn::Tag.where(condition).pluck(:name)
end || []

render :json => @tags


Say that on average i resort to arel a couple of times for each rails application i wrote.

I would like to ask you if it's only an historical problem of people learning bad manners around the internet or if ActiveRecord querying interface could be improved somehow in order to make it the first choice in place of Arel (and possibly also deprecate the direct use of Arel).

Going on like this would probably lead to problems in case you choose to drop Arel in future.

(I hate to admit it but sorry for my english I still can not express myself well, despite how hard I try)

Maurizio

[Rodrigo Rosenfeld Rosas]

I don't remember any open discussions in Rails-core on the plans for the
future of ActiveRecord when Arel was chosen.

I'd have suggested Sequel instead. Great documentation and performed
much better than current AR when I switched a while back...

I'm not sure if it was even considered by that time...

I'd really appreciate that goals like this were discussed in this group
before they got implemented.

This is just a suggestion, of course.

Cheers,
Rodrigo.

[Ken Collins]

On Jun 3, 2012, at 7:35 AM, Andrew White wrote:

At some point (I don't know when) it was decided that rather than constructing SQL out of string fragments, Rails 3 would use Arel to build SQL but with a separate query interface that maintained backwards compatibility but added functionality (ActiveRecord::Relation). However because Arel was already quite well known by that point some people got the wrong end of the stick, especially with the protracted development of Rails 3.

Coming from the perspective of an ActiveRecord adapter like SQL Server, I wish that both ActiveRecord and plugins alike used Arel more heavily. I know this is a doubled edged sword. You want end users of the framework to stay as high level as possible and not dip down to Arel. However, I see a need to expose Arel more for plugin developers to stop constructing SQL fragments from strings.

The biggest pain from my perspective are order strings and select fragments. The SQL Server adapter's visitor needs to leverage real Arel ordering nodes since the visitor will have to eventually remove duplicates as TSQL does not allow "ORDER BY foo ASC, foo DESC". 

Other issues always come back to the way our adapter has to handle LIMIT and OFFSET using a paging function like ROW_NUMBER(). When plugin authors even use basic of SQL fragments like a SELECT... vs projections, it means we have to end up doing the parsing instead as we have to re-compose a complex statement to work with our window functions. 

The way I see, the more ActiveRecord and its plugins use Arel, the better off other adapters and engines can make decisions lower down the stack. I hope Rails core considers this as we move forward.

 - Ken

[Maurizio Casimirri]

If a similar problem exists I purpose instead the adoption of a middle layer of abstraction for common complex queries in a way similar to what built-in helpers are for views.

A very stupid API stub could be like that

module ASetOfCommonComplexQueries

  def a_very_common_complex_query(*params)

    # .... Use a lot of AREL here

    # then return an ActiveRecord::Relation

  end

end

It's just an example, you can probably do much better

This layer could also provide the necessary abstraction to these queries that for some reason depends on a particular database:

module Pg::ASetOfCommonComplexQueries

  def a_very_common_complex_query(*params)

    # a_very_common_complex_query implemented with pg related features

  end

end  

module Mysql::ASetOfCommonComplexQueries

  def a_very_common_complex_query(*params)

    # a_very_common_complex_query implemented with mysql related features

  end

end  

Obviously these implementations are responsibilities for adapters more than for Rails but a common convenience API could be a way to approach the problem.

I really dislike the idea that client code should depend on an implementative component.

Maurizio

[Bruce Perens]

Xavier wrote:
> though the API should not only have more content than the guide, but should be the comprehensive reference. That should be fixed.

I can't over-emphasize this. If you want users to make use of an API preferentially over another, the API documentation must be present. Not documenting a method will confuse users: is it internal? Is it meant for our use at all? For joins and where to be undocumented this way really is failure to communicate your intention to users.

Rails Guides like http://guides.rubyonrails.org/active_record_querying.html are tutorial rather than canonical. We don't expect them to explain the full functionality of an API, and we are left with either experimentation or overly time-consuming deep dives into source as the only way to determine what the API really does, and then we have no guarantee that it will still do that in the next version.

And much as we appreciate the good books on Rails, they aren't supposed to be the core developers official statements on the API.

Andy White wrote:
> well how many web apps have to be database independent to the degree where you can just flip a switch.

My code is intended for widespread distribution on the net and installation and operation by others without the developer's support. Not every Rails app is a single-site thing. So, database independence is important.

Maurizio wrote:
> I would like to ask you if it's only an historical problem of people learning bad manners around the internet or if ActiveRecord querying interface could be improved somehow in order to make it the first choice in place of Arel (and possibly also deprecate the direct use of Arel).

The original concept of Arel was to duplicate the operators of SQL without departing from the programming language we were already using, by proving a query algebra in Ruby. That's a powerful concept. It's unfortunate that Aaron had to remove some operations to make it fast, but it's still an algebra. The current Active Record Querying interface is, in contrast, a shorthand for SQL which, combined with some SQL fragments in strings, can generate 80% of the queries that people will need.

So, what is our recommended path for users once they are writing those 20% of queries that Active Record Querying operators aren't designed to cover? You can expect them to have to do so at least once in every large application.

It's either fall into SQL, or use Arel.

In Arel's favor, it's portable across databases, it covers most of the ground (although not everything that can be stated in SQL for a particular database), it's in the language we were already writing in, and it is arguably a clearer syntax than SQL because SQL is little changed since Codd's work in the late '70's.

In SQL's favor, it's well known and it can express everything that the connected database can do.

[Rodrigo Rosenfeld Rosas]

Em 04-06-2012 16:03, Bruce Perens escreveu:

...

So, what is our recommended path for users once they are writing those 20% of queries that Active Record Querying operators aren't designed to cover? You can expect them to have to do so at least once in every large application.

That was exactly the point where I moved from AR to Sequel. I reached one of the 20% of the queries (that happens to be more than that in my case) where AR interface is too limiting for my application:

https://groups.google.com/forum/?fromgroups#!topic/rubyonrails-core/XxmaSvOkgO4

Lack of documentation, support, and a good-enough API. All of that I've found in Sequel and it further performed better than AR, so yay! ;)

Pretty happy with the move and I sincerely wish the best of luck for AR to get over the "80%" it currently supports...

Cheers,
Rodrigo.

[Bruce Perens]

Sequel looks interesting. At first glance, it looks more mature than AR, and I guess shows how AR might evolve. Of course, I know nothing about its performance, etc. I might try it in my next application.

[Maurizio Casimirri]

Il giorno 04/giu/2012, alle ore 23.27, Bruce Perens ha scritto:

Sequel looks interesting. At first glance, it looks more mature than AR, and I guess shows how AR might evolve. Of course, I know nothing about its performance, etc. I might try it in my next application.

Yes, AR might evolve. Sequel has intrigued me as well, but relying on unofficial solutions scares me a little, even if it has a good support.

My concern stems more from the fact that the design of Rails sometimes exposes not-too-defined boundaries (ACTIVERECORD/AREL/SQL is only an example), that sooner or later will be defined, i hope.

Maybe this arises from the fact that Ruby does not have a built-in construct for interfaces, but i can't say that for sure. In any case I do not know if replacing a key component of the framework with a third party one is a wise choice in the long run. I can't be sure that pieces will fit always good.

Maurizio

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To view this discussion on the web visit https://groups.google.com/d/msg/rubyonrails-core/-/Dcgs-Xq1guMJ.
To post to this group, send email to rubyonra...@googlegroups.com.
To unsubscribe from this group, send email to rubyonrails-co...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.

[Allen Madsen]

Squeel (not to be confused with Sequel) does a pretty good job of covering the other 20%.

https://github.com/ernie/squeel

Allen Madsen
http://www.allenmadsen.com

[Maurizio Casimirri]

Rodrigo Rosenfeld Rosas:

Lack of documentation, support, and a good-enough API. All of that I've found in Sequel and it further performed better than AR, so yay! ;)

Pretty happy with the move and I sincerely wish the best of luck for AR to get over the "80%" it currently supports...

Allen Madsen:

Squeel (not to be confused with Sequel) does a pretty good job of covering the other 20%.

https://github.com/ernie/squeel

Allen Madsen
http://www.allenmadsen.com

I can agree that there may be better solutions to replace / complement ActiveRecord but I believe that now is the moment to define a query interface that meets the requirements of our applications, in a way that is not intended to change for a long time and that can be shared between different gems / libraries / ORMs, so that the rest of the implementation can safely change across the time (Arel, Squeel, Sequel or any other brilliant solution that is right for our).

Maurizio

[Rodrigo Rosenfeld Rosas]

Sorry, but I didn't get it. Sequel shares nothing with AR as far as I can tell you and hasn't significantly changed its interface for years I guess...

For example, Sequel is not likely to support dynamic methods or method_missing for one. You shouldn't expect to find something like record.created_at_changed? in Sequel core, for example, among other "magic" methods, like the now deprecated find_by_xxx dynamic finders.

So, it doesn't rely on AR API and it won't change if AR does.

Did I miss something?

Best,
Rodrigo.

[Rodrigo Rosenfeld Rosas]

Em 04-06-2012 19:24, Maurizio Casimirri escreveu:

Il giorno 04/giu/2012, alle ore 23.27, Bruce Perens ha scritto:

Sequel looks interesting. At first glance, it looks more mature than AR, and I guess shows how AR might evolve. Of course, I know nothing about its performance, etc. I might try it in my next application.

Yes, AR might evolve. Sequel has intrigued me as well, but relying on unofficial solutions scares me a little, even if it has a good support.

I can totally understand your point of view as I also share it and taking the decision to migrate from the default AR to another solution was not that easy as you might expect.

I found that MongoDB is better supported by the Rails community than Sequel, so I had to write some integration code by my own, like the sequel-devise gem. And I had to look at the source code of Devise as it wasn't clearly documented what such an adapter should provide and it took some time, but not that much as you could expect. I've also had to add a "save!" method to Sequel::Model for being able to use FactoryGirl and a few other adjustments but I wouldn't say it was a pain to get my preferred tools (like RSpec) to gracefully integrate to Sequel, although it certainly took some time.

I've wrote about some of the solutions I've written, but I'm planning to post some more interesting features when I find some time:

http://rosenfeld.herokuapp.com/en/articles/2012-04-18-getting-started-with-sequel-in-rails

Among the features I haven't written about yet are the possibility of running my development (or any other) environment inside a transaction inspired by the "rails c --sandbox" option.

But actually the "--sandbox" is half-baked while my solution is not:

https://groups.google.com/forum/#!topic/rubyonrails-core/fAhmhaevKQY

Other than that I was able to use transactions and PostgreSQL savepoints even in RSpec beforeAll/afterAll hooks, which I just can't do in out-of-the-box RSpec + AR setup.

Using transactions this way is not supported by Sequel as RSpec doesn't provide an aroundAll hook, but I could get fast and great support from Jeremy Evans, the current main maintainer of Sequel. He helped me to get an unsupported solution that would help me with the sandbox and transaction support in beforeAll/afterAll.

This not only made my integration specs run much faster (truncate is real slow in PostgreSQL for testing purposes and transactions are much faster then lots of DELETE statements that must be issued in a certain order due to foreign key constraints). It also made my specs easier to read.

And running "SANDBOX=1 rails s" is really helpful when I'm experimenting with a new feature or trying to reproduce a bug and have a copy of the production database that I don't want to mess in my development environment. So every single change would be rolled back when I hit Ctrl+C. This is awesome.

Of course, I have only tested this on PostgreSQL which also happens to be the database vendor of choice of most Sequel users, differently from AR that seems to be better target at MySQL. I don't know if MySQL would support savepoints, so maybe my solution won't fit everyone's need.

But this is another reason why I chose Sequel. I don't like MySQL and I don't think AR gets some things right. For example, Sequel thinks that the database should be used in full, with ACID support and that your constraints are very important. It also encourages to write triggers when appropriate among other things. And I've always shared those same ideas. This was always something I disliked in AR and how it doesn't seem so much worried about such constraints in the sense that you write "t.references :something" and it won't create a foreign key to it as well as an index. At least it is how it has been over the last years. And I think this happens because the original authors prefer MySql and in the last years it bundled with MyISAM engine set to the default and this situation has just changed this year.

This tells me a lot. For example, in my humble opinion, it shows me that the original authors didn't know about databases as much as the database expert people I admire. Or they wouldn't have chosen MySql in the first place. Also it tells me that they don't care about database constraints, indices and transactions as much as I do. It seems that they started writing something and when they got more clients and the queries got slower they realized that they should create an index, for example. Maybe I'm just guessing here, but this is definitely not the way I see databases.

Sequel just seems more aligned with my own ideas and even if I had to take some time to adapt it to the gems I'm using (Devise, FactoryGirl, RSpec) I can say it totally worthed my little time spent on this integration and it has saved me a lot of work trying to figure out how to do some tasks with AR, specially when you can't find much documentation about it.

I'm not advocating everyone should replace AR with Sequel. It will really depend on what project your ideas are more aligned with. I can say that Rails itself integrates very well to Sequel, although you might have to write some code to integrate to other gems. And I don't think it would be likely to break your application due to some Rails upgrade because Sequel would be no longer supported as it doesn't require any integration to Rails itself.

But such decision is up to you and I'd advice you to spend some time reflecting on it for your next project because the ORM is a key library in most applications, specially usual web ones.

I'm not currently considering using anything other than Sequel for any serious new project I decide to create and can take this decision. If you decide for Sequel though, just let me know if you need any assistance and I'll gladly help you. You can also send any questions to sequel-talk and it's likely that you get a response in within 24h.

Cheers,
Rodrigo.

[Maurizio Casimirri]

No you didn't. In fact i was not speaking of AR changes but the way rails interacts with it. If I'm not wrong Sequel needs different plugins from AR's ones or at least alternate versions of AR plugins. It would be probably a good thing if AR would have a stable and definitive interface capable to cover any reasonable use case, then probably you can use an universal adapter for any AR plugins in a way you can't even notice it and thus filling the gaps between AR support and Sequel.

Maurizio

[Rodrigo Rosenfeld Rosas]

Sorry, but I have no idea what you're talking about. Could you please give me an example of one of those AR plugins you're referring to?

[Maurizio Casimirri]

Sorry, but I have no idea what you're talking about. Could you please give me an example of one of those AR plugins you're referring to?

Sorry, you are right i was referring to gems that involve querying the database through AR, Devise to say one that yourself mentioned.

As in a past thread in which we discussed together, in brief my thought is: there are probably some good reasons that prevent rails to drop ActiveRecord, but  with a well thought and well designed interface the fact that ActiveRecord is the default for rails would not seal the giant support AR has to the other ORMs or persistence solutions.

The situation with Sequel is probably similar to what i faced once with mongoid. At first glance all seemed good to me, but then i realized i was spending a lot of time on the framework (patching something, choosing  alternative gems working with mongoid ...) more that on my business (and I probably liked this as i 'm writing here now ;) )

I think that this is a case in which a bit more of engineering would not hurt.

Maurizio

[Rodrigo Rosenfeld Rosas]

I guess I got what you mean, although I don't agree.

Rails 3 is a pretty solid and modular framework in my opinion, lacking just a decent documentation.

It doesn't rely on AR for anything. It is just included by default but if you run "rails new app -O" then AR won't be included as a dependency (although installing rails will also install AR, which seems wrong to me as AR shouldn't be a dependency for Rails).

But you can't just define a common API that all gems will be able to use regardless of which ORM solution you chose. The ORMs can take very different approaches when dealing with API design and that is what make them so interesting.

The problem relies in the gems in my opinion. They should follow the SRP instead of just assuming things to get a broader range of users. Even if there are any interface "keyword" in Ruby, you can still write interfaces in a number of ways.

The easiest one would be to document what methods you expect from the ORM for it to work and how they should respond.

An even better approach would be to provide a test-suite that all conforming adapters would pass when correctly implemented.

But if Devise had it only documented I wouldn't need to investigate the source to understand what to add to Sequel models.

For example, FactoryGirl had it documented that "save!" was the sole dependency and documented how it works, so it only took one minute for implementing it and getting it to work with Sequel.

DatabaseCleaner was supposed to support Sequel but it seems no one was using it with Sequel as it was broken and was just fixed recently:

https://github.com/bmabey/database_cleaner/pull/120

RSpec had also good documentation, although no example or built-in support for Sequel, but there were already lots of examples on how to set-up transactional examples with Sequel with some quick search in the web.

I've only took a considerable amount of time to be able to do an unsupported usage of transactions with Sequel. Sequel doesn't provide methods like DB.start_transaction and DB.stop_transaction. You need to use it like:

DB.transaction do
  ...
end

This works with RSpec around(:each):

around(:each) do |example|
    DB.transaction do
        example.run
    end
end

But RSpec doesn't provide an around(:all), so I had to call some private methods on DB for getting transactions and save points to work in before(:all) and after(:all). The same technique was used for adding support for a sandbox environment.

Knowing which private methods to call and in which order took some time and a lot of help from Jeremy Evans.

But the way I see, replace AR with Sequel should be much easier than replacing it by some non relational database solution, like MongoDB. They have completely different approaches and you shouldn't expect for lots of gems that aim in adding some kind of support for relational databases to be easily integrated to MongoDB as well.

So, this time I think Rails is currently very well designed and I'd like to thank a lot Yehuda, José Valim, Piotr Sarnacki and all core developers for the excelent job they did on making Rails more modular and easy to extend with Engines. A lot of unnecessary dependency was cut off and Rails just doesn't seem to be the issue anymore for adopting alternative solutions like testing framework, orm and JavaScript library. This was really a great engineering design and the Rails community should be proud of their code base.

Educating gem authors to follow Rails' example should be much easier and less painful.

Cheers,
Rodrigo.

[Maurizio Casimirri]

Il giorno 05/giu/2012, alle ore 13.55, Rodrigo Rosenfeld Rosas ha scritto:

Em 05-06-2012 00:17, Maurizio Casimirri escreveu:

Sorry, but I have no idea what you're talking about. Could you please give me an example of one of those AR plugins you're referring to?

Sorry, you are right i was referring to gems that involve querying the database through AR, Devise to say one that yourself mentioned.

As in a past thread in which we discussed together, in brief my thought is: there are probably some good reasons that prevent rails to drop ActiveRecord, but  with a well thought and well designed interface the fact that ActiveRecord is the default for rails would not seal the giant support AR has to the other ORMs or persistence solutions.

The situation with Sequel is probably similar to what i faced once with mongoid. At first glance all seemed good to me, but then i realized i was spending a lot of time on the framework (patching something, choosing  alternative gems working with mongoid ...) more that on my business (and I probably liked this as i 'm writing here now ;) )

I think that this is a case in which a bit more of engineering would not hurt.

I guess I got what you mean, although I don't agree.

Rails 3 is a pretty solid and modular framework in my opinion, lacking just a decent documentation.

Yes it is, but this does not mean it is perfect. The limits of Rails design are more evident on the long run, when you start dealing with changes and expecially with unexpected ones. Also they are not a problem in general, but only for certain applications, and some of them fall in the use cases of rails framework. 

It doesn't rely on AR for anything. It is just included by default but if you run "rails new app -O" then AR won't be included as a dependency (although installing rails will also install AR, which seems wrong to me as AR shouldn't be a dependency for Rails).

Of course rails does not rely on AR but it is the default, and so it has the major support from the community.

But you can't just define a common API that all gems will be able to use regardless of which ORM solution you chose. The ORMs can take very different approaches when dealing with API design and that is what make them so interesting.

I don't want a generic api for all ORMs but i wish that the default (AR) has a better and more generic API so that we can drop / integrate it without drop the support from community gems. 

The problem relies in the gems in my opinion. They should follow the SRP instead of just assuming things to get a broader range of users.

You are right, but you can say now that all the gem creators take this into account?

Even if there are any interface "keyword" in Ruby, you can still write interfaces in a number of ways.

You are right again, i'm not saying that the missing of an interface keyword is a problem by itself but i'm concerned to the fact that due to this many times ruby libraries expose the implementation to the client code (eg. we need to call #arel_table). Also and probably more often, libraries API will change when implementation changes. 

The easiest one would be to document what methods you expect from the ORM for it to work and how they should respond.

An even better approach would be to provide a test-suite that all conforming adapters would pass when correctly implemented.

Documentation is probably enough to express interfaces but would probably be better to enforce the separation between interface (what does not change) and implementation (what can safely change) by delegating to a void implementation in the top level layer, and then injecting the real implementation into the stubbed one.

This way we would be sure that the top level interface will stay independent from the concrete implementation code (as an example think of AREL case again).

So, this time I think Rails is currently very well designed and I'd like to thank a lot Yehuda, José Valim, Piotr Sarnacki and all core developers for the excelent job they did on making Rails more modular and easy to extend with Engines. A lot of unnecessary dependency was cut off and Rails just doesn't seem to be the issue anymore for adopting alternative solutions like testing framework, orm and JavaScript library. This was really a great engineering design and the Rails community should be proud of their code base.

Educating gem authors to follow Rails' example should be much easier and less painful.

Well here i see a misunderstanding i don't think that Rails has a bad design but that it can move to an approach that is a bit more friendly to enterprise applications where it doesn't make things too complicated in order to make our applications more maintainable and age resistant. 

Probably i need to cite someone that is better than me: http://www.languageparallax.com/wordpress/?p=39

I had the feeling that a more enterprise-oriented approach can be achieved in rails when i looked at play framework, (even if the current version is a nearly unusable mess due to its young age) it proves that there are ways to fill the gap between enterprise requirements and simplicity and quickness belonging to agile frameworks like rails. 

I'know that you don't like it but sometimes i prefer data mapper and domain driven design over active record and database driven design, the same way i would like to see a two-step view approach for views.

If for the latter there are no problems (i can quickly implement it with a few lines of code and only my views will depend on it) for the first one i've a lot of trouble due to the lack of solid support, since i don't think that rails will drop AR my proposal is to have at least a complete querying/mapping interface that can bridge the gap from ORMs.

In any case i will look at Sequel, now i'm too curious! ;)

Maurizio

 

[Rodrigo Rosenfeld Rosas]

Em 05-06-2012 10:10, Maurizio Casimirri escreveu:

Il giorno 05/giu/2012, alle ore 13.55, Rodrigo Rosenfeld Rosas ha scritto:

...

Em 05-06-2012 00:17, Maurizio Casimirri escreveu:The problem relies in the gems in my opinion. They should follow the SRP instead of just assuming things to get a broader range of users.

You are right, but you can say now that all the gem creators take this into account?

That is just how open-source and community-driven development work :)

Even if there are any interface "keyword" in Ruby, you can still write interfaces in a number of ways.

You are right again, i'm not saying that the missing of an interface keyword is a problem by itself but i'm concerned to the fact that due to this many times ruby libraries expose the implementation to the client code (eg. we need to call #arel_table).

If you just need the table name, you can just use YourModel.tablename I guess:

http://api.rubyonrails.org/classes/ActiveRecord/ModelSchema/ClassMethods.html#method-i-table_name

Man, it was really hard to get this specific link from the API. I needed to show the frame source in Chrome and then extract the "source:" prefix... Having an API based on frames already sucks, but not being able to right or midle-click the link in the left panel makes it even worse.

Also and probably more often, libraries API will change when implementation changes.

Having an interface does not provide any guarantees per se. I maintain a Grails application and their API has changed in a minor upgrade (2.0.1 -> 2.0.2) despite Groovy and Java providing "interface" as a keyword. Having a void implementation won't help here. If you don't want something to change in the Rails community that usually means writing a test for it.

...

So, this time I think Rails is currently very well designed and I'd like to thank a lot Yehuda, José Valim, Piotr Sarnacki and all core developers for the excelent job they did on making Rails more modular and easy to extend with Engines. A lot of unnecessary dependency was cut off and Rails just doesn't seem to be the issue anymore for adopting alternative solutions like testing framework, orm and JavaScript library. This was really a great engineering design and the Rails community should be proud of their code base.

Educating gem authors to follow Rails' example should be much easier and less painful.

Well here i see a misunderstanding i don't think that Rails has a bad design but that it can move to an approach that is a bit more friendly to enterprise applications where it doesn't make things too complicated in order to make our applications more maintainable and age resistant.

Well, this argument often pops out in the Rails community. I won't repeat myself on my own opinions:

http://rosenfeld.herokuapp.com/en/articles/ruby-rails/2012-03-04-should-we-move-forward-or-remain-backward-compatible
https://groups.google.com/forum/#!topic/rubyonrails-core/fTvmRxtFGcE/discussion

Probably i need to cite someone that is better than me: http://www.languageparallax.com/wordpress/?p=39

I had the feeling that a more enterprise-oriented approach can be achieved in rails when i looked at play framework, (even if the current version is a nearly unusable mess due to its young age) it proves that there are ways to fill the gap between enterprise requirements and simplicity and quickness belonging to agile frameworks like rails.

Yes, this is possible, but I don't want that in Rails. That is because if you want to keep backward-compatibility at all costs you'll end up with a code base that is hard to maintain and inconsistent and will prevent you from moving forward sometimes.

For example, I don't make any effort for making my own gems compatible with Ruby 1.8 because I simply don't like to write hashes like {:symbol => value}, preferring {symbol: value} much more. When you have to remain backward-compatible to Ruby 1.8 support, it means more code to maintain which can be really a pain.

This is valid for most other backward compatible decisions.

I'know that you don't like it but sometimes i prefer data mapper and domain driven design over active record and database driven design, the same way i would like to see a two-step view approach for views.

All of them are valid engineering decisions and I don't believe there is a right way of doing things. I like to say that Rails is just Ruby. It is a glue that will help you start coding your applications helping you with decisions you don't care about, like what database or ORM or template engine or test framework or JavaScript framework you should work. If you don't care about them, let Rails choose a default one for you. If you do care, then change the defaults. For example, I opted for jQuery when Prototype was the default. I opt for RSpec for testing as well as Sequel for ORM. And opting for other tools in Rails is damn simple nowadays.

There is no reason why someone should really try to keep with Rails defaults nowadays IMO. You should keep thinking on your app as an application like you would do in Java or any other language/framework. Rails will just help taking some unimportant decisions for you by default unless you care about them.

So, if DataMapper makes more sense for you, I'd go for it if I were you. The same for your views although I have no idea what two-step views mean. Personally I use a SimpleDelegator class as a presenter for my JSON representations, for example if that is what you mean...

If for the latter there are no problems (i can quickly implement it with a few lines of code and only my views will depend on it) for the first one i've a lot of trouble due to the lack of solid support, since i don't think that rails will drop AR my proposal is to have at least a complete querying/mapping interface that can bridge the gap from ORMs.

The lack of a well defined and documented API is a blocker issue for me and probably the main reason I stopped using it. So, I heartedly agree with you on that one. Also, any internal implementation detail should be hidden from the API documentation.

In any case i will look at Sequel, now i'm too curious! ;)

Let me know (outside this list) if you have any questions about it.

Cheers,
Rodrigo.