[Announcement] Reactive jQuery DataTables for MeteorJS

[Austin Rivas]

Hello Everyone,

I've spent some time over the past few weeks putting together a reactive DataTables component for MeteorJS that allows you to page, sort, and filter millions of records while maintaining meteor's awesome server / client reactivity.

The packages is hosted on Meteor's package repository [ AtmosphereJS ]( https://atmospherejs.com/package/luma-datatables )

[ Here is a Live Example ]( http://luma-datatables.meteor.com/ )

[ Here is the Annotated Source ]( http://lumapictures.github.io/luma-datatables/ )

Any input is appreciated, and pull requests are more than welcome!

Setting up the component is simple :

1. Define the component in your template

[code]

{{> dataTable

    selector=browsers.selector

    columns=browsers.columns

    collection=browsers.collection

    options=browsers.options

    subscription=browsers.subscription

    query=browsers.query

}}

[/code]

2. Setup the data in your controller or as template helpers

[code]

if Meteor.isClient

  Template.home.browsers = 

      columns: [

        {

          sTitle: "Engine"

          mData: "engine"

        }

        {

          sTitle: "Browser"

          mData: "browser"

        }

        {

          sTitle: "Platform"

          mData: "platform"

        }

        {

          sTitle: "Version"

          mData: "version"

          sClass: "center"

        }

        {

          sTitle: "Grade"

          mData: "grade"

          sClass: "center"

          mRender: ( dataSource, call, rawData ) ->

            rawData ?= ""

            switch rawData.grade

              when "A" then return "<b>A</b>"

              else return rawData.grade

        }

        {

          sTitle: "Created"

          mData: "createdAt"

          mRender: ( dataSource, call, rawData ) ->

            rawData.createdAt ?= ""

            if rawData.createdAt

              return moment( rawData.createdAt ).fromNow()

            else return rawData.createdAt

        }

        {

          sTitle: "Counter"

          mData: "counter"

        }

      ]

      selector: "dataTable-browsers"

      collection: Browsers

      subscription: "all_browsers"

      options:

        oLanguage:

          sProcessing: "You must build construct additional pylons!"

      query:

        grade: "A"

[/code]

3. Publish data on the server ( coffeescript )

[code]

if Meteor.isServer

  DataTable.debug = "true"

  DataTable.publish "all_browsers", Browsers

[/code]

[Austin Rivas]

@avital I fixed the memory leak, turn out you were right, my observers were not being stopped when the subscription was terminated. The culprit was an unnoticed indention when I was refactoring my code that placed my onStop() inside of a conditional!

Here is the updated code https://github.com/LumaPictures/luma-datatables/blob/master/lib/datatables.server.coffee#L96

Let me know when you can re-enable luma-datatables.meteor.com.

[Avital Oliver]

Got it. The site is no longer stopped.

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Austin Rivas]

UPDATE

The package name has been changed to `jquery-datatables` and the repo is now https://github.com/lumapictures/jquery-datatables

[Pascal Richier]

Thanks for this, can be very usefull.
We've talk about this issue "DataTable + Blaze" and collection updates this week-end at DEVOXX France. Cool to see this solution today ;) !

--
Pascal

[Austin Rivas]

Thanks Pascal, I appreciate the feedback.

We are using this package for several apps we are developing internally ( we plan on open sourcing those as well ), and plan on supporting this package for the foreseeable future.

Some of the features I plan on having in place by the end of the week are :

* Individual Column Filtering

* Column Reorder

* User configuration settings for column display

* Reactive table base query and configuration ( update base query from session for example )

* Rendering reactive templates in a cell ( already in place, just needs to be tested further )

If you would like to contribute at all the extras and plugins sections of the official jquery-datatables is a great place to start ( https://datatables.net/extras/ )

I could also use a hand testing the table under use cases and load conditions I haven't anticipated.

So far from my testing this table structure can support full text search on a collection with 2 million rows, as long as the searchable columns are kept reasonably low ( 5 - 10 max ). Pagination and sorting scales much higher as long as all of sortable fields are indexed.

If you know of a good way to test blaze components using tiny test I'm all ears.

- Austin

[Julián Álvarez]

We are waiting for this :D

Great work Austin !

Thanks, thanks a lot

[Austin Rivas]

Thank Julian,

You are most welcome. Displaying, paging, sorting, and filtering large datasets has been a recurring issue in almost all of my projects, and I figured others could benefit from some the the tricks I've learned.

I feel like abstracting away a lot of the complex pub / sub logic and providing clean api's for user interface components is going to help a lot with making meteor more accessible to people new to the framework or programming in general.

I have a pretty long hit list of components at http://luma-ui.meteor.com/ that I will also be turning into meteor packages over the next few weeks. If a page looks broken, it prob is, just refresh the page. luma-ui is a really shitty prototype of what I am aiming for, a comprehensive front end framework composed of discrete Blaze components working together.

Please don't hesitate to report any bugs or ideas you have while using it. This is by no means a finished product, but the goal is to make this the most full featured table component around for meteor.

Here's what I'm working on for the next release https://github.com/LumaPictures/meteor-jquery-datatables/issues/milestones

- Austin

[Adrian Lanning]

Very cool!  Regarding paging/filtering, I didn't see it in your roadmap but any plan to integrate with Meteor's publish to avoid pushing all data to the client?

Also, your example site is really impressive!  Fast and really neat to see all those disparate components pulled together.  I think Blaze is living up to its name.  Modal dialogs in particular used to be a pain with Spark so its nice to see what appears to be stock jquery-ui just work.

I did notice one small thing, the text color of the modal form's input fields appears to be the same as the background.  Either that or typing isn't working for them in Chrome on OSX.  http://luma-ui.meteor.com/interface-components/visuals#form_modal

(Be sure to include a license in your luma-ui repo)

[Austin Rivas]

Hi Adrian,

Thanks for the feedback, its always appreciated. The current iteration of the package already accounts of paginating and searching over very large collections.

If you check out in the example, this is where I am using a server side datasource that is auto paged / sorted / and filtered via the datatables package.

https://github.com/LumaPictures/meteor-jquery-datatables/blob/master/example/example.coffee#L63

I've tested this with collections containing hundreds of thousands of rows, and as long as you are keeping the number of fields you are searching low.

Ideally though you will want to shrink the dataset as much as possible using the query option. This way you can show a lot of columns, and still have live search.

If you are more interested in how the pub / sub logic works you can check out these methods

Template.dataTable.setSubscriptionAutorun ( https://github.com/LumaPictures/meteor-jquery-datatables/blob/master/lib/datatables.client.coffee#L249 )

and on the server

DataTable.publish ( https://github.com/LumaPictures/meteor-jquery-datatables/blob/master/lib/datatables.server.coffee#L13 )

 This whole project is still very much a work in progress, so feel free to give input on anything that seems clunky or unnecessarily confusing to implement.

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/nhulj4Zh1fU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

[Adrian Lanning]

My bad, I was looking through the example but read, "DataTable.publish" as "Meteor.publish..." so thought you were publishing everything.  Glad you already worked it out and thanks for the links to the relevant code.

On Saturday, April 26, 2014 7:21:21 PM UTC-4, Austin Rivas wrote:

Hi Adrian,

Thanks for the feedback, its always appreciated. The current iteration of the package already accounts of paginating and searching over very large collections.

If you check out in the example, this is where I am using a server side datasource that is auto paged / sorted / and filtered via the datatables package.

https://github.com/LumaPictures/meteor-jquery-datatables/blob/master/example/example.coffee#L63

I've tested this with collections containing hundreds of thousands of rows, and as long as you are keeping the number of fields you are searching low.

Ideally though you will want to shrink the dataset as much as possible using the query option. This way you can show a lot of columns, and still have live search.

If you are more interested in how the pub / sub logic works you can check out these methods

Template.dataTable.setSubscriptionAutorun ( https://github.com/LumaPictures/meteor-jquery-datatables/blob/master/lib/datatables.client.coffee#L249 )

and on the server

DataTable.publish ( https://github.com/LumaPictures/meteor-jquery-datatables/blob/master/lib/datatables.server.coffee#L13 )

 This whole project is still very much a work in progress, so feel free to give input on anything that seems clunky or unnecessarily confusing to implement.

On Fri, Apr 25, 2014 at 5:55 PM, Adrian Lanning <alan...@gmail.com> wrote:

Very cool!  Regarding paging/filtering, I didn't see it in your roadmap but any plan to integrate with Meteor's publish to avoid pushing all data to the client?

Also, your example site is really impressive!  Fast and really neat to see all those disparate components pulled together.  I think Blaze is living up to its name.  Modal dialogs in particular used to be a pain with Spark so its nice to see what appears to be stock jquery-ui just work.

I did notice one small thing, the text color of the modal form's input fields appears to be the same as the background.  Either that or typing isn't working for them in Chrome on OSX.  http://luma-ui.meteor.com/interface-components/visuals#form_modal

(Be sure to include a license in your luma-ui repo)

--

[Austin Rivas]

No problem at all, let me know if you have any issues.

[Caleb LeNoir]

Is there a way to access the dataTable after it has rendered? I would like to add a click event to each row that uses data from that row.

[Austin Rivas]

Hey Caleb,

You can access the datatable after it has been initialized, it is stored in the data context of the instantiated datatable component. However this is not the best way to extend the tables features.

You have 3 main options ( going from simplest to most flexible )

1. simply attach events via jquery ( you must set selector )

{{> dataTable selector="example" columns=pages.columns rows=pages.rows }}

$( '#example tbody' ).on 'click', 'tr', ->
  name = $( 'td', @ ).eq( 0 ).text()
  console.log "You clicked on #{ name }'s row"

The major drawback of this method is that you have to track the table state externally. Good for simple events, but I wouldn't recommend it for anything complex. 

2. Attach events through the initialization options

Set options in your controller or via a template helper

options =
  initComplete: ->
    api = @api()
    api.$( 'td' ).click -> api.search( @innerHTML ).draw()

{{> dataTable options=options columns=pages.columns rows=pages.rows }}

Now datatables is handling the data for you. Here is a ton of api method example https://datatables.net/examples/api/

This option will cover most of your use cases.

3. Create a plugin

This is a great option for when you need to add events and some dom components that rely on your data. This is how I am implementing all of my drill down filters and export tools.

Datatables just released a huge update and the plugins api has changed slightly, I'll update with a working pattern later.

[Austin Rivas]

jquery-datatables v1.0.0

I have finally been able to incorporate the following features and upgrade jquery-datatables to 1.0.0 status. Your feedback is always welcome!

Reactive Query

Live Example

The query parameter for reactive tables is now reactive ( duh ). My goal was to have the table impose no structure on the query and just use raw mongoDB selector.

The basic idea is that you set a session variable ( or some other reactive datasource ) to your initial query and then use that var as the query parameter for the table component. Whenever the query parm changes the table will automagically rerender using the new query to fetch its dataset.

You can see a basic implementation of this there. In this example the table controls just extend the query object with whatever value they are set to. I tried to include examples of all the common control types, but if you think of any that I missed feel free to let me know.

Currently applying the filter can cause a somewhat janky table reload depending on what its contents looks like, this is something I plan on addressing ASAP.

Server Base Query

I changed the way that the DataTable class publishes data. Instead of publish() being a static method it is now an instance method that requires a DataTable instance. I think this is a more sustainable structure that will allow for a lot more flexibility in the future.

Here is how you instantiate a server DataTable component.

RowsTable = new DataTableComponent
  subscription: "rows"
  collection: Rows
  query:
    someProperty: "someValue"
    otherProperty: $regex: "value"
    nested:
      property: $gte: 0

The query you define on the server will limit your dataset to the client while preserving all of the pagination, sorting, and filtering.

Breaking Changes

The DataTableComponent is now classy cased along with the actual template name.

{{> DataTable options }}

instead of

{{> dataTable options }}

​

[Julián Álvarez]

Great I'll be testing tomorrow :D

--

[Austin Rivas]

thanks Julián, another user was experiencing issues with nested properties, if you see anything like that please let me know

[Larry Weru]

How do we set up and use the Column Filter? Can you point me to the documentation on how to set it up?

I'm trying to get column filtering set up with your package, and for some reason it won't let me use the api().column([columnnumber]).search(query).draw() to filter through individual columns (nothing happens, it just says processing and then when it's finished processing nothing changes). I can only filter the entire table using api().search(query).draw()

[Austin Rivas]

Hey Larry,

Do you mind posting a more complete example so that I can get sense of the context you are calling the api methods from?

[Larry Weru]

Sure, Austin. Thanks for the quick response! Basically I'm setting the option in my controller. Excuse the noise:

// $('input.search_init') are the search inputs. each column has one input.search_init

options = {

  initComplete: function() {

    var api = this.api();

    $('input.search_init').on('change keyup keypress blur', function(e){

        var columnIndex = $('input.search_init').index(this);

        api.columns([columnIndex]).search(this.value).draw();

    });

  }

};