Custom Handler

[Craig Swank]

Hello,
I am interested in creating a custom handler for some data sets I have
at work. They are stored in a mongodb database, and I would like to
be able to make some kind of adaptor that can enable serving the data
up in opendap. Do you have any suggestions as to where to start?

Thanks,

Craig

[Roberto De Almeida]

Hi, Craig.

The documentation on creating handlers is still outdated. There's and old document written for Pydap 2.x here:

  http://pydap.org/2.x/docs/plugins.html

Basically, a handler is a subclass of ``pydap.handlers.lib.BaseHandler`` with a method ``parse_constraints``. This method should inspect the query string of the request, and return a dataset object (an instance of ``pydap.model.DatasetType``) matching the request. The NetCDF handler is a good starting point as an example.

If your data is not too big your handler will only need to create a dataset and populate it with your data (all of it). You can see an example here of how a dataset with a Sequence should be created:

  http://code.google.com/p/pydap/source/browse/tests/constrain.txt

Once you've created your dataset you can have Pydap automatically parse the query string and constrain the dataset:

  from pydap.handlers.helper import constrain
  dataset = constrain(complete_dataset, environ['QUERY_STRING'])
  return dataset

Of course ideally you would want to parse the query string yourself and retrieve only the necessary data from the database. I've just started porting the SQL handler from pydap 2.x to 3.0, you could use it as a template for mongodb. The SQL handler parses the query string and builds a SQL query that fetches only the required data from the database.

Please let me know if you need more help with this.
Cheers,
--Rob

--
Dr. Roberto De Almeida
http://dealmeida.net/
http://lattes.cnpq.br/1858859813771449
:wq

[Craig Swank]

Thanks Roberto,
Is the sql handler you mentioned in a branch somewhere? I couldn't
find in on the google code site.

Thanks,

Craig

[Roberto De Almeida]

Hi, Craig.

I've commited it to the repos, in pydap/handlers/sql. It's still very basic, but you can't get the feel of it. I still need to handle slices in the query string (mapped as OFFSET and LIMIT on the SQL query), and converting the data properly (specially with timestamps), but the handler already works.

Cheers,
--Rob

[Roberto De Almeida]

Hi, Craig.

I just commited a new (finished) version of the SQL handler:

  http://code.google.com/p/pydap/source/browse/handlers/sql/pydap/handlers/sql/__init__.py?spec=svnc6e123f20e920cf727ad0ae9233d44370e43dc41&r=c6e123f20e920cf727ad0ae9233d44370e43dc41

This version uses a special SQLProxy object that reads the data from the database iteratively. The former version at

  http://code.google.com/p/pydap/source/browse/handlers/sql/pydap/handlers/sql/__init__.py?spec=svnc6e123f20e920cf727ad0ae9233d44370e43dc41&r=bb094fe64b0708a53d2ab98469f712d5e8337f2c

was reading everything into memory, a suboptimal solution. The SQL handler should be a good start for your custom handler.

I also plan on releasing a CSV handler soon.

Best,
--Rob