Aggregation Framework and support for a $transform type capability in the future?

[Darin]

Thank you for the videos on the site.  By watching the aggregation framework video and looking at the test cases on git, I was able to easily understand how to use the new aggregation framework.

One question I do have is about the future of the aggregation framework and the possibilities it might open.  For example, is there any consideration for supporting a $transform type function in the future?   In this rough idea, one could specify the root node (json name) and a stylesheet to use for the transform.  There are certainly a lot of details that would need to be worked out.

I can simulate a similar thing today by returning json from Mongo, parsing the json back to xml, and then running this xml through a stylesheet for a final transform (to perhaps xml, xhtml, etc.).  But, it would be nice to have the option of doing this server side.  Of course, this could very well violate every fundamental principle of Mongo.  Anyway, would be interested in your thoughts.

Darin.

[Dan Crosta]

There is no way to use javascript in aggregation pipelines, though
depending on what exactly you are trying to do, you may be able to
achieve it with a combination of the existing operators ($project in
particular). If you can explain in greater detail what you're trying
to accomplish, we might be able to suggest a solution using the
existing tools.

If not, you can log a feature request in our issue tracker at
https://jira.mongodb.org/. When doing so, specific examples, suggested
APIs, and use cases will go a long way.

- Dan

[Darin McBeath]

True, one might be able to do this with existing operators (such as $project) but it would likely be a bit complex.

Let me try and explain a bit more.  Feel free to say I'm crazy and that this is not appropriate within Mongo.

Assume that I'm able to load a fairly complex (heavily nested) structure into Mongo.  This document is initially xml and I've applied some assumptions/conventions when doing the transform to json prior to loading the document into Mongo.  For example, how to record namespaces, handle attributes, handle mixed content, maintain document order, etc. in the generated json document so that I could potentially roundtrip back to the original document (xml).  Actually, I've been able to successfully do this.

What I was thinking might be useful is if Mongo provided essentially a built-in transform capability (much like Saxon, xalan, etc.) where you could $transform a specific node (and it's children).  So, I might do a query based on a subset of metadata for the json document, but then would want to $transform part of the document using a stylesheet (and the specified node).  Whether the transformation could happen directly from json  or would have to be first reconstituted as xml would be a decision.  Might be easier to reconstitute the original xml but wouldn't be as performant.  It could look something like the following:

$transform : { name : stylesheet}

Where name is the root node (I guess a name in the json structure such as a.b or a) and stylesheet would be the name of a stylesheet to apply (or perhaps an in-line stylesheet).  If a stylesheet name, then it would need to have been previously loaded so it could be found.

Something could probably be done with $project, but I think this would be pretty ugly and very complex.  Plus, as far as I understand you need to know the full name (a.b.c) as something like (a.*.c) is not allowed.  When transforming content (such as apply-templates in xsl) then something like this would be needed.

Hope this helps clarify what I'm thinking.

Darin

---

From: Dan Crosta <dcr...@10gen.com>
To: mongodb-user <mongod...@googlegroups.com>
Sent: Thursday, May 3, 2012 11:00 AM
Subject: [mongodb-user] Re: Aggregation Framework and support for a $transform type capability in the future?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com.
To unsubscribe from this group, send email to mongodb-user+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.

[Chris Westin]

We don't have any plans at this time to do XSL-style transformations.  The goal here is to provide the kinds of capabilities people expect from traditional RDBMS aggregation facilities like group-by, along with other things that simple dataflow engines enable.  From your second post, I see you're describing something that looks more like an XPath reference of a kind we can't currently do.  That's not the first time I've seen a request for something like that, so I can imagine it appearing eventually, but that still would provide a general transform capability such as you're suggesting.

Chris

[Dan Crosta]

Ah, I see what you're trying. You can log a feature request to add
something like this, but I think you'll need to spend some time first
speccing out how it would work, what it could do or couldn't.

More immediately, I think the right approach here is to do this
transformation application-side. Even setting aside that MongoDB can't
do it yet (except possibly through some complicated projections, as
you suggest), it will generally be more flexible and scalable, since
it will be outside of the db. If you know that you are only operating
on a subset of the document, you can use field selection to limit the
returned documents like:

> db.foo.find({/* criteria */}, {field1:1, field2:1, field3:1,
_id: 0})

Which will return documents containing only field1, field2, and
field3. For more on field selection, see
http://www.mongodb.org/display/DOCS/Retrieving+a+Subset+of+Fields

- Dan

On May 3, 11:20 am, Darin McBeath <ddmcbe...@yahoo.com> wrote:
> True, one might be able to do this with existing operators (such as $project) but it would likely be a bit complex.
>
> Let me try and explain a bit more.  Feel free to say I'm crazy and that this is not appropriate within Mongo.
>
> Assume that I'm able to load a fairly complex (heavily nested) structure into Mongo.  This document is initially xml and I've applied some assumptions/conventions when doing the transform to json prior to loading the document into Mongo.  For example, how to record namespaces, handle attributes, handle mixed content, maintain document order, etc. in the generated json document so that I could potentially roundtrip back to the original document (xml).  Actually, I've been able to successfully do this.
>
> What I was thinking might be useful is if Mongo provided essentially a built-in transform capability (much like Saxon, xalan, etc.) where you could $transform a specific node (and it's children).  So, I might do a query based on a subset of metadata for the json document, but then would want to $transform part of the document using a stylesheet (and the specified node).  Whether the transformation could happen directly from json  or would have to be first reconstituted as xml would be a decision.  Might be easier to reconstitute the original xml but wouldn't be as performant.  It could look something like the following:
>
> $transform : { name : stylesheet}
>
> Where name is the root node (I guess a name in the json structure such as a.b or a) and stylesheet would be the name of a stylesheet to apply (or perhaps an in-line stylesheet).  If a stylesheet name, then it would need to have been previously loaded so it could be found.
>
> Something could probably be done with $project, but I think this would be pretty ugly and very complex.  Plus, as far as I understand you need to know the full name (a.b.c) as something like (a.*.c) is not allowed.  When transforming content (such as apply-templates in xsl) then something like this would be needed.
>
> Hope this helps clarify what I'm thinking.
>
> Darin
>
> ________________________________

>  From: Dan Crosta <dcro...@10gen.com>

> To: mongodb-user <mongod...@googlegroups.com>
> Sent: Thursday, May 3, 2012 11:00 AM
> Subject: [mongodb-user] Re: Aggregation Framework and support for a $transform type capability in the future?
>
> There is no way to use javascript in aggregation pipelines, though
> depending on what exactly you are trying to do, you may be able to
> achieve it with a combination of the existing operators ($project in
> particular). If you can explain in greater detail what you're trying
> to accomplish, we might be able to suggest a solution using the
> existing tools.
>

> If not, you can log a feature request in our issue tracker athttps://jira.mongodb.org/. When doing so, specific examples, suggested

> APIs, and use cases will go a long way.
>
> - Dan
>
> On May 3, 8:00 am, Darin <ddmcbe...@yahoo.com> wrote:
>
>
>
>
>
>
>
>
>
> > Thank you for the videos on the site.  By watching the aggregation
> > framework video and looking at the test cases on git, I was able to easily
> > understand how to use the new aggregation framework.
>
> > One question I do have is about the future of the aggregation framework and
> > the possibilities it might open.  For example, is there any consideration
> > for supporting a $transform type function in the future?   In this rough
> > idea, one could specify the root node (json name) and a stylesheet to use
> > for the transform.  There are certainly a lot of details that would need to
> > be worked out.
>
> > I can simulate a similar thing today by returning json from Mongo, parsing
> > the json back to xml, and then running this xml through a stylesheet for a
> > final transform (to perhaps xml, xhtml, etc.).  But, it would be nice to
> > have the option of doing this server side.  Of course, this could very well
> > violate every fundamental principle of Mongo.  Anyway, would be interested
> > in your thoughts.
>
> > Darin.
>
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.

> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.

[Darin McBeath]

ok, that helps clarify in my mind what you are all thinking.

I actually do this on the 'client' side currently (the server talking to mongo).  I store metadata about my xml document in a json document and then store the xml (using the same 'id') in GridFS.  So, I use mongo to query the metadata to identify the document(s) and then just pull the document from GridFS (which is xml).  Yes, xml could have been stored in the json document, but there is that 16mb limit I run into; so, my decision to leverage GridFS.

I guess another option (perhaps a lot less painless) would be to extend the mongo api for retrieving a document from GridFS.  Perhaps, one could specify a stylesheet in addition to the file to retrieve.  Just a thought.

Or, I just keep doing it on the client side.

Thanks for your insight.

Darin.