Processing Documents on Couchbase Server to Produce New Documents

[s.ande...@gmail.com]

I am working on an app where data (in the form of Couchbase documents) is received by the server, potentially lots of data. I need to process the data to produce a smaller subset of new documents which could then be synced with the phone.

My question, could I use map-reduce to produce this smaller subset of new documents or would I need to write some sort of server side process (in Java, C#, Ruby, Node.js, etc) to do this?

[Jens Alfke]

On Sep 1, 2014, at 7:19 PM, s.ande...@gmail.com wrote:

I am working on an app where data (in the form of Couchbase documents) is received by the server, potentially lots of data.

How is it being received exactly? Is it written into a Couchbase Server bucket, or replicated into the Sync Gateway?

My question, could I use map-reduce to produce this smaller subset of new documents or would I need to write some sort of server side process (in Java, C#, Ruby, Node.js, etc) to do this?

You'll need a server-side process. Map/reduce alone won't do it because Couchbase and CouchDB consider its output (view rows) different from documents.

—Jens

[s.ande...@gmail.com]

On Wednesday, 3 September 2014 04:03:39 UTC+12, Jens Alfke wrote:

On Sep 1, 2014, at 7:19 PM, s.ande...@gmail.com wrote:

I am working on an app where data (in the form of Couchbase documents) is received by the server, potentially lots of data.

How is it being received exactly? Is it written into a Couchbase Server bucket, or replicated into the Sync Gateway?

The data would be submitted by app users, so it would be via Sync Gateway.

My question, could I use map-reduce to produce this smaller subset of new documents or would I need to write some sort of server side process (in Java, C#, Ruby, Node.js, etc) to do this?

You'll need a server-side process. Map/reduce alone won't do it because Couchbase and CouchDB consider its output (view rows) different from documents.

I could just sync the data received by the server to the phone, but it could potentially grow very large, so that's not really an option.

I need to 1) exclude outliers and 2) calculate the median from the remaining data. Does the following sound like a reasonable approach?

1) Have a map/reduce function to calculate the interquartile range and determine the outlier boundaries (sorry not sure what the correct terminology is).
2) Using the outlier boundary values from step 1, have a map function to return values inside the outlier boundaries and a reduce function to return the median.

—Jens

[Jens Alfke]

On Sep 2, 2014, at 6:17 PM, s.ande...@gmail.com wrote:

I need to 1) exclude outliers and 2) calculate the median from the remaining data. Does the following sound like a reasonable approach?

Sure, but you'll also need an external process to query the view and create/update a document in the Sync Gateway db with the reduced data in it.

—Jens