Calling JS scipts or functions with params and returning results from checks

[monosij...@gmail.com]

Hi all -

I had posted this before and had gotten some pointers but still trying to find a definitive answer.

I am trying to see if there is a way to execute JS scripts or functions that allow me to call them with a parameter and return results.

trying to see if someone could point me to some example. I was pointed to the JS tests for MongoDB but those are standalone scripts that cannot be called with a param or return results of sanity checks.

...

I am writing the scripts to be executable from Python or JS or independently from within Mongo shell.

Regardless, the scripts need to be parameterized as they are initialization scripts - initializing a database / collection with values.

So I want to be able to call the scripts with the database / collection name and also the values to be initialized with.

I want the script to terminate with a a return value of the number of items and such sanity checks in each of the collections. 

Python is for testing and development for now. I did not want the code / logic to be 'stuck' inside Python if I can avoid it.

...

Any pointers to example scripts will be much appreciated.

Thanks for your help.

Mono

[Stephen Steneker]

On Thursday, 17 April 2014 05:30:04 UTC+10, monosij...@gmail.com wrote:

I had posted this before and had gotten some pointers but still trying to find a definitive answer.

I am trying to see if there is a way to execute JS scripts or functions that allow me to call them with a parameter and return results.

Hi Mono,

It would be helpful to include a link to your previous post in case anyone wants to reference the previous discussion: https://groups.google.com/forum/#!topic/mongodb-user/ivkFUNhd5rM.

I think the documentation reference you are looking for is: http://docs.mongodb.org/manual/tutorial/write-scripts-for-the-mongo-shell/.

This includes an important reference table on differences between interactive and scripted interaction with the `mongo` shell. For example, "use <db>" is an interactive shell command so for scripted interaction you would need to use the equivalent "db = db.getSiblingDB('<db>')".

The `mongo` shell in MongoDB 2.4 and 2.6 is an ES5-compatible interpreter (v8) so you can use standard JavaScript function() definitions for parameters and return values, eg:

function countDocs(dbName, collName) {
   return db.getSiblingDB(dbName).getCollection(collName).count();
}

Calling this from the shell:

> countDocs("twitter","tweets")
51428

If you want the value to be displayed (rather than returned to a calling function) you can use print() or printjson().

As Asya noted in the previous thread, you can use load() to include additional JavaScript files into the current context.

If you want to invoke this script from Python, you will have to open a `mongo` shell (probably using `subprocess`) and capture any values you need from stdout.

For some further examples of scripts written specifically for the `mongo` shell, see:

 https://github.com/TylerBrock/mongo-hacker/

Regards,

Stephen

[Monosij Dutta-Roy]

Hi Stephen -

Thanks for your reply and the details. Next time I will provide a link to the original post, but since it had gone deep and I had not heard back from Asya or anyone else, I thought start a new thread.

Asya pointed me in the right direction but then the scripts did not do what I was looking for.

I will try this put from the mongo shell and Python, and I am using 2.6 so all should work.

...

But to elaborate and confirm a little more, I was trying to find a way to run some 'init' scripts on some collections.

While with a DBMS I can write atomic SQL stmts for this that can be executed by any client, I believe this would be the only way to do it for Mongo?

Ie write an 'atomic' self-contained script that can be executed for any collection with the data and collection name passed into it, that can be tested by multiple clients, and results returned as well.

...

Would this be the only way to do it? is there a better way? i don't understand why I can't use eval from PyMongo? Isn't eval for executing functions?

Hope the problem I am stating is clear - I am wanting to initialize collections with some data and then return sanity check results. I want to do this from different clients, Python, JavaScript, Mongo shell possibly.

I am new to Mongo - so thank you for your patience with my, possibly convoluted, SQL-minded, questions.

Your clarifications are much appreciated.

Monosij

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/1V0920ULUQ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/896d3e22-983f-4539-b509-c2e3b80886ef%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Stephen Steneker]

On Wednesday, 23 April 2014 13:36:20 UTC+10, Monosij Dutta-Roy wrote:

But to elaborate and confirm a little more, I was trying to find a way to run some 'init' scripts on some collections.

While with a DBMS I can write atomic SQL stmts for this that can be executed by any client, I believe this would be the only way to do it for Mongo?

Ie write an 'atomic' self-contained script that can be executed for any collection with the data and collection name passed into it, that can be tested by multiple clients, and results returned as well.

...

Would this be the only way to do it? is there a better way? i don't understand why I can't use eval from PyMongo? Isn't eval for executing functions?

Hi Mono,

You can use the eval command to execute JavaScript code on the server, but in general this is a more limited and less scalable approach than writing a client-side script (i.e. the example I provided for the `mongo` shell): http://docs.mongodb.org/manual/reference/method/db.eval/.

Some relevant limitations on server-side db.eval():

 - it blocks other read/write operations to the same database

 - you cannot open a connection to another database from within an eval (you need to provide the database context for the command)

 - if authentication is enabled, you need to have all actions to all resources in order to run db.eval()

 - as at MongoDB 2.6, you cannot load() external scripts from a db.eval()

I think db.eval() is not a great fit for your use case. If you implement this code on the client side it can also be modularised and properly managed via version control.

Hope the problem I am stating is clear - I am wanting to initialize collections with some data and then return sanity check results. I want to do this from different clients, Python, JavaScript, Mongo shell possibly.

Yes, believe I understand what you are after. You are initialising collections and want to load and validate from multiple client environments. If you only want to write that code once you need to decide what works best .. but one approach would be Javascript to load your data fixtures via the `mongo` shell.

Depending on the data you need to insert, you could also consider storing the data in a language-independent format (eg. JSON). In this case you'd have to add a bit of code into each of your clients to load and validate the data.

If you still have doubts, can you provide an example of the sort of initialisation script you are trying to set up?

I am new to Mongo - so thank you for your patience with my, possibly convoluted, SQL-minded, questions.

No worries .. we're here to help :).

Regards,

Stephen 

[Monosij Dutta-Roy]

Hi Stephen -

Thank you again for your details explanations. Now I have a few more clarifications I seek.

...

1. Regards db.eval() I was of the understanding that it ran JS code as in scripts that would be local in a file system. And if it was stored on the server, it would run it there as well.

2. I understand now db.eval() will not work. I am trying to run JS scripts, and I may not have authorization on all resources.

3. Regards the getSiblingDB, I looked it up, curious why it is called getSiblingDB and not just getDB?

4. Is it because I have to first get a db using a connection to a db that is on the particular Mongo instance - and then get sibling DBs on that server? ie change the connection?

5. To run a script file locally on the remote server, do I just need a connection as in: connection = pymongo.Connection("mongodb://localhost", safe=True)

6. You wanted example of a script I would run.

Essentially an initialization script that would:

a. create a db if it does not exist;

b. create the collection;

c. validate the JSON data using JsonSchema; - may be skipped

d. insert data in collection;

e. run a set of sanity checks and return the result;

f. calling program writes test in a file;

...

Essentially I want to test the scripts from Python but eventually this will ideally be part of an automated build and I want to make sure the scripts work.

Example I will likely use SaltStack to call the scripts which can be extended from Python so possibly from there. But they could be called directly from the shell as well.

Again this something I have done for SQL DBMS for initializations.

Hope that helps.

Thanks again for your patience. Just wanted to understand my options before I start down a path.

On thing I did not understand you wrote above was: 'to load your data fixtures via the `mongo` shell.'

I keep thinking as you mentioned before as well:

'If you want to invoke this script from Python, you will have to open a `mongo` shell (probably using `subprocess`) and capture any values you need from stdout.'

Is this the approach you are thinking I should focus on? I do not understand why I need a shell using 'subprocess'.

As you mentioned, on mongo-hacker: https://github.com/TylerBrock/mongo-hacker/

is there a particular script I could look at that would best satisfy these requirements.

Thanks again. Just trying to get as much perspective as possible before trying things out.

Mono

--

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/1V0920ULUQ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/739be74f-98c4-4b11-9a37-aad68480df37%40googlegroups.com.

[Stephen Steneker]

On Thursday, 24 April 2014 08:42:24 UTC+10, Monosij Dutta-Roy wrote:

Thank you again for your details explanations. Now I have a few more clarifications I seek.

...

1. Regards db.eval() I was of the understanding that it ran JS code as in scripts that would be local in a file system. And if it was stored on the server, it would run it there as well.

Hi Mono,

The db.eval() command accepts a string of JavaScript code and evaluates it on the MongoDB server: http://docs.mongodb.org/manual/reference/method/db.eval/. There are usage caveats (as per my earlier reply). As at MongoDB 2.6, if you want to load() local scripts from a filesystem you have to do this on the client side. The use of load() from within a db.eval() to read files on the server side has been disallowed as potential security risk.

 

2. I understand now db.eval() will not work. I am trying to run JS scripts, and I may not have authorization on all resources.

OK

 

3. Regards the getSiblingDB, I looked it up, curious why it is called getSiblingDB and not just getDB?

getSiblingDB() is a helper in the `mongo` shell. If you execute it without the parentheses you can see there is an underlying command getDB() which is called on the current connection:

> db.getSiblingDB

function ( name ){

    return this.getMongo().getDB( name );

}

 

4. Is it because I have to first get a db using a connection to a db that is on the particular Mongo instance - and then get sibling DBs on that server? ie change the connection?

I included that in my example function because you were asking about parameters. In my arbitrary countDocs() function I decided I wanted to accept a database and collection name as parameters.

If you just need to access a single db (i.e. the one you are already connected to) than the getSiblingDB() helper is unnecessary.

 

5. To run a script file locally on the remote server, do I just need a connection as in: connection = pymongo.Connection("mongodb://localhost", safe=True)

If you want to evaluate JavaScript code via eval(), you need a connection as well as appropriate user/role permissions if auth is enabled.

If you want to run JavaScript code via the `mongo` shell, the code will be loaded from a path local to where you start the shell.

6. You wanted example of a script I would run.

Essentially an initialization script that would:

a. create a db if it does not exist;

b. create the collection;

c. validate the JSON data using JsonSchema; - may be skipped 

d. insert data in collection;

e. run a set of sanity checks and return the result;

f. calling program writes test in a file;

I think your approach here is a holdover from your RDBMS experience as you are trying to do much more than is required/useful :).

MongoDB has a dynamic schema, which means that: 
 - you do not have to pre-create your databases or collections; they will automatically be created when you insert data

 - there is currently no server-side enforcement of schema, so any validation should be done in your client

 - the docs go into more detail: http://docs.mongodb.org/manual/data-modeling/

Similarly, as long as you are using Acknowledged writes (which is the default) your application will know the data has been inserted and you should not have to do extra validation.

On thing I did not understand you wrote above was: 'to load your data fixtures via the `mongo` shell.'

I keep thinking as you mentioned before as well:

'If you want to invoke this script from Python, you will have to open a `mongo` shell (probably using `subprocess`) and capture any values you need from stdout.'

Is this the approach you are thinking I should focus on? I do not understand why I need a shell using 'subprocess'.

I was thinking that you were trying to load a set of data fixtures to seed a new database from different client apps which may be using different languages. If you are running client-side JavaScript via the `mongo` shell, you need some way to call that from Python.

If most of what you want to do is covered in your list of points for #6, then I think the rest of the discussion is fairly academic and you probably don't need to "initialise" the database/collections as you are anticipating.

I would suggest doing some more testing with MongoDB so you get a better idea of what is actually needed for your use case.

 

As you mentioned, on mongo-hacker: https://github.com/TylerBrock/mongo-hacker/

is there a particular script I could look at that would best satisfy these requirements.

No. I pointed at this as an example of scripts written around the `mongo` shell.

Regards,
Stephen

[Monosij Dutta-Roy]

Hi Stephen -

Thanks for the detailed email again. Need to try out the approaches you suggest and see how it goes.

I am aware regards the schema-less approach of Mongo. I have been using it directly from Python. Just wanted to set the same basic guidelines for the SQL DBMS for the initialization process if I could. It may not be necessary however, but would still like to try.

I will keep you and Asya updated on this forum. Thank you both again for your detailed help and pointers.

Have a good weekend.

Monosij

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/1V0920ULUQ0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/4fc4d895-5d8d-4101-9869-352eaf8e5f13%40googlegroups.com.