Is scripted extensible? Crazy idea to use scripted to manage MongoDB Shell

[Angelo zerr]

Hi scripted team,

I would like manage mongo shell with browser and I tell me if I could extend scripted for that?

My idea is to provide a mongo shell in the browser and as shell script is javascript, why not to use scripted?

Here a screen with my idea: 

Here I have created a JS mongo shell script with this content : 

----------------------------------------------

use infocenter;

db.logs.find();

----------------------------------------------

If you don't know Mongo, this script connect to the "infocenter" database and returns the all JSON document stored in the "logs" collection.

I would like manage several feature : 

1) manage completion after use to show the database list.

2) manage completion after db. to show the collection list.

3) manage completion after logs to show the whole method of the collection (find, etc)

4) execute this script by calling the installed Mongo.

I would like to manage MapReduce (executing MapReduce on webbrowser to use Chrome, FF Debugger to debug MapReduce), 

I'm working on that for the moment, I will explain you if you are interested.

So my question is : is scripted is extensible (or could be extensible) to plug this feature : 

1) when completion opens, it calls Node.js (server side) to connect to mongo (mongo provides a mongo module for node) to retrieve the list of databses, collections etc.

2) add new toolbar button to execute a script and returns the result in the console.

SO my question is : is scripted id an IDE like to add new features.

Many thank's for your answer.

Regards Angelo

use infocenter;

db.logs.find();

use infocenter;

zaza

db.logs.find();

[Andrew Eisenberg]

So, it sounds like you just need to hook into the content assist mechanism.

One of our major goals for 0.4 is to define and implement the plugin model.  And a portion of that is to ensure that content assist is extensible.  For now, though you will have to hack scripted to make it work.  You can take a look at templateContentAssist.js and esprimaJsContentAssist.js for how to create a content assistant.  The relevant API function is computeProposals.

And then to hook in the content assistant, see the scriptedEditor.js file.  Please let us know how this goes and if you have any pain points with the extension.  We plan on tweaking things as we expose a real public API.

However,

use infocenter;

computeProposals.

is not syntactically correct and will show errors in the editor.  It may even cause parsing problems for the parser.

--
 
 

[Kris De Volder]

Andrew already explained a bit about contents assist.

You also seem to want something to run a script from scripted. One thing that is fairly easy right now (current version of scripted) is add keybindings to run shell scripts.
You could probably use this mechanism to bind a key to run the script in the editor.

If you haven't already, have a look at the docs for this feature here:

https://github.com/scripted-editor/scripted/wiki/Executing-Shell-Commands

Kris

[Angelo zerr]

Hi Andrew, Kris,

Many thank's for your answer. I have tried quicly to start something with Mongo in scripted (see attached screenshot). I have just created a new completions\mongo.scripted-completions like this: 

{

"scope": "source.js",

"completions": [

{ "trigger" : "db (Mongo)", "contents" : "db" },

{ "trigger": "addUser() (Mongo)", "contents": "addUser(${1:username}, ${2:passwd})$0" },

{ "trigger": "getCollectionNames() (Mongo)", "contents": "getCollectionNames()$0" },

{ "trigger": "getCollection() (Mongo)", "contents": "getCollection(${1:name})$0" }

]

}

and completion works!

The only thing which is important is to kill the server and restart it when you change this file (I know you know that, but I tell that if somebody whish to manage that).

I have started to create my own client\scripts\plugins\mongo\mongoContentAssist.js to manage completion with collection name (ex: db.collectionN...), but I think I will have a lot of problem.

This completion should request the Mongo Server with Ajax, but as this server is not in the same host than Node.js, I will have to use JSONP. But using JSONP gives you not the capability to manage ajax with synchronysous.

I'm afraid that completion will not work.

I have too several problems : 

1) 'db' variable is not known so there are syntax errors. I have tried to investiage to see how to 'document' variable is managed but it's hard for me (try to add my own Mongo DB definition in the \client\scripts\plugins\esprima\types.js but it doesn't work).

2) I think template completion should be improved, because it doesn't manage context. When you type directly 'getElementById', completion displays this function although this method belongs to 'document'.

3) it should  be cool if completion could displays a second window to displays some description (like Java Editor of Eclipse). For Javascript you could displays JSDoc. In th etemplates you could add some comments (db.getCollectionNames() returns the collection names of Mongo, please see the link xxxx).

4) I think the template completion should depends on a sub-scope (I have noticed that you have a scope for cs, for js), because for instance jQuery methods appears, but if you don't use jQuery, I think it's shame. In my case when I type db. (Ctrl+Space), getElementById appears (it should not).

For the moment I hack the scripted, but if I have understood, you have the intention to improve scripted to become extensible. 

Hope you will understand my comments (none offense with your great work!)

Many thank's for your help.

Regards Angelo

2013/1/17 Kris De Volder <kris.de...@gmail.com>

Kris

--
 
 

[Andrew Eisenberg]

Glad you got something working!

> The only thing which is important is to kill the server and restart it when
> you change this file (I know you know that, but I tell that if somebody
> whish to manage that).

We haven't optimized for that. We need to tell the server to drop its
template cache somehow and the easiest way is to just restart the
server. We could listen in the completions directory for new files
coming and changing, but since most people will be consuming existing
templates rather than writing their own, I feel this is lower
priority. I could be convinced otherwise, though...

> I have started to create my own
> client\scripts\plugins\mongo\mongoContentAssist.js to manage completion with
> collection name (ex: db.collectionN...), but I think I will have a lot of
> problem.
> This completion should request the Mongo Server with Ajax, but as this
> server is not in the same host than Node.js, I will have to use JSONP. But
> using JSONP gives you not the capability to manage ajax with synchronysous.
> I'm afraid that completion will not work.

Yes, the completions engine is currently synchronous. We haven't put
much thought into accepting asynchronous completions. However, doing
a server call for each completion invocation seems too expensive even
if asynchronous and server is local. I would recommend doing this
work in the background on the client when the page loads (eg-
webworker) so all the information is cached prior to any invocation.
Some of this info could be stored in local storage. This is exactly
how we handle indexing of referenced js files and it seems to work
quite well.

>
> I have too several problems :
>
> 1) 'db' variable is not known so there are syntax errors. I have tried to
> investiage to see how to 'document' variable is managed but it's hard for me
> (try to add my own Mongo DB definition in the
> \client\scripts\plugins\esprima\types.js but it doesn't work).

Two solutions here:

1. add a line like this at the top of each script that uses it:

/*global db */

2. add a .jshint file in your project and add a section for predefs.
See an example at the end of this gist:
https://gist.github.com/3708560

> 2) I think template completion should be improved, because it doesn't manage
> context. When you type directly 'getElementById', completion displays this
> function although this method belongs to 'document'.

Our templates could use a bit of a clean. But yes, templates do not
recognize context. That is by design to a certain extent. Templates
are just a snippet that gets added based on a prefix. More complex
analysis requires our regular completion assistant
(esprimaJsContentAssist).

For 0.4.0, I will be removing some of the existing templates since
they don't add much (except noise).

> 3) it should be cool if completion could displays a second window to
> displays some description (like Java Editor of Eclipse). For Javascript you
> could displays JSDoc. In th etemplates you could add some comments
> (db.getCollectionNames() returns the collection names of Mongo, please see
> the link xxxx).

This is something that I have thought about a lot and this would be
very useful. Would you mind raising an issue for this? I would like
to see this happen, and plan to look at this, but I don't think I can
get to it for 0.4.0.

> 4) I think the template completion should depends on a sub-scope (I have
> noticed that you have a scope for cs, for js), because for instance jQuery
> methods appears, but if you don't use jQuery, I think it's shame. In my case
> when I type db. (Ctrl+Space), getElementById appears (it should not).

See my comment above about cleaning up the templates. I think we
would be better served by having a more typed understanding of jquery
rather than using templates. I'll be removing most of the jquery
templates (or maybe somehow make them optional);

> For the moment I hack the scripted, but if I have understood, you have the
> intention to improve scripted to become extensible.

Yes, this is our intention. Please be explicit as to what extension
points you require. There are many possibilities and we won't be able
to get to everything for 0.4.0 and so we need to prioritize.

> Hope you will understand my comments (none offense with your great work!)

Thanks for the feedback. It is helpful.

>
> Many thank's for your help.
>
> Regards Angelo
>
>
>
> 2013/1/17 Kris De Volder <kris.de...@gmail.com>
>>
>> Andrew already explained a bit about contents assist.
>>
>> You also seem to want something to run a script from scripted. One thing
>> that is fairly easy right now (current version of scripted) is add
>> keybindings to run shell scripts.
>> You could probably use this mechanism to bind a key to run the script in
>> the editor.
>>
>> If you haven't already, have a look at the docs for this feature here:
>>
>> https://github.com/scripted-editor/scripted/wiki/Executing-Shell-Commands
>>
>> Kris
>>
>> --
>>
>>
>
>

> --
>
>

[Angelo zerr]

Hi Andrew

2013/1/19 Andrew Eisenberg <andrew.e...@gmail.com>

Glad you got something working!

> The only thing which is important is to kill the server and restart it when
> you change this file (I know you know that, but I tell that if somebody
> whish to manage that).

We haven't optimized for that.  We need to tell the server to drop its
template cache somehow and the easiest way is to just restart the
server.  We could listen in the completions directory for new files
coming and changing, but since most people will be consuming existing
templates rather than writing their own, I feel this is lower
priority.  I could be convinced otherwise, though...

No problem with that, I agree with you it's not important. I  tell you that just if you wish write some wiki page about "how to write template" and tell that it's important to kill/restart the server.

> I have started to create my own
> client\scripts\plugins\mongo\mongoContentAssist.js to manage completion with
> collection name (ex: db.collectionN...), but I think I will have a lot of
> problem.
> This completion should request the Mongo Server with Ajax, but as this
> server is not in the same host than Node.js, I will have to use JSONP. But
> using JSONP gives you not the capability to manage ajax with synchronysous.
> I'm afraid that completion will not work.

Yes, the completions engine is currently synchronous.  We haven't put
much thought into accepting asynchronous completions.  However, doing
a server call for each completion invocation seems too expensive even
if asynchronous and server is local.  I would recommend doing this
work in the background on the client when the page loads (eg-
webworker) so all the information is cached prior to any invocation.
Some of this info could be stored in local storage.  This is exactly
how we handle indexing of referenced js files and it seems to work
quite well.

Ok I understand your idea. I'm just know when I can do this job. Could you tell me which JS files you manage index and store it to local storage (I have never used that). 

>
> I have too several problems :
>
> 1) 'db' variable is not known so there are syntax errors. I have tried to
> investiage to see how to 'document' variable is managed but it's hard for me
> (try to add my own Mongo DB definition in the
> \client\scripts\plugins\esprima\types.js but it doesn't work).

Two solutions here:

1. add a line like this at the top of each script that uses it:

/*global db */

2. add a .jshint file in your project and add a section for predefs.
See an example at the end of this gist:
https://gist.github.com/3708560

I tell me that to manage 'db' like 'document'. In mongo shell, db is a the context (like 'document' or 'window' in HTML).

It seems that 'document' is not managed with your 2 suggestions, so I'm trying to manage 'db' like 'document', but perhaps it's a bad idea?

> 2) I think template completion should be improved, because it doesn't manage
> context. When you type directly 'getElementById', completion displays this
> function although this method belongs to 'document'.

Our templates could use a bit of a clean.  But yes, templates do not
recognize context.  That is by design to a certain extent.  Templates
are just a snippet that gets added based on a prefix.  More complex
analysis requires our regular completion assistant
(esprimaJsContentAssist).

For 0.4.0, I will be removing some of the existing templates since
they don't add much (except noise).

Ok, so I give up to manage mongo, with template and I will manage to with my own  mongoContentAssist

to manage context. Is that?

> 3) it should  be cool if completion could displays a second window to
> displays some description (like Java Editor of Eclipse). For Javascript you
> could displays JSDoc. In th etemplates you could add some comments
> (db.getCollectionNames() returns the collection names of Mongo, please see
> the link xxxx).

This is something that I have thought about a lot and this would be
very useful.  Would you mind raising an issue for this?  I would like
to see this happen, and plan to look at this, but I don't think I can
get to it for 0.4.0.

It's difficult today for me, to tell you, I need this API. I must play with scripted to tell you what I need.

But I tell me that we need : 

1) extends contentAssist bu creating an new xxxContentAssist

2) extends existing contentAssist. For example contributing to  esprimaJsContentAssist to manage my 

custom 'db' case (to avoid to reparse the buufer in a new mongoContentAssist)

But I must test that.

> 4) I think the template completion should depends on a sub-scope (I have
> noticed that you have a scope for cs, for js), because for instance jQuery
> methods appears, but if you don't use jQuery, I think it's shame. In my case
> when I type db. (Ctrl+Space), getElementById appears (it should not).

See my comment above about cleaning up the templates.  I think we
would be better served by having a more typed understanding of jquery
rather than using templates.  I'll be removing most of the jquery
templates (or maybe somehow make them optional);

Ok. 

> For the moment I hack the scripted, but if I have understood, you have the
> intention to improve scripted to become extensible.

Yes, this is our intention. Please be explicit as to what extension
points you require.  There are many possibilities and we won't be able
to get to everything for 0.4.0 and so we need to prioritize.

I must develop something to tell you. 

> Hope you will understand my comments (none offense with your great work!)

Thanks for the feedback.  It is helpful.

Cool:)

Many thank's to taking time to answer me.

Regards Angelo