My javascript mistake

[Edward K. Ream]

This post documents the biggest mistake I have made in the last decade or so.

This mistake is akin to my initial misunderstanding of emacs.  I didn't realize that tab completion makes it unnecessary to remember or type full emacs command names.  I would have seen my error had I seen an emacs user in action. BH (LeoUser) opened my eyes with his prototype of Leo's minibuffer.

Similarly, javascript's real problems have blinded me to the huge amount of work done in the the javascript "space", namely the internet.  Joe Orr's LeoVue project has set me straight. Packages such as node.js, vue.js, react, etc. establish their own "coding conventions" that set de facto standards and minimize the problems and deficiencies of "bare" javascript.  I would have seen my error had I paid more attention to what has been happening.

I have started my study of node.js and vue.js.  This will continue indefinitely.  Just today I have started to study the leovue/src folder.  It's a new world.  I find it exciting.  It's going to be a hobby of mine while I continue to work on the desktop version of Leo.  I recommend close study of the code to anyone wanting to come up to speed on web technologies.

Edward

P. S. A node of caution.  If I were designing a hothouse for malware, I would choose something like the npm package warehouse.  A search for "npm malware" yields this page.  Presumably Joe Orr is aware of the risks.

The node.js people do not seem even remotely concerned enough. Hiding behind the halting problem is pathetic nonesense. They might start by studying the notion of trusted notebooks in the Jupyter world.

Edward

[Edward K. Ream]

On Tuesday, January 16, 2018 at 7:00:43 AM UTC-6, Edward K. Ream wrote:

P. S. A node of caution.  If I were designing a hothouse for malware, I would choose something like the npm package warehouse.  A search for "npm malware" yields this page.  Presumably Joe Orr is aware of the risks.

The node.js people do not seem even remotely concerned enough. Hiding behind the halting problem is pathetic nonsense.

Later npm blog posts indicate that the npm folks are taking reasonable steps to enforce a notion of trusted packages, independent of the halting problem ;-)

EKR

[Edward K. Ream]

On Tuesday, January 16, 2018 at 7:00:43 AM UTC-6, Edward K. Ream wrote:

> Packages such as node.js, vue.js, react, etc. establish their own "coding conventions" that set de facto standards and minimize the problems and deficiencies of "bare" javascript.

babel.js is a compiler that allows advanced syntax by compiling to more basic syntax.  It seems like a good idea, but it means that Leo's js importer will have to keep up.  The fix for #651 adds support for the backtick string syntax.

Edward

[Terry Brown]

Probably want to support JSX as well :-)

Cheers - Terry

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Edward K. Ream]

On Tue, Jan 16, 2018 at 6:08 PM, Terry Brown <terry...@gmail.com> wrote:

Probably want to support JSX as well :-)

​Which jsx do you mean?

- https://jsx.github.io/

- https://facebook.github.io/jsx/

- https://reactjs.org/docs/jsx-in-depth.html​

Edward

[rengel]

​Which jsx do you mean?

- https://jsx.github.io/

- https://facebook.github.io/jsx/

- https://reactjs.org/docs/jsx-in-depth.html​

Edward

Probably the TypeScript created by Microsoft:

http://www.typescriptlang.org/docs/handbook/jsx.html

[vitalije]

Regarding js importing, after some research about the topic, I concluded it is not possible/practical to make perfect js importer that can fully import javascript code on its own. Doing it right would require building complete AI for this problem space and perhaps dealing with neural networks feeding them with big data collected from github and other source code repositories.

And even if we do make such a clever code and intelligent importer, javascript syntax is growing fast, and that would require enormous effort to keep up.

Instead, I believe, what would suffice (at least for my own needs) is bunch of new simple commands that can help human to orchestrate import process. 

Right now, I have other duties and can't dedicate enough time to realize this idea. In case someone else is willing to make it, I wouldn't mind.

Blueprint for importing_tools plugin

Make a bunch of command buttons (ideally with well designed icons) and place them in Log tab pane as a new pane. Every button should perform one of the following commands:

• place-at-others-here
  

  places at-others directive before cursor. If cursor is indented then the directive is indented by the same amount. If body already contains this directive, command is noop

• extract-to-new-node

  expects cursor to be placed after at-others line. Extracts all lines from at-others to cursor in new node and starts editing headline. After editing headline, returns focus to parent body.

• extract-to-previous-node
  extracts all lines from at-others line to cursor and append them to the body of the last child node
  

• split-body
  splits body on the cursor position in two nodes. Current node keeps first part, and new node is inserted as the following sibling to the current node and body is set to second part
  

• shift-lines-to-previous-sibling
  extracts first lines from the current body up to the cursor and append them to the previous sibling node
  

• shift-lines-to-next-siblings
  extracts lines from cursor to the end of current body, and prepends them to the next sibling node
  

• advance-js-block
  moves cursor down to the end of next block. If line at cursor starts multiline expression, moves cursor at the end of that expression, else moves cursor to the next line that starts multiline expression. Executing this command several times moves cursor several blocks ahead.
  

• extract-current-block-in-section
  extracts all lines from the current cursor position to the end of current syntax block and put them in new section node. User is expected to type name of the section.

This is what I made as a note to myself to do when I find the time. Feel free to rename commands anyway you like. The only challenging command is advance-js-block. I believe I have solved this already and wrote about it HERE. It is not the final version published there. There were some tweaks necessary to better support regex skipping. I haven't got time to publish new improved version, but if there is interest I can send you a copy. 

Also, similar command advance-js-block, can be made for other languages and buttons can be shown/hidden depending on the file extension or language directive. I thought to make a plugin importing_tools from this commands and gui pane. 

Vitalije

PS: I haven't explained how these commands should be used. User would import whole file in one node, and then start reading code and chopping it up, using these commands. Hope this is more or less obvious. 

[Edward K. Ream]

On Wed, Jan 17, 2018 at 3:54 AM, vitalije <vita...@gmail.com> wrote:

Regarding js importing, after some research about the topic, I concluded it is not possible/practical to make perfect js importer that can fully import javascript code on its own.

​I have reached the opposite conclusion.  Yesterday I fixed two minor bugs in the importer.  One was a rare special case in the base linescanner.py code, the other added support for the new backtick syntax to the javascript scanner.

As a result, all the .js files in LeoVue import correctly and reasonably. I know of no javascript files that can not be imported correctly.  Please report any problems.

It's important to keep up with javascript, and the present code base is easily up to the task.  As for jsx, a variant of the js importer should be easy enough to do.

Doing it right would require building complete AI for this problem space and perhaps dealing with neural networks feeding them with big data collected from github and other source code repositories.

​The present code does the required AI.  I documented the code last year on leo-editor.  The javascript importer is remarkably simple.  It consists only of three overrides:

1. js_i.scan_line & helpers

2. js_i.starts_block

3. js_i.clean_headline

js_i.scan_line handles the crazy complexities of javascript token scanning.  All scan_line methods return the state at the end of the scanned line.  Typically a function or class can only be recognized if the end of the previous line is not in a state (comment, string, etc.), but as I re-read the code it appears that this is done implicitly.

js_i.starts_block returns true if the line should start a new node. This is tricky code, but it appears to work.

js_i.clean_headline is easy code, with no implications for other code.

The other part of the puzzle is the JS_ScanState class.  It keeps track of context and counts of parents and curly brackets.

Edward

[Edward K. Ream]

On Wed, Jan 17, 2018 at 4:41 AM, Edward K. Ream <edre...@gmail.com> wrote:

The other part of the puzzle is the J

​​

S_ScanState class.  It keeps track of context and counts of parents and curly brackets.

​Oops.  ​The ​JS_ScanState class keeps track of counts of parens and curly brackets.

Edward

[Edward K. Ream]

On Wednesday, January 17, 2018 at 4:41:30 AM UTC-6, Edward K. Ream wrote:

On Wed, Jan 17, 2018 at 3:54 AM, vitalije <vita...@gmail.com> wrote:

Regarding js importing, after some research about the topic, I concluded it is not possible/practical to make perfect js importer that can fully import javascript code on its own.

...

Doing it right would require building complete AI for this problem space and perhaps dealing with neural networks feeding them with big data collected from github and other source code repositories.

I want to emphasize that js_i.starts_block and js_i.scan_line are the only code involved in the AI. If there are errors in the present code, only these two methods will change.

The code looks for whole lines of a certain form, namely lines whose parens and/or curly brackets counts change and which also have 'function' in the line.  As I re-read the code, I see that it might be possible to contrive an example where this strategy fails.

If real (not imagined) problems arise, they can be dealt with in js_i.starts_block and js_i.scan_line. As an extreme measure, js_i.scan_line could set an ivar that tells whether "function" appears outside of any string.

In short, the present importer framework is robust and flexible.  I will reject other approaches unless real, impossible-to-fix problems are found.

Edward

[vitalije]

Well, obviously we don't agree on what "reasonably" imported code look like. I have had encountered dozens of js files that *are* imported correctly in terms that they can be written again with the file content unchanged, but they all suffered from creating too many or too few nodes and I had to import them by hand. 

Attached to this message are two files one Leo file trying to import the other js file. Put them in the same folder and try import. It fails, it leaves a lot of created nodes with arguably ugly names. No developer would ever name those nodes like importer does, nor he would structure the tree like that if he were creating that file using Leo from the beginning.  For me, reasonably imported file must be something close to what a developer would have if he wrote the same source file directly in Leo.

So, I can partially agree that current implementation of js importer can import some of the source files, but I personally find those imports useless most of the time. I have to reshape the tree so much that it turned to be more efficient to import whole file in one node and then to extract pieces by hand.

YMMV, Vitalije

PS: it seems that the js file attachment is refused. It can be found on github. I have just renamed it to jqterm.js.

https://raw.githubusercontent.com/jcubic/jquery.terminal/master/js/jquery.terminal.js

[vitalije]

In the file from the previous post, there few special regex patterns that make js importer fail. If you delete those lines 2595-2613 from jquery.terminal.js, importer will successfully import the file. Looking in generated tree here are few example nodes:

//headline:if (typeof define === 'function' && define.amd)

//and the content of body is
    if (typeof define === 'function' && define.amd) {
        // AMD. Register as an anonymous module.
        define(['jquery'], factory);
    } else if (typeof module === 'object' && module.exports) {
        // Node/CommonJS
        @others
    }
})(function($, undefined) {
    'use strict';
    // -----------------------------------------------------------------------
    // :: Replacemenet for jQuery 2 deferred objects
    // -----------------------------------------------------------------------

//or another one
//headline:everyTime: function(interval, label, fn, times, belay)

//and the body

everyTime: function(interval, label, fn, times, belay) {

@others

},

// and in at-others is following code

return this.each(function() {

jQuery.timer.add(this, interval, label, fn, times, belay);

});

If one wants to read and understand function `everyTime`, he must look into other node to see what it does. And here is little bit of context around that function in source file:

    // -----------------------------------------------------------------------

    // :: jQuery Timers

    // -----------------------------------------------------------------------

    var jQuery = $;

    jQuery.fn.extend({

        everyTime: function(interval, label, fn, times, belay) {

            return this.each(function() {

                jQuery.timer.add(this, interval, label, fn, times, belay);

            });

        },

        oneTime: function(interval, label, fn) {

            return this.each(function() {

                jQuery.timer.add(this, interval, label, fn, 1);

            });

        },

        stopTime: function(label, fn) {

            return this.each(function() {

                jQuery.timer.remove(this, label, fn);

            });

        }

    });

Arguably, the whole this code can fit nicely in one body. After all it is just 20 lines long. Instead Leo importer split this piece of code into 7 nodes. The worse is that the beginning of the above code that has three lines of comment as small heading is placed at the bottom of totally unrelated node whose headline is `function dc(n)` and the body is as follows:

function dc(n) { return wc(n, '', -1); } /** * Public API * $.Storage.set("name", "value") * $.Storage.set({"name1":"value1", "name2":"value2", etc}) * $.Storage.get("name") * $.Storage.remove("name") */ var localStorage; if (!hasCookies() && !isLS) { localStorage = {}; $.extend({ Storage: { set: wls, get: rls, remove: dls } }); } else { if (isLS) { localStorage = window.localStorage; } $.extend({ Storage: { set: isLS ? wls : wc, get: isLS ? rls : rc, remove: isLS ? dls : dc } }); } // ----------------------------------------------------------------------- // :: jQuery Timers // ----------------------------------------------------------------------- var jQuery = $; jQuery.fn.extend({

Now, if this examples are not enough to prove my point, please show imported tree to any js developer and ask him what the code does or if he finds that this particular view of source code is better then looking in a flat file. 

We may pretend that those are small glitches but, I really doubt that anyone will buy it. Leo should be superior tool for studying others people code, but when we speak about imported javascript files it fails (IMHO) miserably. And I do believe it can be done much better. 

I am sorry to be just criticizing your solution without providing any other, and at the moment I really can't do anything more to provide the solution. I shared my idea hoping that it would help. When I find more free time, hopefully, I will made that idea come true. Also I was hoping that once I have this plugin, and start to use it for importing js projects, perhaps I would find that some sequence of the above explained commands need to be manually repeated over and over until the whole file is imported. In that case it would be simple enough to make that sequence of steps in one new command or in some kind of recipe for importing files of some specific coding style. Perhaps it would lead to a small collection of recipes for importing js files depending on code style. Then perhaps, Leo could execute several recipes, show the result tree and user just chose what recipe suites best for particular project.

 Vitalije

[Terry Brown]

On Wed, 17 Jan 2018 01:50:37 -0800 (PST)
rengel <reinhard...@gmail.com> wrote:

> On Wednesday, January 17, 2018 at 9:56:27 AM UTC+1, Edward K. Ream

> wrote:
> >
> > On Tue, Jan 16, 2018 at 6:08 PM, Terry Brown <terry...@gmail.com
> > <javascript:>> wrote:
> >
> > Probably want to support JSX as well :-)
> >

> > ​Which jsx do you mean?
> >
> > - https://jsx.github.io/
> > - https://facebook.github.io/jsx/
> > - https://reactjs.org/docs/jsx-in-depth.html​
> >
> > Edward
>

> http://www.typescriptlang.org/docs/handbook/jsx.html

Yep, the "embeddable XML-like syntax.", as in the typescriptlang.org
and facebook links. That's what reactjs.org is talking about too, but
in this context it's the extension to JS syntax that's relevant.

Cheers -Terry

[Edward K. Ream]

On Wed, Jan 17, 2018 at 5:30 AM, vitalije <vita...@gmail.com> wrote:

Well, obviously we don't agree on what "reasonably" imported code look like. I have had encountered dozens of js files that *are* imported correctly in terms that they can be written again with the file content unchanged, but they all suffered from creating too many or too few nodes and I had to import them by hand. 

​There are many styles of writing javascript, including many styles of writing javascript classes.  So your comment about needing​ AI to do the import makes sense.

Imo this is not such a big deal, because the initial import can always be massaged.  Indeed, the import code is used only in the these cases:

1. The explicit import commands.

2. When reading @auto nodes.

3. When executing the parse-body command.

4. When executing c.recursiveImport.

After the import, no matter how good or bad, you can always convert the tree to @file or @clean, and make any desired change permanently.

Attached to this message are two files one Leo file trying to import the other js file. Put them in the same folder and try import. It fails,

​I'll open a new issue for this.

​

it leaves a lot of created nodes with arguably ugly names. No developer would ever name those nodes like importer does,

​This can be fixed by changing js_i.clean_headline.  In fact, changing "if 1:" to "if 0:" will probably result in something more to your liking.  Perhaps a way of personalizing this method would be good.​

nor he would structure the tree like that if he were creating that file using Leo from the beginning.  For me, reasonably imported file must be something close to what a developer would have if he wrote the same source file directly in Leo.

​That may be true, but now we are in the realm of true javascript expertise, something that on purpose I did not build into the js importer.

All of Leo's importers (with one or two exceptions) now use a pipeline architecture.  See importers/linescanner.py, the node "i.The pipeline". There are 5 stages in this pipeline, stages 0 through 4.  Stage 2 is a post pass, executed after all nodes have been created.

Again, it might be good to allow per-user extensions to the per-language post pass.  In the meantime, you could write your own AI-inspired postpass after, say, calling c.recursiveImport.

So, I can partially agree that current implementation of js importer can import some of the source files, but I personally find those imports useless most of the time. I have to reshape the tree so much that it turned to be more efficient to import whole file in one node and then to extract pieces by hand.

​Are you sure your own post-pass wouldn't be able to clean things up to your liking?

Edward

[Edward K. Ream]

On Wed, Jan 17, 2018 at 6:09 AM, vitalije <vita...@gmail.com> wrote:

In the file from the previous post, there few special regex patterns that make js importer fail.

​Please file a separate issue for this. Problems here can almost certainly be fixed in js_i.scan_line and its helper, js_i.skip_regex.  As noted in comments in js_i.scan_line, recognizing the start of a regex requires context-dependent state.  The present comment is:

QQQ

Distinguishing the the start of a regex from a div operator is tricky:
http://stackoverflow.com/questions/4726295/
http://stackoverflow.com/questions/5519596/
(, [, {, ;, and binops can only be followed by a regexp.
), ], }, ids, strings and numbers can only be followed by a div operator.

QQQ

These comments might be incomplete or just wrong.  Any help with this code would be appreciated.

​

If you delete those lines 2595-2613 from jquery.terminal.js, importer will successfully import the file. Looking in generated tree here are few example nodes:

​Perhaps the present code can be improved.  I would suggest experimenting with an improved post pass.

Edward

[Edward K. Ream]

On Thu, Jan 18, 2018 at 5:13 AM, Edward K. Ream <edre...@gmail.com> wrote:

On Wed, Jan 17, 2018 at 5:30 AM, vitalije <vita...@gmail.com> wrote:

Attached to this message are two files one Leo file trying to import the other js file. Put them in the same folder and try import. It fails,

​I'll open a new issue for this.

​Done: #653.

Edward

[Edward K. Ream]

On Wednesday, January 17, 2018 at 5:30:17 AM UTC-6, vitalije wrote:

Well, obviously we don't agree on what "reasonably" imported code look like.

I have just closed #653, relating to the jqterm.js.  Notes:

- I did not detect any problem parsing js regex expressions.

- The fix was to i.cut_stack, which now issues a log message without crashing when the stack can't be cut back as expected.  This is enough to import jqterm.js perfectly.

- js_i.clean_headline is now simpler and more useful.  Imo, it reports the minimum useful information about each function.  This is a straightforward method.  All contributions gratefully accepted.

- I have just created #655: The js importer could remove @others when a node has only one child. This is a separate, relatively low priority item.  It would improve only a few imported nodes in jqterm.js.

Imo, jqterm.js shows how good Leo's present javascript importer is.  It handles several js patterns for defining functions, using only the nesting level of parens and curly brackets as a guide.  The essential fact is that the (complex!) perfect import checks pass.  As a result, a post-pass, as envisioned by #655, would be relatively straightforward.

Edward

[Edward K. Ream]

On Sat, Jan 20, 2018 at 7:04 AM, Edward K. Ream <edre...@gmail.com> wrote:

- I have just created #655: The js importer could remove @others when a node has only one child. This is a separate, relatively low priority item.  It would improve only a few imported nodes in jqterm.js.

​Imo, we need HI (Human Intelligence) ​rather than AI for general javascript imports :-)  Happily, both @file and @clean allow arbitrary reorgs of the sources while retaining compatibility with the original.  For safety, we can manage the sources with git, so we can diff the changes easily.

Edward