.js importer still has problems

[tfer]

I've hung out with Leo since early days, discovered it when exploring Literate Programming, always liked the focus management outline provides, but I've yet to get the benefits I was looking for from it.  Maybe that is due to what my envisioned use case is, I'm not looking to write my own original code, (yet), I want to understand various existing code, pick it apart, figure out what the author is doing, all with the eye to modifying it, by changes or plugins.

Edward's recent posts show a lot of progress on flows for Python development, but I find myself in the weird world of JavaScript these days, which brings me to topic of the importer,  I want to make a plugin for the html presenter reveal.js I'm tentatively calling nthD, the intent of which is break the slides out of their 2D "Flatland", and give you the ability to use as many dimensions as you need to organize them, (think table of contents, outlines...). 

To this end I tried import the main engine of reveal, the file reveal.js, Leo creates an @auto node, and creates a subtree. I get the following in the cmd shell I started Leo from:

-----------------------------------------------------------------------------------------------

===== PERFECT IMPORT FAILED ===== reveal.js

len(s1): 4810 len(s2): 4810

first mismatched line: 3174

s1...

  3172 'el.controls = true;\n'

  3173 '// Once the video does start playing, hide the controls a...(62)\n'

* 3174 "el.addEventListener( 'play', function() {\n"

  3175 'el.controls = false;\n'

  3176 '} );\n'

s2...

  3172 'el.controls = true;\n'

  3173 '// Once the video does start playing, hide the controls a...(62)\n'

* 3174 '} );\n'

  3175 '}\n'

  3176 '}\n'

imported C:/VC/git/reveal.js/js/reveal.js

----------------------------------------------------------------------------------------------------

,

and this in the Log panel:

----------------------------------------------------------------------------------------------------

Leo Log Window

Leo 6.2-b1-devel, devel branch, build b51bcaa8b5

2020-01-05 19:47:27 -0500

Python 3.6.5, PyQt version 5.9.4

Windows 10 AMD64 (build 10.0.18362) SP0

tabs found with @tabwidth -4 in reveal.js

changed leading tabs to blanks in 4800 lines in reveal.js

@auto failed: reveal.js

imported C:/VC/git/reveal.js/js/reveal.js

-----------------------------------------------------------------------------------------------------

Opening reveal.js in vscode I find the line numbers from the cmd shell don't match the actual line numbers.

It probably shouldn't be surprising that this file choked import, looking at it, it appears most of the code is a complex nesting of objects, variables, and functions in a top level anonymous function.

With by cursor in the offending line, I noticed something I hadn't before, vscode's top line in the buffer was identical to the outline path Leo once had, (or some plugin gave).  From this you could see how the nodes the import created should have been nested, (there weren't, came out much flatter).

Anyway, vscode can be use as a check on the import via this line. Note: if you click on the purple cube prefixing functions, you get that function outline neighborhood.

Tom

[Edward K. Ream]

On Wed, Jan 22, 2020 at 3:36 PM 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

I've hung out with Leo since early days, discovered it when exploring Literate Programming, always liked the focus management outline provides, but I've yet to get the benefits I was looking for from it.

Thanks, Tom, for checking in.

Maybe that is due to what my envisioned use case is, I'm not looking to write my own original code, (yet), I want to understand various existing code, pick it apart, figure out what the author is doing, all with the eye to modifying it, by changes or plugins.

That's usually a better plan than starting from scratch :-)

Edward's recent posts show a lot of progress on flows for Python development, but I find myself in the weird world of JavaScript these days, which brings me to topic of the importer,  I want to make a plugin for the html presenter reveal.js I'm tentatively calling nthD, the intent of which is break the slides out of their 2D "Flatland", and give you the ability to use as many dimensions as you need to organize them, (think table of contents, outlines...). 

I have been impressed with your design work in "flatland". I look forward to nthD.

To this end I tried import the main engine of reveal, the file reveal.js, Leo creates an @auto node, and creates a subtree. I get the following in the cmd shell I started Leo from:

-----------------------------------------------------------------------------------------------

===== PERFECT IMPORT FAILED ===== reveal.js

len(s1): 4810 len(s2): 4810

first mismatched line: 3174

[Snip]

> It probably shouldn't be surprising that this file choked import, looking at it, it appears most of the code is a complex nesting of objects, variables, and functions in a top level anonymous function.

I have just created #1481 for this. Am I correct that the sources for reveal.js are here? If not, please send the sources that failed for you.

The JS importer should have a full set of proper/traditional unit tests.

I have forgotten all the details of the JS importer. That's a good thing, because I'll be able to assess the existing code with new eyes.

Edward

[Edward K. Ream]

On Wednesday, January 22, 2020 at 3:36:08 PM UTC-5, tfer wrote:

To this end I tried import the main engine of reveal, the file reveal.js, Leo creates an @auto node, and creates a subtree. I get the following in the cmd shell I started Leo from:

-----------------------------------------------------------------------------------------------

===== PERFECT IMPORT FAILED ===== reveal.js

len(s1): 4810 len(s2): 4810

first mismatched line: 3174

I am not able to reproduce this failure.  I git cloned reveal.js from https://github.com/hakimel/reveal.js/, then used the perfect import script. It warned that it converted tabs to blanks, but otherwise all went well.

The line that starts "// Once the video does start playing, hide the controls" appears at line 4069 of my file, not line 3173 as in your file. Please send your copy of reveal.js to me at edre...@gmail.com.

Edward

[tfer]

I wrote an earlier reply, but it did not post, ah web interfaces...

The error with the mixed up line numbers below, is from the console from which I'm running Leo and have to do with the import, the don't exist in any file.

On Sunday, January 26, 2020 at 4:48:56 AM UTC-5, Edward K. Ream wrote:

On Wednesday, January 22, 2020 at 3:36:08 PM UTC-5, tfer wrote:

To this end I tried import the main engine of reveal, the file reveal.js, Leo creates an @auto node, and creates a subtree. I get the following in the cmd shell I started Leo from:

-----------------------------------------------------------------------------------------------

===== PERFECT IMPORT FAILED ===== reveal.js

len(s1): 4810 len(s2): 4810

first mismatched line: 3174

You are using the same file as I am, strange that you are getting only a warning.  I'm am running under an up to date devel branch.

 

I am not able to reproduce this failure.  I git cloned reveal.js from https://github.com/hakimel/reveal.js/, then used the perfect import script.

As to this warning, the import should be under the effect of a "@tabwidth 2" directive, which actually was hard to have happen, I post that in another thread. 

It warned that it converted tabs to blanks, but otherwise all went well.

The line that starts "// Once the video does start playing, hide the controls" appears at line 4069 of my file, not line 3173 as in your file. Please send your copy of reveal.js to me at edre...@gmail.com.

As I mentioned above, there is no such file, I can send my "nthD-exploration.leo" file which preserves the import, but I'm planning to copy that node an hand curate it into an @auto node as it should have been imported, change the filename in the node so it will generate the file on save and allow me to diff it against the original.  Might be more useful to wait for that.

Tom

[Edward K. Ream]

​On Mon, Jan 27, 2020 at 1:22 AM 'tfer' via leo-editor wrote:

​> ​the import should be under the effect of a "@tabwidth 2" directive, which actually was hard to have happen​.

I do see an import failure with @tabwidth 2.​​ I did so as follows:

dir_ = r'C:\leo.repo\reveal.js\js'
c.tab_width = 2
try:
    c.recursiveImport(
        dir_ = dir_,
        kind = '@clean',
        recursive = True,
        safe_at_file = True,
        theTypes = ['.js',],
    )
finally:
    c.tab_width = -4

I see the same message as you reported. This should be enough to fix the problem.

Edward

[Edward K. Ream]

On Monday, January 27, 2020 at 5:22:41 AM UTC-6, Edward K. Ream wrote:

​I see the same message as you reported. This should be enough to fix the problem.

I now remember the general problem: it's difficult/impossible to use section references in some situations, but without section references it's not possible to put code in the proper place.

In this case, the node

"toArray( element.querySelectorAll( 'video, audio' ) ).forEach function"

needs a section reference to properly place the child node,

"if( promise && typeof promise.catch === 'function' && el.controls === false )"

and its child,

"el.addEventListener( 'play', function".

In other words, the actual import machinery is probably working correctly. The problem arising in the strategy used to allocate lines to nodes. This is a tricky task. A rethink may be needed. One possibility would be to disallow child nodes in some situations...

Edward

[tfer]

On Monday, January 27, 2020 at 6:57:46 AM UTC-5, Edward K. Ream wrote:

On Monday, January 27, 2020 at 5:22:41 AM UTC-6, Edward K. Ream wrote:

​I see the same message as you reported. This should be enough to fix the problem.

I think I can see how the following could occur: 

I now remember the general problem: it's difficult/impossible to use section references in some situations, but without section references it's not possible to put code in the proper place.

In this case, the node

"toArray( element.querySelectorAll( 'video, audio' ) ).forEach function"

needs a section reference to properly place the child node,

"if( promise && typeof promise.catch === 'function' && el.controls === false )"

and its child,

"el.addEventListener( 'play', function".

Changing the @auto node to an @file node and manually doing the sectioning is acceptable, but we probably want to yield an at auto import to give us a result that, upon being saved, still generates a diff equivalent file to the original.  Disallowing some children is a good idea, as long as that noted in the doc's.

In other words, the actual import machinery is probably working correctly. The problem arising in the strategy used to allocate lines to nodes. This is a tricky task. A rethink may be needed. One possibility would be to disallow child nodes in some situations...

I suspect part of the problem maybe due to some of the lines that are tagged to be nodes, (and thus be the first line of the nodes body text), actually contains a leading fragment that should be the end of the prior node's body.  I have an idea for a new type of section reference that allows for the first line of  its body text to be joined into the line in which it occurs.  i.e. say <|section|>., or <|section>> to indicate the side of the join.

Tom

[Edward K. Ream]

On Mon, Jan 27, 2020 at 8:14 AM 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

> Changing the @auto node to an @file node and manually doing the sectioning is acceptable, but we probably want to yield an at auto import to give us a result that, upon being saved, still generates a diff equivalent file to the original.  Disallowing some children is a good idea, as long as that noted in the doc's.

I suspect the problem might persist with @file nodes.

>> In other words, the actual import machinery is probably working correctly. The problem arising in the strategy used to allocate lines to nodes. This is a tricky task. A rethink may be needed. One possibility would be to disallow child nodes in some situations...

> I suspect part of the problem maybe due to some of the lines that are tagged to be nodes, (and thus be the first line of the nodes body text), actually contains a leading fragment that should be the end of the prior node's body.  I have an idea for a new type of section reference that allows for the first line of  its body text to be joined into the line in which it occurs.  i.e. say <|section|>., or <|section>> to indicate the side of the join.

Thanks for this comment. I am going to investigate this issue next. Imo, new kinds of section references aren't likely to help. The new importers always and only assign whole lines to nodes. The problem surely must be that whole lines are assigned to the wrong nodes, which is equivalent to saying that lines are misplaced.

Edward