New Script To Render @jupytext files With VR3

[Thomas Passin]

Now that we have a more stable import format for @jupytext nodes, I redid my script to produce a version that renders the file with VR3.  For this version, you need to select the head node of the @jupytext tree.  I will make the script able to walk up the file tree to find the @jupytext top node but I wanted to get the script out there for testing as soon as possible. Note that this script will not modify the @jupytext file in any way.

To use, copy the script to a node somewhere in the outline in which you have an @jupytext external file. Select the command's node and use the "script-button" button to create a button for the command. Then highlight the @jupytext node and press the button.

The command will create a new node below your @jupytext tree and will select it.  VR3 may even open automatically and render it. If you activate the command and you haven't selected an @juptext node nothing will happen except a message about it.

Please give this script a good workout so we can fix up any edge cases.

"""Convert an @jupytext node tree for rendering with VR3.

The selected node must be the head of the tree.
"""

MD_MARKER = '# %% [markdown]'
CODE_MARKER = '# %%'
PREFIX = '<< prefix >>'

AT_MD = '@language md'
AT_CODE = '@language python'
MD_LITERAL_FENCE = "```"

JUPYTER_POSITION = c.p

def process_first_line(line:str) -> tuple(str, bool):
    """Convert the first line of a cell for VR3."""
    is_code = False
    if line.startswith(MD_MARKER):
        line = AT_MD
    elif line.startswith(CODE_MARKER):
        line = AT_CODE
        is_code = True
    if line.startswith('# '):
        line = firstline[2:]
    return (line, is_code)


jupy_file = JUPYTER_POSITION.h.split('@jupytext')
if len(jupy_file) == 1:
    g.es('Not an @jupytext node')
else:
    cells = [AT_MD]
    for p in JUPYTER_POSITION.subtree():
        is_code = False
        headline = p.h
        is_prefix = headline == PREFIX
        if is_prefix:
            cells.append(MD_LITERAL_FENCE)
        lines = p.b.split('\n')
        firstline = lines[0]
        firstline, is_code = process_first_line(firstline)
        cells.append(firstline)
        for line in lines[1:]:
            i = 0 if is_code else 2
            cells.append(line[i:])
        if is_prefix:
            cells.append(MD_LITERAL_FENCE)

    body = '\n'.join(cells)

    vr3_jupytext_node = c.p.insertAfter()
    vr3_jupytext_node.h = 'Rendering for ' + jupy_file[1]
    vr3_jupytext_node.b = body

    c.selectPosition(vr3_jupytext_node)
    c.redraw()

[Thomas Passin]

As promised, here is a version of the script that will work if you select either the @jupytext node or any of its immediate children. Since the tree will only be one level deep, clicking on any node in the tree will do.

"""Convert an @jupytext node tree for rendering with VR3.

Select any node of the @jupytext tree before running this script.

"""

MD_MARKER = '# %% [markdown]'
CODE_MARKER = '# %%'
PREFIX = '<< prefix >>'

AT_MD = '@language md'
AT_CODE = '@language python'
MD_LITERAL_FENCE = "```"

def process_first_line(line:str) -> tuple(str, bool):
    """Convert the first line of a cell for VR3."""
    is_code = False
    if line.startswith(MD_MARKER):
        line = AT_MD
    elif line.startswith(CODE_MARKER):
        line = AT_CODE
        is_code = True
    if line.startswith('# '):

        line = line[2:]
    return (line, is_code)

found_head = False
count = 0
for p0 in c.p.self_and_parents():
    count += 1
    if count > 2:  # Only look one step up the tree
        break
    jupy_file = p0.h.split('@jupytext')
    if len(jupy_file) > 1:
        found_head = True
        JUPYTER_POSITION = p0
        break

if not found_head:
    g.es('Cannot find an @jupytext node')

else:
    cells = [AT_MD]
    for p in JUPYTER_POSITION.subtree():
        is_code = False
        headline = p.h
        is_prefix = headline == PREFIX
        if is_prefix:
            cells.append(MD_LITERAL_FENCE)
        lines = p.b.split('\n')
        firstline = lines[0]
        firstline, is_code = process_first_line(firstline)
        cells.append(firstline)
        for line in lines[1:]:
            i = 0 if is_code else 2
            cells.append(line[i:])
        if is_prefix:
            cells.append(MD_LITERAL_FENCE)

    body = '\n'.join(cells)

    vr3_jupytext_node = JUPYTER_POSITION.insertAfter()

    vr3_jupytext_node.h = 'Rendering for ' + jupy_file[1]
    vr3_jupytext_node.b = body

    c.selectPosition(vr3_jupytext_node)
    c.redraw()

[HaveF HaveF]

Thank you, Thomas. The script works.

 

The command will create a new node below your @jupytext tree and will select it.  

But I don't quite understand why you need to do this? Why do we need a new node? Is it because you don't want to put the `#` logic in the jupytext node into vr3?

In addition, I think a possible use case is that after selecting a jupytext child node, only the content of the current child node is rendered, and the rendering is performed while changing. It may be more convenient for latex-related things.

 

[Thomas Passin]

I create a new node for rendering because VR3 understands how to render markdown with the @language directives but the @jupytext tree isn't markdown. If the leading comment signs were removed and the code sections fenced then VR3 would be able to render the entire tree, but not the way things stand now. I don't want to change the actual @jupytext tree because then it wouldn't save correctly with the current save code. 

The importer could have created the file tree such that VR3 could render it directly, but things went in another direction.

When I incorporate the conversion code into VR3, there won't be a need for this script - or for creating a new node - but until I do that, this script allows you to view the rendered file. Getting the script out there to be used can also help me make sure the conversion is solid when I do build it into VR3.

On Thursday, October 31, 2024 at 10:29:31 PM UTC-4 iamap...@gmail.com wrote:

In addition, I think a possible use case is that after selecting a jupytext child node, only the content of the current child node is rendered, and the rendering is performed while changing. It may be more convenient for latex-related things.

I can do that when I get the code into VR3 itself. Thanks for the suggestion. I'm not sure how it relates to latex, though. VR3 doesn't know how to render latex code per se.  If it's in a fenced code or literal block, it will be rendered as source code.  If it's MathJax, it will render as equations - they look really good. VR3 is probably never going to render latex code except as literal source code, because the whole process is too complex and would be much too slow to be practical.  Sure would be nice, though!

I have not started to modify VR3  to render @jupytext files directly because I wanted to have some confidence that the format will be stable.

[HaveF HaveF]

When I incorporate the conversion code into VR3, there won't be a need for this script - or for creating a new node - but until I do that, this script allows you to view the rendered file. Getting the script out there to be used can also help me make sure the conversion is solid when I do build it into VR3.

Get it!

 

I'm not sure how it relates to latex, though. VR3 doesn't know how to render latex code per se.  If it's in a fenced code or literal block, it will be rendered as source code.  If it's MathJax, it will render as equations - they look really good. 

Oh, the latex I mentioned is the formula in Markdown, I didn’t express it clearly, sorry

 

[Edward K. Ream]

On Wed, Oct 30, 2024 at 6:40 PM Thomas Passin <tbp1...@gmail.com> wrote:

Now that we have a more stable import format for @jupytext nodes, I redid my script to produce a version that renders the file with VR3.

Very cool. I see intermixed @python and @md directives. Does your script insert those directives?

Edward

[Thomas Passin]

Yes it does. VR3 can have any number of mixed code language sections in a tree or even in a single node. Each code section will get colorized by a colorizer for its language.  Colorizing is (I think for all three of RsT, MD, Asciidoc) by pygments which isn't the greatest but that's out of my hands.  Of course, if the code language sections are for different programming languages, code execution will be highly unlikely to work, since all the code blocks are collected into a single string that gets executed.

BTW, VR3 has a menu item that will render only the code blocks.

Those directives are why I claimed so confidently that the MD and code sections of the jupytext file could be recovered.

[Thomas Passin]

On Thursday, October 31, 2024 at 10:29:31 PM UTC-4 iamap...@gmail.com wrote:

Thank you, Thomas. The script works.

 

The command will create a new node below your @jupytext tree and will select it.  

But I don't quite understand why you need to do this? Why do we need a new node? Is it because you don't want to put the `#` logic in the jupytext node into vr3?

Ask and you shall receive. Here is a modified version of the script that renders the notebook on the fly - without creating a new node.  I suggest that you open VR3 before running it since I have seen once in a while that the splitter doesn't open up the VR3 pane.  It's there, and it even has a splitter handle to drag its pane larger, but that's hard to see.  I think this only happens right after startup but I can' reproduce the behavior yet. After 6.8.3 ...

Note that after rendering the notebook, VR3 will be left in its "freeze" state.  That means it won't render other nodes when they get selected.  This is intentional because otherwise you would lose the rendered notebook by mistake all the time.  The top left-hand menu in VR3 has a checkable item "Freeze".  Click it and the freeze state will be removed.

Oh, and you can export the rendered view to the system browser by clicking on VR3's "Export" button.

"""Convert an @jupytext node tree for rendering with VR3.

   Select any node of the tree before running this script.
   Note that VR3 will be set into its "freeze" mode.
"""
import leo.plugins.viewrendered3 as v3

MD_MARKER = '# %% [markdown]'
CODE_MARKER = '# %%'
PREFIX = '<< prefix >>'

AT_MD = '@language md'
AT_CODE = '@language python'
MD_LITERAL_FENCE = "```"

pc = g.app.pluginsController
vr3_enabled = pc.isLoaded('viewrendered3.py')

class MdNode:
    def __init__(self, h:str = '', md:str = ''):
        self.h = h  # headline
        self.b = md  # body

def process_first_line(line:str) -> tuple(str, bool):
    """Convert the first line of a cell for VR3."""
    is_code = False
    if line.startswith(MD_MARKER):
        line = AT_MD
    elif line.startswith(CODE_MARKER):
        line = AT_CODE
        is_code = True
    if line.startswith('# '):

        line = line[2:]
    return (line, is_code)

if not vr3_enabled:
    g.es('viewrendered3.py must be enabled in settings to render this @jupytext file.')
else:

    found_head = False
    count = 0
    for p0 in c.p.self_and_parents():
        count += 1
        if count > 2:  # Only look one step up the tree
            break
        jupy_file = p0.h.split('@jupytext')
        if len(jupy_file) > 1:
            found_head = True
            JUPYTER_POSITION = p0
            break

    if not found_head:

        g.es('Cannot find an @jupytext node')

    else:
        cells = [AT_MD]
        for p in JUPYTER_POSITION.subtree():
            is_code = False
            headline = p.h
            is_prefix = headline == PREFIX
            if is_prefix:
                cells.append(MD_LITERAL_FENCE)
            lines = p.b.split('\n')
            firstline = lines[0]
            firstline, is_code = process_first_line(firstline)
            cells.append(firstline)
            for line in lines[1:]:
                i = 0 if is_code else 2
                cells.append(line[i:])
            if is_prefix:
                cells.append(MD_LITERAL_FENCE)

        body = '\n'.join(cells)

        h = c.hash()
        vr3 = v3.controllers.get(h)
        c.selectPosition(c.p)
        c.redraw()

        headline = 'Rendering of ' + jupy_file[1].split('/')[-1]
        md_node = MdNode(headline, body)
        vr3.set_freeze()
        vr3.update_md([md_node], {})