Lexer node hierarchy

[Nabil Khoury]

Hi, I'm a recent and happy mako adopter. 

I went with mako templates (v. 1.0.8) to house my business logic in XML form. Something along these lines:

t = BungoTemplate("""

<%namespace name="me" module="__main__" import="rows,row"/>

<%me:rows one="I" two="II" three="III">

    <%me:row four="IV" five="V" six="VI"/>

    <%me:row four="IV" five="V" six="VI">

        <%me:field src="src" src_regex="src_regex" src_call="src_call"/>

    </%me:row>

</%me:rows>

""")

I can define these custom tags somewhere above it in the same .py and it'll just call them no problem and I can get it to call the capture on the caller's body to effectively recreate the XML hierarchy in my call-stack:

@supports_caller

def rows(context, *, one=1, two=2, three=3):

    child = capture(context, context['caller'].body)

    return ("<rows one='{}' two='{}' three='{}'>"

            "{}"

            "</rows>"

            ).format(one, two, three, child)

This all works well so far and I'm sure I can automate the XML conversion so I don't have to do it in each of my supports_caller-decorated functions. 

I guess my question would be: Is there a better way to grab the current call-stack in XML form or just plain stack/list form?

I thought for sure it would be in context.caller_stack but although I can get the stack of Namespace objects as shown below, I'm not sure how to interrogate/inspect the Namespace objects properly in order to retrieve a piece of information that identifies the caller without ambiguity so I can re-create the calling XML shown above.

list of callers via 'caller_stack':  [None, <mako.runtime.Namespace object at 0x7f36e5ac8dd8>, <mako.runtime.Namespace object at 0x7f36e5ac89e8>] [(None, 'caller', <mako.runtime.Context object at 0x7f36e5af2358>, {'body': <function render_body.<locals>.ccall.<locals>.body at 0x7f36e5b361e0>}), (None, 'caller', <mako.runtime.Context object at 0x7f36e5af2358>, {'body': <function render_body.<locals>.ccall.<locals>.body.<locals>.ccall.<locals>.body at 0x7f36e5b36158>})]

I can do something very similar to what I need but statically. I can, for instance, rely on the TemplateNode.parent and TemplateNode.nodes construct found in Lexer to get the XML structure of my custom tags conveniently before the render has been called like so:

class BungoLexer(Lexer):

    glexer = None

    def __init__(self, *args, **kwargs):

        BungoLexer.glexer = self

        super().__init__(*args, **kwargs)

    def get_action_tree(self, _node=None, _xml=None):

        if _xml is None:

            _xml = {id(None):ET.Element("bungo")}

        if _node is None:

            _node = self.template

        print("child nodes: ", [(id(n.parent), n.expression)

               for n in _node.nodes 

               if n and isinstance(n, CallTag) or isinstance(n, CallNamespaceTag)])

        for n in _node.nodes:

            if isinstance(n, CallTag) or isinstance(n, CallNamespaceTag):

                if n.parent:

                    print("node:{} with expression:{} has parent:{}".format(id(n), n.expression, id(n.parent)))

                else:

                    print("node:{} with expression:{} has no parent".format(id(n), n.expression))

                safe_tag = n.keyword.replace(':','-')

                try:

                    parent_xml = _xml[id(n.parent or None)]

                    print("parent_xml=", parent_xml)

                except KeyError as ke:

                    # this should never happen

                    ke.args += ("This is really bad! Parent key could not be found in the xml cache!",)

                    raise ke

                _xml[id(n)] = ET.SubElement(parent_xml,

                                     safe_tag,

                                     attrib=n.attributes,

                                     parent_id=str(id(_node)),

                                     id=str(id(n))

                                     )

            else:

                continue

            self.get_action_tree(_node=n, _xml=_xml)

        return _xml[id(None)]

class BungoTemplate(Template):

    def __init__(self, *args, **kwargs):

        # override lexer_cls=Lexer with BungoLexer

        super().__init__(*args, lexer_cls=BungoLexer, **kwargs)

    def render(self, *args, **kwargs):

        return super().render(*args, **kwargs)

t = GinsuTemplate("""

<%namespace name="me" module="{myself}" import="slices,slice"/>

<%me:slices one="I" two="II" three="III">

    <%me:slice four="IV" five="V" six="VI"/>

    <%me:slice four="IV" five="V" six="VI">

        <%me:dice src="src" src_regex="src_regex" src_call="src_call"/>

    </%me:slice>

</%me:slices>

""".format(myself=__name__))

_xml = GinsuLexer.glexer.get_action_tree()

print(minidom.parseString(ET.tostring(_xml)).toprettyxml(indent='  '))

Which gives me something similar to what I'm trying to generate dynamically during run-time invocation of custom-tag function:

<?xml version="1.0" ?>

<ginsu>

  <me-slices id="139873758424200" one="I" parent_id="139873758620696" three="III" two="II">

    <me-slice five="V" four="IV" id="139873758424536" parent_id="139873758424200" six="VI"/>

    <me-slice five="V" four="IV" id="139873758249704" parent_id="139873758424200" six="VI">

      <me-dice id="139873758250096" parent_id="139873758249704" src="src" src_call="src_call" src_regex="src_regex"/>

    </me-slice>

  </me-slices>

</ginsu>

I know this is not your typical mako usecase so I really can't expect it to have all the conveniences I may require. I would, still, love to get some input on alternative approach with my gratitude.

Thanks a bunch in advance.

Nabil Khoury

[Mike Bayer]

On Sat, Sep 22, 2018 at 5:30 PM Nabil Khoury <nabilg...@gmail.com> wrote:
>
> Hi, I'm a recent and happy mako adopter.
>
> I went with mako templates (v. 1.0.8) to house my business logic in XML form. Something along these lines:
>
> t = BungoTemplate("""
> <%namespace name="me" module="__main__" import="rows,row"/>
> <%me:rows one="I" two="II" three="III">
> <%me:row four="IV" five="V" six="VI"/>
> <%me:row four="IV" five="V" six="VI">
> <%me:field src="src" src_regex="src_regex" src_call="src_call"/>
> </%me:row>
> </%me:rows>
> """)
>
>
> I can define these custom tags somewhere above it in the same .py and it'll just call them no problem and I can get it to call the capture on the caller's body to effectively recreate the XML hierarchy in my call-stack:
>
> @supports_caller
> def rows(context, *, one=1, two=2, three=3):
> child = capture(context, context['caller'].body)
> return ("<rows one='{}' two='{}' three='{}'>"
> "{}"
> "</rows>"
> ).format(one, two, three, child)
>
>
> This all works well so far and I'm sure I can automate the XML conversion so I don't have to do it in each of my supports_caller-decorated functions.
>
> I guess my question would be: Is there a better way to grab the current call-stack in XML form or just plain stack/list form?
> I thought for sure it would be in context.caller_stack but although I can get the stack of Namespace objects as shown below, I'm not sure how to interrogate/inspect the Namespace objects properly in order to retrieve a piece of information that identifies the caller without ambiguity so I can re-create the calling XML shown above.

so....you need a call that's nested deeply inside to know something
about who called it? what's the information you're looking to see?
can you pass it along explicitly ?

> --
> You received this message because you are subscribed to the Google Groups "Mako Templates for Python" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mako-discuss...@googlegroups.com.
> To post to this group, send email to mako-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mako-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Nabil Khoury]

Mike, It's a privilege talking to you! 

Ideally, I'd like to maintain access to the XML element of the calling parent so that I can append to it in the child's body when it's called in the Template:

import xml.etree.ElementTree as ET

@supports_caller
def root_tag(context, **kwargs):
    parent_xml = ET.Element('root_tag", attrib=kwargs)
    context['parent_xml'] = parent_xml
    child = capture(context, context['caller'].body)
    return parent_xml

@supports_caller
def child_tag(context, **kwargs):
    parent_xml = context['parent_xml']
    parent_xml = ET.SubElement(parent_xml, 'child_tag', attrib=kwargs)
    context['parent_xml'] = parent_xml
    child = capture(context, context['caller'].body)
    return parent_xml
   

I thought of it but I was concerned that context['parent_xml'] may not always be the xml of the caller but simply the last caller's SubElement, which may be a sibling child_tag. 

Is this the case where I should be attaching to context['caller'] instead of just context? I noticed there's a context.push_caller and context.pop_caller so perhaps that will guarantee that my context['caller']['_xml'] will always hold the callers SubElement ?

Is that roughly what you have in mind?

I hope I'm not completely off.

Nabil

[Mike Bayer]

are you trying to build an XML document as the template renders?
That seems really hard. I would probably go the other way around
(xml document builds a template). Mako is kind of presentation
oriented and doesn't have a real power API for analyzing the structure
of the template (but of course, that you made your own Lexer, then you
do).

Like I think if you passed data between your tags you could work with
that, but I see the building of an XML tree in memory, I probably
wouldn't repurpose the template language that way.

[Nabil Khoury]

That's indeed what I'm trying to do: create an XML equivalent of the template as it renders. By overriding Lexer, I managed to do this statically but the "as the template renders" has been an issue. 

 

Incidentally, I understand this is not the typical mako use case and do plan on using mako for presentation too later on during the project. 

You mentioned that I can pass the XML argument between tags. Is there a way to pass via the capture(context['caller'].body) call? or perhaps I can leave it on the caller's context? How do you suggest I pass the XML argument? I tried this:

child = capture(context, context['caller'].body, _xml)

among other things but that gives me an error because body doesn't take any positional parameters.

Is there any way that I can communicate the _xml part to the child tag without knowing in advance what the child tag will be?

I hope I'm not making too much of a pest out of myself and thanks again for your two-cents.
Nabil

[Nabil Khoury]

In case someone else might stumble upon a similar use-case. Here's one way to reconstitute the XML of custom tags during template render:

from mako.template import Template

from mako.runtime import capture, supports_caller

from functools import wraps

from typing import Callable

callables = dict(doubleit=lambda src: src*2)

def autotag(action):

    @wraps(action)

    @supports_caller

    def autotagged(context, **kwargs):

        ret = capture(context, context['caller'].body)

        action_ret = action(**kwargs)

        kwargs.update(action_ret)

        action_attrib = " ".join(["{}='{}'".format(k,v) for k, v in kwargs.items()])

        action_xml = "<{n} {a}>{r}</{n}>".format(

                n=action.__name__, a=action_attrib, r=ret

                )

        return action_xml

    return autotagged

@autotag

def slices(*, one: str='1', two: str='2', three: str='3'):

    return dict(one=one, two=two, three=three)

@autotag

def slice(*, four: int=4, five: int=5, six: int=6):

    return dict(four=four, five=five, six=six)

@autotag

def dice(*, src: str, src_regex: str, src_call: Callable):

    return dict(src=src, src_regex=src_regex, src_call=src_call)

t = Template("""

<%namespace name="me" module="{this_module}" import="slices,slice"/>

<%me:slices one="I" two="II" three="III">

    <%me:slice four="4" five="5" six="6"/>

    <%me:slice four="44" five="55" six="66">

        <%me:dice src="src" src_regex="^(?P<src>.*)$" src_call="doubleit"/>

    </%me:slice>

</%me:slices>

""".format(this_module=__name__))

print(t.render())

Output:

>>> import scratch_autotag

<slices one='I' two='II' three='III'>

    <slice four='4' five='5' six='6'></slice>

    <slice four='44' five='55' six='66'>

        <dice src='src' src_regex='^(?P<src>.*)$' src_call='doubleit'></dice>

    </slice>

</slices>

<module 'scratch_autotag' from '/home/nkhoury/scratch_autotag.py'>