[ANN] lua-resty-template - (HTML) Templating Engine for OpenResty (and Lua)

[Aapo Talvensaari]

Hi,

I wanted to start a new thread here for lua-resty-template that I created last weekend (based on tirtemplate.lua). Mainly because I do not want to pollute the another thread ("simple and fast Lua templating engine") with this.

The project can be found from here:

https://github.com/bungle/lua-resty-template

I think that the code is now quite stable, and the current functionality is fairly well documented (on Github). The code base is relatively small, just around 100 lines of Lua. So it is easy to understand, and fork. What I wish here is that I could get some feedback. Do you find lua-resty-template useful? Do you think that it is missing something important?

Something that I have been thinking about:

- Implementing variable modifiers (http://www.smarty.net/docs/en/language.modifiers.tpl)?

- Somekind of a plugin architecture?

- Do you think that it is better for render just to return a string instead of outputting it aka template.compile(file)(context) == template.render(file, context) or should I drop render alltogether?

I kinda like how small the whole thing is, and I do not personally have needs to add functionality beyond what is currently provided (I'd rather remove code if it can be simplified). But I surely would like to know if there are any bugs, edge cases, or possibilities to optimize things (LuaJIT string buffer sounds great when it is ready, what else?).

Kind Regards

Aapo

[Tor Hveem]

Looks good!

I will be testing it when I have a chance to.

--

Best Regards,

Tor Hveem

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Aapo Talvensaari]

Great!

And thanks for that tirtemplate. I wouldn't have started this without it.

Today I made some furher contributions. Main things:

1. Metatables of context objects are not modified anymore
2. You can also use strings as templates (in addition to files)
3. You can call tostring for tables that are created with template.new
4. Linefeeds are now better handled especially when using code blocks, i.e. {% for _, name in ipairs(names) do %} ... {% end %}
5. Empty content is not added to __r[#__r + 1] anymore
6. Documentation is updated
7. Original context object is provided as context in templates, i.e. {{context.contextvariable}} == {{contextvariable}}
8. Some bugs were fixed too (nothing alarming, though)

Regards

Aapo

[jerome lafon]

Hello,

It seems great!

Did you do some benchmarks with your templating engine with a simple template just to get an idea on the expected performance ?

Regards

Jérôme

[Yichun Zhang (agentzh)]

Hello!

On Wed, Feb 26, 2014 at 2:39 PM, Aapo Talvensaari wrote:
> I really do not know why my posts get deleted in this message board.
>

Your last few posts were marked as spam by Google and were waiting for
my moderation. They were not deleted.

> There seems to be some regressions with LuaJIT (Lua 5.2.2 seems to work
> little bit better on large iteration counts).

Several suggestions on performance tuning:

1. Try running your Lua scripts with luajit's -jv option to see what
is aborting JIT compilation. Work-around the NYI items shown in the
output.

2. Try running your Lua scripts with the builtin low-overhead
profiler, for example, with the -jp=vf20 option. See
http://repo.or.cz/w/luajit-2.0.git/blob_plain/refs/heads/v2.1:/doc/ext_profiler.html
for details. (Ensure your Lua scripts run long enough like seconds
because the profiler is based on sampling).

3. Try generating a C-land on-CPU flame graph for the luajit process
running your scripts:
https://github.com/agentzh/nginx-systemtap-toolkit#sample-bt This
graph can show CPU time distribution on a lower level, the VM level,
than LuaJIT's builtin profiler (which works on the Lua code level).

Best regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Thu, Feb 27, 2014 at 6:19 AM, Aapo Talvensaari wrote:
> Hi Agentzh,
>

Please do not capitalize my nick; I hate that. If you insist or your
mail client insists, please call me Yichun instead, because that is my
proper first name.

> There are still some regressions with LuaJIT, but overall it got a whole lot
> better with this. I think the next optimization points are in template.parse
> which uses string.gmatch that is not JIT compiled.
>

Yeah, getting most of your hot Lua code paths compiled and reduce
dynamic allocations of GC objects like tables and strings can make a
big difference.

> Thanks for your suggestions.
>

My pleasure.

> I'm still amuzed that my perfectly valid messages got marked as a spam. I
> did write 4 mails, all of which were marked as spam.
>

Blame Google for it ;) It seems like Google does not like you. Alas.

Regards,
-agentzh

[Aapo Talvensaari]

On Fri, Feb 28, 2014 at 9:12 PM, Yichun Zhang (agentzh) <age...@gmail.com> wrote:

Blame Google for it ;) It seems like Google does not like you. Alas.

I already did that. Nothing worked. All my messages go deleted immediately. Can you please check if someone has banned me or something. It also seems that no one is moderating this group. So all the deleted messages will stay deleted forever. Very frustrating for OpenResty users/community. It's not only Google's problem, it's also a problematic for OpenResty community.

Regards

Aapo 

[Yichun Zhang (agentzh)]

Hello!

On Mon, Mar 3, 2014 at 10:34 AM, Aapo Talvensaari wrote:
> Right now, lua-resty-template caches compiled template functions is a
> lua-table, but I'm not sure if that is a way to go.

You can either let your user cache the template himself or cache
automatically in your template engine.

You can implement a simple LRU cache atop Lua tables (and FFI cdata).
This can be a separate Lua library I suppose.

> And even bigger problem with
> large number of workers (not sharing cache). That's why I'm looking for
> alternatives.
>

You cannot share Lua code across the process boundary (or the Lua VM
boundary). It will not be cheaper than compiling your template string
again.

> I see there is this:
> http://wiki.nginx.org/HttpLuaModule#ngx.shared.DICT
>

No, you need serialize and deserialize the compiled Lua closure
yourself, which has its own overhead. Keep in mind you're going across
the process boundary. And compiled Lua closures are essentially a
bunch of binary data strucutres (GCproto, GCfunc, and even with
GCtrace and machine code area) interleaved with the rest of the VM.

> All my messages go deleted immediately.

From your UI, they may appear as "deleted", but they are actually not.
They're in my moderation queue. For example, your last 2 messages just
appear due to my manual approval.

> Can you please check if someone has banned me or
> something.

No, you're not banned. Your messages are just always marked as "spam" by Google.

> It also seems that no one is moderating this group.

I've been actively moderating this group. Otherwise most of your posts
would never show up.

> Very frustrating for OpenResty users/community. It's
> not only Google's problem, it's also a problematic for
> OpenResty community.

Well, that's not fair. I moderate posts in the lists almost everyday.

BTW, Google does provide a "post and always allow future messages from
author(s)" button in the group moderation UI, I always click on that
button for your posts, but it never works.

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Mon, Mar 3, 2014 at 8:36 PM, Yichun Zhang (agentzh) wrote:
> You can implement a simple LRU cache atop Lua tables (and FFI cdata).
> This can be a separate Lua library I suppose.
>

Forgot to mention that you should use the Lua-module-level tables for
the cache, so that the cache can be shared by all the requests served
by the current NGINX worker process, as documented here:

https://github.com/chaoslawful/lua-nginx-module#data-sharing-within-an-nginx-worker

This should be good enough. NGINX users usually do not configure
hundreds or even thousands of workers anyway, unlike Apache prefork
mpm.

Regards,
-agentzh

[Aapo Talvensaari]

Okay, after 280 commits this thing is starting to take an usable shape. I have done some perfomance tweaks (parser code is rewritten for example). The engine is quite fast, especially when cached. I have also added things like template_root, and template_location Nginx variables that are documented here:

https://github.com/bungle/lua-resty-template#nginx--openresty-configuration

The API feels quite stable now, and I do not have anything on my mind that would somehow break the API in the near future. I think that I have also documented this thing quite well now on Github's readme.

So feel free to try it out, and please report any bugs you find. I have personally tested this quite well (not automated yet, though), and it feels stable.

If you have any guestions on your mind, please do not hesitate to ask.

One thing I still have on my mind is the cache implementation. Right now it uses module level Lua table, and caches everything. I think that there might be a need to have a pluggable cache (you can already write your own, and disable automatic caching with template.caching(false)), the default implementation could be changed to act as a weak table: __mode = "kv", or I may implement LRU-type of cache. But right now it works ok for normal sites (not too many templates). But if you have tens of thousands templates, then you should look the memory usage too (or disable caching, and optionally implement your own).

Regards

Aapo

[Simon]

I've tested this module and run online,  it is wonderful and powerful, thanks.

[Aapo Talvensaari]

I've tested this module and run online,  it is wonderful and powerful, thanks.

Great to hear that! Have you hit any bugs, performance problems or other

inconsistencies? Or do you miss any features?

I'm thinking about adding LuaRocks support, and busted/Nginx tests,

as soon as I have time to do that. I also need to start collecting change

logs, and publish version 1.0 (as a release).

Regards

Aapo

[Jerome Lafon]

I have tried lua-resty-template and benchmarked it.

It's very powerful, mostly with the possibility to separate the content over multiple files and using a layout file to embed the all the parts (header, footer, view....)

It's faster than etlib.

I don't know if you can improve even more the speed but congratulations and thank you!

Jérôme

[Aapo Talvensaari]

I have added one small feature to lua-resty-template. That is a support for "blocks" (think about ASP.NET master pages and it's placeholders, and content tags in a views). Well, if that concept is not familiar to you, see the updated documentation for it here (it's pretty self-explatonary):

https://github.com/bungle/lua-resty-template#using-blocks 

Latest sources are in Github too.

[Jerome Lafon]

Thank you, that a very nice feature!