Generating C++ code

[Jean-Michel Pichavant]

Greetings,

I'm trying to generate C++ code from an XML file. I'd like to use a template engine, which imo produce something readable and maintainable.
My google search about this subject has been quite unsuccessful, I've been redirected to template engine specific to html mostly.

Does anybody knows a python template engine for generating C++ code ?

Here's my flow:

XML file -> nice python app -> C++ code

>From what I know I could use Cheetah, a generic template engine. I never used it though, I'm not sure this is what I need.
I'm familiar with jinja2 but I'm not sure I could use it to generate C++ code, did anybody try ? (maybe that's a silly question)

Any advice would be appreciated.

JM

[Andrea Crotti]

I think you can use anything to generate C++ code, but is it a good idea?
Are you going to produce this code only one time and then maintain it
manually?

And are you sure that the design that you would get from the XML file
actually makes sense when
translated in C++?

[Etienne Robillard]

> --
> http://mail.python.org/mailman/listinfo/python-list

You can build nice python app with Cython/Pyrex too if you got sufficient knowledge in C/C++ programming
to extend the app in C. :-)

Otherwise that seem like a little counter-productive to produce a python template only for generating C stubs when python
is good for many things without requiring advanced C++ skills like memory management, etc.

HTH,
E
--
Etienne Robillard
Green Tea Hackers Club
Fine Software Carpentry For The Rest Of Us!
http://gthc.org/
er...@gthcfoundation.org

[Jean-Michel Pichavant]

Well, the C++ code will end up running on a MIPS on a SOC, unfortunately, python is not an option here.
The xml to C++ makes a lot of sense, because only a small part of the code is generated that way (everything related to log & fatal events). Everything else is written directly in C++.

To answer Andrea's question, the files are regenerated for every compilation (well, unless the xml didn't change, but the xml is highly subject to changes, that's actually its purpose)

Currently we already have a python script that translate this xml file to C++, but it's done in a way that is difficult to maintain. Basically, when parsing the xml file, it writes the generated C++ code. Something like:
if 'blabla' in xml:
h_file.write("#define blabla 55", append="top")
c_file.write("someglobal = blabla", append="bottom")

This is working, but the python code is quite difficult to maintain, there's a lot of escaping going on, it's almost impossible to see the structure of the c files unless generating one and hopping it's successful. It's also quite difficult to insert code exactly where you want, because you do not know the order in which the xml trees are defined then parsed.

I was just wondering if a template engine would help. Maybe not.

JM

[Etienne Robillard]

On Wed, 10 Oct 2012 11:59:50 +0200 (CEST)
Jean-Michel Pichavant <jeanm...@sequans.com> wrote:

> Well, the C++ code will end up running on a MIPS on a SOC, unfortunately, python is not an option here.
> The xml to C++ makes a lot of sense, because only a small part of the code is generated that way (everything related to log & fatal events). Everything else is written directly in C++.

sorry but i don't get what you mean with a "MIPS on a SOC". Is not Python well supported on MIPS ?

> Currently we already have a python script that translate this xml file to C++, but it's done in a way that is difficult to maintain. Basically, when parsing the xml file, it writes the generated C++ code. Something like:
> if 'blabla' in xml:
> h_file.write("#define blabla 55", append="top")
> c_file.write("someglobal = blabla", append="bottom")

Don't do that! This is a good example of ambigous coding (to say the least..) and you'll make C++ programmers eyes to bleed
at this.

> This is working, but the python code is quite difficult to maintain, there's a lot of escaping going on, it's almost impossible to see the structure of the c files unless generating one and hopping it's successful. It's also quite difficult to insert code exactly where you want, because you do not know the order in which the xml trees are defined then parsed.

Its maybe working but why then are you stuck asking for help ? I suggest you either write plain C++ code or learn to
use Python more efficiently...


> JM
> --
> http://mail.python.org/mailman/listinfo/python-list

Kind regards,
Etienne

[Stefan Behnel]

Jean-Michel Pichavant, 10.10.2012 11:59:

> Well, the C++ code will end up running on a MIPS on a SOC,
> unfortunately, python is not an option here. The xml to C++ makes a lot
> of sense, because only a small part of the code is generated that way
> (everything related to log & fatal events). Everything else is written
> directly in C++.
>

> To answer Andrea's question, the files are regenerated for every
> compilation (well, unless the xml didn't change, but the xml is highly
> subject to changes, that's actually its purpose)
>

> Currently we already have a python script that translate this xml file
> to C++, but it's done in a way that is difficult to maintain. Basically,
> when parsing the xml file, it writes the generated C++ code. Something
> like:
>
> if 'blabla' in xml:
> h_file.write("#define blabla 55", append="top")
> c_file.write("someglobal = blabla", append="bottom")
>

> This is working, but the python code is quite difficult to maintain,
> there's a lot of escaping going on, it's almost impossible to see the
> structure of the c files unless generating one and hopping it's
> successful. It's also quite difficult to insert code exactly where you
> want, because you do not know the order in which the xml trees are
> defined then parsed.
>

> I was just wondering if a template engine would help. Maybe not.

Depends. Template engines are great for injecting small data snippets into
large static code blocks. They are less good for finely structured code
with conditional insertions and varying code order all over the place.

In Cython, we use a combination of both: a template engine for large code
blocks with small adaptations and line-by-line generated C code for highly
varying code. Works nicely.

As for a template engine, we use Tempita for the more complex stuff
(because the implementation is small and can be embedded) and
string.Template for the simple stuff. If you need something more advanced
straight away, I'd say go for Cheetah.

In case you decide not to use a template engine at all, given that your
input format is XML, consider using lxml to build up your code tree for you
by using custom element classes. Then, generate the code recursively top-down.

http://lxml.de/element_classes.html

Might or might not be a suitable approach, depending on the complexity of
your mapping from XML to code.

Stefan

[andrea crotti]

2012/10/10 Jean-Michel Pichavant <jeanm...@sequans.com>:

> Well, the C++ code will end up running on a MIPS on a SOC, unfortunately, python is not an option here.
> The xml to C++ makes a lot of sense, because only a small part of the code is generated that way (everything related to log & fatal events). Everything else is written directly in C++.
>
> To answer Andrea's question, the files are regenerated for every compilation (well, unless the xml didn't change, but the xml is highly subject to changes, that's actually its purpose)
>
> Currently we already have a python script that translate this xml file to C++, but it's done in a way that is difficult to maintain. Basically, when parsing the xml file, it writes the generated C++ code. Something like:
> if 'blabla' in xml:
> h_file.write("#define blabla 55", append="top")
> c_file.write("someglobal = blabla", append="bottom")
>
> This is working, but the python code is quite difficult to maintain, there's a lot of escaping going on, it's almost impossible to see the structure of the c files unless generating one and hopping it's successful. It's also quite difficult to insert code exactly where you want, because you do not know the order in which the xml trees are defined then parsed.
>
> I was just wondering if a template engine would help. Maybe not.
>

> JM
> --
> http://mail.python.org/mailman/listinfo/python-list


I think it depends on what you're writing from the XML, are you
generating just constants (like the #define) or also new classes for
example?

If it's just constants why don't you do a generation from XML -> ini
or something similar and then parse it in the C++ properly, then it
would be very easy to do?

You could also parse the XML in the first place but probably that's
harder given your requirements, but I don't think that an ini file
would be a problem, or would it?

[Ulrich Eckhardt]

Am 09.10.2012 18:00, schrieb Jean-Michel Pichavant:
> I'm trying to generate C++ code from an XML file. I'd like to use a
> template engine, which imo produce something readable and
> maintainable.

> [...]

> Here's my flow:
>
> XML file -> nice python app -> C++ code

There is one question that you should answer (or maybe decide?) first:
How close is the XML structure to C++ semantically?

The syntactic level is obviously very different, as one uses XML as
metaformat while the other is C++. The semantic level is rather about
the question if there is e.g. a "<class name='foo'>" that directly
translates to a "class foo {" in C++. If that is the case, the SAX API
should help you, as it basically invokes callbacks for every XML element
encountered while parsing the input stream. In those callbacks, you
could then generate the according C++ code in a way that should be
readable and maintainable with plain Python or some template engine.

You you need to skip back-and-forth over the input, reading the whole
XML as DOM tree would probably be a better approach. Still, the
processing of input is separate from output generation, so you could at
least divide your task before conquering it.

Notes:
- There is also XSLT which can generate pretty much anything from XML,
but it is can't do much more than text replacements triggered by input
matching. The more the output differs semantically from the input, the
more difficult it becomes to use. Also, XSLT tends to become write-only
code, i.e. unreadable.
- I think there was a feature in GCC that allows generating XML from
C++ input, maybe even the reverse. Maybe you could leverage that?


Good luck!

Uli

[Jean-Michel Pichavant]

> sorry but i don't get what you mean with a "MIPS on a SOC". Is not
> Python well supported on MIPS ?

Sorry, SOC means system on chip. The binary runs on the MIPS, there's no file system, no operanding system, except for one very basic task scheduler.

That's why everything is done at compile time.

JM

[Michael Torrie]

On 10/09/2012 10:00 AM, Jean-Michel Pichavant wrote:
> Greetings,

>
> I'm trying to generate C++ code from an XML file. I'd like to use a template engine, which imo produce something readable and maintainable.

> My google search about this subject has been quite unsuccessful, I've been redirected to template engine specific to html mostly.
>

> Any advice would be appreciated.

There are two ways I can see to do this.

One is to use XSLT. XML is designed to be transformed easily, and XSLT
is the standard vehicle to do this transformation. You can use XSLT to
translate an XML document to HTML, for example, or to another schema of
XML, or to something non-XML like plain text, Tex, or even C++ code.
XSLT itself is turing complete, so complicated, logical transformations
are possible. Since I know of no existing transformation templates for
emitting C++, or any other language, you'll have to write it yourself.
Yes it won't be python code per se, though you can drive the XML XSLT
engine from python using a module called lxml. Anyway, XSLT is
complicated (I have done nothing in it myself), but it is designed for
this kind of thing.

Secondly, you could just use a python XML module of some kind, read the
document in, traverse the nodes and emit code. This is basic compiler
theory, though your source code is likely already in an abstract syntax
tree (XML). And I bet you could make it one pass, as you can rely on
the compiler to check symbol names for you. Compilers are not simple
beasts, but they are fun to write, and very educational. You get to
learn about grammars and recursive descent parsing (though the XML
library basically does both for you).

[Grant Edwards]

On 2012-10-10, Etienne Robillard <anime...@gmail.com> wrote:
> On Wed, 10 Oct 2012 11:59:50 +0200 (CEST)
> Jean-Michel Pichavant <jeanm...@sequans.com> wrote:
>
>> Well, the C++ code will end up running on a MIPS on a SOC,
>> unfortunately, python is not an option here. The xml to C++ makes a
>> lot of sense, because only a small part of the code is generated that
>> way (everything related to log & fatal events). Everything else is
>> written directly in C++.
>
> sorry but i don't get what you mean with a "MIPS on a SOC". Is not
> Python well supported on MIPS ?

SoC == System On a Chip.

It's a single-chip micro-controller embedded inside something that's
not a general purpose computer (e.g. it's in a router, or piece of
industrial equipment, or whatever). It may only have a couple MB of
memory, it might have only a minimal RTOS (non-Linux/Unix,
non-Windows), or it may actually have no OS at all. It almost
certainly doesn't have a hard drive.

Many years ago, there was a "deeply embedded Python" project that was
attempting to get Python running on such platforms, but it's been
abandoned for ages. IIRC, it was using Python 1.50 as a base version.

--
Grant Edwards grant.b.edwards Yow! My vaseline is
at RUNNING...
gmail.com

[Jean-Michel Pichavant]

Have a look at http://code.google.com/p/python-on-a-chip/
Last news on 2011/09/26, I'm not sure the project is still alive.

JM

[Stefan Behnel]

Jean-Michel Pichavant, 10.10.2012 17:05:

>> SoC == System On a Chip.
>>
>> It's a single-chip micro-controller embedded inside something that's
>> not a general purpose computer (e.g. it's in a router, or piece of
>> industrial equipment, or whatever). It may only have a couple MB of
>> memory, it might have only a minimal RTOS (non-Linux/Unix,
>> non-Windows), or it may actually have no OS at all. It almost
>> certainly doesn't have a hard drive.
>>
>> Many years ago, there was a "deeply embedded Python" project that was
>> attempting to get Python running on such platforms, but it's been
>> abandoned for ages. IIRC, it was using Python 1.50 as a base
>> version.
>>

>> Grant Edwards

>
> Have a look at http://code.google.com/p/python-on-a-chip/
> Last news on 2011/09/26, I'm not sure the project is still alive.

For FLOSS projects, it's usually better to look at the revision history,
which, in this case, names September 3 as last entry, *this* year.

Pretty much alive, I'd say.

Stefan

[Tim Roberts]

Jean-Michel Pichavant <jeanm...@sequans.com> wrote:
>
>I'm trying to generate C++ code from an XML file. I'd like to use a template engine, which imo produce something readable and maintainable.
>My google search about this subject has been quite unsuccessful, I've been redirected to template engine specific to html mostly.
>
>Does anybody knows a python template engine for generating C++ code ?

I'm a big fan of Cheetah. It's simple but flexible enough to be useful.
Besides the many web projects I've done with it, I also I use it in one
project to generate PHP code (it generates data access objects from a live
database schema).
--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.

[Etienne Robillard]

> --
> http://mail.python.org/mailman/listinfo/python-list

Also take a look at IDL, for a proper way to handle interface generation in C++. No python or cheetah
required as by definition your interfaces should be portable. What your describing is more or less look
like a hack or something which would be a pain to maintain without SWIG or something more suited for
this purposes than XML.

http://en.wikipedia.org/wiki/Interface_description_language
http://en.wikipedia.org/wiki/SWIG