Embeddable and Extensible Languages
PlayDough
evaluating several embeddable and extensible languages. However, I've
yet to find one that really strikes my fancy for our project.
I'm working on a project that would embed a language into our
simulation system for FPGA and ASIC design. This language would be
used to control a testbench through the PLI/FLI provided by the
simulator. Preferrably, it would be C (or C like) since our end users
are most familiar with C. Also, we would like the ability to have
multiple source files or packages that would allow our tools team to
develop support tools.
I've looked at Lua, Python, Pike, Ici, Eic, Ch, Cint, S-Lang, Small,
elastiC, Simkin, and even some small, homegrown projects. However,
they all have one problem or another. I really like Ch, but it is
quite expensive (~$2k). Or Small, but it doesn't provide basic I/O and
support is sparse. Or S-Lang and issues with syntax for our customers
(internal).
So far, I've narrowed it down to Lua and Python. I like the Lua
syntax, but the support base isn't as big as Python. But with Python,
I fear that the syntax could be problem for the customers of our
project. Both have the problem of not being capable of running
multiple contexts (outside of the interpreter).
I often think a virtual machine approach is the best because it allows
me to have several contexts (realized in separate virtual machines).
At worst, I can get by with a single context but it limits the
generality I'm looking for. On the virtual machine tack, I looked into
Java or the CLR, but interfacing such a beast with our simulator
doesn't seem feasible. I've even thought of writing our own virtual
machine to execute code, but then we'd have to invest in a compiler.
Years ago I created an interpreted version using lex and yacc, but that
was for a different company and I don't have it anymore. And
re-creating it doesn't sound like fun, nor was it feature rich.
So, given this huge lead-in, I'm wondering if anyone has tackled a
similar problem and what did they do? Or, can anyone recommend a
language that is embeddable, extensible, has a good set of features
(file and console I/O, math operations, even if through
packages/add-ons). Perferably it has a reasonable footprint, it
portable across platforms (Linux and Win32), and support for multiple
contexts.
I know this is probably a tall order, but before I settle with either
Lua or Python, I'd like to get more opinions. Over the years I've
found the newsgroups an indispensible source and I'd appreciate any
responses.
Pete
Georgios Petasis
by far the most advanced language when it comes to portability &
maturity of code (stable, easy to understand & extend).
It has also a graphical toolkit, its easy to extend through packages
(either in Tcl or C/C++), it has good I/O capabilities, but it can be
quite expensive. I know that exist version that even run in handheld
devices, but hou may have to investigate the details yourself.
I think that the best approach is to repeat this post in
comp.lang.tcl, where you can get answers from people who can better
understand you post (I am not sure what "multiple contexts" means :-))
George
"PlayDough" <pet...@gocougs.wsu.edu> wrote in message
Luiz Henrique de Figueiredo
>So far, I've narrowed it down to Lua and Python. [...]
>Both have the problem of not being capable of running
>multiple contexts (outside of the interpreter).
I'm not sure what you mean by the last sentence, but Lua can run multiple
independent states and multiple related states, with support for coroutines.
--lhf
Jean-Marc Bourguet
> I'm working on a project that would embed a language into our
> simulation system for FPGA and ASIC design. This language would be
> used to control a testbench through the PLI/FLI provided by the
> simulator. Preferrably, it would be C (or C like) since our end
> users are most familiar with C. Also, we would like the ability to
> have multiple source files or packages that would allow our tools
> team to develop support tools.
>
> I've looked at Lua, Python, Pike, Ici, Eic, Ch, Cint, S-Lang, Small,
> elastiC, Simkin, and even some small, homegrown projects.
Considering your target domain, I'd consider TCL as it is more or less
a de-facto standard extension langage used by EDA applications and so
there is a good chance that your users are already familiar with it.
> So far, I've narrowed it down to Lua and Python. I like the Lua
> syntax, but the support base isn't as big as Python. But with
> Python, I fear that the syntax could be problem for the customers of
> our project. Both have the problem of not being capable of running
> multiple contexts (outside of the interpreter).
I'm not sure what you want but multiple TCL interpreters can be run in
the same executable.
Another language you didn't name but which seems to have a signigicant
user base is Ruby.
Yours,
--
Jean-Marc
Thant Tessman
> I've spent quite a bit of time over the last couple of months
> evaluating several embeddable and extensible languages. However, I've
> yet to find one that really strikes my fancy for our project.
[...]
Have you looked at Scheme? Its syntax is as far from C as you can get,
but if you can get beyond that, I'm pretty sure at least one
implementation (Chez) will do exactly what you need. There might be
others.
-thant
Jürgen Kahrs
> So, given this huge lead-in, I'm wondering if anyone has tackled a
> similar problem and what did they do? Or, can anyone recommend a
> language that is embeddable, extensible, has a good set of features
> (file and console I/O, math operations, even if through
> packages/add-ons). Perferably it has a reasonable footprint, it
> portable across platforms (Linux and Win32), and support for
> multiple contexts.
Tcl
http://www.tcl.tk/customers/success/edacad.html
http://www.tcl.tk/customers/success/testautomation.html
Tom Verbeure
> its by far the most advanced language when it comes to portability &
> maturity of code (stable, easy to understand & extend).
From what I understand of the original email, the intention is to use
have language to write behavioral models that interact with the
simulator cycle-by-cycle.
(Disclaimer: Despite my intense distaste for Tcl,) I agree that Tcl is
the best choice for a *command* language, that is, to control
simulator, set breakpoints etc. But I don't think it's a good language
to write functional hardware models. Python has a number of
constructs, like generators, that make it much easier to model a
cycle-based environment. (You may have a look at
http://jandecaluwe.com/Tools/MyHDL/Overview.html for some
inspiration.)
I know of a company that linked Python to a VHDL simulator and used
the python environment to mimic an external or internal processor.
(Basically, the python environment was driving an AHB bus.) I don't
know if they used multiple, concurrent, threads though.
Tom
PlayDough
Georgios Petasis wrote:
> It seems odd that you haven't look Tcl (http://www.tcl.tk). For me
> its by far the most advanced language when it comes to portability &
> maturity of code (stable, easy to understand & extend).
I did look at Tcl, as well as several others, but it's more of a syntax
issue there. Even as much work as I've done in the EDA land and their
love of Tcl, I'm still forced to frequent a Tcl manual to figure out
syntax. Also, our internal customers have been using GUI based
simulation and hand-writing force files, then verifying functionality
visually. Automated testbenches are not common, and my attempt is to
make it automated. As such, Tcl is quite likely to be unfamiliar as
well.
Thanks for the suggestion.
[ next message ]
Jean-Marc Bourguet wrote:
> Another language you didn't name but which seems to have a
> signigicant user base is Ruby.
I don't know why I didn't look at Ruby. I've used it in the past, but
I completely forgot about it here. I'll be sure to look into it more.
Thanks for the suggestion!
[ next message ]
Luiz Henrique de Figueiredo wrote:
> PlayDough <pet...@gocougs.wsu.edu> wrote:
> >So far, I've narrowed it down to Lua and Python. [...]
> >Both have the problem of not being capable of running
> >multiple contexts (outside of the interpreter).
> I'm not sure what you mean by the last sentence, but Lua can run
> multiple independent states and multiple related states, with
> support for coroutines.
I just perused the manual, and you are correct. I misstated the issue
with Lua that I was having.
The system I'm creating uses SystemC. (I over-generalized earlier by
saying FLI/PLI, thinking it wasn't important. Turns out it is quite
relavent to the problem.) The SystemC implementation I am using
requires that I put the Lua state as a member of the class. When
extending Lua, I need to get a pointer to the state. However,
callbacks from the Lua interpreter must be implemented as static
member functions, so "this" doesn't exist and I can't get a pointer to
the state (required to gather passed parameters and to return values).
I can use userdata, but the amount of userdata is limited, and I want
the system to be as expandable as possible.
Further discussion of this issue belongs on a Lua (or related)
newsgroup, and I will post there to try and resolve this issue.
Thanks for the information,
Pete
PlayDough
> packages/add-ons). Perferably it has a reasonable footprint, it
> portable across platforms (Linux and Win32), and support for
> multiple contexts.
I wasn't abundantly clear on the above point (others have asked "what
do I mean exactly"), so I want to clarify the above point.
I would like to find a language with a reentrant interpreter. But
even this isn't totally accurate. Perhaps a C++ example might clear
up what I'm trying to do.
For example, I create a class:
class X
{
public:
X(const std::string& script_file);
void Run(void);
private:
Interpreter_context ctx;
};
And in the constructor, I do something like:
X::X(const std::string& script_file)
{
interp_init();
interp_create_context(&ctx);
load_script(&ctx, script_file);
}
void X::Run(void)
{
run_script(&ctx);
}
Now in the application, I would instantiate class X several times,
however, in separate threads. I don't have control over the thread
creation mechanism, nor the instantiation of the classes. I only have
control over the class I create.
However, as an example, I can do the following:
void* thread1(void* arg)
{
X x1("script1");
x1.Run();
}
void* thread2(void* arg)
{
X x2("script1");
x2.Run():
}
int main(void)
{
pthread_t thr1, thr2;
pthread_create(&thr1, 0, thread1, 0);
pthread_create(&thr2, 0, thread2, 0);
pthread_join(thr1, 0);
pthread_join(thr2, 0);
}
Nearly all the languages I've looked at have some soft of global,
static storage that prevents this type of mechanism from working. For
example, Ruby has ruby_init(). Ignoring issues of callback (language
extensions that call class methods), is there a language that
guarantees that x1's interpreter and x2's interpreter will not collide
in threads?
This is why I'm leaning towards Lua. It seems well behaved in this
respect.
Thanks,
Pete
Ken Rose
Thant Tessman
[...]
> Nearly all the languages I've looked at have some soft of global,
> static storage that prevents this type of mechanism from working. For
> example, Ruby has ruby_init(). Ignoring issues of callback (language
> extensions that call class methods), is there a language that
> guarantees that x1's interpreter and x2's interpreter will not collide
> in threads?
See Chez Scheme's notion of 'waiters' and 'cafes'. Also see
"call-with-current-continuation." It's a little hard to get your head
around at first, but with it you can build whatever kind of threading
suits your application. Really powerful stuff.
-thant
--
"When fascism comes to America, it will be wrapped in the flag and
carrying the cross." -- Sinclair Lewis, "It Can't Happen Here" (1935)
Jean-Christophe Le Lann
> I've looked at Lua, Python, Pike, Ici, Eic, Ch, Cint, S-Lang, Small,
> elastiC, Simkin, and even some small, homegrown projects. However,
> they all have one problem or another. I really like Ch, but it is
> quite expensive (~$2k).
As far as I know, there are 2 distinct versions of Ch (standard and
professional). Standard edition seems to be free for both
non-commercial and commercial applications. Let me know if I am wrong.
I don't know if it is reentrant, however.
Jean-Christophe Le Lann
George Neuner
wrote:
>PlayDough wrote:
>
>... is there a language that guarantees that x1's interpreter
>and x2's interpreter will not collide in threads?
What you want is a sandbox.
I second Thant Tessman 's recommendation for a Scheme based solution.
Using syntax macros or input parsers you can give your users just
about any input syntax you wish. Scheme limits code to interaction
with a particular execution environment and R5RS implementations have
built-in support for multiple environments. By controlling the input
syntax, you can also restrict the ways in which users are allowed to
extend their environment.
There are several embeddable Schemes. Not all are based on the R5RS
spec (necessary if you wanted to roll your own sandboxing), but every
implementation I have seen has some kind of extension for multiple
environments even if it isn't based on R5RS.
George
PlayDough
> On 15 Jan 2005 20:57:49 -0500, "PlayDough" <pet...@gocougs.wsu.edu>
> wrote:
>
> >PlayDough wrote:
> >
> >... is there a language that guarantees that x1's interpreter
> >and x2's interpreter will not collide in threads?
>
> What you want is a sandbox.
>
> I second Thant Tessman 's recommendation for a Scheme based solution.
> Using syntax macros or input parsers you can give your users just
> about any input syntax you wish. Scheme limits code to interaction
You mention syntax macros and input parsers. You seem to imply that
it is possible to create my own syntax, and have it "converted" or
"interpreted" by scheme. Is that correct?
Could you expand on this a bit? Or point me to some good references?
I've digged around a bit and found things such as "define-syntax name
transformer-spec" and "let-syntax ((name transformer-spec) ...) body".
Is this along the lines you are speaking of?
> There are several embeddable Schemes. Not all are based on the R5RS
> spec (necessary if you wanted to roll your own sandboxing), but every
> implementation I have seen has some kind of extension for multiple
> environments even if it isn't based on R5RS.
I'll look more into them. Thanks for the pointer.
Pete
PlayDough
> Le Wed, 12 Jan 2005 23:00:41 -0500, PlayDough a écrit :
>
> > I've looked at Lua, Python, Pike, Ici, Eic, Ch, Cint, S-Lang, Small,
> > elastiC, Simkin, and even some small, homegrown projects. However,
> > they all have one problem or another. I really like Ch, but it is
> > quite expensive (~$2k).
>
> As far as I know, there are 2 distinct versions of Ch (standard and
> professional). Standard edition seems to be free for both
> non-commercial and commercial applications. Let me know if I am wrong.
There are two versions of the Ch environment. However, we needed
Embedded Ch, and that is not free. Nor is the Ch SDK we would need.
> I don't know if it is reentrant, however.
Also it isn't (yet) reentrant. I've chatted with their sales people
and they have confirmed with me that the current version isn't yet, but
they intend on it eventually being.
For our use however, the cost is currently prohibitive.
Thanks,
Pete
Lex Spoon
> On 15 Jan 2005 20:57:49 -0500, "PlayDough" <pet...@gocougs.wsu.edu>
> wrote:
>
> >PlayDough wrote:
> >
> >... is there a language that guarantees that x1's interpreter
> >and x2's interpreter will not collide in threads?
>
> What you want is a sandbox.
This kind of thing is exactly what the E language was designed for.
The language has some similarity to Scheme. It's a small orthogonal
language, using let a lot, and it has a rich list syntax which is akin
to Scheme's backquote syntax for making s-expressions.
E supports independent threads via "vats". To create a new
independent thread, you simply create a new vat. Code inside one vat,
can create a new vat if it wants and interact with it. Each object in
the system belongs to exactly one vat. Objects in one vat only
interact with objects in another vat by passing messages, in the
network sense; OO messages are simulated on top of this by using
"eventual sends" and "promisses". Objects within one vat can interact
in the call-and-return semantics of most OO languages.
These guys have thought carefully about what it means to create a vat.
A vat starts out *completely* independent of the rest of the world,
and in fact can do nothing useful. It can't create a file, open a
window, or veen talk to any other vat that already exists. In order
for a vat to communicate with the outside world, someone outside the
vat must pass in some object references for the vat to talk to.
-Lex
George Neuner
wrote:
>You mention syntax macros and input parsers. You seem to imply that
>it is possible to create my own syntax, and have it "converted" or
>"interpreted" by scheme. Is that correct?
That's exactly right. Scheme programs can generate and execute code
on the fly. Some implementations can only interpret such code, so if
the code is going to reused a lot or execution speed is important you
need to choose a version that can also compile on the fly.
>Could you expand on this a bit? Or point me to some good references?
>I've digged around a bit and found things such as "define-syntax name
>transformer-spec" and "let-syntax ((name transformer-spec) ...)
>body". Is this along the lines you are speaking of?
You're on the right track.
The syntax "transformers" are used to extend the language with custom
keywords and control constructs. They don't change the look however,
their input and ouput are both still Lispy.
As you typically want to control user interaction with the scripting
engine, the easiest route is simply to embed a parser for the input
syntax into your application. There are a number of good parsers
available for Scheme including general regex, LL and LR (lex/yacc
workalikes). Your program can take user input in whatever form it
likes, generate Scheme code and directly execute the results.
If you are very familar with Scheme (or Lisp), many implementations
allow customization of the system "reader", which is the standard
input facility. By manipulating the reader, you can input code
directly to Scheme using your favorite syntax rather than the normal
Lisp syntax. Typically this is used to created internal APIs or to
allow programming in domain specific terminology - AFAICT it isn't
something generally offered directly to users.
Reader customization is implementation dependent in Scheme. It's a
feature borrowed from Common Lisp, and most/all high quality Schemes
offer a way to do it ... but there is no standard for it yet. Most
are similar to CL but not all offer all the options. If you think you
want to be able to do this, you need to find out what your Scheme
allows and how exactly to do it.
As for Scheme references, you can check out comp.lang.scheme and visit
www.schemers.org. Schemers maintains a good list of available
versions, libraries and reference materials. As for implementations,
I would start by looking at some of these:
Chez - http://www.scheme.com/
Chez Scheme is a commercial native compiler version. There is also a
freely available threaded interpreter version called Petite Chez which
is fully compatible with the commercial version but does not include
the compiler.
PLT - http://www.plt-scheme.org/
PLT (aka DrScheme) is a free, mostly open source university teaching
version which was built around a core known as mzScheme. mzScheme is
a byte code compiler, is commercial grade, separable from PLT and
fully embeddable.
Guile - http://www.gnu.org/software/guile/
Elk - http://sam.zoy.org/projects/elk/
TinyScheme - http://tinyscheme.sourceforge.net/home.html
These are library implementations designed for scripting and
embedding. Guile and Elk are both well known embeddable versions,
TinyScheme is a well regarded experimental version.
Chez and PLT are both up to date and have extensive library support.
They are also most compatible with 3rd party code libraries you may
find. Scheme has no standard module facility so foreign libraries
usually need some spit and polish to run.
The other versions are lighter weight (by design) and don't come with
extensive libraries. They also tend to be a behind state of the art
and thus will have more gotcha's to porting code. However, they *are*
lighter weight and that may be important to your application.
Also I should mention that Common Lisp can do all of this as well and
you can pick up a lot of useful information on these techniques in
comp.lang.lisp. I didn't mention CL before it is not typically used
in an embedded setting (I am only aware of 2 embeddable versions) and
is much more heavyweight than Scheme. However, as a reference source,
there's a lot of mindshare common to both Scheme and CL and though
they are different languages, there is quite a bit of crossover in
applicable programming technique.
Hope some of this helps.
George