thread-safe flex/bison
nicholas black
class, aside from being compilers-related; i posted this there only
because there may be some compilers god in your midst who knows this]
i'm using flex and bison to generate bytecode on the fly for a virtual
machine. i've integrated the call to yyparse() into my code, but there
are multiple threads which need to use the compiler. looking at the
code generated by flex and bison, it is utterly non-threadsafe (static
tables and global variables). i've wrapped my calls to yyparse() in a
mutex, which is saving me now, but this is causing unacceptable lossage.
(well, not a mutex, but linux-specific read/write semaphores, but that
doesn't matter.)
hand-coding a lexer and parser is not an acceptable solution: this is a
large language.
does anyone have other ideas? i'm looking at a postprocessor to modify
the generated code to pass an allocated context around, which shouldn't
be too hard, but it's kludgey and (imho) error-prone.
--
[nick black ][ a night you don't sleep means one less ]
[da...@angband.org ][ morning when you have to wake up. ]
[http://www.angband.org/~dank ][ -nick bronn ]
[head developer, trellis security][ ]
Xellos
>i'm using flex and bison to generate bytecode on the fly for a virtual
>machine. i've integrated the call to yyparse() into my code, but there
>are multiple threads which need to use the compiler. looking at the
>code generated by flex and bison, it is utterly non-threadsafe (static
>tables and global variables). i've wrapped my calls to yyparse() in a
>mutex, which is saving me now, but this is causing unacceptable lossage.
>does anyone have other ideas? i'm looking at a postprocessor to modify
>the generated code to pass an allocated context around, which shouldn't
>be too hard, but it's kludgey and (imho) error-prone.
I read this from the bison info pages:
...
Normally, Bison generates a parser which is not reentrant. This is
suitable for most uses, and it permits compatibility with YACC. (The
standard YACC interfaces are inherently nonreentrant, because they use
statically allocated variables for communication with `yylex',
including `yylval' and `yylloc'.)
Alternatively, you can generate a pure, reentrant parser. The Bison
declaration `%pure_parser' says that you want the parser to be
reentrant. It looks like this:
%pure_parser
...
And it goes on from there to describe the changes that occur when the
above is used...
Does this help?
--
Stevie Strickland - CS 2130 TA | sstr...@angband.org
Official Debian GNU/Linux Developer | http://angband.org/~sstrickl
"Ahhh... having Stevie around is better than manpages every day."
-- Dan Baisley
Brian McNamara!
>i'm using flex and bison to generate bytecode on the fly for a virtual
>machine. i've integrated the call to yyparse() into my code, but there
>are multiple threads which need to use the compiler. looking at the
>code generated by flex and bison, it is utterly non-threadsafe (static
>tables and global variables). i've wrapped my calls to yyparse() in a
>mutex, which is saving me now, but this is causing unacceptable lossage.
ISTR a bison which generates C++ rather than C. Check out, for example,
http://www.cs.cmu.edu/~AUIS/bison-a26.html
which seems to maybe do something similar to the postprocessor strategy
you suggest. flex has a "-+" option you might try.
--
Brian M. McNamara lor...@acm.org : I am a parsing fool!
** Reduce - Reuse - Recycle ** : (Where's my medication? ;) )
nicholas black
> declaration `%pure_parser' says that you want the parser to be
> reentrant. It looks like this:
> %pure_parser
> ...
> And it goes on from there to describe the changes that occur when the
> above is used...
no, pure_parser alone will not make this threadsafe. for one, the lexer
is still non-reentrant :). pure_parser primarily gets rid of the use
of global variables used by bison.
it doesn't matter; i've found another way to do it (hurrah for the Lexer
class).
--
[nick black ][ "i thought well as well him as another ]
[da...@angband.org ][ and then i asked him with my eyes to ask ]
[http://www.angband.org/~dank ][ again yes and then he asked me" ]
[head developer, trellis security][ -james joyce, ulysses ]
Xellos
>no, pure_parser alone will not make this threadsafe. for one, the lexer
>is still non-reentrant :). pure_parser primarily gets rid of the use
>of global variables used by bison.
Yeah, I didn't think about checking flex... was busy with a class
project :)
>it doesn't matter; i've found another way to do it (hurrah for the Lexer
>class).
Ah, great :)
Jason Lunz
>does anyone have other ideas? i'm looking at a postprocessor to modify
>the generated code to pass an allocated context around, which shouldn't
>be too hard, but it's kludgey and (imho) error-prone.
I haven't played with bison yet, but how much of the yylex() function
isn't thread-safe? It didn't seem from looking at it that it was more
than a handful of static variables. It might not be that hard to modify
flex and bison themselves to write versions of yylex/yyparse that can
use a private context to do their thing. Then you can run multiple
versions concurrently.
I made a postprocessor once to hack up a bunch of generated java code.
It sucked, I don't recommend it. Hack the generator.
Jason