Announcing Ox release 1.10.1

[Tom Shields]

Ox version 1.10.1 is now available on SourceForge
(sourceforge.net/projects/ox-attribute-grammar-compiler/).

Ox generalizes the function of Yacc in the way that an attribute grammar
generalizes a context-free grammar. Ordinary Yacc and Lex specifications may
be augmented with definitions of synthesized and inherited attributes written
in C/C++ syntax. Ox checks these specifications for consistency and
completeness, and generates from them a program that builds and decorates
attributed parse trees. The user may specify post-decoration traversals for
easy ordering of side effects, such as code generation. Ox handles the
tedious and error-prone details of writing code for parse-tree management, so
its use eases problems of security and maintainability associated with that
aspect of translator development.


Ox NEWS

* Noteworthy changes in release 1.10.1 (1 Feb 2022)

** Internals

This release continues the upgrade of the Ox source from C99 to C++11.

The Ox internal 'stringTable []' implementation was changed to use the
'std::unordered_map' container. The generation of 'yyyStringTab [] from the
contents of 'stringTable []' was changed to eliminate empty entries.

The code generation architecture was restructured:
- The C 'typedef union target * targetT' pun (aka hack) was replaced with a
class hierarchy, with abstract base class 'outMgr'.
- The use of C global 'FILE *' variables and C standard library file
management functions was replaced by a 'std::iostream *' member in class
'outMgr' and the respective member functions for output.
- Classes derived from 'outMgr' unify output to files (using a 'std::fstream
*' member) and in-memory buffers (using a 'std::stringstream *' member).
- The generation of '#line' directives was simplified, replacing the 'targetT'
pun's set of 'lex* ()' and ‘gen* ()' text output functions with a set of
overloaded 'put ()' text output functions.
- Calls to 'put ()' are preceded by a call to 'lex ()' (for input file '#line'
directives) or 'gen ()' (for output file '#line' directives), as required.
- The deferred blank line ring buffer complexity was simplified to just count
blank lines submitted for output, but not yet written.
- Blank lines (up to a maximum) are actually written to the output when
switching between writing non-blank *as-is* input text and generated text,
with '#line' directives written, as required.
- Other code generation function name prefixes 'gen' and 'print' were replaced
with 'put' for unifority.

The lexer specifications are now processed by 'flex++':
- The use of C global 'FILE *' variables and C standard library file
management functions was replaced with 'std::ifstream' objects and the
respecitive member functions for input.
- The custom 'YY_INPUT ()' macro used with 'flex'-generated lexer source was
replaced with an overriden 'LexerInput ()' implementation to handle the
Windows "\r\n" EOL sequence.
- The custom input buffer stack management was replaced with the 'flex'
standard input buffer stack management.

The following additional changes were made to the lexer specifications:
- A number of individual pattern rules with the same action were combined into
a single pattern rule using the '|' pattern alternative operator, for
clarity.
- The use of 'unput ()' was replaced with 'yyless ()', for efficiency.
- A number of trailing context pattern rules were rewritten to eliminate the
trailing context, for efficiency.


** Bug fixes

Occurrences of unmatched ` and ' characters in // comments, which cause
'flex++' version 2.6.2 to run off the end of the specification file, were
either removed or paired to avoid the issue.

The error message string created for failure to erase a file was left in a
global 'std::stringstream' buffer if the erase operation succeeded.
Subsequent use of that global buffer for an error message contained the erase
failure message text.

The Y-file lexer was corrected to return the 'RBRACE' token for a '}'
character in the Yacc definition section (not associated with a "%<directive>
{ ... }" C code block, which would normally never occur); it was incorrectly
returning the 'LBRACE' token.

The tree visualization skeleton was missing "#include <cassert>" (for a C++
compile) or "#include <assert.h>" (for a C compile).

If Ox is configured for a single target parser (e.g., with the option
"--enable-target=bison"), only that corresponding target parser header file
(e.g., '#include "agc-bison.hh"') is now processed when compiling
'./source/main.cc'.

A typographical error in an attribute evaluation expression (a + operator in
the expression should have been a - operator) was corrected in the '*calc*'
test cases.

The test suite cleanup configuration where the 'bison' version is < 3.5.4 was
corrected.

The './configure' script check for the presence of the C++ header
'FlexLexer.h' in the MinGW environment was corrected to look in the
'/usr/include' system directory. If 'flex++' is used in the MinGW
environment, the 'AM_CPPFLAGS' Automake variable is augmented with "-idirafter
/usr/include" so the MinGW C++ compilers can find 'FlexLexer.h' at compile
time.

The './configure' script was corrected to extract the 'flex' version number
regardless of the executable's name.


** Build system

The C++ header 'FlexLexer.h' from the 'flex' distribution used to generated
the distributed lexer C++ source was added to the './source' distribution
directory. The 'FlexLexer.h' file installed in other environments may not
match the distributed source, causing C++ compile failure.

The './configure' script check for C++ compiler support for the
"-std=gnu++11|14|17" option was changed to "-std=c++11|14|17".

The Automake 'BUILT_SOURCES' variable is now only set if Ox is configured with
the "--enable-maintainer-mode" option.

The lexer generation Make rule is now only enabled if Ox is configured with
the "--enable-maintainer-mode" option.

[Tom Shields]

Howdy, Roger, responses to your questions below:
> On Feb 7, 2022, at 1:21 PM, Roger L Costello <cost...@mitre.org> wrote:
>
> Hello Tom,
>
> I have a few questions about Ox:
>
> 1. Is there a complete, up-to-date, easy-to-read Ox manual?

Kurt Bischoff, the original (primary?) Ox developer at Iowa State University,
wrote a tutorial (dtd 5 Nov 1993) and a reference manual (dtd 14 Nov 1993).

I have attempted to keep both documents current as I’ve expanded on the
original Ox functionality - the reference manual more extensively than the
tutorial. Both documents are in the ‘./docs’ distribution directory, in
both PDF and in LaTeX source form.

I believe that both documents are current, although it is certainly possible
that I’ve missed something. To date, this is a one-man labor of love ;-)

As to whether the documents are “easy-to-read”, you will have to be the
judge of that.

I suspect that the reference manual might be considered a bit cryptic if one
isn’t already familiar with attribute grammars. Furthermore, the
limitations of the current Ox translator (in particular, allowing arbitrary
C/C++ code in an attribute's definition), make it easy to create hidden
attribute dependencies that will randomly break the Ox-generated code.

I’m open to constructive comments from users, but so far no one has said
anything to me directly.


> 2. Are there any published books on Ox?

No … do you want to write one?

The original Ox distribution includes an example compiler for a small
programming language, GPPL, built using Ox (and originally Yacc and Lex). The
compiler generates C source code. I have maintained the GPPL compiler, but
only to the extent required to enable it to compile, link & execute.

Kurt Bischoff wrote a report describing the compiler: "GPPL: A Small
Block-Structured Imperative Programming Language Implemented using Ox”, Iowa
State University TR#92-32, December, 1992. A Postscript file of that document
from Kurt’s last known Ox distribution is in the ‘./gppl’ distribution
directory.


> 3. Is there an Ox group on Stack Overflow?

Not that I am aware of.

I did a search just now on the Stack Overflow web and only found 2 questions
that were obviously about Ox, both from the same user in April & May of 2021.
I guess I should investigate how to set up a trigger to get notified of future
Ox questions.


> 4. Are there any commercial products that were developed using Ox?

Not to my knowledge. I don’t get much user feedback, save for the
infrequent bug report.

According to the SourceForge server, there have been a total of 472 downloads
of the various Ox distribution files since December 2018, which was when I
first registered the project. That tells me is there is a non-trivial level
of interest, so I’ve kept at it.

Regards,

Tom Shields