Asciidoc version 8.6.6 launches errors when it is called from JYTHON (org.python.util.PythonInterpreter)

59 views
Skip to first unread message

Jorge Marín G

unread,
Feb 27, 2012, 4:05:17 AM2/27/12
to asciidoc
Hello , I used asciidoc version 8.5.3 to create HTML documents from
txt documents calling ascidocapi.py from java
(org.python.util.PythonInterpreter).

It worked very well until asciidoc version 8.6.5, but when I have
tried to use the newest version (8.6.6) an exception is launched.

I call the asciidocapi.py script in this way:

1- PythonInterpreter python = new PythonInterpreter();
2- python.exec("import imp");
3- python.exec("asciidocapi = imp.load_source('AsciiDocAPI', [PATH
where
is asciidocapi.py in my machine])");

4- python.exec("asciidoc = asciidocapi.AsciiDocAPI()");
5- python.exec("asciidoc.asciidoc.execute(''file.txt')");

When I did that, I got a file.HTML using previous asciidoc versions,
but now, with version 8.6.6 I get the next exception:

asciidoc: FAILED: malformed [titles] underlines entry

I think there is a BUG in this version, something has been changed and
now this version (8.6.6) is not compatible with JYTHON.

If anybody know how to fix this error, please, answer me.

Thanks.

Lex Trotman

unread,
Feb 27, 2012, 8:04:17 AM2/27/12
to asci...@googlegroups.com
On 27 February 2012 20:05, Jorge Marín G <jmg...@gmail.com> wrote:
> Hello , I used asciidoc version 8.5.3 to create HTML documents from
> txt documents calling ascidocapi.py from java
> (org.python.util.PythonInterpreter).
>
> It worked very well until asciidoc version 8.6.5, but when I have
> tried to use the newest version (8.6.6) an exception is launched.
>
> I call the asciidocapi.py script in this way:
>
> 1- PythonInterpreter python = new PythonInterpreter();
> 2- python.exec("import imp");
> 3- python.exec("asciidocapi = imp.load_source('AsciiDocAPI', [PATH
> where
> is asciidocapi.py in my machine])");
>
> 4- python.exec("asciidoc = asciidocapi.AsciiDocAPI()");
> 5- python.exec("asciidoc.asciidoc.execute(''file.txt')");
>
> When I did that, I got a file.HTML using previous asciidoc versions,
> but now, with version 8.6.6 I get the next exception:
>
> asciidoc: FAILED: malformed [titles] underlines entry

This looks like a normal error message due to an incorrect input.

Try running asciidoc with standard python on the command line with the
same input file and see if it gets the same error.

If it does its the input file, if it does not then possibly it is due
to some difference between Jython and CPython.

As far as I know Stuart does not have the resources to test asciidoc
on any Python implementations other than the standard one.

Cheers
Lex

Stuart Rackham

unread,
Feb 27, 2012, 4:58:53 PM2/27/12
to asci...@googlegroups.com

Have just done a basic tests against the distributed example article with Jython
2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_20

and come up with the same error:

$ jython asciidoc.py doc/article.txt


asciidoc: FAILED: malformed [titles] underlines entry

I poked around and *suspect* that the error was introduced with this changeset:

http://code.google.com/p/asciidoc/source/detail?r=66543730504e5dc07a56fcd89b238ef784cdeac4

which uses the Python compiler to remove eval()s.

I would like AsciiDoc to work with Jython so if anyone can figure out exactly
what's causing it and come up with at patch that would be great.


Cheers, Stuart

>
> Thanks.
>

Lex Trotman

unread,
Feb 27, 2012, 7:25:06 PM2/27/12
to asci...@googlegroups.com
[...]

> I poked around and *suspect* that the error was introduced with this
> changeset:
>
> http://code.google.com/p/asciidoc/source/detail?r=66543730504e5dc07a56fcd89b238ef784cdeac4
>
> which uses the Python compiler to remove eval()s.
>
> I would like AsciiDoc to work with Jython so if anyone can figure out
> exactly what's causing it and come up with at patch that would be great.
>

Hi Stuart,

Perhaps one solution would be to use the parser module instead of the
compiler module, this is messier and harder to use since you have to
refer to the detailed grammar, but is available for all Python 2
versions. I don't think the grammar for "safe" literal structures (ie
expr) has changed across Python versions so the result should be
portable.

literal_eval() would need to be re-written to use the output of the
parser module, but that should be fairly mechanical.

Cheers
Lex

>
> Cheers, Stuart
>
>
>>
>> Thanks.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "asciidoc" group.
> To post to this group, send email to asci...@googlegroups.com.
> To unsubscribe from this group, send email to
> asciidoc+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/asciidoc?hl=en.
>

Stuart Rackham

unread,
Feb 27, 2012, 8:47:00 PM2/27/12
to asci...@googlegroups.com

On 28/02/12 13:25, Lex Trotman wrote:
> [...]
>> I poked around and *suspect* that the error was introduced with this
>> changeset:
>>
>> http://code.google.com/p/asciidoc/source/detail?r=66543730504e5dc07a56fcd89b238ef784cdeac4
>>
>> which uses the Python compiler to remove eval()s.
>>
>> I would like AsciiDoc to work with Jython so if anyone can figure out
>> exactly what's causing it and come up with at patch that would be great.
>>
>
> Hi Stuart,
>
> Perhaps one solution would be to use the parser module instead of the
> compiler module, this is messier and harder to use since you have to
> refer to the detailed grammar, but is available for all Python 2
> versions. I don't think the grammar for "safe" literal structures (ie
> expr) has changed across Python versions so the result should be
> portable.
>
> literal_eval() would need to be re-written to use the output of the
> parser module, but that should be fairly mechanical.

Thanks Lex, a bit more digging and it turns out to be the Jython
compiler.parse() function in parse_to_list() e.g.

>>> compiler.parse("[" + '"==","--","~~","^^","++"' + "]", mode='eval')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/srackham/local/bin/jython2.5.2/Lib/compiler/transformer.py", line
55, in parse
return Transformer().parseexpr(buf)
File "/home/srackham/local/bin/jython2.5.2/Lib/compiler/transformer.py", line
134, in parseexpr
return self.transform(parser.expr(text))
NameError: global name 'parser' is not defined
>>>

A similar issue reported here http://bugs.jython.org/issue1417 comes with the
comment:

``Jython doesn't fully support it and never will, because a) it generates
.pyc bytecode that we don't use b) it heavily relies on the parser
module which we will never support either c) is pending deprecation and
was removed in Python 3

The parser module provides the low level innards of CPython's parser
(which is really nothing like ours), and the way to deal with AST going
forward is via the _ast module, which we support and was originally
added in CPython 2.5. We also support the ast (no underscore) module, as
well as the ability to pass modified/generated AST to the compile
function, which were both added in CPython 2.6''

The implication being that the parser module will suffer the same problems.

As far as I'm concerned the cure is worse than the disease and we should just
roll back
http://code.google.com/p/asciidoc/source/detail?r=66543730504e5dc07a56fcd89b238ef784cdeac4


Cheers, Stuart

Stuart Rackham

unread,
Feb 27, 2012, 9:02:12 PM2/27/12
to asci...@googlegroups.com
I missed the fact that the ast code is already there, just ensure Jython uses
it, this seems to work:

diff -r 2d79f221db30 asciidoc.py
--- a/asciidoc.py�������Tue Feb 28 13:14:28 2012 +1300
+++ b/asciidoc.py�������Tue Feb 28 14:59:58 2012 +1300
@@ -375,7 +375,7 @@

if float(sys.version[:3]) < 2.4:
pass # No compiler
-elif float(sys.version[:3]) < 2.6:
+elif float(sys.version[:3]) < 2.6 and sys.platform[:4] != 'java':
import compiler
from compiler.ast import Const, Dict, Expression, Name, Tuple, UnarySub,
Keyword


Cheers, Stuart

On 28/02/12 13:25, Lex Trotman wrote:

Stuart Rackham

unread,
Feb 27, 2012, 9:06:16 PM2/27/12
to asci...@googlegroups.com
Off topic: CPython is around 3x faster than Jython compiling the User Guide:

$ time jython asciidoc.py doc/asciidoc.txt

real 0m53.406s
user 1m2.240s
sys 0m7.132s

$ time asciidoc doc/asciidoc.txt

real 0m21.821s
user 0m20.469s
sys 0m1.168s

On 28/02/12 13:25, Lex Trotman wrote:

Lex Trotman

unread,
Feb 27, 2012, 9:07:48 PM2/27/12
to asci...@googlegroups.com
[...]

> The parser module provides the low level innards of CPython's parser
> (which is really nothing like ours), and the way to deal with AST going
> forward is via the _ast module, which we support and was originally
> added in CPython 2.5. We also support the ast (no underscore) module, as
> well as the ability to pass modified/generated AST to the compile
> function, which were both added in CPython 2.6''

Sigh, typical developer response, "here is our solution going forward,
oh by the way it means your code is incompatible with old versions"
</rant>

>
> The implication being that the parser module will suffer the same problems.

If they can create ast they can create parser, but anyway...they won't.

>
> As far as I'm concerned the cure is worse than the disease and we should
> just roll back
> http://code.google.com/p/asciidoc/source/detail?r=66543730504e5dc07a56fcd89b238ef784cdeac4
>

I'll admit I don't see it as a likely attack vector, but everybody is
security consious these days so total removal might be a bad look.

How does Jython report itself, I couldn't find anything in the docs on
what version it reports,?

Can it use the ast code in the <2.6 else clause, it appears to imply that?

Or can it be identified some other way so it can fall back to an
unsafe eval? Better than total removal of the security code.

Cheers
Lex

Lex Trotman

unread,
Feb 27, 2012, 9:08:35 PM2/27/12
to asci...@googlegroups.com
Our emails crossed :)

Cheers
Lex

Stuart Rackham

unread,
Feb 27, 2012, 9:24:49 PM2/27/12
to asci...@googlegroups.com
Don't we just waste an awful amount of time and energy because of a few bad
apples (cyber attacks, war on terror, war on drugs...), but I think I'm getting
off topic here :-)

Cheers, Stuart

Stuart Rackham

unread,
Feb 27, 2012, 10:35:27 PM2/27/12
to asci...@googlegroups.com
The following commit to the trunk should fix the regression, please test:

http://code.google.com/p/asciidoc/source/detail?r=701003befc60eefc2cfe9482af254d85f812d8a3


Cheers, Stuart


On 27/02/12 22:05, Jorge Mar�n G wrote:

Stuart Rackham

unread,
Feb 29, 2012, 7:10:05 PM2/29/12
to asci...@googlegroups.com

Jorge M G

unread,
Mar 7, 2012, 4:56:23 AM3/7/12
to asci...@googlegroups.com
Hello Stuart, sorry about my delayed response, but I have just tried
your modified files just replacing theses new files by the old files
of the 8.6.6 version and the result is OK, and without modifying
anything else.

Your fixed changes have been successful :)

I hope you collect this improvements in the next version of asciidoc ;)

Thank you very much Stuart.

Best regards.

2012/3/1 Stuart Rackham <srac...@gmail.com>:

> On 27/02/12 22:05, Jorge Marín G wrote:
>>
>> Hello , I used asciidoc version 8.5.3 to create HTML documents from
>> txt documents calling ascidocapi.py from java
>> (org.python.util.PythonInterpreter).
>>
>> It worked very well until asciidoc version 8.6.5, but when I have
>> tried to use the newest version (8.6.6) an exception is launched.
>>
>> I call the asciidocapi.py script in this way:
>>
>> 1- PythonInterpreter python = new PythonInterpreter();
>> 2- python.exec("import imp");
>> 3- python.exec("asciidocapi = imp.load_source('AsciiDocAPI', [PATH
>> where
>> is asciidocapi.py in my machine])");
>>
>> 4- python.exec("asciidoc = asciidocapi.AsciiDocAPI()");
>> 5- python.exec("asciidoc.asciidoc.execute(''file.txt')");
>>
>> When I did that, I got a file.HTML using previous asciidoc versions,
>> but now, with version 8.6.6 I get the next exception:
>>
>> asciidoc: FAILED: malformed [titles] underlines entry
>>
>> I think there is a BUG in this version, something has been changed and
>> now this version (8.6.6) is not compatible with JYTHON.
>>
>> If anybody know how to fix this error, please, answer me.
>>
>> Thanks.
>>
>

> --
> You received this message because you are subscribed to the Google Groups
> "asciidoc" group.
> To post to this group, send email to asci...@googlegroups.com.
> To unsubscribe from this group, send email to
> asciidoc+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/asciidoc?hl=en.
>

--
Ingeniero Superior de Telecomunicación  |  Ingeniero de Desarrollo Software Java

Móvil:  +34 668 873 780

Reply all
Reply to author
Forward
0 new messages