[Python 3 Runtime] Release 4.10 breaks support for any Python >= 3.5

[Derzart Ampere]

It seems like in the latest 4.10 release, the package info is for Python2 not Python3.

And this breaks version check for any python version >= 3.5.

Was this a mistake made when releasing 4.10 for Python 3?

Thank you so much!

[Derzart Ampere]

To provide more context, I'm using Python 3.8. The package would install fine.

But when I run my program. It would raise an error:

```

Traceback (most recent call last):
  File "./solver", line 11, in <module>
    from solving import DPLL, text_to_cnf_rep, bnf_to_cnf_rep, cnf_rep_to_text
  File "***/solving/__init__.py", line 1, in <module>
    from .bnf import *
  File "***/solving/bnf.py", line 3, in <module>
    from .bnf_parser import bnf_text_to_trees, BNFParserError
  File "***/solving/bnf_parser/__init__.py", line 2, in <module>
    from .bnf_tree import *
  File "***/solving/bnf_parser/bnf_tree.py", line 6, in <module>
    from .BNFLexer import BNFLexer
  File "***/solving/bnf_parser/BNFLexer.py", line 37, in <module>
    class BNFLexer(Lexer):
  File "***/solving/bnf_parser/BNFLexer.py", line 39, in BNFLexer
    atn = ATNDeserializer().deserialize(serializedATN())
  File "***/lib/python3.8/site-packages/antlr4/atn/ATNDeserializer.py", line 28, in deserialize
    self.checkVersion()
  File "***/lib/python3.8/site-packages/antlr4/atn/ATNDeserializer.py", line 50, in checkVersion
    raise Exception("Could not deserialize ATN with version " + str(version) + " (expected " + str(SERIALIZED_VERSION) + ").")
Exception: Could not deserialize ATN with version  (expected 4).

```

[Terence Parr]

wow. Sorry about that. I'm sure I messed up somehow... I was doing a huge amount of cutting and pasting. Yep, fixed and pushed to dev, master.

But, the description should not change anything right?  Oh, damn. I'm not sure why the typing thing says < 3.5. Hmm.. That's commit looks like it what's the one that added that:

https://github.com/antlr/antlr4/commit/2a3530f513346f497fae13d603d162d1bc829c32

Open to suggestions about what that should be!

Ter

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/antlr-discussion/04424aa4-0981-4a67-8b7d-c0897558a643n%40googlegroups.com.

--

Dictation in use. Please excuse homophones, malapropisms, and nonsense.

[Terence Parr]

btw, you need to regenerate your parts are from the grammar.

Ter

On Mon, Apr 18, 2022 at 10:11 AM Derzart Ampere <der...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/antlr-discussion/04424aa4-0981-4a67-8b7d-c0897558a643n%40googlegroups.com.

[the-moog]

Similar problem here.

As per instructions on GITHUB https://github.com/antlr/antlr4, I am posting here first ask to raise the issue.

I am using Python 3.9.  Wtih ANTLR 4.10

There are two issues in generated code.

(Note PyPi only has 4.10, not 4.10.1)

The issue is with other code, but to validate this issue I used the 01-Hello example.

Issue 1:

The import line for the HelloParser and HelloLexer is incorrect.

The line

from typing.io import TextIO

Fix: should read

from typing import TextIO

Issue 2:

The generated code puts a stack of binary data in a StringIO buffer, wrapped in a function serializedATN()

I wonder why it's not just a UTF-8 string  (e.g.  "".join(lines)). Though a StringIO buffer is used it's accessed as an array rather than a file so gains nothing

def serializedATN():
with StringIO() as buf:
  buf.write("\3\u608b\ua72a\u8133\ub9ed\u417c\u3be7\u7786\u5964\3\5")
  buf.write("\b\4\2\t\2\3\2\3\2\3\2\3\2\2\2\3\2\2\2\2\6\2\4\3\2\2\2")
  buf.write("\4\5\7\3\2\2\5\6\7\4\2\2\6\3\3\2\2\2\2")
  return buf.getvalue()

could read

def serializedATN():

  buf = (

   "\3\u608b\ua72a\u8133\ub9ed\u417c\u3be7\u7786\u5964\3\5",
   "\b\4\2\t\2\3\2\3\2\3\2\3\2\2\2\3\2\2\2\2\6\2\4\3\2\2\2",
    "\4\5\7\3\2\2\5\6\7\4\2\2\6\3\3\2\2\2\2")

  return "".join(buf)

As this is a string buffer object, the later getvalue() method returns a string, and this is passed to the ATNDeserializer().deserialize(data) method.

This method (like the java basis) expects an array of integers.  Hence it fails to parse.

Fix:

return tuple(ord(x) for x in buf.getvalue())

or even better

return tuple(ord(x) for x in "".join(buf))

According to the docs, the Java library has overloaded functions for string and int, but Python does not support overloaded functions.

Should the deserialize use duck typing and fix it internally?

Issue 3:

Even if I patch the generated code it does not work.  "Exception: Could not deserialize ATN with version 3 (expected 4)."

(You can see above that it IS 3)

The generated code puts the 1st byte with value 3 - that surely makes this a ANTLR3 ATN structure rather than ANTLR4?

Am I calling ANTLR wrong.  The docs say nothing about .  Why is it generating version 3 ATN?

Issue 4:

I've not actually seen this yet, but surely if the character " (double quote) happens to appear in the binary data then it will break, causing a syntax error.

Or is that handled by encoding any dodgy characters as \xxx?

Issue 5:

The type hint for the data parameter to the deserialize method in ATNDeserializer.py is wrong.

Fix:

It is currently an 'int' but it should be Iterable[int]  (from typing import Iterable)

How to proceed?

I can fix the python each time I build but how to correct the code generator to save having to do that?

Is there going to be a new (corrected) PyPi package any time soon?

[Terence Parr]

hiya. That's serialized ATN looks like pre 4.10. Are you sure you're rebuilding your granmmars and so on?
T

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/antlr-discussion/a17ecdf7-f47a-4127-a471-5758f24b3eb1n%40googlegroups.com.