Bug#835271: python-protobuf returns " TypeError: Descriptors should not be created directly, but only retrieved from their parent" message

[Josue Ortega]

Package: python-protobuf
Version: 3.0.0-1
Severity: important

Dear Maintainer,

Currently I am getting the following error trying to use Python protobuf
generated code:

File "/usr/lib/python2.7/dist-packages/google/protobuf/descriptor.py", line 501, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors should not be created directly, but only retrieved from their parent.

At the beginning I thought it was a problem of the piece of the software that is
using the generated python protobuf code, after regenerating the protobuf code
the same error is returned even if I use proto2 or proto3 syntax.

Then I removed python-protobuf package and installed the PyPi version[1] and the
software worked like a charm.

If you want to reproduce this, you can checkout the following repository:
https://github.com/tgalal/python-axolotl.git
and run:
python2.7 -m unittest discover -v

Cheers

-- System Information:
Debian Release: stretch/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=es_GT.UTF-8, LC_CTYPE=es_GT.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages python-protobuf depends on:
ii libc6 2.23-4
ii libgcc1 1:6.2.0-1
ii libprotobuf10 3.0.0-1
ii libstdc++6 6.2.0-1
ii python 2.7.11-2
ii python-pkg-resources 25.2.0-1
ii python-six 1.10.0-3
pn python:any <none>

python-protobuf recommends no packages.

python-protobuf suggests no packages.

-- no debconf information

[Josue Ortega]

Oh, forgot to specify that I used the latest version in PyPi:
https://pypi.python.org/pypi/protobuf/3.0.0

--
Josue Ortega
«Happy Hacking»
http://josueortega.org

[Antonio Ospite]

Package: python-protobuf
Followup-For: Bug #835271

Dear Maintainer,

this bug affects also all the software using python-axolotl like for
instance gajim.

For now I worked around the issue by downgrading to python-protobuf_2.6.1-2
which is still available in the archives:
http://debian.sil.at/debian/pool/main/p/protobuf/

The reason why the OP does not see the issue in PyPy is in the code:
https://github.com/google/protobuf/blob/2fe0556c7abbe31c02147b9171397737a35bfe3b/python/google/protobuf/pyext/descriptor.cc#L94

These new checks in 3.0.x (mainly that the file ends with "_pb2.py") are
not executed when the module is compiled for PyPy.

The problem may as well be in python-axolotl, I see that in
/usr/lib/python2.7/dist-packages/axolotl/protocol/whisperprotos.py both
descriptor and descriptor_pb2 are imported, but apparently with protobuf
3.0.x only the second one is supposed to be used directly.

Any idea about how to proceed from here?
I think this should be fixed ASAP.

Ciao ciao,
Antonio

-- System Information:
Debian Release: stretch/sid
APT prefers unstable

APT policy: (900, 'unstable'), (500, 'unstable-debug')
Architecture: amd64 (x86_64)

Kernel: Linux 4.6.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=it_IT.utf8, LC_CTYPE=it_IT.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Init: systemd (via /run/systemd/system)

Versions of packages python-protobuf depends on:

ii libc6 2.23-5
ii libgcc1 1:6.2.0-1
ii libprotobuf9v5 2.6.1-2

ii libstdc++6 6.2.0-1
ii python 2.7.11-2
ii python-pkg-resources 25.2.0-1

pn python:any <none>

python-protobuf recommends no packages.

python-protobuf suggests no packages.

-- no debconf information

--
Antonio Ospite
http://ao2.it

A: Because it messes up the order in which people normally read text.
See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

[Antonio Ospite]

On Wed, 24 Aug 2016 13:29:01 +0200
Antonio Ospite <a...@ao2.it> wrote:

> Package: python-protobuf
> Followup-For: Bug #835271
>
> Dear Maintainer,
>
> this bug affects also all the software using python-axolotl like for
> instance gajim.
>
> For now I worked around the issue by downgrading to python-protobuf_2.6.1-2
> which is still available in the archives:
> http://debian.sil.at/debian/pool/main/p/protobuf/
>
> The reason why the OP does not see the issue in PyPy is in the code:
> https://github.com/google/protobuf/blob/2fe0556c7abbe31c02147b9171397737a35bfe3b/python/google/protobuf/pyext/descriptor.cc#L94
>
> These new checks in 3.0.x (mainly that the file ends with "_pb2.py") are
> not executed when the module is compiled for PyPy.
>
> The problem may as well be in python-axolotl, I see that in
> /usr/lib/python2.7/dist-packages/axolotl/protocol/whisperprotos.py both
> descriptor and descriptor_pb2 are imported, but apparently with protobuf
> 3.0.x only the second one is supposed to be used directly.
>
> Any idea about how to proceed from here?

OK, the issue is not directly about descriptor_pb2, but about the code
which uses protobuf in axolotl.

Basically newer protobuf version (when NOT using PyPy) check that the
generated protocol files still have the _pb2 suffix, in the case of
axolotl these are the files generated from axolotl/protobuf/*.proto,
namely axolotl/protocol/whisperprotos.py and
axolotl/state/storageprotos.py; renaming these to keep the _pb2
suffix fixes the issue.

See the attached patch, it's against
https://github.com/tgalal/python-axolotl.git and after applying it the
tests pass again.

I am going to submit a bug report to upstream python-axolotl as soon as
you confirm that the fix works for you too.

Thanks,
Antonio

[Antonio Ospite]

On Thu, 25 Aug 2016 10:10:02 +0200
Antonio Ospite <a...@ao2.it> wrote:

[...]

> Basically newer protobuf version (when NOT using PyPy) check that the
> generated protocol files still have the _pb2 suffix, in the case of
> axolotl these are the files generated from axolotl/protobuf/*.proto,
> namely axolotl/protocol/whisperprotos.py and
> axolotl/state/storageprotos.py; renaming these to keep the _pb2
> suffix fixes the issue.
>
> See the attached patch, it's against
> https://github.com/tgalal/python-axolotl.git and after applying it the
> tests pass again.
>
> I am going to submit a bug report to upstream python-axolotl as soon as
> you confirm that the fix works for you too.
>

Well, I went ahead and submitted the bug report upstream:
https://github.com/tgalal/python-axolotl/issues/14

Maybe the Debian bug can be reassigned to python-axolotl if you agree
with my analysis.

Ciao,

[Josue Ortega]

Antonio,

Thanks for your patch, I'll be including it the next python-axolotl revision :)

I have reassigned the bug to python-axolotl

Cheers!

---

[Antonio Ospite]

On Mon, 29 Aug 2016 20:32:15 -0600
Josue Ortega <jo...@debian.org> wrote:

> Antonio,
>
> Thanks for your patch, I'll be including it the next python-axolotl revision :)
>
> I have reassigned the bug to python-axolotl
>

Cool.

Maybe the current patch is not exactly the one that will land in the
upstream repository, but it's good enough to get things working gain.

If you are in contact with the upstream developers, can you remind them
of https://github.com/tgalal/python-axolotl/issues/14 ?

Or you could just comment on the report to keep it "hot" :)

Thanks,