Python package has missing symbols

[Pepijn de Vos]

Hey all,

I'm exploring serialization libraries, so I installed the Python lib and got the following error

ImportError: [...]capnp.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN2kj1_24TransformPromiseNodeBase7onReadyERNS0_5EventE

Which is definitely a thing, and has been for two years: https://github.com/capnproto/capnproto/blame/master/c%2B%2B/src/kj/async-inl.h#L392

It also seems to be linked correctly

$ ldd [...]capnp.cpython-38-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffe675cf000)
    libcapnpc-0.7.0.so => /usr/lib/libcapnpc-0.7.0.so (0x00007f7dc5173000)
    libcapnp-rpc-0.7.0.so => /usr/lib/libcapnp-rpc-0.7.0.so (0x00007f7dc5090000)
    libcapnp-0.7.0.so => /usr/lib/libcapnp-0.7.0.so (0x00007f7dc4ff4000)
    libkj-async-0.7.0.so => /usr/lib/libkj-async-0.7.0.so (0x00007f7dc4f60000)
    libkj-0.7.0.so => /usr/lib/libkj-0.7.0.so (0x00007f7dc4eda000)
    libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f7dc4cf0000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f7dc4cd4000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f7dc4b0d000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f7dc4aeb000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f7dc49a5000)
    /usr/lib64/ld-linux-x86-64.so.2 (0x00007f7dc540e000)

So the system library version is 0.7, and the latest Python library is 0.6.4, not sure if the versions are just mismatched somehow.

I tried installing from git with the same result, is the Python library just outdated?

Or maybe the Arch package is just broken, because it doesn't contain the required symbol?

Cheers,

Pepijn

[Pepijn de Vos]

Actually, it appears that the library contains a pointer, whereas the Python lib tries to use a reference.

I'm not sure where this difference came from, but it's clearly incorrect.

(env) [apicula]$ nm -gD /usr/lib/libkj-async-0.7.0.so | grep TransformPromiseNodeBase
000000000002b1e0 T _ZN2kj1_24TransformPromiseNodeBase7onReadyEPNS0_5EventE
(env) [apicula]$ c++filt _ZN2kj1_24TransformPromiseNodeBase7onReadyEPNS0_5EventE
kj::_::TransformPromiseNodeBase::onReady(kj::_::Event*)
(env) [apicula]$ c++filt _ZN2kj1_24TransformPromiseNodeBase7onReadyERNS0_5EventE
kj::_::TransformPromiseNodeBase::onReady(kj::_::Event&)

[Jacob Alexander]

While I haven't gotten all the tests working yet (there are some issues with some of the timer functions on Windows it seems), I spent a bunch of time getting asyncio working with pycapnp (it was the most reasonable way to get native Python TLS support working).

I've fixed a lot of bugs, removed a lot of the deprecated C++ functions (the warnings were hiding a lot of serious issues).

https://github.com/haata/pycapnp-async

I'll be using this for a cross-platform tool I've been working so I'll at least be maintaining it for basic client functionality (server stuff works as well, minus a few Windows tests). My main use case is a TLS connection between a Rust server and Python clients (this is currently working using tokio-rustls).

It does require Python 3.7 and higher (3.8 also works). I discovered some bugs in asyncio that make porting difficult to 3.5 and 3.6 (though it's likely possible with a bunch of effort).

-HaaTa

[pepijn de vos]

Thanks for the link.

While I'm not actually interested in RPC or async at all, this fork actually works.

Any chance this will be upstreamed?

Pepijn

--
You received this message because you are subscribed to a topic in the Google Groups "Cap'n Proto" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/capnproto/1x8_5_RC9wU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/f8decb16-f304-4d80-9e2c-7df4946df458%40googlegroups.com.

[Jacob Alexander]

I'm happy to upstream (though I've taken some more extreme decisions in some areas in regards to prior compatibility).

Unfortunately, I don't believe there's an active maintainer for the pycapnp github repo so it's currently a bit futile. :(

To unsubscribe from this group and all its topics, send an email to capn...@googlegroups.com.

[Kenton Varda]

Hi Pepijn,

I suspect your problem is because somehow the Python module was built against headers from a different version of Cap'n Proto compared to the .so file that it's being linked against. I seem to recall there being some hackery in the Python implementation where it might try to use a bundled version of Cap'n Proto rather than the system-installed version, but I don't remember the details. Maybe this led to a mismatch.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/304b0818-aeef-48b7-87d8-ab028fd76bdb%40googlegroups.com.

[Kenton Varda]

On Fri, Nov 29, 2019 at 1:28 AM Jacob Alexander <tripl...@gmail.com> wrote:

I'm happy to upstream (though I've taken some more extreme decisions in some areas in regards to prior compatibility).

Unfortunately, I don't believe there's an active maintainer for the pycapnp github repo so it's currently a bit futile. :(

A few months ago, Colin Jermain (https://github.com/cjermain; I don't seem to have his e-mail) offered to help with pycapnp maintenance, so I gave him commit rights. It looks like he did some work but hasn't touched it in a few months, so I'm not sure if he intends to keep working on it. If someone else wants to take over maintainership I'm happy to help get the right permissions set up.

-Kenton

 

You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/812fa7a9-0bf3-4eb4-b483-e20f8ebb33c0%40googlegroups.com.

[ja...@datrium.com]

I don't mind helping maintain pycapnp. It will make it easier for me if I can pull over most/all of my current changes.

Some things that I've done

• Removed most of the references to deprecated capnproto functions (there are a few left that are a bit tricky that have to do with the .capnp file parser imports)
• Added Windows support (there seems to be a bug related to the capnproto timer that's erroring out the last few tests)
• asyncio support for Python (this enables TLS support for both client and servers)
• Python 3.7+ support (earlier versions don't work well with asyncio and will take a bunch of work to make work)
• Updated the minimum version to capnproto 0.7.0
• Moved from Travis to Github Actions
• No more git flow
• Updated to C++14 (fixes lots of issues, including those that used to be there with macOS)

This is my github https://github.com/haata

I don't have many more changes planned. Well, beyond fixing new issues that pop-up, adding support for new Python/OS options and the last few Windows test errors. I also want to cleanup the documentation a bit and add a proper changelog.

I'm also not sure how the package uploads work for https://pypi.org/project/pycapnp/ and user accounts there. I have my forked version here: https://pypi.org/project/pycapnp-async/

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/812fa7a9-0bf3-4eb4-b483-e20f8ebb33c0%40googlegroups.com.

[Kenton Varda]

Hi Jacob,

Sorry for the delay in replying.

This is awesome work!

I'd love to let you take over the official pycapnp repository and merge this back in. I think it's totally fine if you've made breaking API changes in the name of cleaning things up. Depending on the extent of the changes it might make sense to bump the major version number before the next release. The Go implementation is on version 3 now so there's plenty of precedent for that.

I've filed an issue on pycapnp to propose making you a maintainer, to give the current owners a chance to weigh in: https://github.com/capnproto/pycapnp/issues/194

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/49839775-0604-4653-bda9-cb2c5d5f9d41%40googlegroups.com.

[Jacob Alexander]

Cool!

Yeah, I agree with bumping the version.

One of the things I want to figure out is how to properly push binary linux pypi packages to pypi.org. Basically glibc compatibility for Python is a nightmare so there's a need to use a glibc as old as possible (but also not run into issues with distros actually requiring a minimum version of glibc for some things; I've been having issues with glibc 2.12.2 and some newer distros for example). I vaguely recall there are some

Fortunately macOS and Windows are easy in this regard.

-Jacob

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/49839775-0604-4653-bda9-cb2c5d5f9d41%40googlegroups.com.

[Asheesh Laroia]

Hi Jacob and all,

From what I've seen, there are some Docker-based build environments that are designed to overcome the glibc version binary compatibility issues you mentioned.

I'm semi offline right now, so I can't find references, but the thing I'd search for is [manylinux1 docker build].

Docker isn't essential, but it serves as a convenient way to distribute enough of an operating system to do these builds.

I'm enthusiastic about helping you figure this out, so I'll watch this email thread. Also, if you want to remote pair program on it, I'm game the week of December 23, or any week in January.

There's also a newer manylinux2010 build target if you need it.

If you want to avoid a manual build step for you, I'd also be happy to help you set up GitHub Actions, Travis CI, or some other automated system for publishing those wheels (binary packages) to PyPI.

Cheers,

Asheesh.

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/8ef1d626-d327-49aa-a666-f6d56e795884%40googlegroups.com.

[Kenton Varda]

I've now added Jacob as a maintainer of pycapnp.

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/8ef1d626-d327-49aa-a666-f6d56e795884%40googlegroups.com.

[Jacob Alexander]

Thanks! I'll start integrating changes in the next couple of days.

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/8ef1d626-d327-49aa-a666-f6d56e795884%40googlegroups.com.

[Jacob Alexander]

Hi Asheesh,

The manylinux targets look pretty interesting, docker isn't definitely the most straight-forward way of handling glibc versions. Publishing to PyPi with GitHub Actions does sound quite useful as well.

Let's sync up around December 23rd to see where things are at.

-Jacob

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/8ef1d626-d327-49aa-a666-f6d56e795884%40googlegroups.com.

[Jacob Alexander]

Hi Asheesh,

I've set up a GitHub issue to track progress: https://github.com/capnproto/pycapnp/issues/197

-Jacob