Debian Wheezy: ZMQError: Invalid argument

[Antti Kaihola]

I just created this issue on GitHub: https://github.com/saltstack/salt/issues/383

and brought up the problem on the IRC channel. Jeff Schroeder asked me to write about it on this mailing list as well.

Basically I always get a

File "socket.pyx", line 489, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4797)

zmq.core.error.ZMQError: Invalid argument

when salt tries to bind on a TCP port. This is on Debian Wheezy with libzmq1 2.1.10-1 with both python-zmq 2.1.9-1 and pyzmq 2.1.11 from PyPI.

I wrote this test script to emulate what salt does:

import zmq context = zmq.Context() clients = context.socket(zmq.ROUTER) clients.bind('tcp://0.0.0.0:4506')

 

and got the same error:

$ python /tmp/test.py

Traceback (most recent call last):

  File "/tmp/test.py", line 4, in <module>

    clients.bind('tcp://0.0.0.0:1234')

  File "socket.pyx", line 459, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4552)

zmq.core.error.ZMQError: Invalid argument

[Antti Kaihola]

(the port in the test script was also 1234, of course)

[Jeff Schroeder]

jeff@desktopmonster:~$ python

Python 2.6.6 (r266:84292, Sep 15 2010, 16:22:56) 

[GCC 4.4.5] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import zmq

>>> context = zmq.Context()

>>> clients = context.socket(zmq.ROUTER)

>>> clients.bind('tcp://0.0.0.0:4506')

>>> zmq.__version__

'2.1.10'

>>> 

It seems to work fine for me. Can you post versions of some software you are using?
    - python

    - python-zmq

    - libzmq

    - linux distribution

[Antti Kaihola]

ii  libzmq1              2.1.10-1             ZeroMQ lightweight messaging kernel (shared library)

ii  python               2.7.2-9              interactive high-level object-oriented language (default

ii  python-zmq           2.1.9-1              Python bindings for 0MQ library

Distribution: Debian Wheezy http://www.debian.org/releases/wheezy/

Also tried removing python-zmq and installing pyzmq from PyPI, same results (see https://github.com/saltstack/salt/issues/383).

I'll test this on different Debian and Ubuntu versions next.

[Jeff Schroeder]

On Wed, Dec 21, 2011 at 11:36 PM, Antti Kaihola <akai...@gmail.com> wrote:
> ii  python-zmq           2.1.9-1              Python bindings for 0MQ library

There is the problem. You need python-zmq 2.1.10

--
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

[Antti Kaihola]

salt-master does work correctly in Debian Squeeze with libzmq1 from backports.

Installed Debian packages:

python-2.6.6-3+squeeze6

Installed packages as shown by pip freeze -l:

M2Crypto==0.20.1

PyYAML==3.09

pycrypto==2.1.0

pyzmq==2.1.11

salt==0.9.4

However, the installation instructions in the documentation didn't work. Below are the steps I needed to take to actually install Salt in Squeeze. I created a ticket about this: https://github.com/saltstack/salt/issues/385

sudo su
echo "deb http://backports.debian.org/debian-backports squeeze-backports main" >/etc/apt/sources.list.d/backports.list
apt-get update
apt-get install libzmq1 python-crypto python-m2crypto python-yaml python-pip
apt-get install build-essential python-dev libzmq-dev 
pip install pyzmq
pip install salt

[Antti Kaihola]

I did get the same error when pip-installing pyzmq from PyPI. I'll have to try once more and make sure the version installed and used was in fact 2.1.10 in that case.

[Thomas S Hatch]

On Sun, Dec 25, 2011 at 3:37 PM, Antti Kaihola <akai...@gmail.com> wrote:

I did get the same error when pip-installing pyzmq from PyPI. I'll have to try once more and make sure the version installed and used was in fact 2.1.10 in that case.

Generally this is an issue with pyzmq and zeromq, pyzmq is written in cython and when it compiles it should be the same version as the zeromq dev files that are installed, you probably have a pyzmq package that is not compiled against the same libzmq you are running. ZeroMQ is very picky in this regard.

Keep us in the loop, we need to get these types of issues ironed out!

- Thomas S Hatch

[Antti Kaihola]

Thanks for the advice! I retried from the beginning and here's how it went. I had these .deb packages installed:

$ dpkg -l "libzmq*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version                        Description
+++-==============================-==============================-============================================================================
ii  libzmq-dev                     2.1.10-1                       ZeroMQ lightweight messaging kernel (development libraries and header files)

ii  libzmq1                        2.1.10-1                       ZeroMQ lightweight messaging kernel (shared library)

I checked that I don't have pyzmq or salt lurking around:

$ python
Python 2.7.2+ (default, Dec  1 2011, 01:55:02)
[GCC 4.6.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
ImportError: No module named zmq
>>> import salt

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
ImportError: No module named salt

I then did a sudo pip install pyzmq==2.1.10, and saw it compile and install the 2.1.10 version of the C extension.

A sudo pip install salt downloaded and installed Salt 0.9.4.

sudo salt-master still throws the exception I pasted earlier.

So apparently it's impossible to get a compatible combination of libzmq, pyzmq and Salt installed on Debian Wheezy at this point using only Debian and PyPI packages currently available?

[Thomas S Hatch]

This is very odd indeed, someone else just posted problems with the backports packages and said that this issue wend away when zeromq was compiled from source.

I think that we need to get with the upstream zeromq debian maintainers and get these packages fixed!

[Corey Quinn]

...and get them to do python-pyzmq as well, while they're up.

-- Corey

[Jeff Schroeder]

On Thu, Dec 29, 2011 at 1:49 AM, Antti Kaihola <akai...@gmail.com> wrote:
> Thanks for the advice! I retried from the beginning and here's how it went.
> I had these .deb packages installed:
>
> $ dpkg -l "libzmq*"
> Desired=Unknown/Install/Remove/Purge/Hold
> |
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ Name                           Version
> Description
> +++-==============================-==============================-============================================================================
> ii  libzmq-dev                     2.1.10-1                       ZeroMQ
> lightweight messaging kernel (development libraries and header files)
>
> ii  libzmq1                        2.1.10-1                       ZeroMQ
> lightweight messaging kernel (shared library)
>
> I checked that I don't have pyzmq or salt lurking around:
>
> $ python
> Python 2.7.2+ (default, Dec  1 2011, 01:55:02)
> [GCC 4.6.2] on linux2
>
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import zmq
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named zmq

Just for kicks, can you do this and re-run these tests:
sudo python

As you are running salt as root and you might have a different
PYTHONPATH or something set. Also, could you paste the output of "env
| grep PYTHON" as a normal and root user to this email or a pastebin
somewhere just to verify your env looks ok.

>>>> import salt
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ImportError: No module named salt
>
> I then did a sudo pip install pyzmq==2.1.10, and saw it compile and install
> the 2.1.10 version of the C extension.

Can you import the zmq module and print the version of it along with
zmq.__file__ to make sure you are indeed using the right version?

> A sudo pip install salt downloaded and installed Salt 0.9.4.
>
> sudo salt-master still throws the exception I pasted earlier.

> So apparently it's impossible to get a compatible combination of libzmq,
> pyzmq and Salt installed on Debian Wheezy at this point using only Debian
> and PyPI packages currently available?

Can you try installing salt from git for kicks please?

$ pip install git+git://github.com/saltstack/salt.git@develop

You might find better luck doing this as a non-root user using a
virtualenv for testing.

[Antti Kaihola]

Jeff Schroeder:

>>>> import zmq

> ImportError: No module named zmq

Just for kicks, can you do this and re-run these tests:
sudo python

$ sudo pip uninstall pyzmq salt
[un-installs both successfully]
$ sudo python

Python 2.7.2+ (default, Dec  1 2011, 01:55:02)
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named zmq

 

Also, could you paste the output of "env | grep PYTHON" as a normal and root user

akaihola:~$ env|grep PYTHON
akaihola:~$ sudo env|grep PYTHON
akaihola:~$ sudo su
/home/akaihola# env|grep PYTHON
/home/akaihola# exit
akaihola:~$ sudo su -
~# env|grep PYTHON
~#
 

Can you try installing salt from git for kicks please?

$ pip install git+git://github.com/saltstack/salt.git@develop

You might find better luck doing this as a non-root user using a
virtualenv for testing.

akaihola:/tmp$ mkdir salttest;virtualenv --distribute salttest;. salttest/bin/activate
New python executable in salttest/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
akaihola:/tmp$ pip install pyzmq
[pyzmq 2.1.11 installs successfully]
akaihola@papu:/tmp$ pip install git+git://github.com/saltstack/salt.git@develop
[salt installs successfully]
akaihola:/tmp$ salt-master

Traceback (most recent call last):

  File "/tmp/salttest/bin/salt-master", line 17, in <module>
    main()
  File "/tmp/salttest/bin/salt-master", line 13, in main
    master = salt.Master()
  File "/tmp/salttest/local/lib/python2.7/site-packages/salt/__init__.py", line 42, in __init__
    self.opts = salt.config.master_config(self.cli['config'])
NameError: global name 'salt' is not defined

akaihola:/tmp$ sudo salttest/bin/salt-master
[same traceback as above]

[Thomas S Hatch]

I just ran into this, is python-crypto installed?

[Antti Kaihola]

The traceback above was caused by Python not being able to import the yaml library installed globally from the Debian package. This import error was masked by the try: import .. except: pass in salt/__init__.py (an issue in itself, maybe worth fixing?).

I ran the same test again, providing the additional --system-site-packages argument for virtualenv. The result:

(salttest)akaihola:/tmp$ sudo salttest/bin/salt-master
Missing configuration file:
08:38:28,747 [salt.master    ][WARNING ] Starting the Salt Master
Missing configuration file:
Missing configuration file:

Traceback (most recent call last):

  File "salttest/bin/salt-master", line 17, in <module>
    main()
  File "salttest/bin/salt-master", line 14, in main
    master.start()
  File "/tmp/salttest/local/lib/python2.7/site-packages/salt/__init__.py", line 104, in start
    master.start()
  File "/tmp/salttest/local/lib/python2.7/site-packages/salt/master.py", line 136, in start
    reqserv.run()
  File "/tmp/salttest/local/lib/python2.7/site-packages/salt/master.py", line 231, in run
    self.__bind()
  File "/tmp/salttest/local/lib/python2.7/site-packages/salt/master.py", line 204, in __bind
    self.clients.bind(self.uri)

  File "socket.pyx", line 489, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4797)
zmq.core.error.ZMQError: Invalid argument

Process Publisher-2:

Traceback (most recent call last):

  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/tmp/salttest/local/lib/python2.7/site-packages/salt/master.py", line 164, in run
    pub_sock.bind(pub_uri)

  File "socket.pyx", line 489, in zmq.core.socket.Socket.bind (zmq/core/socket.c:4797)

ZMQError: Invalid argument

I also tried doing pip install pyzmq==2.1.10, but still got the same traceback, except that the line number on the second last line is zmq/core/socket.c:4840.

[Antti Kaihola]

Yes, the python-crypto 2.4.1-1 Debian package is installed. I tried removing it and installing with pip install pycrypto instead, but still got the same traceback from salt-master.

[Thomas S Hatch]

Yes, it ties back to the fact that the trace should be picked up in that try/except block. That was put in to fix an issue with the build process and msgpack, I will try to get it out.

This is odd though, since I just installed on Debian Wheezy without incident and following the install doc. This zeromq error is strange. I assume I am missing it from higher up in the thread but I assume you have zeromq 2.1.10.

I will keep trying to reproduce this erorr, thanks for the info

[Antti Kaihola]

I created https://github.com/saltstack/salt/issues/411 about the import issue.

I tried removing the try..except, and python setup.py install still works correctly.

[Antti Kaihola]

I did an apt-get upgrade and a reboot on my Wheezy virtual machine since it was way out of date. I'm pinning the kernel and udev to older versions though because of Xen issues, but I can't imagine that could be the reason for this issue.

Unfortunately upgrading didn't solve the "Invalid argument" issue either. This is indeed strange if your Wheezy box works correctly. Are all its packages up-to-date as well?

[Thomas S Hatch]

On Tue, Jan 3, 2012 at 1:36 AM, Antti Kaihola <akai...@gmail.com> wrote:

I did an apt-get upgrade and a reboot on my Wheezy virtual machine since it was way out of date. I'm pinning the kernel and udev to older versions though because of Xen issues, but I can't imagine that could be the reason for this issue.

Unfortunately upgrading didn't solve the "Invalid argument" issue either. This is indeed strange if your Wheezy box works correctly. Are all its packages up-to-date as well?

Yes, mine is up to date, I just set it up. I installed everything except python-zmq from from apt, and then pip installed python-zmq and installed salt from git. I ran into no issues at all and everything is running smoothly.

How did you install zeromq again?

[Antti Kaihola]

How did you install zeromq again?

With apt-get from Wheezy repos. Here's what I have:

$ dpkg -s libzmq1

Package: libzmq1

Status: install ok installed

Priority: optional

Section: libs

Installed-Size: 382

Maintainer: Martin Lucina <ma...@kotelna.sk>

Architecture: i386

Source: zeromq

Version: 2.1.10-1

Depends: libc6 (>= 2.3.6-6~), libgcc1 (>= 1:4.1.1), libpgm-5.1-0 (>= 5.1.116~dfsg), libstdc++6 (>= 4.2.1), libuuid1 (>= 2.16)

Description: ZeroMQ lightweight messaging kernel (shared library)

 The 0MQ lightweight messaging kernel is a library which extends the

 standard socket interfaces with features traditionally provided by

 specialised messaging middleware products. 0MQ sockets provide an

 abstraction of asynchronous message queues, multiple messaging

 patterns, message filtering (subscriptions), seamless access to

 multiple transport protocols and more.

 .

 This package contains the ZeroMQ shared library.

Homepage: http://www.zeromq.org/

No pyzmq from Debian:

$ dpkg -l python-zmq

Desired=Unknown/Install/Remove/Purge/Hold

| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)

||/ Name                           Version                        Description

+++-==============================-==============================-============================================================================

un  python-zmq                     <none>                         (no description available)

But one installed from PyPI instead in a virtualenv:

$ pip freeze|grep zmq

pyzmq==2.1.10

And I have tried as many combinations as I could. For example, I replaced libzmq1 and libzmq-dev with the ones from squeeze-backports and recreated the virtualenv with pyzmq==2.1.10 from PyPI and Salt from the repository, but still got the same error.

Here are checksums from the Wheezy libzmq1 package:

$ dpkg -L libzmq1|xargs md5sum

md5sum: /.: Is a directory

md5sum: /usr: Is a directory

md5sum: /usr/lib: Is a directory

7e1a44c6890ad4cebb9c66ecfd4678fb  /usr/lib/libzmq.so.1.0.0

md5sum: /usr/share: Is a directory

md5sum: /usr/share/doc: Is a directory

md5sum: /usr/share/doc/libzmq1: Is a directory

7a38533a5b123517e55d70e6e2891fe2  /usr/share/doc/libzmq1/AUTHORS

6ab39bdff83a30e9dafb4798a0fc8658  /usr/share/doc/libzmq1/changelog.Debian.gz

c7d477894679add7c240ef596a8a2cc0  /usr/share/doc/libzmq1/copyright

4b45fc294500060de504003ebfdfe7b8  /usr/share/doc/libzmq1/README

f976b4ce6f6a68567b4591028ac2ec28  /usr/share/doc/libzmq1/README.Debian

1634ed45df0b45a13181296ca93f7044  /usr/share/doc/libzmq1/NEWS.gz

85ba1ceef133ec872583df39023ad6cc  /usr/share/doc/libzmq1/changelog.gz

md5sum: /usr/share/lintian: Is a directory

md5sum: /usr/share/lintian/overrides: Is a directory

f1089e96a843ca4d2c170009ed690377  /usr/share/lintian/overrides/libzmq1

7e1a44c6890ad4cebb9c66ecfd4678fb  /usr/lib/libzmq.so.1

And from libzmq-dev:

$ dpkg -L libzmq-dev|xargs md5sum

md5sum: /.: Is a directory

md5sum: /usr: Is a directory

md5sum: /usr/include: Is a directory

c65d1e777b9c23f3f189a913fd818c44  /usr/include/zmq_utils.h

1de86f8c93cbb40f8636b7a079cdd1c1  /usr/include/zmq.h

050d895c0778241e4308760663bd9252  /usr/include/zmq.hpp

md5sum: /usr/lib: Is a directory

md5sum: /usr/lib/pkgconfig: Is a directory

06e7381f879369e7f7c8ce869cd65d2c  /usr/lib/pkgconfig/libzmq.pc

6126d0ad368bfe45fed820802b3edefd  /usr/lib/libzmq.a

e3422dad66aa01be259465042f552ca2  /usr/lib/libzmq.la

md5sum: /usr/share: Is a directory

md5sum: /usr/share/man: Is a directory

md5sum: /usr/share/man/man3: Is a directory

42c1719b86f23e6f02149750cbbacd0e  /usr/share/man/man3/zmq_recv.3.gz

2fe9b25e783a861bcfd9f19f00ea12f2  /usr/share/man/man3/zmq_msg_close.3.gz

eec0f0cbe44ff8397697f2cd82df577c  /usr/share/man/man3/zmq_strerror.3.gz

ab821809fc12a682d4abfa8827625dd3  /usr/share/man/man3/zmq_term.3.gz

e0b6149cd4983627387a63d450a28b4b  /usr/share/man/man3/zmq_init.3.gz

71df543da4541d4edaac987779f6bd40  /usr/share/man/man3/zmq_msg_init_size.3.gz

7d264737b1eeaa752f668baa44303dc7  /usr/share/man/man3/zmq_bind.3.gz

6df8f4a88deaf8f7a8942c9fdc5111a4  /usr/share/man/man3/zmq_msg_move.3.gz

f07a8645c54dd899b23d985ba6309db0  /usr/share/man/man3/zmq_poll.3.gz

cde5c028fa3d5d96aa7c761b131c9374  /usr/share/man/man3/zmq_msg_data.3.gz

0087f1acd467ff01c24d59efd90d0480  /usr/share/man/man3/zmq_msg_init.3.gz

7df197d6947bd98d4ded263f28aee3d8  /usr/share/man/man3/zmq_errno.3.gz

381faaa26a48c90573bcfb7b7ab43203  /usr/share/man/man3/zmq_setsockopt.3.gz

ce4ab68d71c3e6b73c46c46c8a60e0ff  /usr/share/man/man3/zmq_version.3.gz

fd7cfdf213d3c9165670b881ab7c89bd  /usr/share/man/man3/zmq_close.3.gz

43a9f9c088834d189e8da4b7d16166ed  /usr/share/man/man3/zmq_msg_copy.3.gz

f2de32ab0b2c438093e2637c9e730645  /usr/share/man/man3/zmq_connect.3.gz

29128ed34d499ba1456b9eebbf937f83  /usr/share/man/man3/zmq_getsockopt.3.gz

1d79b5c0ab056c964f2a7d4bda6e8cc4  /usr/share/man/man3/zmq_socket.3.gz

d314886ed95c14918d684a4076665144  /usr/share/man/man3/zmq_msg_size.3.gz

d1d1a55426f8a35f3591c79c90c7151e  /usr/share/man/man3/zmq_send.3.gz

7817808197f77f22cced589397789f43  /usr/share/man/man3/zmq_msg_init_data.3.gz

983cf36ce4cfed095ba14ec15d9ce112  /usr/share/man/man3/zmq_device.3.gz

md5sum: /usr/share/man/man7: Is a directory

e0830fbe869941dcddf37f1b61abb1f5  /usr/share/man/man7/zmq_epgm.7.gz

33de05b7742c5a0c4b753e12afa88c3c  /usr/share/man/man7/zmq_tcp.7.gz

ba7b16738e7ead623b222b39e5e34090  /usr/share/man/man7/zmq.7.gz

739bcbe9c8ca5b1f0e7f4364502cef16  /usr/share/man/man7/zmq_ipc.7.gz

a12e652bdfb4910b32527257e158f082  /usr/share/man/man7/zmq_inproc.7.gz

e0830fbe869941dcddf37f1b61abb1f5  /usr/share/man/man7/zmq_pgm.7.gz

ef7461d1bf22193bfca14bf3672756d3  /usr/share/man/man7/zmq_cpp.7.gz

md5sum: /usr/share/doc: Is a directory

md5sum: /usr/share/doc/libzmq-dev: Is a directory

6ab39bdff83a30e9dafb4798a0fc8658  /usr/share/doc/libzmq-dev/changelog.Debian.gz

c7d477894679add7c240ef596a8a2cc0  /usr/share/doc/libzmq-dev/copyright

85ba1ceef133ec872583df39023ad6cc  /usr/share/doc/libzmq-dev/changelog.gz

md5sum: /usr/share/lintian: Is a directory

md5sum: /usr/share/lintian/overrides: Is a directory

36611f067acb799f30452c25766b9bc6  /usr/share/lintian/overrides/libzmq-dev

7e1a44c6890ad4cebb9c66ecfd4678fb  /usr/lib/libzmq.so

What else could we compare to help find the issue?

[Thomas S Hatch]

I have been debating about this for a while now, but not sure what to check next. I does look like a zeromq issue though. Have you tried with  fresh debian install?