pika.exceptions.StreamLostError: Stream connection lost: IndexError('pop from an empty deque')

[João Conde]

Hello all,

I am running pika 1.1.0, python 3.8.5, RabbitMQ 3.8.9, and Erlang 23.1.4.

I have multiple threads publishing messages to one exchange. Since pika is not thread-safe, instead of directly using `BlockingChannel.basic_publish` I am using the `BlockingConnection.add_callback_threadsafe` to add a `basic_publish` callback.

However, sometimes, when the number of concurrent publish callbacks increases, I get the

`pika.exceptions.StreamLostError: Stream connection lost: IndexError('pop from an empty deque')` which I think might be related with this multi-threaded scenario.

I can post code chunks but honestly its just this, multiple publish attempts on an exchange.

Thank you in advance 

[Luke Bakken]

Hello,

What is logged (if anything) by RabbitMQ at the same time as this error?

[João Conde]

First, the complete application logs are attached. I get a
`pika.exceptions.StreamLostError` followed by `AssertionError:
('_AsyncTransportBase._produce() tx buffer size underflow', -81464,
1)` and then for there on out a
`pika.exceptions.ConnectionWrongStateError:
BlockingConnection.add_callback_threadsafe() called on closed or
closing connection`.

As for the RabbitMQ logs, I have 3 nodes. Attached I send the logs
from rabbit1, 2 and 3.

> --
> You received this message because you are subscribed to the Google Groups "Pika" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pika-python...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pika-python/dd3a8940-501f-4d73-a1ab-4f89261ff7f0n%40googlegroups.com.

[Luke Bakken]

Hello,

It looks as though you may be sending a very large message. Is that true?

> I have multiple threads publishing messages to one exchange. Since pika is not thread-safe, instead of directly using `BlockingChannel.basic_publish` I am using the `BlockingConnection.add_callback_threadsafe` to add a `basic_publish` callback.

Please change your code so that each thread has its own instance of BlockingConnection for publishing messages, instead of using add_callback_threadsafe. This should fix the issue.

If you can provide code to reproduce this error we could track it down.

Thanks,

Luke

[João Conde]

Hello Luke,

I now have one Blocking Connection per thread, and it works with no
issues (as expected).

Thank you

> --
> You received this message because you are subscribed to a topic in the Google Groups "Pika" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/pika-python/GeUPpexSB5Q/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to pika-python...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pika-python/ab4b8a81-96c3-4cc2-8fd7-f292a587cf2cn%40googlegroups.com.

[Luke Bakken]

Great! Thank you for following up. It will probably perform a bit better, if that matters for your use-case.