How to keep producer connection alive during inactivity periods

[Marcos Modenesi]

Hi!

Take a look at this simple example:

while True:

     new_files = get_new_files_to_process()

     if new_files:

         message = build_message(new_message)

         producer_channel.basic_publish(exchange, routing_key, message)

 

Suppose for a long period of time there are no new files, hence, nothing to publish. Then I notice the publisher connection ioloop does not get the chance to send heartbeats to the server.

In this case I work around it catching the dropped connection error and reestablishing it, but it feels hacky (and produces a lot of error logs).

Is there a way of ensuring that the producer has the chance to send heartbeats during those periods when basic_publish is not called?

Thanks in advance

Marcos

[Luke Bakken]

Hi Marcos,

get_new_files_to_process should be changed to not block forever, or you should run it on a separate thread.

Thanks,

Luke

[Marcos Modenesi]

Hi Luke,

thanks for your feedback!

I'm still missing some clue on how to behave when there is actually nothing to process for a sufficiently long period of time.

That is, what to do when there is no reason to call basic_publish. Something like

while True:

    new_files = get_new_files_to_process(timeout_s=30)  # when timeout is reached, returns an empty list

    if new_files:

         ...

    else:

        producer_channel.i_have_nothing_to_say_but_i_want_you_to_know_that_i_am_still_here()

    

--
You received this message because you are subscribed to a topic in the Google Groups "Pika" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pika-python/btDQCjt1Aak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pika-python...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pika-python/c86a10dd-2ee3-429f-8c77-a79d5238f266n%40googlegroups.com.

--

Marcos Modenesi

---

This email and any files transmitted with it are for the use of the intended recipient(s) only. The information contained in this communication is confidential and may be subject to non-disclosure provisions. This email (and any attachments) may also include material nonpublic information (“MNPI”) relating to Satellogic Inc., including information relating to its business, operations or financial performance. Any MNPI included in this email is being provided to you as our customer, supplier or strategic partner in the ordinary course of our business.  Neither you nor any officer, employee or affiliate of yours who is given access to the MNPI may use or disclose this information other than in connection with the furtherance of our business relationship. If you have received this email in error, please notify the sender immediately and then delete it. Please do not disclose, copy, distribute or otherwise use this email or its contents without the author’s permission.

[Luke Bakken]

Hi Marcos,

You use this method -

https://pika.readthedocs.io/en/stable/modules/adapters/blocking.html#pika.adapters.blocking_connection.BlockingConnection.process_data_events

Again, ideally you would run  get_new_files_to_process in a different thread, because when that method runs you will block Pika's I/O loop.

If you'd like to start work on your code and add them to a git repo I could clone (like via GitHub or GitLab), I'm happy to assist.

Thanks,

Luke

[Marcos Modenesi]

Thanks so much, Luke, that's exactly the concept I was missing.

Thanks for your assistance.

--

You received this message because you are subscribed to a topic in the Google Groups "Pika" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pika-python/btDQCjt1Aak/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pika-python...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pika-python/91be555f-5645-42a6-9e48-4efa2a4dafaen%40googlegroups.com.