work_queue_graph_workers appears to be missing in ndcctools 7.5.4

[Igor Khomyakov]

Hello, work_queue_graph_workers appears to be missing in ndcctools 7.5.4. Anyone knows where it could be obtained? 

[Douglas Thain]

Hello Igor -

Hmm, the script is still there, but we were neglecting to install it for some reason.

I made a quick fix which will go out in the next release:

https://github.com/cooperative-computing-lab/cctools/pull/3327

In the meantime, you can just download the script directly from here:

https://github.com/cooperative-computing-lab/cctools/blob/master/work_queue/src/work_queue_graph_workers

Let us know if you need anything else.

Sincerely,

Doug

On Tue, May 9, 2023 at 5:01 AM Igor Khomyakov <ikhom...@gmail.com> wrote:

Hello, work_queue_graph_workers appears to be missing in ndcctools 7.5.4. Anyone knows where it could be obtained? 

--
You received this message because you are subscribed to the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/c042dc29-4dc9-42ea-967e-633025b93ebcn%40googlegroups.com.

[Igor Khomyakov]

Dear Doug,

Thank you for your prompt reply and your valuable contributions to the CCTools software and DISC lectures. We recently discovered your software and found it relevant to our needs, and closely aligned with our requirements. In addition, we would like to express our gratitude for your insightful and valuable DISC lectures, which have been incredibly helpful to us.

We are in the process of selecting a workload system for our satellite startup, where we plan to process thermal and hyperspectral images from our own and partner satellites. We are planning to try out Work Queue and possibly TaskVine.

We are curious, based on your experience, is there a limit on the number of workers that can connect to one manager to work with one queue? What is the largest number of workers you have tried, and did it affect performance?

Thank you once again for your contributions to CCTools and for sharing your knowledge through your lectures.

Best regards,

Igor

[Douglas Thain]

Hello Igor -

Sounds like an exciting application, we would be happy to talk with you more to understand your needs

and see how our software can be part of the solution.

Regarding scale, some of our largest applications have sustained 10K tasks running concurrently on 1000+ multi-core workers.  So it's possible to go fairly large.

That said, it's important to remember that it is really the nature of the tasks and the underlying hardware that are the primary constraint upon scale.  The ultimate scale of most distributed applications is a function of how much data needs to move, and how fast the network can move it.

Sincerely,

Doug

To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/613ad19b-dce1-463c-b0cc-19a38adac35en%40googlegroups.com.

[Igor Khomyakov]

Hello Doug,

Thank you for offering your help and advice. We are thrilled to collaborate with you and your team.

The 10K parallel tasks on 1000+ multi-core workers seem to be more than sufficient for our current application. Our plan is to create the full DAG incrementally based on the metadata of the incoming satellite images from various sources. However, we will only trigger the image processing for subsets of the graph based on client demand. This means that our Work Queue manager will be somewhat similar to Makeflow, with the key difference being that our graph will be huge and built incrementally, and the targets will be triggered per client order on demand. We were thinking of using Makeflow, but due to the differences, we are currently leaning towards implementing our own Work Queue manager around the DAG implemented in the Postgres database. This way, we will be able to add workload subgraphs as new images come and trigger subgraphs that intersect with client orders. It should also be very convenient for us to track the progress of our workload using the database.

Regarding data transfer, we envision a system where minimal data needs to be exchanged either between the Work Queue manager and the workers or directly among the workers themselves. We plan to utilize S3 object storage to store the data, and have the tasks retrieve all data inputs directly from S3 and output their data products back to S3. We are planning to run the workload close to the S3 data within the same AWS region, and we do need to test the network bandwidth that AWS can provide to us to ensure it doesn't limit our scalability.

Thank you once again for your cooperation and support.

Best regards,

Igor

[Igor Khomyakov]

Dear Doug,

I'm very happy to inform you that our satellite startup has successfully implemented a prototype of a workflow and archiving system. This system draws inspiration from Makeflow but has a distinct feature: it persists the DAG in a Postgres database. This allows for the DAG's incremental expansion and contraction. Notably, it employs cryptographic hashing for automatic DAG deduplication even when multiple independent clients inject potentially duplicate sub-DAGs into the database simultaneously. Our plan is to utilize this system for both workflow orchestration and product archiving.It will handle serving product catalog metadata queries from the same repository. Furthermore, it enables clients to specify demand (activate specific target nodes), with the system producing only the requested nodes. The repository can store DAGs for data artifacts that may never be demanded and thus never produced. However, these DAGs are not useless; they hold metadata and act as product catalogs for items that could be produced upon demand. The persistent repository also provides provenance information, detailing how product artifacts will or have been produced and the software versions of the processors (tasks) used. The system can process the DAG simultaneously across multiple compute clusters, each supported by a different backend batch system. The first integrated batch system was Workqueue, which so far has been performing exceptionally well.

We have a few questions where your or the community's expertise would be invaluable:

1. Our software, acting as a Workqueue manager, is currently a single-threaded Python application interfacing between the Postgres database and Workqueue. Its logic could be more elegantly and robustly implemented using either asynchronous coroutines or separate threads. We have two questions: 

a) Is there a plan to introduce an asynchronous API for Workqueue? 

b) Is the current queue API MT-safe at both C and Python levels?

2. We've encountered an issue with AWS Network Load Balancer (NLB) dropping connections after a short (around 5 minutes) idle period, as detailed in this article: https://medium.com/tenable-techblog/lessons-from-aws-nlb-timeouts-5028a8f65dda. Despite our attempts to activate Workqueue's keepalive feature and adjust system settings as suggested, connections between workers and the manager still become stale approximately every 5 minutes. We wonder: 

a) Could there be an issue with the Workqueue keepalive or the way how we activate it? We tried several methods: by default, using "specify_keepalive_interval/timeout", and “tune.” 

b) Have you faced similar issues with NLB and found any solutions or workarounds?

3. Regarding Work_queue_status, the manual states it can query a manager directly for detailed information about tasks and workers when given an address and port. However, we haven't been able to retrieve any information when specifying the IP and port of a running Workqueue manager. Can you provide any insights into this issue?

Doug, thank you once again for the remarkable software your team has developed. We eagerly anticipate applying them in our satellite image processing applications and greatly value your advice and insight.

Best regards,

Igor 

You received this message because you are subscribed to a topic in the Google Groups "Cooperative Computing Tools" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cctools-nd/iUCy__TEi8U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cctools-nd+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/db07b714-7029-49d2-9e2c-be236bc508efn%40googlegroups.com.

[Douglas Thain]

Hello Igor -

Glad to hear things are going well, I would certainly like to learn more about the workflow-in-database concept.

To answer your questions -

1 - We are not planning to make the WQ interface asynchronous or multi-threaded.  At core, it's a shared data structure that needs to be handled with some degree of sequential consistency.

That said, we have interfaced WQ with more concurrent systems (like Parsl) by placing an interface process between the two that does the translation.  Happy to talk more about that.

2 - There is an application-level keep-alive however (taking a quick look) it is provoked by the manager sending a "check" message and the worker responding with an "alive" message.   If you run the worker with the "-d all" option, you can see a log of the messages going back and forth which may clarify what's happening.  It might be more practical for the worker to send unsolicited keepalives just to keep the TCP connection open, and that's something we can fix up if it would help you.

3 - The direct connection from work_queue_status to the manager should work, although it can be delayed if the manager is working hard.  Let me do a little lock checking here to make sure something hasn't broken recently.

Cheers,

Doug

To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/305492BC-3C28-4254-93D7-8C4EC03A83B8%40gmail.com.

[Igor Khomyakov]

Hello Doug,

I would be happy to provide additional information about the workflow-in-database concept and discuss it in greater detail. Please feel free to contact me at i...@ikhsoftware.com to arrange a meeting, or we can discuss this further via email, whichever is more convenient for you.

1- Regarding concurrency, we aim to set up three parallel threads:

a) The first thread sleeps on a Postgres connection, awaiting Postgres notifications via the listen/notify mechanism. Once a notification is received, it retrieves runnable tasks from the database one by one, feeds them to the queue as long as it is 'hungry', and then goes back to sleep.

b) The second thread sleeps on the work queue and, upon task completion, removes results from the queue, updates the database, checks if the queue is 'hungry', and if so, sends a notification to wake up the first thread before returning to sleep on the queue.

c) The third thread sleeps on a timer, wakes up every 60 seconds, retrieves all work queue stats (bandwidth, capacity, efficiency, task counters, workers, etc.), and writes this data into the database. This is mainly for convenience, as we're implementing a dashboard that will read all information from the database, and it's useful to have queue stats in one place.

We are curious if these three threads might face any concurrency issues while simultaneously submitting new tasks, retrieving results of completed tasks, and reading current queue stats. The problem is that we can’t really protect these using a mutex because we want to use blocking workqueue.wait to retrieve the task results. 

2 - Regarding keep-alive: 

We were also thinking that initiating keep-alive checks from the worker side might be beneficial: If the worker does not receive an "alive" response within the keepalive timeout, it can attempt to re-establish the connection or, alternatively, exit. If the manager does not receive any messages from a worker within, for instance, twice the keepalive interval, it could then remove the worker and close the connection. 

Alternatively, the initiation of keep-alive checks could remain on the manager side, but the worker could be enhanced to monitor the time since the last message was received from the manager. If this period exceeds, for instance, twice the keepalive interval, the worker could then drop the connection and either attempt to re-establish it or exit.

We were puzzled as to why the NLB connection didn't stay active with keepalive checks every two minutes from the manager. The next item seems to provide an answer to this question.

3 - Thanks for investigating the work_queue_status. We believe we have identified the issue, which also seems to relate to the NLB timeout problem. 

It appears that when our Python manager program neglects the queue for too long, specifically by not periodically calling workqueue.wait, background workqueue functions cease to operate. This includes halting check message transmissions, not accepting new workers, stopping stats and debug log updates, and failing to respond to work_queue_status requests.

We suspect this happens because these activities are managed by the main thread. When the Python program is predominantly sleeping or waiting for database notifications (which occurs in our main loop when the workqueue is empty), it suspends the background queue tasks, leading to various issues. To remedy this, we've adjusted our main loop to spend most of the time inside workqueue.wait(1), waking up every second to check the database and returning to workqueue.wait(1) for another second if there was no new work. 

We would appreciate it if you could reconfirm the accuracy of our findings and suspicions. Additionally, if you have any better suggestions for remedying this situation, please let us know.

Additionally, we observed that running the manager with a password leads to the rejection of work_queue_status, as it does not support password authentication (auth: peer is not using password authentication). This isn't a significant issue, as we are operating on an internal network and can run without the password. However, we wanted to highlight this in case we overlooked a method to set the password with work_queue_status.

  

Thank you again for dedicating your time and attention to our questions.

Best regards, 

Igor

To view this discussion on the web visit https://groups.google.com/d/msgid/cctools-nd/CAGrksMBb7WEruXJObHCiEeZCiZ6mmjfP-95_9pptKiBjnAzGZA%40mail.gmail.com.