Status of TaskGroup or something like it?

[Justin Mayfield]

I'm curious if any work has happened towards supporting a grouping concept like Yury has discussed with TaskGroup?   I've been away from python for a while and I'm finding that the use of gather and wait is trickier than I remember it.   My code tends to be inelegant when done as the docs recommend (using create_task first, etc).  I think I benefit from a class based context manager primitive like TaskGroup.

Side note, I find I'm missing Javascript's Promise.race() quite a bit too (as_completed is often an aesthetic mismatch for my code), so it would be neat if the TaskGroup behaved like a set of outstanding work with methods like `race()` and `wait()`.

Somewhat related, I recently watched Nathaniel's talk on the happy eyeballs algo (https://www.youtube.com/watch?v=oLkfnc_UMcE) and I enjoyed the use of this very easy to understand algo as the test basis for concurrency designs.  However when he described the problem my first instinct was that he would the write algo in a top-down style where the outer blocks handled scheduling and the inner tasks simply tried to do work or failed.  It seemed to me the use of events objects, scheduling the next task from within an existing task (with conditions) and handling cleanup manually via cancel calls was less than perfect and I dare say not dissimilar to the reviled GOTO.  Could this algo be written in even simpler terms?  Namely, without calls the any sort of cancel, without the tasks themselves knowing anything about the outside scheduling or work set, and perhaps most importantly without the use of Event objects which are champions of the preemptive world IMO.

Please permit me to indulge in some hypothetical code I would like to be able to write someday to perhaps highlight my wishlist for asyncio...

async def might_fail_or_be_slow():
  await asyncio.sleep(random.random() * 10)
  if random.random() > 0.5:
    raise Exception("job failed")
  return "job worked"


async def happy_eyeballs(max_job_wait=0.200):
  async with asyncio.group() as ag:
    for i in range(10):
      ag.add_task(might_fail_or_be_slow())
      try:
        task = await ag.race(timeout=max_job_wait)
      except asyncio.TimeoutError:
        print("Nothing is ready yet, add another..")
      else:
        try:
          return await task
        except Exception:
          print("Somebody failed, add another..")
  raise Exception("nobody worked")

   

Some notes.  This is obviously just pseudo code and I've not thought this through very carefully so it's obviously rough and incomplete.  For one, I'm making the assumption here that the TaskGroup will be somewhat queue like.  E.g. if you call `race()` the result is removed from the group;  This might be silly or need to be more explicit, or maybe there could be a multitude of grouping classes for each use-case TaskQueue, TaskGroup, TaskRace, etc (spitballing).  LIkewise it might be useful for the above TaskGroup class to have `remove_task(task)` , `wait(timeout=None) -> [tasks...]`, and perhaps even something like `results(timeout) -> [results from tasks...]` which is a shortcut method for `results = [await t for t in await group.wait()]`.

Cheers.

[Yury Selivanov]

Hi Justin,

Yes, we want to have TaskGroups in asyncio, and the only reason we didn't add them to Python 3.8 is that we also need to support ExceptionGroups (or MultiErrors).  Nathaniel and I plan to start working on that.

Here's am implementation of TaskGroups that I created for EdgeDB https://github.com/edgedb/edgedb/blob/master/edb/common/taskgroup.py — you can use it in the meantime, it works just fine.

Yury

--
You received this message because you are subscribed to the Google Groups "python-tulip" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tulip+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-tulip/54f410f6-b9b6-4af3-be29-c2e133fb7fe7%40googlegroups.com.

[Justin Mayfield]

Excellent,

I'll have a look, thanks for the response.

To unsubscribe from this group and stop receiving emails from it, send an email to python-tulip...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python-tulip/54f410f6-b9b6-4af3-be29-c2e133fb7fe7%40googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "python-tulip" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-tulip/dsv_PDcqvsw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to python-tulip...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python-tulip/k3gf1sm6.3a8e58ab-eb46-4304-95b2-69f5e3418733%40we.are.superhuman.com.