Writing multiple files with with-context

[Shunichi Wakabayashi]

To write onto multiple files on the same time (a number of files are variable),
I'd like to code as follows, for example, IF I can do,

LIST_LEN = 4
with [ open('list_%d.txt' % i, 'w') for i in range(LIST_LEN) ] as fobjlist:
for i in range(1000):
fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

or using a dict object,

DICT_KEYS = ('foo', 'bar', 'baz')
with { k: open('dict_%s.txt' % k, 'w') for k in DICT_KEYS } as fobjdict:
for i in range(1000):
fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

However, list and dict don't has __exit__ method and so they cannot run.
One idea is using contextlib.nested(),

from contextlib import nested

with nested(*[open('list_%d.txt' % i, 'w') for i in range(LIST_LEN)]) as fobjlist:
for i in range(1000):
fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

with nested(*[open('dict_%s.txt' % k, 'w') for k in DICT_KEYS]) as fobjlist:
fobjdict = dict(zip(DICT_KEYS, fobjlist)) #convert list to dict
for i in range(1000):
fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

On Python2.x, this is OK. but 3.x warns that nested() is deprecated.
Moreover, on using dict, it is required to convert list to dict.

Another idea is to make container classes having __exit__() myself.

class MyList(list):
def __enter__(self):
return [ v.__enter__() for v in self ]
def __exit__(self, exc_type, exc_value, traceback):
ret = False
for v in self:
if v.__exit__(exc_type, exc_value, traceback):
ret = True
exc_type = exc_value = traceback = None
return ret

class MyDict(dict):
def __enter__(self):
return { k: v.__enter__() for k, v in self.items() }
def __exit__(self, exc_type, exc_value, traceback):
ret = False
for v in self.values():
if v.__exit__(exc_type, exc_value, traceback):
ret = True
exc_type = exc_value = traceback = None
return ret

with MyList( open('list_%d.txt' % i, 'w') for i in range(LIST_LEN) ) as fobjlist:
for i in range(1000):
fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")

with MyDict( (k, open('dict_%s.txt' % k, 'w')) for k in DICT_KEYS ) as fobjdict:
for i in range(1000):
fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")

I think this is smarter a little than others,
but it cannot guaranteed to call __exit__() of members in containers
if members are changed during with-context.

So, do you have another, more smart and pythonic way?

Thanks

[Ian Kelly]

On Sun, May 22, 2011 at 8:48 PM, Shunichi Wakabayashi
<shunichi_w...@yahoo.co.jp> wrote:
> One idea is using contextlib.nested(),
>
> from contextlib import nested
>
> with nested(*[open('list_%d.txt' % i, 'w') for i in range(LIST_LEN)]) as fobjlist:
>  for i in range(1000):
>    fobjlist[random.randrange(LIST_LEN)].write(str(i)+"\n")
>
> with nested(*[open('dict_%s.txt' % k, 'w') for k in DICT_KEYS]) as fobjlist:
>  fobjdict = dict(zip(DICT_KEYS, fobjlist)) #convert list to dict
>  for i in range(1000):
>    fobjdict[random.choice(DICT_KEYS)].write(str(i)+"\n")
>
> On Python2.x, this is OK. but 3.x warns that nested() is deprecated.

Not merely deprecated. It has already been removed in 3.2.

> Another idea is to make container classes having __exit__() myself.
>
> class MyList(list):
>  def __enter__(self):
>    return [ v.__enter__() for v in self ]
>  def __exit__(self, exc_type, exc_value, traceback):
>    ret = False
>    for v in self:
>      if v.__exit__(exc_type, exc_value, traceback):
>        ret = True
>        exc_type = exc_value = traceback = None
>    return ret

This has a number of subtle bugs in it:

1) Each context manager's __exit__ method is not loaded before the
corresponding __enter__ method is invoked.
2) If the second context manager's __enter__ method raises an
exception, the first context manager's __exit__ method is never
called, breaking the with statement guarantee.
3) The __exit__ methods are called in the same order that the
__enter__ methods were called. Since they form a stack, they should
be called in the reverse order.

These highlight the complexity of handling context managers correctly,
which I think suggests that a custom implementation is probably a bad
idea.

> So, do you have another, more smart and pythonic way?

Copy the implementation of contextlib.nested to your own custom module
and use that. The last revision prior to its removal is here:

http://hg.python.org/cpython/file/45506be44514/Lib/contextlib.py