Problems writing to file (always thinks file is closed)

[Joseph Coffey]

Has anyone else encountered issues writing to a file in reinteract?

When I try to do this in a worksheet, I get a Value error, telling me
that I'm writing to a closed file. It works fine in a library.

[Robert Schroll]

I'm seeing this too. It's a bit weird, but I think I've worked out
what's going on.

Firstly, if you enclose the file commands in another block, everything
works just fine. For example, this always works for me:
if True:
f = file('temp.txt', 'w')
f.write('Hi!\n')
f.close()
Enclosing the output commands in a try block, with f.close() in the
finally block, is probably not a bad idea anyway, so this could be a
usable work-around.

I also noticed that if I capture the output of the write command in
another variable, I never get a ValueError. I think this is due to how
Reinteract tracks modification of objects. I believe that when
Reinteract sees something like "f.write('blah')", it assumes that the
command will modify f. Therefore, before executing it, Reinteract makes
a copy of f and executes write() on that copy. However, copy.copy() on
an open file object returns a closed, uninitialized file object. Trying
to write to this object raises the ValueError. On the other hand, when
Reinteract sees a statement like "g = f.write('blah')", it assumes that
write() does not modify f, so no copy is made.

(It may be worth noting that the same thing happens when reading files.
f = file('temp.txt', 'r')
f.read()
raises a ValueError, while
f = file('temp.txt', 'r')
s = f.read()
does not. The same logic applies.)

What I don't understand is that sometimes
f = file('temp.txt', 'w')
g = f.write('Hi!\n')
f.close()
fails to actually write to the file temp.txt, even though it executes
without error. (Instead, I get an empty file.) I haven't been able to
figure out why it works sometimes and not other times. Opening the file
for both reading and writing and conducting a read operation seems to
ensure that the write succeeds. Perhaps someone who actually
understands file operations can explain this part.

Anyway, my feeling is that the best work-around is to enclose all the
file output commands in a block. Besides fixing this problem, it will
also help ensure that the file was opened before attempting to write to it.

Hope this helps,
Robert

[Owen Taylor]

On Thu, Jul 16, 2009 at 6:58 PM, Robert Schroll <rsch...@gmail.com> wrote:
>
> Joseph Coffey wrote:
>> Has anyone else encountered issues writing to a file in reinteract?
>>
>> When I try to do this in a worksheet, I get  a Value error, telling me
>> that I'm writing to a closed file.  It works fine in a library.

I think it's worth pointing that there is a conceptual issue related
to fitting file I/O into the Reinteract framework.

If I have:

f = open("somefile.txt", "w");
f.write("a\n");
f.write("b\n");
f.close();

there's no way I can execute just some of those lines and have things
work out right. If I change f.write("b\n"); to f.write("c\n"); then
the file needs to end up with:

a
c

Not:

a
b
c

> I'm seeing this too.  It's a bit weird, but I think I've worked out
> what's going on.
>
> Firstly, if you enclose the file commands in another block, everything
> works just fine.  For example, this always works for me:
>        if True:
>            f = file('temp.txt', 'w')
>            f.write('Hi!\n')
>            f.close()
> Enclosing the output commands in a try block, with f.close() in the
> finally block, is probably not a bad idea anyway, so this could be a
> usable work-around.

This would be my suggestion as well - make writing the file atomic
from Reinteract's perspective. It also bypasses the problem with being
unable to copy a file since if 'f' isn't a defined variable before the
block is executed, Reinteract won't try to copy it.

> I also noticed that if I capture the output of the write command in
> another variable, I never get a ValueError.  I think this is due to how
> Reinteract tracks modification of objects.  I believe that when
> Reinteract sees something like "f.write('blah')", it assumes that the
> command will modify f.  Therefore, before executing it, Reinteract makes
> a copy of f and executes write() on that copy.  However, copy.copy() on
> an open file object returns a closed, uninitialized file object.  Trying
> to write to this object raises the ValueError.  On the other hand, when
> Reinteract sees a statement like "g = f.write('blah')", it assumes that
> write() does not modify f, so no copy is made.

Exactly. But see above for why this isn't really a very satisfactory
workaround for file I/O. File I/O has side effects on the outside
world (the magnetization patterns on the hard disk), so neither
copying nor not copying actually works out.

> What I don't understand is that sometimes
>        f = file('temp.txt', 'w')
>        g = f.write('Hi!\n')
>        f.close()
> fails to actually write to the file temp.txt, even though it executes
> without error.  (Instead, I get an empty file.)  I haven't been able to
> figure out why it works sometimes and not other times.  Opening the file
> for both reading and writing and conducting a read operation seems to
> ensure that the write succeeds.  Perhaps someone who actually
> understands file operations can explain this part.

I think it's just that f.close() is also a mutation, so again the
f.close() is happening on a copy, so the f.write() is never flushed.
It seems to work reliably (with the limitations discussed above) for
me if I do:

h = f.close();

I think the long-term solution here is to have some way of
"annotating" standard methods, so Reinteract would know:

>>> m = re.search("b+", "abbbb")
>>> m.group(0) # Not a mutation, no need to copy

>>> f = file('temp.txt', 'w')

>>> f.write('Hi!\n') # A mutation, can't copy, need to re-execute from the point f was initially assigned

- Owen

[Robert Schroll]

FYI: I opened Trac ticket #91 about this. Although the ticket links to
this thread, I forgot to link the thread back to the ticket. So here it
is: http://www.reinteract.org/trac/ticket/91

Robert