Issue 17 in pylibtiff: tests fail on all os.remove() on Windows

[pyli...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 17 by matej.s...@gmail.com: tests fail on all os.remove() on
Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

Temporary files are locked on Windows, so they can not be removed without
closing beforehand.

Log file of nosetests on Windows 7, VS2008, Python 2.7 attached.

I tried to fix the error by calling .close() method on objects that opened
the files, deleting mmaped variables and/or TIFFfile or TIFFimage instances
before calling os.remove(). It didn't solve the issue unfortunately. The
mmaped numpy arrays are somehow still open.

Attachments:
pylibtiff_nosetests.txt 3.0 KB

[pyli...@googlecode.com]

Updates:
Status: Accepted

Comment #1 on issue 17 by pearu.peterson: tests fail on all os.remove() on
Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

A possible workaround would be enclose .close() method calls
within try-except block. This is to see whether the tests
pass under Windows.

Next workaround would be to remove temporary files using atexit module.
Perhaps the memmap files are closed at the exit time of Python.

[pyli...@googlecode.com]

Comment #2 on issue 17 by matej.s...@gmail.com: tests fail on all

os.remove() on Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

You meant to enclose os.remove() calls within try-except block, don't you?

[pyli...@googlecode.com]

Comment #3 on issue 17 by pearu.peterson: tests fail on all os.remove() on
Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

Yes.

[pyli...@googlecode.com]

Comment #4 on issue 17 by matej.s...@gmail.com: tests fail on all

os.remove() on Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

With this patch tests proceed without error. But it's only a workaround,
few thousands of temporary files are created...

Attachments:
pylibtiff_tests_windows.patch 2.0 KB

[pyli...@googlecode.com]

Comment #5 on issue 17 by matej.s...@gmail.com: tests fail on all

os.remove() on Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

hundreds not thousands...

[pyli...@googlecode.com]

Comment #6 on issue 17 by pearu.peterson: tests fail on all os.remove() on
Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

I just committed a patch that uses atexit to cleanup temporary files.
Let me know whether it works under Windows.

[pyli...@googlecode.com]

Comment #7 on issue 17 by matej.s...@gmail.com: tests fail on all

os.remove() on Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

It doesn't work.

c:\...\libtiff\tests>nosetests -s
.....277:292,[[(250, 0, 0) (0, 251, 0) (0, 0, 252)]
[(250, 0, 0) (0, 251, 0) (0, 0, 252)]]
[array([[[250, 0, 0],
[250, 0, 0]]], dtype=uint8), array([[[ 0, 251, 0],
[ 0, 251, 0]]], dtype=uint8), array([[[ 0, 0, 252],
[ 0, 0, 252]]], dtype=uint8)]
...
----------------------------------------------------------------------
Ran 8 tests in 4.486s

OK
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
WindowsError: [Error 32] The process cannot access the file because it is
being used by another
process: 'c:\\users\\smid\\appdata\\local\\temp\\tmpw3kmc0.tif'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
WindowsError: [Error 32] The process cannot access the file because it is
being used by another
process: 'c:\\users\\smid\\appdata\\local\\temp\\tmptwma83.tif'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Python27\lib\atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
...
...
...

Maybe this can give a clue:
http://stackoverflow.com/questions/1519956/numpy-and-memmap-errno-24-too-many-open-files
but I do not have time to do it myself just now.

[pyli...@googlecode.com]

Comment #8 on issue 17 by pearu.peterson: tests fail on all os.remove() on
Windows
http://code.google.com/p/pylibtiff/issues/detail?id=17

Ok, I'll take a look at it later. Occasionally, I have seen
the too many open files issue when working over nfs and
the proposed clue is very interesting to me too.

Thanks for feedback so far!