Windows \ 2.9.12 \ Error3 \ cache

[Kiran Subbaraman]

Downloaded web2py src, 2.9.12, and extracted to a folder. Am on Windows 8.1
Added an existing application in it to check if it works. I see this error.
Have cleared up the cache, sessions, etc.

web2py™	Version 2.9.12-stable+timestamp.2015.01.17.06.11.03
Python	Python 2.7.9: D:\programs\open\python27\python.exe (prefix: D:\programs\open\python27)

Traceback (most recent call last):
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\restricted.py", line 224, in restricted
    exec ccode in environment
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\applications\my_app\views\default/index.html", line 149, in <module>
  File "applications\my_app\modules\userdata.py", line 177, in get_cities_filter
    update_cities_filter()
  File "applications\my_app\modules\userdata.py", line 286, in update_cities_filter
    filter_rows = _get_cities_filter_from_db()
  File "applications\my_app\modules\userdata.py", line 340, in _get_cities_filter_from_db
    return db(db.institute.id > 0).select(db.institute.permanent_city_or_town, distinct=True, **cache_settings)
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\dal\objects.py", line 2066, in select
    return adapter.select(self.query,fields,attributes)
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\dal\adapters\base.py", line 1230, in select
    time_expire)
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\cache.py", line 417, in __call__
    self.storage[key] = (now, value)
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\cache.py", line 324, in __setitem__
    self.replace(tmp_path, fullfilename)
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\cache.py", line 286, in replace_windows
    os.rename(src, dst)
WindowsError: [Error 3] The system cannot find the path specified

Some more details:

def replace_windows(src, dst):
    if not ReplaceFile(dst, src, None, 0, 0, 0):
        os.rename(src, dst) <--- this code is highlighted as the point where error is throw

Variables
src     r'C:\Users\subbaraman\Downloads\web2py_src_001\web2py\applications\my_app\cache\tmpkjuyyo'
dst     r'C:\Users\subbaraman\Downloads\web2py_src_001\web...FEBLUQRKSIUQCQ2LOON2GS5DVORSS42LEEA7CAMBJHM======'
os.rename     <built-in function rename>
global os     <module 'os' from 'D:\programs\open\python27\lib\os.pyc'>

Any suggestions?

[Leonel Câmara]

Hey,

This is weird, can you get me a minimal application that reproduces the issue on windows 8.1?

[Kiran Subbaraman]

I'll create an app with db support, and also one that uses disk based cache. Will see if the issue is re-creatable there, and share here.

[Kiran Subbaraman]

Here you go. A minimal app to reproduce the issue.
The app has a table t1, with 'name' as a column.
The view just retrieves all the rows in that table and displays it, via a cache.
The db is postgres. I did not encounter the problem when I used the sqllite db.
Not sure what is going on here.

On Monday, January 19, 2015 at 6:18:34 PM UTC+5:30, Leonel Câmara wrote:

[Leonel Câmara]

Humm that test application gave no error for me. Are you sure you cleared everything in the cache folder before upgrading?

[Kiran Subbaraman]

Weird.
I downloaded a fresh copy of web2py-src, and copied an existing app into it. Started web2py, then using the windows explorer I deleted the contents of the application's cache folder. I had to resort to a manual delete, because when I used the admin to "clean" the application, I encountered a "Access denied" exception.
So, yes, the cache is clean. I tried the app without a cache, and with cache.ram, and both those work as expected. The cache.disk is the one that causes the issues mentioned.

If you are curious about the access-denied exception, then here are the details:

<type 'exceptions.WindowsError'> [Error 5] Access is denied: 'C:/Users/subbaraman/Downloads/web2py_src_001/web2py/applications/cache2912_issue/cache/092'
Version

web2py™     Version 2.9.12-stable+timestamp.2015.01.17.06.11.03
Python     Python 2.7.9: D:\programs\open\python27\python.exe (prefix: D:\programs\open\python27)
Traceback

Traceback (most recent call last):
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\restricted.py", line 224, in restricted
    exec ccode in environment

  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\applications\admin\controllers/default.py", line 1945, in <module>
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\globals.py", line 393, in <lambda>
    self._caller = lambda f: f()
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\applications\admin\controllers/default.py", line 448, in cleanup
    clean = app_cleanup(app, request)
  File "C:\Users\subbaraman\Downloads\web2py_src_001\web2py\gluon\admin.py", line 124, in app_cleanup
    if f[:1] != '.': os.unlink(os.path.join(path, f))
WindowsError: [Error 5] Access is denied: 'C:/Users/subbaraman/Downloads/web2py_src_001/web2py/applications/cache2912_issue/cache/092'

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

On Mon, 19-01-2015 9:59 PM, Leonel Câmara wrote:

Humm that test application gave no error for me. Are you sure you cleared everything in the cache folder before upgrading?

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leonel Câmara]

You did spot a bug in admin as that needs to be changed from os.unlink to recursive_unlink I will make a pull request with the fix.

However, manually deleting the cache folder should have solved your problem. I will keep trying to reproduce your problem as it's hard to solve it if I can't.

[Kiran Subbaraman]

Thanks for confirming that the "access-denied" is an issue, and for the fix.
The original issue that I mentioned - I do not see it being reproduced on a linux instance. Am investigating the issue.

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

On Tue, 20-01-2015 1:22 AM, Leonel Câmara wrote:

You did spot a bug in admin has that needs to be changed from os.unlink to recursive_unlink I will make a pull request with the fix.

However, manually deleting the cache folder should have solved your problem. I will keep trying to reproduce your problem as it's hard to solve it if I can't.

[Kiran Subbaraman]

(Updated with correct code)

Well, as it turns out, the src and dst file names were hitting a Windows specific 256 character limit. This was the issue: http://stackoverflow.com/a/23230380/348142
The fix is to introduce the following code (or something more refined):
cache.py Line: 284

                def replace_windows(src, dst):
                    windows_prefix = "\\\\?\\"
                    if not ReplaceFile(windows_prefix+dst, windows_prefix+src, None, 0, 0, 0):
                        os.rename(windows_prefix+src, windows_prefix+dst)

I do not know where else this sort of code needs to be introduced.
Also, will see if I can issue a pull-request (haven't tried to unit-test web2py with this fix; need to understand that process before issuing a pull-request)



On Tuesday, January 20, 2015 at 1:22:56 AM UTC+5:30, Leonel Câmara wrote:

You did spot a bug in admin as that needs to be changed from os.unlink to recursive_unlink I will make a pull request with the fix.

[Leonel Câmara]

Great detective work Kiran. No wonder I couldn't reproduce it, for some reason it wasn't going over the limit for me.

Some suggestions for your pull request.

May I suggest that instead of putting the fix in replace_windows, you put it in key_filter_in_windows and then remove the prefix in key_filter_out_windows the reason for this is that the filenames are actually used as the keys so we need to be able to convert them to exactly the key the user put in (mainly for regex based removal from cache).

Also add some comment, something like \\?\ activates extended-length path names in NTFS. The reason this needs a comment is that this is the kind of thing no one would know the reason for in a couple of weeks. Hopefully one day it can even be removed.

Sigh NTFS...

[Kiran Subbaraman]

Leonel,
Am confused as to where to place the prefix code (this is more so because am not too familiar with the cache code)
My understanding is this:

• The NTFS prefix is applicable for only accessing/manipulating a file (with a long file name), from the file system. This prefix does not get added to the file name, when it is stored on the file system.
  
• The key in/out methods are applicable only to manage the keys within a cache (Storage)
• I didn't see the keys themselves being used as names, rather I see this on line https://github.com/web2py/web2py/blob/master/gluon/cache.py#L321, where the key is joined with a parent path.
  

This is the reason why I decided to add the key prefix code only in the replace* method, and nowhere else.
I tested this locally (manually testing with my sample app), to see if the cache contents are retrieved correctly. And then used the http://127.0.0.1:8000/myapp/appadmin/ccache to inspect the cache, and also remove keys from it. It seemed to work fine.
Am I missing something?

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

[Leonel Câmara]

Am I missing something?

Nope. I was! Particularly this:

 This prefix does not get added to the file name, when it is stored on the file system.

Which should be obvious given that ? isn't even a valid character in a filename for windows. Your fix is correct then. Maybe just add the explanatory comment and go ahead with the pull request.

[Kiran Subbaraman]

Thanks, will do.
Opened an issue to track this to closure: https://code.google.com/p/web2py/issues/detail?id=2035

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscribe@googlegroups.com.

[Niphlod]

my bad. rebuilding now.

On Tuesday, January 20, 2015 at 7:22:54 PM UTC+1, Kiran Subbaraman wrote:

Thanks, will do.
Opened an issue to track this to closure: https://code.google.com/p/web2py/issues/detail?id=2035

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

On Tue, 20-01-2015 11:25 PM, Leonel Câmara wrote:

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.

[Niphlod]

try this please https://www.dropbox.com/s/jmdy2qtt71i0snb/web2py_win.zip?dl=0 .

[Kiran Subbaraman]

Thanks, just tried it.
I do not see the error reported in Issue 2034: https://code.google.com/p/web2py/issues/detail?id=2034, but I see the error being addressed in this current discussion thread (Issue 2035).
If it is possible for you to bundle in the fix for Issue 2035 (PR: https://github.com/web2py/web2py/pull/593), will make it easier for me to can confirm this works fine.

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

[Niphlod]

easy....
- stop the executable
- download https://www.dropbox.com/s/vf83tc5kasv5o6k/cache.pyc?dl=0
- replace library.zip/gluon/cache.pyc with the downloaded file
- start the executable

[Kiran Subbaraman]

Done. Works :-)

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/