HTML files locked during autocompile
M.S.
first of all: Great tool. I use this for some weeks now and it is
really very very helpful to me.
But what I am wondering about is, that the autocompile command seems
to lock the html files (in content dir) so that I need to force
overwriting the files during saving.
Is this usual or even desired behavior?
My setup is:
* Windows XP Pro
* Ruby 1.8.6
* nanoc 3.0.2
Greets
Marco
Denis Defreyne
Hello Marco,
You’ll have to excuse me for not having a lot of experience with Windows XP. What do you mean by “lock” the files (a shared read lock)? The files are simply written using IO.open(filename, 'wb') { |io| io.write(data) } so I’m not sure what could go wrong here. Are there any other processes that could have taken a lock on the files in the output directory?
Regards,
Denis
--
Denis Defreyne
denis.d...@stoneship.org
M.S.
thanks for your answer.
unfortunately I don't know anything about Ruby.
But i found out following behavior:
* start nanoc3 autocompile
* open (e.g.) content/index.html in editor (I use Vim)
* change file and save
* --> works fine
* start browser with http://localhost:3000
* change file again in editor
* --> editor claims that file is readonly from now on
Maybe the problem is not the usage of the command IO.open(...) but the
embedded mini web server used by the autocompile command.
I tried to reproduce this problem on Mac OS (10.6.2) without luck. So
I think this problem might be specific to windows.
Maybe there is any Windows&Ruby expert here in this community who can
(and want) to help solving this problem.
Greets
Marco
Eric Sunshine
I have been using nanoc3 on Vista to develop a site, but have never
run into this problem with the autocompiler. nanoc3 should only ever
be reading files from the 'content' directory, thus not locking them.
It writes to files in the 'output' directory, but even then a file
should be closed automatically each time nanoc3 finishes writing to
it. Since the file you are editing is in the 'content' directory, it
is odd indeed that something is locking it. (Mac OS X is Unix-based:
its locking/file modification model is completely different, so it can
not be compared meaningfully against Windows.)
What is the exact error emitted by vim? Out of curiosity, have you
tried a different editor to see if the problem is universal or
specific to vim? Another question: Does the file become unlocked if
you stop the autocompiler?
You might also want to use Process Explorer
(http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx) to
determine definitively which program has the file locked.
-- ES
M.S.
to your questions:
Error in Vim is: E505: "index.html" is read-only (add ! to overwrite)
This error message occurs, when I want to save the content/index.html
via command ":w". However, I can force overwriting with command ":w!"
successfully.
I checked it with SuperEdi (http://www.wolosoft.com/de/superedi/)
additionally and this editor is not able to save the file at all with
some similar error description. So it seems not to be a Vim only
issue.
Process Explorer told me that content/index.html is used by a running
ruby.exe process (as the autocompiler is running currently).
When I start the autocompiler and haven't access the http://localhost:3000
yet, I have no problem in writing to this file. From the first browser
access on, the file is and keeps locked, even if the browser finished
loading the page. When I then quit the autocompiler, the file is not
locked anymore.
I did some deeper research with Process Monitor (also Sysinternals)
and it seems that ruby.exe process executes a CreateFile operation on
content/index.html (yes: not only on output/index.html) even if I just
update the browser (during running autocompiler). This happens even if
I haven't changed anything on the file content/index.html. But - and
this is what I am really wondering about - Process Monitor does not
show a CloseFile operation after the last CreateFile operation on that
file.
Maybe Vista can recognize and handle those problems (on OS level)
somewhat better than Win XP. But it would be interesting, if you could
see the same problem in Process Monitor on your machine too.
In Process Monitor I added following conditions to the filter:
Process Name is ruby.exe (include)
Path contains content\index.html (include)
Greets
Marco
> > * start browser withhttp://localhost:3000
Eric Sunshine
On 1/8/2010 1:53 AM, M.S. wrote:
> Process Explorer told me that content/index.html is used by a running
> ruby.exe process (as the autocompiler is running currently).
I can confirm this. In fact, ruby.exe has the file opened multiple
times, and keeps it opened. My observation is that every page load
(http://localhost:3000/) increases the number of open handles to
content/index.html by three, whether the file was modified or not.
> When I start the autocompiler and haven't access the http://localhost:3000
> yet, I have no problem in writing to this file. From the first browser
> access on, the file is and keeps locked, even if the browser finished
> loading the page. When I then quit the autocompiler, the file is not
> locked anymore.
Oddly, in my tests, even the most crude editor (notepad.exe) did not
have trouble writing to the file, though ruby.exe held it open.
> I did some deeper research with Process Monitor (also Sysinternals)
> and it seems that ruby.exe process executes a CreateFile operation on
> content/index.html (yes: not only on output/index.html) even if I just
> update the browser (during running autocompiler). This happens even if
> I haven't changed anything on the file content/index.html. But - and
> this is what I am really wondering about - Process Monitor does not
> show a CloseFile operation after the last CreateFile operation on that
> file.
It might be necessary to trace through the nanoc code to find out what
is triggering this anomalous condition. Perhaps Denis can give some
pointers as to where to start looking.
> Maybe Vista can recognize and handle those problems (on OS level)
> somewhat better than Win XP. But it would be interesting, if you could
> see the same problem in Process Monitor on your machine too.
> In Process Monitor I added following conditions to the filter:
> Process Name is ruby.exe (include)
> Path contains content\index.html (include)
Using Process Monitor, on each page access (http://localhost:3000/),
ruby.exe invoked CreateFile() six times but only invoked CloseFile()
three times, hence the above observation about it leaking three handles
on each occasion. (For readers not familiar with Windows API,
CreateFile() is employed here merely to open the file for reading. It is
not creating the file.) This could very well be a bug in ruby.exe or
some extension used by nanoc, or nanoc itself.
-- ES
Eric Sunshine
On 1/8/2010 4:26 PM, Eric Sunshine wrote:
> Using Process Monitor, on each page access (http://localhost:3000/),
> ruby.exe invoked CreateFile() six times but only invoked CloseFile()
> three times, hence the above observation about it leaking three handles
> on each occasion.
Further observations which may help narrow down where to look for the
problem. Following is the trace of a single http://localhost:3000/
access with autocompiler running. I present only the first one-third of
the trace since it merely repeats itself three times upon each access
(with the exception that QueryStandardInformationFile() appears only in
the first one-third of the trace).
CreateFile (OK) [open generic-read]
QueryStandardInformationFile (OK) [query file size]
QueryInformationVolume (OK) [query creation time]
QueryAllInformationFile (BUFFER OVERFLOW) [query creation access
write change times, file size, open mode -> read]
ReadFile (OK) [return all bytes(file-size)]
ReadFile (END OF FILE) [return 0 bytes(end of file)]
UnlockFileSingle (RANGE NOT LOCKED) [0-UINT_MAX]
CloseFile (OK)
QueryDirectory (OK) [filter:index.html, result:index.html]
CreateFile (OK) [open generic-read]
The first part of the trace seems fine. It opens the file, retrieves
information (modification time, etc.), reads the entire content, and
then closes the file. The next part, however, is where the leak occurs.
The directory is queried, and then the file (content/index.html) is
opened but never closed, hence the leak of three file handles upon each
http://localhost:3000/ access.
-- ES
Denis Defreyne
I’ve just pushed a possible fix for this issue. You can find it at <http://projects.stoneship.org/hg/nanoc/rev/914f541e560a> or <http://github.com/ddfreyne/nanoc/commit/40835b6c6740103876d9e8e6e03dc9769f6921c0> if you prefer git. This fix ensures that FileProxy closes files immediately after they are used; previously the File instances would stick around until the GC is run.
Can you check whether this change fixes the issue you are experiencing? You can clone the hg or the git repository; run the “./bin/nanoc3” executable in there instead of the normal “nanoc3” executable and check whether the issue’s gone.
Regards,
Denis
--
Denis Defreyne
denis.d...@stoneship.org
M.S.
I wasn't able to get the repository clone to work. It claims about "no
such file to load -- cri (LoadError)" when I start autocompile with
"ruby ..\..\Temp\nanoc\bin\nanoc3 autocompile" from the homepages root
path.
But I've patched file_proxy.rb in the nanoc3 default installation
inside the Ruby directory tree. After the change, it works fine now.
Now I haven't any problems with locked files during autocompile
anymore.
Thanks for the fix, Denis. And thanks to Eric too, for helping
isolating the problem.
Greets
Marco
On 9 Jan., 13:42, Denis Defreyne <denis.defre...@stoneship.org> wrote:
> Hi,
>
> I’ve just pushed a possible fix for this issue. You can find it at <http://projects.stoneship.org/hg/nanoc/rev/914f541e560a> or <http://github.com/ddfreyne/nanoc/commit/40835b6c6740103876d9e8e6e03dc...> if you prefer git. This fix ensures that FileProxy closes files immediately after they are used; previously the File instances would stick around until the GC is run.
>
> Can you check whether this change fixes the issue you are experiencing? You can clone the hg or the git repository; run the “./bin/nanoc3” executable in there instead of the normal “nanoc3” executable and check whether the issue’s gone.
>
> Regards,
>
> Denis
>
> --
> Denis Defreyne
Denis Defreyne
> Hi Denis,
>
> I wasn't able to get the repository clone to work. It claims about "no
> such file to load -- cri (LoadError)" when I start autocompile with
> "ruby ..\..\Temp\nanoc\bin\nanoc3 autocompile" from the homepages root
> path.
Hi,
You probably need `ruby -rrubygems ...` in order for this to work.
> But I've patched file_proxy.rb in the nanoc3 default installation
> inside the Ruby directory tree. After the change, it works fine now.
> Now I haven't any problems with locked files during autocompile
> anymore.
Ahh, good to know! This fix will be included in 3.0.5. :)
Regards,
Denis
--
Denis Defreyne
denis.d...@stoneship.org
Eric Sunshine
On 1/9/2010 7:42 AM, Denis Defreyne wrote:
> I�ve just pushed a possible fix for this issue. This fix ensures that
> FileProxy closes files immediately after they are used; previously
> the File instances would stick around until the GC is run. Can you
> check whether this change fixes the issue you are experiencing?
Indeed, I had also arrived at FileProxy as being the culprit, and
testing showed, as you found, that #respond_to? was being invoked and
leaking the file descriptor (until GC).
This auto-magical behavior of FileProxy seems essentially a hack. For
nanoc 3.1, perhaps consider retiring it altogether or at least requiring
that the client request a File object explicitly (possibly in the
context of a block so that it can be closed automatically). Alternately,
publish only the pathname of the file, again forcing the client to deal
with the file explicitly rather than implicitly.
-- ES
Denis Defreyne
> This auto-magical behavior of FileProxy seems essentially a hack. For nanoc 3.1, perhaps consider retiring it altogether or at least requiring that the client request a File object explicitly (possibly in the context of a block so that it can be closed automatically). Alternately, publish only the pathname of the file, again forcing the client to deal with the file explicitly rather than implicitly.
Hi,
You are correct. I should have removed Nanoc3::FileProxy from the initial 3.0 release, but I kept around for backward compatibility with nanoc 2.x (a stupid excuse, really). It is necessary because I wanted to have a :file attribute for each Nanoc3::Item object; this attribute would contain a File pointing to the content file. With lots of pages, using raw File objects is impossible due to the limit on the number of file descriptors you can have open at the same time, so Nanoc3::FileProxy was born.
I’m hesitant to introduce backwards-incompatible changes in a minor (x.Y.z) release, but I can provide alternatives to the :file attribute, which I can now deprecate in 3.1. Instead of providing a :file attribute that points to the content file, nanoc 3.1 should probably provide :content_filename and :meta_filename strings containing (absolute) paths to the relevant files. Something like this, perhaps: http://projects.stoneship.org/hg/nanoc/rev/1acd60eef807.
Eric Sunshine
On 1/10/2010 5:15 AM, Denis Defreyne wrote:
> It is necessary because I wanted to have a :file attribute for each
> Nanoc3::Item object; this attribute would contain a File pointing to
> the content file. With lots of pages, using raw File objects is
> impossible due to the limit on the number of file descriptors you can
> have open at the same time, so Nanoc3::FileProxy was born.
Given that the Item[:file] property does not seem to be documented
anywhere (tutorial, user manual, API, 2.x to 3.x migration), I wonder if
it can in fact be considered part of the 3.0.x API.
> I�m hesitant to introduce backwards-incompatible changes in a minor
> (x.Y.z) release, but I can provide alternatives to the :file
> attribute, which I can now deprecate in 3.1. Instead of providing a
> :file attribute that points to the content file, nanoc 3.1 should
> probably provide :content_filename and :meta_filename strings
> containing (absolute) paths to the relevant files. Something like
> this, perhaps:
> http://projects.stoneship.org/hg/nanoc/rev/1acd60eef807.
It might be worthwhile to emit a deprecation warning. If you feel
reasonably confident that FileProxy is used only internally, then
FileProxy#method_missing() could print a warning that Item[:file] is
deprecated. Alternately, it would be possible to augment the attributes
Hash constructed by the data source so that access to :file emits a
warning. A final approach, if one assumes that :file is not heavily used
(since undocumented), is to print nothing at all. :-)
-- ES
Denis Defreyne
> Given that the Item[:file] property does not seem to be documented anywhere (tutorial, user manual, API, 2.x to 3.x migration), I wonder if it can in fact be considered part of the 3.0.x API.
You may be right here, but I’d still rather not change the behaviour of functionality that is already present in nanoc. Recently, nanoc 3.0.3 introduced some backwards-incompatible changes because I saw something as a bug, while others were used to this functionality and saw it as a feature instead. So, to avoid such issues and keep 3.1 entirely backwards compatible with 3.0, I’d rather not remove the :file attribute; some people may rely on it.
> It might be worthwhile to emit a deprecation warning. If you feel reasonably confident that FileProxy is used only internally, then FileProxy#method_missing() could print a warning that Item[:file] is deprecated.
Good idea. I implemented exactly that. Here’s what it looks like:
“WARNING: The :file attribute is deprecated and will be removed in a future version of nanoc. Instead of using this :file attribute, consider creating a File object using the :content_filename or :meta_filename attributes (for filesystem_verbose and filesystem_compact) or :filename (for filesystem_combined).”
The :file attribute itself will be removed in the next major version of nanoc, which will probably be 4.0--but it’ll be a long time before nanoc 4.0 appears on the radar. :)