FW: Rev 1217 of Qubit - more mysteries, login details

[Peter Van Garderen]

Hey Guys, I hope you don't mind but I'm steering this discussion back onto
qubit-dev so that the details are searchable for anyone dealing with similar
issues in the future.

> Anyways, here's what's in Filesystem.php now from lines 189 through to 199
>
> if (!@chmod($this->_dirPath . '/' . $filename,
> self::$_defaultFilePermissions)) {
> var_dump($this->_dirPath . '/' . $filename);
> {
> ini_set('track_errors', $trackErrors);
> require_once 'Zend/Search/Lucene/Exception.php';
> throw new Zend_Search_Lucene_Exception($php_errormsg);
> }
> ini_set('track_errors', $trackErrors);
>
> return $this->_fileHandlers[$filename];
> }
>
> What I get now, after trying to create a new information object is this
> error:
>
> Parse error: syntax error, unexpected T_PUBLIC in
>
/home/david07/public_html/qubit/plugins/sfLucenePlugin/lib/vendor/Zend/Searc
> h/Lucene/Storage/Directory/Filesystem.php on line 209

No problem - you have two opening braces: one on line 189 and one on
line 191 - you still need to delete line 191

> Lines 202 through to 209 say
>
> /**
> * Removes an existing $filename in the directory.
> *
> * @param string $filename
> * @return void
> * @throws Zend_Search_Lucene_Exception
> */
> public function deleteFile($filename)
>
> So, what I'm now wondering, as there's something I hadn't mentioned, is
> whether this has to do with an Apache ownership issue. I noticed in some
of
> the directories that there's a mixture of owners on some of the files,
some
> are david07/david07 and others are nobody/nobody. I can't recall offhand
in
> which directories I saw this.

Ahh! This is probably it! I saw the mixed ownership, but the permissions
looked fine, so I didn't understand why it complains. I bet your files
are world readable / writable, but only the file owner can change
permissions? So when you installed Qubit, it was probably as the user
"david07", but Apache and PHP are probably running as the user "nobody".

The weird thing is that the call to chmod() is in a function
(createFile()) which creates a brand new file. So that file should be
owned by whatever user PHP is running as, hence PHP should have enough
permission to chmod().

The exception is probably thrown when calling createFile() on
data/index/en/read.lock.file - like you guessed - but until I see the
output of var_dump(), we're guessing. If it is
data/index/en/read.lock.file, then the directory listing you sent shows
me it's not only world readable / writable, it's also owned by nobody /
nobody, which is probably what PHP is running as. So there's no way PHP
should not have enough permission to chmod()

So I'm thinking it *must* be trying to chmod() some other file. If you
can send the var_dump() output, we'll know.

[dmat...@shaw.ca]

On Thu, 2008-07-10 at 13:22 -0700, D.H. Mattison wrote:
> July 10, 2008
>
> Hi Jack,
>
> Sorry to report, but this is still not working the way I think you
> think it's supposed to work. I think the best thing is if you could
> modify the Filesystem.php file for me to what you think it should be
> to work, e-mail it to me and I'll upload it. Should have thought of that first.
>
> Here's the latest error, still not showing, I don't think, the
> var_dump output you want. What I'm doing is editing and uploading the
> Filesystem.php file, clearing the symfony cache, and trying to add a
> new information object, which then results in this error screen.

No - you totally got it : ) I navigated to your site, tried creating a
test information object, and up in the top left corner I see:

string(60)
"/home/david07/public_html/qubit/data/index/en/read.lock.file"

So that *is* the file it is trying to chmod - weird...

I guess we're not quite done - you can try adding these two lines
after the var_dump() on line 190:

exec('whoami', $output);
var_dump($output);

- so the file should now look like:

global $php_errormsg;
$trackErrors = ini_get('track_errors');
ini_set('track_errors', '1');

if (!@chmod($this->_dirPath . '/' . $filename, self::
$_defaultFilePermissions)) { var_dump($this->_dirPath . '/' .

$filename); exec('whoami', $output); var_dump($output);

ini_set('track_errors', $trackErrors);
require_once 'Zend/Search/Lucene/Exception.php';
throw new Zend_Search_Lucene_Exception($php_errormsg);

I can attach the file if you prefer.

This should show you the user PHP is running as, in addition to the
file it's trying to chmod.

Finally, lets take another look at:

ls -la /home/david07/public_html/qubit/data/index/en

That will show you the owner and permissions of read.lock.file

I know you sent the directory listing before - I just wanna check it
against the user running PHP, in case the filesystem permissions have
changed since the last directory listing you sent...

[remainder deleted by DavidM]

[dmat...@shaw.ca]

Directory list as requested in previous message; this is AFTER a
couple of days ago, I chown'ed all files & directories from
david07:david07 to nobody:nobody and back again to david07:david07:

ls -la /home/david07/public_html/qubit/data/index/en
total 16
drwxrwxrwx 2 david07 david07 4096 Jul 8 20:15 .
drwxrwxrwx 7 david07 david07 4096 Jul 7 19:47 ..
-rw-rw-rw- 1 david07 david07 0 Jul 11 18:57 read.lock.file
-rw-rw-rw- 1 david07 david07 20 Jul 7 22:10 segments.gen
-rw-rw-rw- 1 david07 david07 20 Jul 7 22:10 segments_1
-rw-rw-rw- 1 david07 david07 0 Jul 8 20:19 write.lock.file

[dmat...@shaw.ca]

To brace or not to brace, that is the question:

My PSPad editor is pointing out that there's an extra, unpaired brace
if I follow the file contents you included in your earlier message.
This is what yours is (note the brace before var_dump):

global $php_errormsg;
$trackErrors = ini_get('track_errors');
ini_set('track_errors', '1');
if (!@chmod($this->_dirPath . '/' . $filename, self::
$_defaultFilePermissions)) { var_dump($this->_dirPath . '/' .
$filename); exec('whoami', $output); var_dump($output);
ini_set('track_errors', $trackErrors);
require_once 'Zend/Search/Lucene/Exception.php';
throw new Zend_Search_Lucene_Exception($php_errormsg);

This is what I've used for this section

public function createFile($filename)
{
if (isset($this->_fileHandlers[$filename])) {
$this->_fileHandlers[$filename]->close();
}
unset($this->_fileHandlers[$filename]);
$this->_fileHandlers[$filename] = new
Zend_Search_Lucene_Storage_File_Filesystem($this->_dirPath . '/' .
$filename, 'w+b');

global $php_errormsg;
$trackErrors = ini_get('track_errors');
ini_set('track_errors', '1');
if (!@chmod($this->_dirPath . '/' . $filename, self::
$_defaultFilePermissions))
var_dump($this->_dirPath . '/' . $filename);
exec('whoami', $output);
var_dump($output);
ini_set('track_errors', $trackErrors);
require_once 'Zend/Search/Lucene/Exception.php';
throw new Zend_Search_Lucene_Exception($php_errormsg);

ini_set('track_errors', $trackErrors);

return $this->_fileHandlers[$filename];
}

Thanks,

DavidM

[dmat...@shaw.ca]

Here is the output of the additional two lines to the Filesystem.php
file:

string(60) "/home/david07/public_html/qubit/data/index/en/
read.lock.file" array(1) { [0]=> string(6) "nobody" }
Warning: Cannot modify header information - headers already sent by
(output started at /home/david07/public_html/qubit/plugins/
sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Storage/Directory/
Filesystem.php:190) in /home/david07/public_html/qubit/lib/vendor/
symfony/lib/exception/sfException.class.php on line 70

[dmat...@shaw.ca]

In an effort to resolve this, I chown'ed recursively from
david07:david07 to nobody:nobody in data/index/en/, cleared the
symfony cache, tried creating a new information object, but this had
no effect and I'm still getting an error. Instead of complaining about
not being able to access the en directory, though, this is all that's
being output:

array(1) { [0]=> string(6) "nobody" }
Warning: Cannot modify header information - headers already sent by
(output started at /home/david07/public_html/qubit/plugins/
sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Storage/Directory/

Filesystem.php:192) in /home/david07/public_html/qubit/lib/vendor/
symfony/lib/exception/sfException.class.php on line 70

I chown'ed back that directory hierarchy back to david07:david07,
cleared the symfony cache, and tried creating a new information
object. The error that results is:

string(60) "/home/david07/public_html/qubit/data/index/en/
read.lock.file" array(1) { [0]=> string(6) "nobody" }
Warning: Cannot modify header information - headers already sent by
(output started at /home/david07/public_html/qubit/plugins/
sfLucenePlugin/lib/vendor/Zend/Search/Lucene/Storage/Directory/
Filesystem.php:190) in /home/david07/public_html/qubit/lib/vendor/
symfony/lib/exception/sfException.class.php on line 70

Thanks,

DavidM

[Jack Bates]

Hmm... On my machine the brace is required, or I get a syntax error.
Here is my version of the file: http://cgi.sfu.ca/~jdbates/tmp/qubit/200807120/Filesystem.phps

(The .phps extension is so you can to download it, instead of the
server trying to evaluate it)

The different behavior depending on whether files are recursively
owned by "david07" or "nobody" is interesting: This indicates that in
the case of "david07" the "if" statement is evaluating true, and in
the case of "nobody" it is evaluating false. You see an exception in
both cases because, without a brace the "if" statement only applies to
the immediately following statement: The first var_dump(). The second
var_dump() is evaluated and the exception thrown in all cases. This
leads me to believe that if you "chown -R nobody:nobody data/index"
and "svn revert plugins/sfLucenePlugin/lib/vendor/Zend/Search/Lucene/
Storage/Directory/Filesystem.php" (to discard our debugging edits) the
problem should be solved.

On closer thought, this seems like a bug in Zend_Search_Lucene: If
Qubit can read / write to index files, it should not complain about
not being able to chmod() them.

I see it has already been reported twice:

http://framework.zend.com/issues/browse/ZF-3500

http://framework.zend.com/issues/browse/ZF-2779

I'll add a reference to this thread to the bug report...

[Jack Bates]

Added: http://framework.zend.com/issues/browse/ZF-2779?focusedCommentId=22864#action_22864

[dmat...@shaw.ca]

Thanks Jack. No need to download your file at this point. I chown -R
nobody:nobody data/index. I already had an unmodified, renamed version
of Filesystem.php, so I renamed the modified version and changed the
unmodified one back to its original name. I cleared the symfony cache
and tried add a new information object. Progress! The error I'm
getting is now:

Fatal error: Maximum execution time of 30 seconds exceeded in /home/
david07/public_html/qubit/plugins/sfLucenePlugin/lib/vendor/Zend/
Search/Lucene/Storage/File/Filesystem.php on line 179

The item-level description I created was, however, added. I was logged
in under qubit_dev.php.

When I try to add a new information object via /qubit/web/index.php, I
end up with a blank screen at the URL http://www.davidmattison.ca/qubit/web/informationobject/update

I haven't had this timeout problem before.

I think you've partially solved the problem though in that I'm now
able to delete "extra", duplicate information objects that got added
during my experiments when I refreshed the error page. Previously,
with this problem, I wasn't able to delete these duplicate information
objects.

[dmat...@shaw.ca]

My guess on this timeout problem is that there's now a conflict
between the data/index and the data/index/en + index files. Here's the
directory listing for data/index/en:

ls -l data/index/en
total 68
-rw-rw-rw- 1 nobody nobody 714 Jul 12 20:54 _0.cfs
-rw-rw-rw- 1 nobody nobody 6 Jul 12 20:54 _0.frq
-rw-rw-rw- 1 nobody nobody 6 Jul 12 20:54 _0.prx
-rw-rw-rw- 1 nobody nobody 35 Jul 12 20:54 _0.tii
-rw-rw-rw- 1 nobody nobody 75 Jul 12 20:54 _0.tis
-rw-rw-rw- 1 nobody nobody 714 Jul 12 20:58 _1.cfs
-rw-rw-rw- 1 nobody nobody 6 Jul 12 20:58 _1.frq
-rw-rw-rw- 1 nobody nobody 6 Jul 12 20:58 _1.prx
-rw-rw-rw- 1 nobody nobody 35 Jul 12 20:58 _1.tii
-rw-rw-rw- 1 nobody nobody 75 Jul 12 20:58 _1.tis
-rw-rw-rw- 1 nobody nobody 761 Jul 12 21:05 _2.cfs
-rw-rw-rw- 1 nobody nobody 8 Jul 12 21:05 _2.frq
-rw-rw-rw- 1 nobody nobody 8 Jul 12 21:05 _2.prx
-rw-rw-rw- 1 nobody nobody 35 Jul 12 21:05 _2.tii
-rw-rw-rw- 1 nobody nobody 107 Jul 12 21:05 _2.tis
-rw-rw-rw- 1 nobody nobody 0 Jul 12 21:05 optimization.lock.file
-rw-rw-rw- 1 nobody nobody 0 Jul 12 21:05 read.lock.file
-rw-rw-rw- 1 nobody nobody 20 Jul 7 22:10 segments.gen
-rw-rw-rw- 1 nobody nobody 20 Jul 12 21:05 segments_1
-rw-rw-rw- 1 nobody nobody 0 Jul 12 21:05 write.lock.file

How would I go about regenerating or updating the index? Is this even
possible once the initial index has been created?

I tried finding this information at http://trac.symfony-project.org/wiki/sfSearchPlugin/QuickStart
but it only tells you how to populate the index initially, not how to
update it.

[Tristan]

Hi, there is a patch for this issue here under attachments:

http://framework.zend.com/issues/browse/ZF-2779

Basically Zend_Lucene tries to chmod the file even if its already been
created, this patch solves this.

Cheers, Tristan.

On Jul 11, 5:52 pm, "Peter Van Garderen" <pe...@artefactual.com>
wrote:

>  signature.asc
> 1KDownload

[dmat...@shaw.ca]

Thanks very much for pointing out the patch. I patched the
Filesystem.php file by hand, cleared the symfony cache, and tried
adding a new information object, but am still getting the timeout
error. Here's what my patched Filesystem.php looks like for that
section:

public function createFile($filename)
{
$fileExists = file_exists($this->_dirPath . '/' . $filename);

if (isset($this->_fileHandlers[$filename])) {
$this->_fileHandlers[$filename]->close();
}
unset($this->_fileHandlers[$filename]);
$this->_fileHandlers[$filename] = new
Zend_Search_Lucene_Storage_File_Filesystem($this->_dirPath . '/' .
$filename, 'w+b');

global $php_errormsg;
$trackErrors = ini_get('track_errors');
ini_set('track_errors', '1');

if (!$fileExists && !@chmod($this->_dirPath . '/' . $filename,
self::$_defaultFilePermissions)) {

ini_set('track_errors', $trackErrors);
require_once 'Zend/Search/Lucene/Exception.php';
throw new Zend_Search_Lucene_Exception($php_errormsg);
}

I've tried various combinations of chown between david07:david07 and
nobody:nobody on data/index/en and their files, but none have removed
the timeout problem. The information objects, as they're called in the
Qubit Toolkit, do get added though.

[dmat...@shaw.ca]

Since we seem to be at an impasse over this problem, do you think I
should just do a new install given that I've mucked around with Apache
owner:group permissions?

Can anyone in the Qubit team confirm whether the patch has been
applied and will stick to the Filesystem.php file?

Thanks,

DavidM

[dmat...@shaw.ca]

Because of problems with directory permissions using the Web installer
and leaving my installation completely readable, I went back to the
svn checkout routine in order to try and resolve the issue around the
creation of a new information object and ending up with an error
screen around a directory permission in the search index that looks
like this at the start:

[Zend_Search_Lucene_Exception]
chmod() [<a href='function.chmod'>function.chmod</a>]: Operation not
permitted
stack trace

at ()
in SF_ROOT_DIR/plugins/sfLucenePlugin/lib/vendor/Zend/Search/Lucene/
Storage/Directory/Filesystem.php line 192 ...
if (!@chmod($this->_dirPath . '/' . $filename, self::

$_defaultFilePermissions)) {
ini_set('track_errors', $trackErrors);
require_once 'Zend/Search/Lucene/Exception.php';
throw new Zend_Search_Lucene_Exception($php_errormsg);
}

ini_set('track_errors', $trackErrors);
at Zend_Search_Lucene_Storage_Directory_Filesystem-
>createFile('read.lock.file')

in SF_ROOT_DIR/plugins/sfLucenePlugin/lib/vendor/Zend/Search/Lucene/
LockManager.php line 88 ...
*/
public static function
obtainReadLock(Zend_Search_Lucene_Storage_Directory $lockDirectory)
{
$lock = $lockDirectory->createFile(self::READ_LOCK_FILE);
if (!$lock->lock(LOCK_SH)) {
throw new Zend_Search_Lucene_Exception('Can\'t obtain shared reading
index lock');

}

I determined that the Filesystem.php file, which is supposed to be the
offending one that's causing this error, has not yet been patched in
Qubit rev 1313, so I applied the patch manually.

Without clearing the symfony cache, I created a new information object
and this is the error that resulted, which is different than above,
and definitely shows that it's a permissions issue on the /data/index/
en/:

[Zend_Search_Lucene_Exception]
fopen(/home/david07/public_html/qubit2/data/index/en/_4o.sti) [<a
href='function.fopen'>function.fopen</a>]: failed to open stream:
Permission denied
stack trace

* at ()
in SF_ROOT_DIR/plugins/sfLucenePlugin/lib/vendor/Zend/Search/
Lucene/Storage/File/Filesystem.php line 69 ...
66.
67. if ($this->_fileHandle === false) {
68. ini_set('track_errors', $trackErrors);
69. throw new
Zend_Search_Lucene_Exception($php_errormsg);
70. }
71.
72. ini_set('track_errors', $trackErrors);

One of the previous suggestions in another thread was to chown to
nobody:nobody that directory.

Clearing the symfony cache and trying to create a new information
object has no effect on this problem.

Thanks,

DavidM

[dmat...@shaw.ca]

I've now resolved this error

[Zend_Search_Lucene_Exception]
fopen(/home/david07/public_html/qubit2/data/index/en/_4o.sti) [<a
href='function.fopen'>function.fopen</a>]: failed to open stream:
Permission denied

by chmod -R /data/index/en . I did not chown -R the files in /data/
index/en so they are nobody:nobody. I am still getting an error/
warning screen about headers being sent, etc., but I am returned to
the newly created information object.
Hooray!

Thanks,

DavidM

[David Juhasz]

Hi David M.

I've opened Issue #389 to add a section to the installer documentation on setting directory permissions so other users can benefit from your issues with getting the permissions right.

Thanks for your patience and persistence in working through the permissions, I'm sure it's been pretty frustrating at times.

Regards,
David J

[dmat...@shaw.ca]

Frustrations are over for the time being as thanks to your
instructions in Issue 375 I was able to complete a successful Web
installation of the 1.0.1 ICA-AtoM tarball at http://www.davidmattison.ca/subatomic101/web