Uploading through staticsync, no files written to filestore

[BRUDmon]

Hi everyone,

We are evaluating RS for internal usage (within our network) so we set up a test box for this (Win7 x86, IIS7,5) but we ran into a strange issue. No Previews, thumbs other than the place holder graphic. We have absolutely no idea why it isn't working.

Our installtion process followed the wiki article for installation in win7 32 bit.

We have the filestore in:
C:\inetpub\wwwroot\resourcespace\filestore

Our syncdir is in:
C:\Pictures (this is just for the test setup)

Our folder matching is working as desired, we run staticsync through CLI.
The folders are read and all our desired fields are populated as expected.

BUT, there are NO preview-images, thumbs nothing besides the fields... all it shows is the placeholder graphic for missing files
I don't really know what's going on and need help. We've been hunting through the group for close to a day with now viable clue.

The installation check gives an error for ffmpeg, however I thought that doesn't affect us, since we're only working with graphics for now.
The error is as follows:
" Unexpected output when executing "c:/ffmpeg/bin\ffmpeg.exe" -version command. Output was ''. "
Everything else is ok, at least the check isn't telling anything other than the described error.

Do you have an idea what could be wrong?


t.i.a.

[Allison Stec]

a few things to check:

- make sure filestore is writable (this shows in the installation check, so I'm assuming this is OK)

In config.php:

- check to see if your file extensions are included in $no_preview_extensions

- check to see if $enable_thumbnail_creation_on_upload = true (this is the default, but if it's set to false you need to run create_preview.php to generate previews for recently uploaded images)

Do you use watermarks? Are all images set to be restricted? If yes then verify your path to the watermark is correct

--
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[BRUDmon]

Hi there,

a few things to check:

- make sure filestore is writable (this shows in the installation check, so I'm assuming this is OK)

Schreibzugriff auf C:\inetpub\wwwroot\resourcespace\include/../filestore    OK
(rough transl.: write acces to C:\inetpub\wwwroot\resourcespace\include/../filestore    OK)

Seems ok,  as I said the only thing not being ok is ffmpeg install and the 2Gb restriction due to 32bit php (which is fine for now).
 

In config.php:

- check to see if your file extensions are included in $no_preview_extensions

is set to =array("icm","icc"); and still in default.config.php
 

- check to see if $enable_thumbnail_creation_on_upload = true (this is the default, but if it's set to false you need to run create_preview.php to generate previews for recently uploaded images)

this is still in default.config.php, so no changes there 
 

Do you use watermarks? Are all images set to be restricted? If yes then verify your path to the watermark is correct

No watermarks in use.
Any other suggestions on this?


Thanks

[Allison Stec]

you need to check for those config settings in config.php NOT config.default.php.

config.php is meant to include any customized settings for your install.

config.default.php should not be changed because it will be overwritten during updates and you will lose your custom settings.

I would try using imagemagick from the command line and see if you get any errors.

--

[BRUDmon]

Hi Allison,

I'm using RS since 2010, I'm not new to it, just new to having it on a windows box (no we cannot use anything linux at our company).

you need to check for those config settings in config.php NOT config.default.php.

Yes we know that, and we also have our custom settings in config.php, and since we're not using "$no_preview_extensions" it remains in config.default.

I would try using imagemagick from the command line and see if you get any errors.

Imagemagick works as expected, though I only tried "convert XYZ.jpg XYZ.png" and that conversion worked. So no errors there.
 

[Allison Stec]

I did not mean to imply that you were a novice at anything. However, it is not enough to look at config.default.php for those settings, as they are overwritten by the settings in config.php.

have you tried to convert an image via the command line?

have you checked your error logs?

--

[BRUDmon]

 

I did not mean to imply that you were a novice at anything. However, it is not enough to look at config.default.php for those settings, as they are overwritten by the settings in config.php.

No offense, it's all good, I just wanted to point out that I'm aware of the relations between config.default and config.php, and I checked both files just in case it would have found its way into config.php, which it hasn't
 

have you tried to convert an image via the command line?

Yes, it worked. converted "XYZ.jpg" to "XYZ.png"

 

have you checked your error logs?

Which error logs do you mean? So far I only checked the server's log and that doesn't give any clue :(

Thanks

[Allison Stec]

you can check php's log, and i'd check the system log as well

--

[BRUDmon]

you can check php's log, and i'd check the system log as well

I have the windows box logging php to Windows Event Log.

The only error that shows up is an application error regarding ffmpeg.exe and I'm clueless as to why ffmpeg isn't working.
From my understanding the system log = windows event log, did I get it wrong?

[BRUDmon]

Alright, forget what I said about ffmpeg.exe, after using the one that comes with imagemagick, the error went away... still no previews/thumbs though. 

[David Dwiggins]

One thing to keep in mind is that staticsync runs from the command
line or the task scheduler, whereas images uploaded through the web
interface are created under the user that is running the web server.
So there are two different security contexts involved. If one has
access to the filestore and the other doesn't, this could cause a
problem.

You might figure out what user the staticsync script is running under,
start a command line session as that user, and then verify that you
can save copy files to the filestore.

-DD

[BRUDmon]

Hi David,
 

One thing to keep in mind is that staticsync runs from the command
line or the task scheduler, whereas images uploaded through the web
interface are created under the user that is running the web server.
So there are two different security contexts involved. If one has
access to the filestore and the other doesn't, this could cause a
problem.

All users (IUSR, Everyone) have read/write, admin has full rights on all directories.
 

You might figure out what user the staticsync script is running under,
start a command line session as that user, and then verify that you
can save copy files to the filestore. 

 

Well I can convert a file in the filestore (the one I uploaded manually) through imagemagick runing cmd.exe as administrator. So I assume staticsync uses the same user (admin).

Any other thoughts?

[David Dwiggins]

If you enable

# Log developer debug information to the debug log (filestore/tmp/debug.txt)?
$debug_log=true;

The system will create a debugging log in your filestore temp folder.
You can do this, run staticsync, and then examine the log file. Among
other things, it will log the actual commands sent to the command line
to execute imagemagick. You can then try running these yourself from
the command line to see if they work. If not, you may get more helpful
errors back that would point to the source of the problem.

Once you have resolved the issue, you should probably turn this
logging off again, since it creates quite a lot of output.

-David

[BRUDmon]

If you enable

# Log developer debug information to the debug log (filestore/tmp/debug.txt)?
$debug_log=true;

Right, that's probably the "system log" Allison meant.
 

The system will create a debugging log in your filestore temp folder.
You can do this, run staticsync, and then examine the log file. Among
other things, it will log the actual commands sent to the command line
to execute imagemagick. You can then try running these yourself from
the command line to see if they work. If not, you may get more helpful
errors back that would point to the source of the problem.

All I get from this file is tons of SQL commands, how do I distinguish those from imagemagick.
One thing I caught was this:

SQL: select *,mapzoom from resource where ref='328'
(ref=328,thumbonly=,extension=jpg,previewonly=,previewbased=,alternative=-1)
SQL: update resource set file_checksum='' where ref='328'
SQL: update resource set preview_tweaks = '0|1' where ref = '328'
File source is C:\inetpub\wwwroot\resourcespace\include/../filestore/3/2/8_824210fa2d00c45/328_e0e13230bc275d5.jpg

But that tells me nothing :(

[David Dwiggins]

Look for lines that start

CLI command:

and

CLI output:
or
CLI errors:

The first should show you command line commands that were run, and the
second and third should show you any output or errors that the command
produced.

[BRUDmon]

Look for lines that start

CLI command:

and

CLI output:
or
CLI errors:

 
Sadly there's nothing like that in the debug.txt, can't find anything with "CLI", "errors" can't even find "command".

I thought maybe it's a database problem because we're testing with a small set of our data (deleting resources/purging filestore/dropping all resource tables after each run), but that didn't help either.

[BRUDmon]

Hi everyone,

We are one step closer to a solution, but it remains strange.

Today we tried something else, we used another subdir for staticsync (one that has been excluded through nogo"" up until now, it does have the same sturcture and similar content) and those files do show up, with all resolutions.

Until today we were testing with one subdir, during our testing we deleted the files from RS and filestore and dropped the resource tables several times. Could it be that RS keeps relations or checksums somewhere else?
I somehow have the impression that it kinda "knows" those files and doesn't like them.

Furthermore staticsync now put out some new entries at the end of the run, stating:
"Looking for deleted files.... file no longer exists...C:path"...
and it lists the broken resources/files that only showed place holder graphics.

Look for lines that start

CLI command:

and

CLI output:
or
CLI errors:

Now I can find those, but only for the new subdir, the old one doesn't show in the log. Also, the log has no entries in the error-lines. 

[BRUDmon]

On further checking, we found that German Umlauts could be responsible for the strange behavior of staticsync.

Out of curiosity we changed all subdirs that contained Umlauts in their name to substitutes like "Oe" for "ö", after that change everything ran fine.

So the new question would be: What do we have to do, to get staticsync to work correctly with subdirs that contain Umlaut-characters?
And please be specific, just telling me "set charset to utf-8" won't help as I guess there will be more than one file to adjust.


t.i.a.

[BRUDmon]

One more thing is, that setting

$mysql_charset="utf8" in config.php

also prevents files from being uploaded, but if we comment it out it works.
With the slight problem that Umlauts cannot be displayed in the browser, wich is a big no no for us.

What else can we do, any ideas?

[Benjamin Bailes]

Out of curiosity, is your database configured as Latin1 or UTF-8?  We had terrible trouble with special characters until we converted our database to UTF-8.  The default in mySQL is Latin1, and sadly there remains no mention of this in the RS documentation.

I don't know if it will fix your problem with Staticsync or not, but your mention of umlauts reminded me of my experience with this.  The utf8 designation in config.php doesn't change the character set of your mySQL database.

A shot in the dark (from a user, not a developer), but I'm just trying to be helpful.

Ben

[BRUDmon]

Hi Ben,
 

Out of curiosity, is your database configured as Latin1 or UTF-8?  We had terrible trouble with special characters until we converted our database to UTF-8.  The default in mySQL is Latin1, and sadly there remains no mention of this in the RS documentation.

We already converted all tables/the database to utf-8, but the behavior remains the same.

We know that the designation in config.php doesn't change the charset of the db.
But again, if we set "mysql_charset=utf8" nothing gets uploaded/converted (though the metadata-fields are written, just without any pictures), if we comment it out it loads/converts all files correctly. But our metadata is displayed with funny looking characters  (no umlauts displayed).

I don't know if it will fix your problem with Staticsync or not, but your mention of umlauts reminded me of my experience with this.  The utf8 designation in config.php doesn't change the character set of your mySQL database.

A shot in the dark (from a user, not a developer), but I'm just trying to be helpful.

Doesn't matter whether you're a dev or not, we appreciate it. 
A workaround would be to disable mysql_charset during the staticsync run and enabling it once everything is in place, but that's hardly a feasible workflow to show RS's capabilities.

Anyways, thank you for the suggestion (I should have mentioned that we already tried that). But as I said, every bit of information is appreciated.

[Brian Rossmajer]

On Thursday, February 14, 2013 11:08:23 AM UTC-5, BRUDmon wrote:

Doesn't matter whether you're a dev or not, we appreciate it.  

A workaround would be to disable mysql_charset during the staticsync run and enabling it once everything is in place, but that's hardly a feasible workflow to show RS's capabilities.

Anyways, thank you for the suggestion (I should have mentioned that we already tried that). But as I said, every bit of information is appreciated.

I had a similar problem on a unix install a while back, and it turned out to be related to how ResourceSpace forms the conversion command line in the case of non-ascii characters, or in how the conversion utility read its command line arguments.  If memory serves, I added a bit of debug to run_command in include/general.php so that I could see what commands were being run to confirm that was the problem.  It was something along the lines of if ( posix_isatty(STDERR) ) { fwrite(STDERR,$command);} 

To fix it, in image_processing.php  I created a hard link to the actual file, but using a temporary ascii filename, making sure to keep the extension the same.  Then I passed the temporary file instead for conversion.

It might not be the problem you're seeing, but I hope this helps!

I made a lot of use of batch/create_previews.php to recreate the missing thumbs, after disabling the threading, which did not work well for me.

[Maria T]

I posted about this a while back and experienced exactly the same problems as you with umlauts:

ghostscript and staticsync

- pdf previews are generated when uploading files by java upload because the file path is scrambled. However when files are imported via static sync it breaks when it encounters an umlaut. I am assuming the problems lie with RS (or perhaps, my installation of RS) as Ghostscript is supposed to support UTF-8 (or at least that's what I read; I have not tested this).

Comparing the debug output, with mysql_charset="UTF8" set to on vs off, it breaks at the following points:

- SQL: select file_path value from resource where ref='1' etc

- SQL: update resource set preview_tweaks (i.e. just before GS is supposed to start)

- it seems to think file source is: filestore/1_[etc]/1_[etc].pdf (i.e. that the file has been ingested, when in actual fact there is nothing in that folder)

With mysql_charset commented out, GS breaks when trying to create previews, e.g.:

- 'PDF multi page preview [....] /var/www/Analysis/hpihh/dradio.pdf', even though it recognises that: 'file source is var/www/Analysis/hpäihh/dradio.pdf'

[BRUDmon]

Hi Brian,

I had a similar problem on a unix install a while back, and it turned out to be related to how ResourceSpace forms the conversion command line in the case of non-ascii characters, or in how the conversion utility read its command line arguments.  If memory serves, I added a bit of debug to run_command in include/general.php so that I could see what commands were being run to confirm that was the problem.  It was something along the lines of if ( posix_isatty(STDERR) ) { fwrite(STDERR,$command);} 

To fix it, in image_processing.php  I created a hard link to the actual file, but using a temporary ascii filename, making sure to keep the extension the same.  Then I passed the temporary file instead for conversion.

It might not be the problem you're seeing, but I hope this helps!

We do not have any real developers here and I don't think I should do anything to the code (not a developer myself).
So that's no real alternative for us, but what that does is point to something like a bug.

Staticsync isn't working with Umlauts, which means staticsync doesn't work with UTF-8.
It is not usable this way, and I have no way of correcting/fixing it myself. 
Could somebody please add this to the bug report?
 

I made a lot of use of batch/create_previews.php to recreate the missing thumbs, after disabling the threading, which did not work well for me.

Recreating the thumbs by means of create_previews is a workaround, but not better than just switching utf-support off-/and on before and after staticsync runs. But again, this is hardly something you'd want to show, while presenting RS as your preferred DAM solution. 

UTF-8 support is crucial, for almost every language other than English and I'm having a hard time believing that nobody else ran into the same problems (it's not like german is the only language to use special characters).

Thanks for the clues!

[BRUDmon]

Hi Maria,

On Friday, February 15, 2013 5:36:37 PM UTC+1, Maria T wrote:

I posted about this a while back and experienced exactly the same problems as you with umlauts:

We read your thread and it seems to be related. Your description matches ours, so I guess there could be a bug.
 

ghostscript and staticsync

- pdf previews are generated when uploading files by java upload because the file path is scrambled. However when files are imported via static sync it breaks when it encounters an umlaut. I am assuming the problems lie with RS (or perhaps, my installation of RS) as Ghostscript is supposed to support UTF-8 (or at least that's what I read; I have not tested this).

Comparing the debug output, with mysql_charset="UTF8" set to on vs off, it breaks at the following points:

- SQL: select file_path value from resource where ref='1' etc

- SQL: update resource set preview_tweaks (i.e. just before GS is supposed to start)

- it seems to think file source is: filestore/1_[etc]/1_[etc].pdf (i.e. that the file has been ingested, when in actual fact there is nothing in that folder)

I can see similar things in the debug.txt, but cannot tell what's really happening.
 

With mysql_charset commented out, GS breaks when trying to create previews, e.g.:

- 'PDF multi page preview [....] /var/www/Analysis/hpihh/dradio.pdf', even though it recognises that: 'file source is var/www/Analysis/hpäihh/dradio.pdf'

IM doesn't seem to be affected, when mysql_charset is commented out, that's why I said it could be a workaround (commenting it out and back in after staticsync run), although a really nasty one.

I could recreate the same behaviour on a VM with the bitnami-stack installed - a vanilla install, with only staticsync configured - with the exact same problem. As soon as it finds umlauts in the path, it breaks.

In my opinion this qualifies as a BUG, but I'm not a developer and cannot do anything about it.

Anybody else an idea?
Would it help if I created some demo-data so that others could try to reproduce this in their (test-)environments?

I'm at a loss here and don't know what to do.

[BRUDmon]

As I am at a loss and don't know how to troubleshoot this issue, I'll post some demo-data.

In the zip-file you'll find:

• a text file with our configuration of staticsync
• a file folder with our folderstructure (including umlauts within paths).

If you happen to have a test system up and running, using utf-8 encoding, please try our demo-data with staticsync (you probably only have to adapt the syncdir) and please report back whether you're as lucky as we are.

Can I do anything else?

[Maria T]

Anyone had a chance to look at this? Thanks, Maria

[BRUDmon]

BUMP.

I didn't have time yet to re-check this under linux. Sadly nobody seems to notice nor confirm this issue, so we'll leave it at that for now. I will try to check the behaviour under linux, but expect it to be the same.

So far our conclusion is: staticsync does not support UTF-8 properly. Do not use, when you have Umlauts in your paths.

[PaulANormanNZ]

In case any one is still following this through...

This sounds like a php/MySql problem rather than just RS as such perhaps?

As you have picked up, looks like RS is telling WIndows 7 to work on folder names that don't match what php is passing to it, as the umlauts have been converted/mangled some where on the fly possibly into multi part Ascii or look like multipart binary to Windows?

1.  I do not work with Windows 7 - so a question from my ignorance - what language character settings does Windows 7 use for German users, for making folder names? utf-8 all the way - or a Latin encoding?

2. Related to above ... What does Windows 7 DOS shell for php CLI, do to utf-8 passed to it for system commands? (There have been issues with previous versions of Windows I think.)

3. Have any html entities shown up in any logs or in MySql?

4. I've struck php/MySql things like this before...

http://webmonkeyuk.wordpress.com/2011/04/23/how-to-avoid-character-encoding-problems-in-php/

http://stackoverflow.com/search?q=umlaut+php+mysql

http://stackoverflow.com/questions/5984647/german-umlauts-in-mysql-insert?rq=1

http://stackoverflow.com/questions/1053258/german-umlaute-in-mysql-phpmyadmin?rq=1

Paul

[Maria T]

On a slightly unrelated note, I am working with Linux and am encountering exactly the same problem... I've posted more about it in this thread:

https://groups.google.com/d/msg/resourcespace/ArQarmhgPtU/1_AH5-TEOYQJ

Thanks, Maria

[Paul A Norman]

Just to clarify on point 2. in previous post:

...2. Related to above ... What does Windows 7 DOS shell for php CLI, do to utf-8 passed to it for system commands? (There have been issues with previous versions of Windows I think.)

chcp 65001  is covered here: 

http://stackoverflow.com/questions/388490/unicode-characters-in-windows-command-line-how

[Paul A Norman]

@Maria T, it appears that this is more widespread than was at first obvious...

Schrödinger's 😻 and outside-the-box naming

http://lwn.net/Articles/545741/

"...The problem, of course, is that unlike previous Fedora release names, "Schrödinger's Cat" contains some characters outside of the basic Latin alphabet: an o with umlaut and an apostrophe. But the specific issue encountered in the wild is even more specific than that; the "apostrophe" in question is frequently typed as the similar-looking but different single-quote character, and quotes can wreak havoc when the release name is processed by a shell script..."

[BRUDmon]

I found this here: http://bugs.mysql.com/bug.php?id=66682 ,but don't have time to check/confirm that we're affected by this.

I will update our MySql and test again, I doubt however that it'll make a difference. Kinda doesn't make sense, after Maria T has a similar problem on a linux install.

It could be several weeks before I can get back to this.

Are there really so few windows-users working with RS?
I thought somebody else would be able to just chip in and you know try it for themselves.

Also not sure what to make of http://lwn.net/Articles/545741/, all I get from this is, that fedora uses utilities that don't support utf-8 fully and that they decided on a workaround so that their tools accept the release-name ("schrödinger´s" Cat instead of "schrödinger's cat").

.

[martin jehnichen]

May be its a bit late to answer - but newer users like me might still look for an answer..
I had the same problem and found out, that resources from subfolders will be processed correctly. Previews are generated and metadata extracted when the images are placed in subfolders.
Only the resources form the basefolder only show the placeholder. I do not know yet why and how to change this behaviour but will post ist as sonn i found out - unles someone else knows why this happens.

By the way: staticsync.php delivered with version 7_1_6513  does not work- maybe missing brackets but the one from 7_0_6244 does work.

Am Mittwoch, 13. Februar 2013 13:17:09 UTC+1 schrieb BRUDmon:

Hi everyone,

We are evaluating RS for internal usage (within our network) so we set up a test box for this (Win7 x86, IIS7,5) but we ran into a strange issue. No Previews, thumbs other than the place holder graphic. We have absolutely no idea why it isn't working.

Our installtion process followed the wiki article for installation in win7 32 bit.

We have the filestore in:
C:\inetpub\wwwroot\resourcespace\filestore

Our syncdir is in:
C:\Pictures (this is just for the test setup)

Our folder matching is working as desired, we run staticsync through CLI.
The folders are read and all our desired fields are populated as expected.

BUT, there are NO preview-images, thumbs nothing besides the fields... all it shows is the placeholder graphic for missing files
I don't really know what's going on and need help. We've been hunting through the group for close to a day with now viable clue.

The installation check gives an error for ffmpeg, however I thought that doesn't affect us, since we're only working with graphics for now.
The error is as follows:
" Unexpected output when executing "c:/ffmpeg/bin\ffmpeg.exe" -version command. Output was ''. "
Everything else is ok, at least the check isn't telling anything other than the described error.

Do you have an idea what could be wrong?


t.i.a.