Resourcespace / NFS / Large batch uploads

[JoshM]

Hello all,

We use resource space at my school to upload collections of video /
audio and image files in batches. Sometimes as large as 200+ files per
collection.

I am running it on a linux box, 4gb ram, dual core intel CPU. I have
mounted the fileshare directory via an NFS volume to a NetApp storage
array.

I can read/write ok, no permission issues i can see.

Trouble is, with these large uploads - it sometimes takes as long as
5+ hours - and I've yet to leave on running that long to see if it
actually completes.

Both my client and server machines are on the same local network - so
the first few files upload at like 10mbps, but the speed dramatically
drops over time to ~80kbps or the java applet finishes the upload,
says ETA = 0seconds, but data continues to be written and i"m not sure
why.

My client computer is a mac pro - running 10.7. I use Firefox 4 and
Chrome. I havn't tried Safari yet but will.

Anyways anyone else experience slow uploads over time with RS?

Great product btw, I used older versions quite extensively and love
it, only since downloading the latest version have these issues
arisen.

JOsh

[JoshM]

I should add i've tried mounting a volume with samba instead of nfs.
and also using the local disk but these don't change the issue.

I also have the latest java version available for OSX. I'm hardpressed
to find a windows machine but i would try that too.

[David Dwiggins]

Is it perhaps trying to transcode/create previews in realtime during
the upload? If so, this might be slowing down the upload because of
the time the server is taking to process the large files.

Have you considered using FTP upload or staticsync in ingest mode to
import the files rather than doing it through the web interface? If
the files are that large, this might be more convenient.

-David

> --
> You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
> To post to this group, send email to resour...@googlegroups.com.
> To unsubscribe from this group, send email to resourcespac...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/resourcespace?hl=en.
>
>

[JoshM]

Thanks for the tips.

I managed to set up an FTP server on another machine - but when I
login through resource space only one of the 15 folders shows up in
the list.

The other folders exist, and can be read by the FTP user (i veified by
logging in from the command line as that user) but for some reason
only one random folder displays in RS.

I also tried the local folder option but i get a php error:

/var/www/html/resourcespace/include/general.php line 2431: mkdir()
[function.mkdir]: Permission denied

I'll try the staticsync plugin if I can find it! So far no luck

Cheers,
Josh

[Jeff Harmon]

You're not serious when you say you're running 10.7 are you? An unreleased beta OS with a browser released a week ago...10.7 changes the Mac OS Java architecture seriously, afaik.

Also, are you running jUpload 5.0.2? You might give that a go.

-jeff

[JoshM]

No luck leaving them over night, the time only gets longer. I think if
ffmpeg crashes jUpload doesn't know about it and waits indefinitely.
I've taken to going one video at a time for now.

My mistake, OS X 10.6.7

I'll check out updating the jUpload applet on the server, thanks!

Josh

On Mar 28, 3:52 pm, Jeff Harmon <jeffreyhhar...@gmail.com> wrote:
> You're not serious when you say you're running 10.7 are you? An unreleased beta OS with a browser released a week ago...10.7 changes the Mac OS Java architecture seriously, afaik.
>
> Also, are you running jUpload 5.0.2? You might give that a go.
>
> -jeff
>

[Paul Manno]

I assume you are uploading image files (jpg, tiff, etc..) is that correct?  If so, have you tried setting this to false in your config.php?

# Allow to disable thumbnail generation during batch resource upload from FTP or local folder.
# In addition to this option, a multi-thread thumbnail generation script is available in the batch
# folder (create_previews.php). You can use it as a cron job, or manually.
$enable_thumbnail_creation_on_upload = true;

Then create thumbnails using the create_previews.php via cron.  It might help.

Although it says "during batch resource upload from FTP or local folder", it appears that before create_previews is called in upload_file this variable is checked and upload_file is called from upload_java.php.

Paul

[JoshM]

That sounds like a great idea. I've already gone through the folders
and converted lossless image formats to png's. I also transcoded all
video to m4v and kept each video under 200mb. So each file is no more
than 200mb in size, but i'll try this suggestion. Thank you!

I'm also running the svn version of resourcespace at the moment to
compare it with the zip i grabbed a few days ago.

I'll let everyone know how it goes.

Cheers,

Josh

[Jeff Harmon]

PNGs are lossless. 

J

[JoshM]

I had more luck with turning off the thumbnail generation during
uploads. I left php running the create_thumbnails.php in a screen
window last night. so far it has not finished, however it is nice to
be able to watch the output in my terminal window.

i wanted to post a couple errors to see if anyone has seen them
before:

identify: Delegate failed `"ffmpeg" -v -1 -i "%i" -vcodec pam -an -f
rawvideo -y "%u0.pam" 2> "%Z"' @ delegate.c/InvokeDelegate/1015

convert: no decode delegate for this image format `/var/www/html/rs/
filestore/1/0/0/3_d92d72c83d36830/1003_30484030ed7c3a2.ds_store' @
constitute.c/ReadImage/526.

Ill post some more, but the scroll back wasnt saved. however these two
messages i get a lot.

Feels like progress :)

Cheers,
Josh

On Mar 29, 1:55 pm, Jeff Harmon <jeffreyhhar...@gmail.com> wrote:
> PNGs are lossless.
>
> J
>

[Jeff Harmon]

The second error has to do with the handling of mac-specific DS_store files and could probably just simply be ignored. You don't want those files handled anyway.

The first one seems serious, like one of the needed codecs for transcoding to FLV is not available to ffmpeg.

J

[Paul Manno]

You've uploaded a ".ds_store" file as a video file and it's trying to process it and the .ds_store file is not a video, it's metadata from Mac:

http://en.wikipedia.org/wiki/.DS_Store

[Jeff Harmon]

Put .ds_store in the ignore list in config and have a go. 

J

[JoshM]

got it - thanks!

For some reason after the create_thumbnails.php completed, all the
media shows up as Offline, or just displays blank images, as shown
here:
http://imgur.com/PSHMr

I even wiped the database and filestore directory and started a fresh
batch - and it's doing the same thing...

output is cleaner now:

Processed resource 125 in 0.59 seconds.
Processing resource n°126.
Processed resource 126 in 2.37 seconds.
Processing resource n°127.
Processed resource 127 in 1.74 seconds.
Processing resource n°128.
Processed resource 128 in 2.02 seconds.
Processing resource n°129.
Processed resource 129 in 2.27 seconds.
Processing resource n°130.
Processed resource 130 in 2.45 seconds.


Please have a gander at the image link above, anyone experience this?

Josh

On Mar 30, 10:21 am, Jeff Harmon <jeffreyhhar...@gmail.com> wrote:
> Put .ds_store in the ignore list in config and have a go.
>
> J
>

> On Mar 30, 2011, at 9:59 AM, Paul Manno <pgma...@gmail.com> wrote:
>
>
>
>
>
>
>
> > You've uploaded a ".ds_store" file as a video file and it's trying to process it and the .ds_store file is not a video, it's metadata from Mac:
>
> >http://en.wikipedia.org/wiki/.DS_Store
>

> > For more options, visit this group athttp://groups.google.com/group/resourcespace?hl=en.

[Paul Manno]

Looks like the web server cannot see the filestore.  You sure that the web-user (www-data or apache) owns the mount, filestore and all subdirs?

[David Dwiggins]

If you ran the create previews script as a privileged user, then it
probably created all the previews as that user, and the webserver
probably can't see them. chmod -R wwwrun filestore should fix this
(where wwwrun is whatever user your web server runs under.)

[JoshM]

[Wed Mar 30 13:15:45 2011] [error] [client xx.xx.xx.xx] File does not
exist: /var/www/html/1, referer: http://website.com/rs/pages/search.php?search=%21collection4

I get this in my apache error log after uploading a fresh batch of
jpegs, and running php create_previews.php

This looks like a mod_rewrite problem to me. It should be looking
below /var/www/html/rs not the web root...

Amirite?
Josh

On Mar 30, 12:25 pm, David Dwiggins <da...@dwiggins.net> wrote:
> If you ran the create previews script as a privileged user, then it
> probably created all the previews as that user, and the webserver
> probably can't see them.  chmod -R wwwrun filestore  should fix this
> (where wwwrun is whatever user your web server runs under.)
>
>
>
>
>
>
>
> On Wed, Mar 30, 2011 at 2:55 PM, Paul Manno <pgma...@gmail.com> wrote:
> > Looks like the web server cannot see the filestore.  You sure that the
> > web-user (www-data or apache) owns the mount, filestore and all subdirs?
>

[JoshM]

well i symlinked /var/www/html/rs/filestore/1 to /var/www/html/1 and
low and behold the images appear...

[JoshM]

this happened because i've been running php create_previews.php
instead of lynx -dump http://website/rs/batch/crea..php etc
ignore!

[JoshM]

resource space used to create streaming videos for us so we didn't
have to download them. not sure why it's generating a single thumbnail
now instead of throwing a little play button... any ideas on this one?

[Paul Manno]

That means ffmpeg is either not being run on those files, or crashing.  Did you check the web server logs for errors?

[JoshM]

thanks, it looks like it's just m4v's...

[Paul Manno]

Add this to your config.php

# List of extensions that can be processed by ffmpeg.
# Mostly video files.
# @see http://en.wikipedia.org/wiki/List_of_file_formats#Video
$ffmpeg_supported_extensions = array(
                'aaf',
                '3gp',
                'asf',
                'avchd',
                'avi',
                'cam',
                'dat',
                'dsh',
                'flv',
                'm1v',
                'm2v',
                'mkv',
                'wrap',
                'mov',
                'mpeg',
                'mpg',
                'mpe',
                'mp4',
                'mxf',
                'nsv',
                'ogm',
                'rm',
                'ram',
                'svi',
                'smi',
                'wmv',
                'divx',
                'xvid',
                'm4v'
        );

I copied it from config.default.php and added m4v to the end of the list.

Paul

[Paul Manno]

Of course, you'll have to recreate previews for those resources after you make this change.

[JoshM]

Everything is working great now, thanks very much for the fast and
detailed responses!

Josh

On Mar 31, 10:52 am, Paul Manno <pgma...@gmail.com> wrote:
> Of course, you'll have to recreate previews for those resources after you
> make this change.
>
>
>
>
>
>
>
> On Thu, Mar 31, 2011 at 12:51 PM, Paul Manno <pgma...@gmail.com> wrote:
> > Add this to your config.php
>
> > # List of extensions that can be processed by ffmpeg.
> > # Mostly video files.

> > # @seehttp://en.wikipedia.org/wiki/List_of_file_formats#Video

> ...
>
> read more »