Upload speed seems slow - does size really matter?

[Edward McCain]

I'm running RS on Ubuntu Precise Pangolin - see configuration info below.

The problem is that we are hoping to use RS as a DAMS for a newsroom photo department and the upload speeds are pretty slow. For example:

135 KB JPG = 48 seconds

4 MB TIFF = 48 sec

6 MB JPG = 50 sec

12 MB JPG = 52 sec

19 MB TIFF = 50 sec

I have tried setting $enable_thumbnail_creation_on_upload = false;

but this doesn't seem to help at all.

I've also tried eliminating all previews, so those don't have to be built. No appreciable difference.

It also seems weird to me that the size of the file doesn't seem to affect the ingest speed very much - in fact, the 19 MB TIFF uploaded faster than the 12 MB JPEG.

Any ideas on how to speed up the upload/ingest process?

Thanks,

Edward McCain

Digital Curator of Journalism

Reynolds Journalism Institute

University of Missouri

ResourceSpace version	6.0.5120
ResourceSpace Build	5120
Server platform	Apache/2.2.22 (Ubuntu)
PHP version	5.3.10-1ubuntu3.9
ExifTool version	8.60
FFmpeg version	ffmpeg 0.8.10-4:0.8.10-0ubuntu0.12.04.1
ImageMagick/GraphicsMagick version	Version: ImageMagick 6.6.9-7 2012-08-17 Q16 http://www.imagemagick.org
Browser user-agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.107 Safari/537.36

[Tom Gleason]

I can't think of any reasons for this behavior, especially without previews.

Is your machine local or remote?

I would say that a slow CPU might produce these kinds of results (preview creation), but you said that's not a factor in your comparisons. 

What happens when you do a speed test on the internet? What does it say your upload speed is?

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.org
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Tom Gleason

Colorhythm LLC
http://www.colorhythm.com

Main Office:  +1 415-399-9921
Fax: +1 415-399-9928
Mobile:  +1 347-537-8465

tgle...@colorhythm.com

[Edward McCain]

RS is running on a server in the building next door. My upload speed tests out at 8.3 Mbps on the Speakeasy test.

Another question I have is why the Upload status quickly shows 100%, but then there is a long pause before the thumbnail shows up and the next file is uploaded?

Could this have anything to do with memory allocations? I'm set to the recommended amounts.

Edward McCain

Reynolds Journalism Institute

[Tom Gleason]

Once the files is uploaded (100%), the previews are being created. What is the CPU speed?

[Edward McCain]

I have RS set not to create previews. Would thumbnail generation take this long? What is a typical speed for these sizes of files on RS?

Here is my CPU info:

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 15

model name : Intel(R) Core(TM)2 Duo CPU     E6750  @ 2.66GHz

stepping : 11

microcode : 0xb6

cpu MHz : 2000.000

cache size : 4096 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 2

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 10

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm lahf_lm dtherm tpr_shadow vnmi flexpriority

bogomips : 5319.77

clflush size : 64

cache_alignment : 64

address sizes : 36 bits physical, 48 bits virtual

power management:

[Allison Stec]

At the point of upload do you also have metadata extractions happening? Are you creating checksums? These processes may be causing slowness. 

Allison Stec
Asset Management Specialist
Colorhythm

http://www.colorhythm.com

Main Office: +1 415-399-9921
Fax: +1 415-399-9928

as...@colorhythm.com

[TimC]

Just as a simple benchmark I have a similar upload speed (around 10mbit max) to a cloud based Debian server around 400 miles distant. It has 4 virtual cores and 4GB Ram.

I uploaded a 40MB TIF and the upload took 36 seconds and the preview creations a further 10. 46 secs in all. A 4MB TIF took a total of 9 seconds including previews.

So can't say what's up with your server but you have a similar speed and performance spec so you should get similar upload speed.

Tim

[Tom Gleason]

In my experience the preview speed is based totally on processor speed. So if Edward has 2 cores and Tim has 4 (of similar GHz), Edward could expect the same previews to take twice as long. The pixel size of the files I believe would be more influential than the filesize. 

[Tom Gleason]

Also, Edward, check that 

convert --version

shows that your Imagemagick features OpenMP, otherwise you'd only be using one core. 

[Edward McCain]

Thanks, Tom,

Here's what I get:

Version: ImageMagick 6.6.9-7 2012-08-17 Q16 http://www.imagemagick.org

Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC

Features: OpenMP

Do I need to configure anything to make sure ImageMagick can use both cores?

[Edward McCain]

Thanks for the benchmark info, Tim. It seems we should be getting better performance than we currently have, so I'll keep working on this situation.

Edward

[Edward McCain]

Thanks Alllison,

Metadata extractions, yes - but very basic and very little.

Not sure about the checksums. Where would I look to check on that?

Edward

[Tom Gleason]

No configuration needed, OpenMP means you're using both.

[Jeff Nova]

With slower machines with less RAM, you might use the Q8 variant of ImageMagick. That makes all the intermediate calculations and images 1/2 the size of Q16.

J

--

Jeff Nova

Chief Executive Officer

Colorhythm LLC

http://www.colorhythm.com

Main Office:  +1 415-399-9921

Fax: +1 415-399-9928

Mobile:  +1 510-710-9590

jn...@colorhythm.com

[Alan Pater]

On my test system, uploads were taking around 20-30 seconds each. I did some looking around and found a mysql 5.5 tweak that dropped those times down to 4 seconds (or less on small files).

In the file /etc/mysql/my.cnf, I wrote the following line:

innodb_flush_log_at_trx_commit = 0

WARNING: This means that data will only be written to the db once per second, not immediately. If your ups dies during this second, you will lose data. If that is not ok, invest in some really fast SSD's instead. Read up on this setting on the Interwebs, there are some good articles about this out there.

I also disabled writing to access logs on the webserver. This test system has the slowest hdd that exists in the world, so every little bit helps.

[Edward McCain]

Alan,

Wow, this seems to have done the trick. I'm seeing a tenfold increase in the speed of ingest now. I don't think the risk of data loss is that serious, but also wonder if there is a way to get a report should there be a power loss or other event. Is there a way to check and report on this kind of thing?

Thank you!

Edward