Video preview length problem

[Chris Blair]

I'm having a problem generating video previews any longer than 120 seconds. If I set the value any higher than that in the config file, the mp4 file starts to get generated but doesn't finish. The error log doesn't give me much in the way of useful information, and I've tried common fixes like increasing script timeouts and execution times in the PHP.ini file. But nothing seems to work.

Any other suggestions for getting longer previews?

Thanks,

Chris Blair

Magnetic Image, Inc.

[Allison Stec]

What are the stats of your system? Creating video derivatives can be challenging for some systems as processing speed plays a large role.

You can try to take RS out of the equation by running the ffmpeg command via command line.

Allison Stec
Senior Developer & Asset Management Specialist
Colorhythm
http://www.colorhythm.com

Main Office: +1 415-399-9921
Fax: +1 415-399-9928

as...@colorhythm.com

--
ResourceSpace: Open Source Digital Asset Management
http://www.resourcespace.org
---
You received this message because you are subscribed to the Google Groups "ResourceSpace" group.
To unsubscribe from this group and stop receiving emails from it, send an email to resourcespac...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Blair]

I'm running it on MediaTemple's Grid Service. So it's a shared hosting environment using Debian Squeeze. There are some limits to amount of memory per script (128MB I think), but there is also quite a bit of customization available via PHP.ini, .htaccess files etc...and it passes all installation tests once I have it setup properly. 

I'll try encoding the file from the command line later today and report back. But, I also have a test installation internally here at our place on an older Linux server running Debian with no limits and I'm seeing the exact same problem. That server isn't a powerhouse but it's plenty powerful enough to encode a 4 minute file using ffmpeg.

Chris Blair

[Chris Blair]

Encoding the same file on the command line using the same settings, including scaling it to 480x270 works fine. No issues. So any suggestions for getting it to work from within Resource Space for longer than 120 seconds? Would it help to put the timeout command somewhere in the PHP file that triggers the video encoding instead of doing it globally in the PHP.ini or in .htaccess?

Chris Blair

[Roger Howard]

A couple of notes:

1. memory per script doesn’t really have an impact here - ffmpeg is executing external to Apache/PHP processes, so it doesn’t add to the memory footprint of the script.

2. that said, it *is* executing simultaneous with Apache/PHP/MySQL, so it does impact the overall memory footprint of the server. MediaTemple Grid is a shared hosting environment and you’re competing for resources with other users. I highly recommend against shared hosting for ResourceSpace - I mentioned this in the past.  For $20/mo (same as Grid) you can get a VPS on Linode or DigitalOcean (or Amazon or Google Cloud) that will give you vastly most processing power and an environment you completely control. 

3. Did your test from the command line take more than 120 seconds to execute? 

4. the max_execution_time setting does not affect the overall execution time of ffmpeg (or other external processes) - it only governs processing happening within the PHP process. In fact, the ResourceSpace code that executes ffmpeg *does* use set_time_limit(0) to disable any PHP timeouts during ffmpeg execution - though I doubt this has any actual affect (it doesn’t hurt though). Ref: http://php.net/manual/en/function.set-time-limit.php

5. I’m not certain whether the Apache timeout setting does affect this - but it may. Whether you can adjust this in MediaTemple’s Grid environment remains to be seen.

6. Just setting a value in PHP.ini, httpd.conf, .htaccess, or other settings files doesn’t guarantee they actually get picked up. MediaTemple may well impose limits on your config values. The only way to know for sure is to check the settings at runtime; e.g., with a phpinfo.php file, which will report the *actual* config values being used.

If I were you, I’d create a simple test case of PHP using shell_exec to run your ffmpeg command. Simply executing it on the command line doesn’t really test this issue, as the execution environment is completely different. Eg;

<?php

set_time_limit(0);

shell_exec(“ffmpeg command here");

?>

Pop this onto your RS server, in the pages/tools/ path, named “ffmpeg_test.php", and then execute it from a browser:

http://yourserver/pages/tools/ffmpeg_test.php

If that works for long encoding sessions (great than 120 seconds) then the problem is indeed somewhere in RS. Otherwise, it’s not.

[Chris Blair]

Thanks Roger...appreciate the info. I've run phpinfo.php to verify what the server is picking up and what it isn't and it does seem to adhere to the directives that are set in the PHP.ini file as well as any placed in the .htaccess file. Of course you're probably right in that MediaTemple could limit memory usage in other ways.

On running it in a shared hosting environment...yes I know it's not ideal. Part of my testing here was to see if it would indeed run in a shared environment, as most people tell you it can't be done, especially using FFMPEG. In fact most people will tell you can't compile ffmpeg on a shared host, but I've done it on several without issue...although I'm currently using a binary that I compiled myself with RS. RS runs surprisingly well in the setup I'm currently using.

More suprising, we've found that the MediaTemple servers are quite powerful and very very fast. Encoding of a nearly 4 minute video file took about 1 minute on the command line, and encoding of files when uploaded through RS is also surprisingly fast. The majority of stuff we upload in this environment is 4 minutes in length or less and is already compressed to manageable sizes. This would be used for client previews mainly, so we won't be uploading large file sizes (like hundreds of gigs or anything like that). Most uploads are 100MB or under...and many are under 50MB.

I'll try the php/ffmpeg test this evening and report back!

Chris Blair

Magnetic Image, Inc.

[Roger Howard]

On Jun 9, 2015, at 12:42 PM, Chris Blair <tophe...@gmail.com> wrote:

Thanks Roger...appreciate the info. I've run phpinfo.php to verify what the server is picking up and what it isn't and it does seem to adhere to the directives that are set in the PHP.ini file as well as any placed in the .htaccess file. Of course you're probably right in that MediaTemple could limit memory usage in other ways.

On running it in a shared hosting environment...yes I know it's not ideal. Part of my testing here was to see if it would indeed run in a shared environment, as most people tell you it can't be done, especially using FFMPEG. In fact most people will tell you can't compile ffmpeg on a shared host, but I've done it on several without issue...although I'm currently using a binary that I compiled myself with RS. RS runs surprisingly well in the setup I'm currently using.

I can’t speak for most people - I can say that shared environments are highly unpredictable by their *very nature*. You may be on an under provisioned node today, and tomorrow end up on a vastly over-provisioned node. You have no control, and no recourse. For lightweight, largely static Web sites, this is usually OK - upstream caching and low traffic usually make that an OK recipe. For business applications, with CPU and i/o intensive components, and an entirely dynamic application layer, this is a recipe for problems.

The fact is, I can spin up a VPS with RS in a few minutes on any of the hosts I’ve mentioned, for the same price, with fully functioning ffmpeg, consistent I/O and CPU resources, and absolutely no outside configuration issues (config is fully within my control), and not encounter the issues you keep running into on MT shared plans. This doesn’t seem like a good environment to run a business application under.

More suprising, we've found that the MediaTemple servers are quite powerful and very very fast.

That’s subjective, and not guaranteed. By its nature, it’s a shared environment. Your performance will vary over time. Seriously - I just don’t understand the reluctance to consider that your system architecture is already costing you far more (if you value your time at all) than the pennies you might be saving (though, realistically, I don’t think you’re even saving pennies here - MT isn’t that cheap).

I'll try the php/ffmpeg test this evening and report back!

Good - that’ll help point things in the right direction.

-R

[Chris Blair]

Points all well taken. But consider that:

The ResourceSpace online demo doesn't work correctly with FFMPEG; We had the EXACT same problems (from an earlier thread) encoding MTS files when we demo'd ResourceSpace prior to trying it ourselves. And If you read this thread from the top you'll notice I also have RS installed on a dedicated web server internally (that I can access externally) and I've run into the SAME problem encoding files over 120 seconds on that dedicated server, which is running nothing but ResourceSpace on it. For that setup I used the Bitnami Installer...so there was very little I had to setup manually. 

So this doesn't seem to be a hosting problem. In fact, NONE of the issues I've run into have been hosting problems. They've been problems with how FFMPEG is compiled, adding the correct config settings to specify MTS files as a valid format, and a bug in a video player php file. So none of the solutions so far has had anything to do with the hosting environment. ResourceSpace is great Open-Source software...the best in the media asset management arena, but compared to proprietary commercial solutions, it has quite a few quirky issues. Ten minutes spent in the forums can confirm that, especially with ffmpeg.

And I've tested no less than 30 commercial solutions over the last 2 years for some of our clients and have implemented solutions for two clients using ThirdLight IMS, so I'm familiar with how the systems generally work. ThirdLight is very similar in functionality and setup to RS. 

Now COULD this problem be a server/hosting issue. Sure it could! But there are plenty of posts from people in the forums who've had the same issue who are running on dedicated servers of one sort or the other. 

I'll report back later.

Chris Blair

Magnetic Image, Inc.

[Chris Blair]

The PHP script has the same problem as RS. It fires ffmpeg and starts creating the mp4 file and stops about 3/4 of the way through. I'll have to turn on the PHP logs and try it again later to see what they say. They're turned off at the moment. So guess it's a PHP or settings issue. Thanks.

Chris Blair

[Chris Blair]

Thought I'd update this...I tried what Roger suggested with Google Cloud and AWS through Bitnami, and WOW...are those complicated setups....especially Google Cloud. You have to be a FULL-TIME IT pro to navigate and use the Google Cloud VPS setup. Amazon (via Bitnami) is much easier and I was able to set it up much faster, but I spent 2 hours just trying to figure out how to change permissions to allow me to edit and change the config.php file (Still haven't figured it out in fact). Bitnami's online documentation for setting permissions didn't work, even after restarting the server. So that's one of the reasons I use hosting services like MediaTemple....they make it incredibly easy to set stuff up and have a VERY good knowledgebase that's written and structured so that non-IT pros can read it and decipher it. While I'm an advanced computer user with loads of experience on Windows, Linux (yes command-line) and some on MacOX...I don't do information technology stuff full-time. I own and run an advertising, marketing and video production company. So the easier a hosting or VPS or VM is, the better.

Google's cloud hosting is unbelievably complicated and, much like their adwords and analytics dashboards, the Developer Console is a mess. Worse still, their documentation doesn't even remotely match the site. Almost every set of instructions and tutorials I read, watched or followed were either dated or just flat our wrong. Their adwords documentation is the SAME WAY. With Google Cloud, just trying to generate and download a public key to allow third-party SSH client use was a nightmare. It simply didn't work when following their instructions. I finally gave up.

So while you CAN get those for the same price as MediaTemple, the user interface (especially Google Cloud) and ease of use are just beyond most people. I'm gonna give AWS another go via Bitnami once they answer my support question about permissions, but I've literally spent 5 hours messing with those two solutions.With Linode and Digital Ocean, sure you get a VPS for $20, but you only get about 30 - 40GB of storage, which isn't enough for asset management for our company...so we'd have to go with the $60-$80/month solutions, which is just too much for us to spend on something that doesn't generate revenue.

Chris Blair

[Roger Howard]

> On Jun 12, 2015, at 11:34 AM, Chris Blair <tophe...@gmail.com> wrote:
>
> Thought I'd update this...I tried what Roger suggested with Google Cloud and AWS through Bitnami, and WOW...are those complicated setups....especially Google Cloud. You have to be a FULL-TIME IT pro to navigate and use the Google Cloud VPS setup. Amazon (via Bitnami) is much easier and I was able to set it up much faster, but I spent 2 hours just trying to figure out how to change permissions to allow me to edit and change the config.php file (Still haven't figured it out in fact). Bitnami's online documentation for setting permissions didn't work, even after restarting the server. So that's one of the reasons I use hosting services like MediaTemple....they make it incredibly easy to set stuff up and have a VERY good knowledgebase that's written and structured so that non-IT pros can read it and decipher it. While I'm an advanced computer user with loads of experience on Windows, Linux (yes command-line) and some on MacOX...I don't do information technology stuff full-time. I own and run an advertising, marketing and video production company. So the easier a hosting or VPS or VM is, the better.

Agreed on all points, Chris. I actually, fundamentally, think RS requires an IT (or devops) person to support. If you look at the archives of this group, it’s packed full of people struggling with systems-related issues - the fact is, whatever you think about RS as a product, deployment/support is a minefield that requires technical knowledge many who try to use it don’t have. I’m not judging anyone, it’s just plain to me.

> Google's cloud hosting is unbelievably complicated and, much like their adwords and analytics dashboards, the Developer Console is a mess. Worse still, their documentation doesn't even remotely match the site. Almost every set of instructions and tutorials I read, watched or followed were either dated or just flat our wrong. Their adwords documentation is the SAME WAY. With Google Cloud, just trying to generate and download a public key to allow third-party SSH client use was a nightmare. It simply didn't work when following their instructions. I finally gave up.

Just for context, I find Google Cloud and AWS a pleasure. It’s a matter of perspective - and that’s why I said what I said above. As an experienced dev and IT operations person, those services work brilliantly for me. I wouldn’t touch a MediaTemple or Dreamhost - I work on large DAM projects, and a MT or DH do not allow me to provide my clients the reliability, scalability or control we need.

>
> So while you CAN get those for the same price as MediaTemple, the user interface (especially Google Cloud) and ease of use are just beyond most people. I'm gonna give AWS another go via Bitnami once they answer my support question about permissions, but I've literally spent 5 hours messing with those two solutions.With Linode and Digital Ocean, sure you get a VPS for $20, but you only get about 30 - 40GB of storage, which isn't enough for asset management for our company...so we'd have to go with the $60-$80/month solutions, which is just too much for us to spend on something that doesn't generate revenue.

Absolutely. I didn’t mean to suggest these are consumer platforms - they aren’t. I would never expect a non-technical user to figure out how to setup and maintain a server stack. What you - and other shops who don’t employ devops folks - need is a simple deployment that works, on a host that is transparent to you.

Well aware of the storage limitations of DH and Linode - I don’t know your requirements. The issue I was responding to wasn’t a question of storage. I can offer solutions there too. The problem you were facing is (especially after running the test I proposed) clearly an issue with the PHP stack on MT and not RS, if you had the same exact issue with the PHP script fragment I suggested you try.

Keep in mind, I don’t get paid for any of this - I have nothing to sell you. I’m just trying to help from experience as a sometimes frustrated user myself.

-R

[Stefan Wild]

If you can, set up $ffmpeg_preview_async=true and try adding " -nostats" to $ffmpeg_preview_options in config.php.

How much space do you need and what is your price limit?

Best,

Stefan

--

www.archivedi.com

ResourceSpace hosting

[Chris Blair]

Roger, Stefan,

Believe me I appreciate the help and suggestions...and I was able to get Bitnami/AWS setup and working without any issues. Bitnami confirmed that permissions weren't working right on the server I setup so suggested I just delete it and create a new one. That worked following the exact same set of instructions and commands. FFMPEG works great on it. I set it up using an Ubuntu OS in hopes of using unoconv/libreoffice to create Office thumbnails but so far, while that works on the command line, it doesn't through RS. RS logs show that it can't find python...which is obviously installed and working since using command line works. But that's off-topic.

AWS and Bitnami are WAAAY more user friendly than Google Cloud and I can certainly see how someone who does this full-time would love the control Google offers as you mention. As a small company of just 6 employees, we have essentially no budget for creating our own asset management solution. We've been using OwnCloud and before that I used Pydio. Both work fine and are easy to deploy...and work fine on shared hosts, but have limited video preview capability and can be really quirky at times. So that's why we're testing RS. Everyone LOVES ResourceSpace but, like I said, we just don't have money for stuff like this. We barely break even most years and last year lost money. But we have a need to be able to archive and find assets more quickly.

I'll try the FFMPEG suggestion on the shared installation to see if that fixes it there.

Chris Blair

On Friday, June 12, 2015 at 5:16:25 PM UTC-4, Stefan Wild wrote:

If you can, set up $ffmpeg_preview_async=true and try adding " -nostats" to $ffmpeg_preview_options in config.php.

How much space do you need and what is your price limit?

Best,

Stefan

www.archivedi.com

ResourceSpace hosting

[Chris Blair]

I finally solved this and the fix is sort of embarrassing. I simply changed the PHP settings on the server to FastCGI. I thought I had tried this before, but I think I may have had a directive in the config.php file that instructed RS to use the latest version of PHP available on the server (it pointed to the folder where it was located on the server). So even if I changed it in the hosting setup, RS was probably overriding it.

I'm going to test more tonight to confirm this fixed this problem....but I switched back and forth on the type of PHP I was using and it worked on FastCGI and didn't on Stable...I'm probably moving to an Amazon/Bitnami solution soon, but it's nice to know that a shared server can indeed work for RS if it's setup right.

Chris Blair

Magnetic Image, Inc.