Chistmas mini-entertainment
Richard Newbould
- Intalio workflow/business process engine (Richard setting up with free help from a kind company called MphasiS). Interaction via web, web services (SOAP) or preferably Apache ActiveMQ for messaging (see below)
- Switched On FTP server for storing photos
- YouTube for hosting videos
- Switched On Christmas code challenge 1: PHP running in Apache on the local machine (in the Switched On training centre) - to be called from a link in an Intalio page that will then receive the file location (video or photo) and store it for the PHP Windows service (below)
- Switched On Christmas code challenge 2: PHP Windows service (not Linux at this stage, but we'll get em migrated!) to process files and upload them reliably in the background, and reliably notify Intalio one completed
- Web page served to our Communications Administrator by Intalio as part of some wider process (Comms Admin enters student profile, messages, photos and video in the Switched On training centres. These will be used in comms with sponsors.) A given web page will include a section to upload photos and/or video. The upload interaction point is simply a hyperlink on the page with a get string URL to pass info to the locally hosted apache along the lines:
<a href="http://localhost:87/upload_photo?filename=103435345&correlation_id=13876234754325" target="NewWin">Click here to upload head and shoulder profile photo</a> - The Comms Admin clicks this link and a new browser window opens that asks the user to browse for the photo using an input field:
<input type="file" name="somename" size="40">
The file should be submitted to the local Apache server using the submit button (though it is local, this will allow us to host the file processing component on another machine in the network in the future). - The local server renames the file to: photo_id.jpeg/gif/png (ending the same as the uploaded file) and saves it in a local folder.
- The local server also appends the correlation_id, file-type and photo_id in a local "work-log" file, to be processed by the PHP Windows Service (Linux daemon in future!)
- The local server window closes and the user can continue entering student profile info in the Intalio service whilst the server is busy processing and uploading files in the background. The intalio server will be waiting for a notification at some stage that the file has been successfully uploaded before sending an email to the sponsor containing links to the photos and video (see below)
- Reads next (top) entry in the work-log file (see above) - including filename and correlation_id. If there are no more items in the worklog, go to sleep for 2 mins and then check again.
- If is a video
- Compresses the video to flash format .flv (bandwidth can be low) to make it significantly smaller. Recommend use the utility ffmpeg (www.ffmpeg.org), naming it with the same filename except with the .flv extension
- Upload it to YouTube using the Direct Upload API documented at: http://code.google.com/intl/en/apis/youtube/2.0/developers_guide_php.html#Direct_Upload
- Test that upload is successful. If fails, try again two more times. Log every attempted upload, with start and end times and success/failure.
- Invoke Intalio to notify of success/failure, passing correlation_id (so Intalio knows which process the file upload corresponds to), filename and success/failure flag. This may be done via a web service (PHP web services are reasonably documented but you need to ensure the necessary extension is loaded), or, preferably via the the Apache ActiveMQ interface for greater resiliency and to save having to code in our own mechanism (remember that there could be a power cut at any moment! Richard can provide the interface details, depending on whether you want a Web Service or messaging interface.
- If is an image
- For the slightly more adventurous: Check if there were previous attempts to upload the file (there may have been a powercut so the file at the top of the work-log may already be partly uploaded) and append the rest of the file if the APPE command or equivalent is available in PHP. Otherwise, just re-attempt and write over.
- Upload it via FTP to the Switched On server (FTP details will be given).
- Resiliency: if fails, re-attempt, appending or overwriting if you prefer. Log all attempts, successful or failed.
- Invoke Intalio to notify of success/failure as for video.
- Move the file to a processed and uploaded file (in case of the video, just keep the original file, not the flv)
- Remove entry from top of work-log and add to it to a process files log, along with the time was uploaded time.
- Go back to step 1.
Richard Newbould
--
You received this message because you are subscribed to the Google Groups "so-class2" group.
To post to this group, send email to so-c...@googlegroups.com.
To unsubscribe from this group, send email to so-class2+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/so-class2?hl=en.
Peter Smit
I would not have any requirements for later integration, except that if a nice Zend Framework structure would be used it is later trivial to integrate it in one application tree. So complete ZF would be nice, but is of course not necessary. And if we need some php extensions later, it will be easy to install that package, without affecting anything else.
I have some comments on the structure of the application
- I would advise to have no long-running php-scripts (as service). php-scripts leak a lot of memory and have a tendency to fail silently. What you better can do is make a php-script that is invoked every 2 minutes (cron, windows scheduler).
- A single work file is quite vurnable for concurrency errors. I would either suggest a small database (mysql / sqllite), or an advanced work file scheme.
An advanced scheme would look like this:
- challenge part 1 would always append to workfile, creating it if it not exists
- everytime a part 2 script is started, it moves the workfile to workfile.timestamp.randomsomethingorproecessid
- part 2 processes it's own workfile. It removes lines from it when that line is ready, and removes the whole file when the file is empty
- part 2 ensures that is does a stat() (changing the access-timestamp) on the workfile at least every hour. (to show that the process is alive)
- every hour another script is started that simply looks to the workfiles.timestamp.randoms and the access-times. If an access time is too long ago, it resubmits the lines in it to the normal workfile
With a database a thing like Zend_Queue can be used.
This holidays I don't have much time I guess, but if nobody picks it up I think this would be a nice thing to do later.
Regards,
Peter
---
+358 44 218 2700
Richard Newbould
Peter
persistence with server outages (is the queue stored on disk?). The
advantage of using Zend_Queue is that it doesn't matter. Just one line
of code change (or configuration file) and another type of queue is
used
2. We are using MySQL for SUM. Sqlite is a server/fileformat that can
be accessed with sql-like statements. (sqlite.org). I think that MySQL
is the easiest.
Merry Christmas!
Peter
On Dec 23, 11:15 am, Richard Newbould <richard...@googlemail.com>
wrote:
> >> 2009/12/22 Richard Newbould <richard...@googlemail.com>
>
> >>> Hello Everybody!
>
> >>> Christmas can have it's quiet periods, and after enough mince pies and
> >>> films it has been known for people to come to the point of boredom.
>
> >>> So, to save you the Christmas gloom, I thought I'd make your coding
> >>> fingers itchy and to have a look at your Switched On tasks and have some
> >>> fun.
>
> >>> In case you don't have anything to work on right now, here's a
> >>> tantalising little challenge, that would actually fill you, me and many
> >>> sponsors and sponsored with Christmas cheer! If you want to take this one
> >>> up (in whole or part), please try let me know beforehand so we can discuss
> >>> it at your convenience (remembering India is 5:30 hours ahead) and ensure
> >>> you're not duplicating someone else's fun!
>
> >>> Thanks for all your enthusiasm and contributions this year!
> >>> Have a very Merry Christmas and a Happy New Year!
>
> >>> God bless,
>
> >>> Richard
>
> >>> -----------
>
> >>> Let me outline the little teaser...
>
> >>> *The problem:*
> >>> Need to upload photos and videos of Switched On sponsored students to the
> >>> Switched On web server and YouTube respectively.
> >>> The bandwidth in the training centres can be low and the network and
> >>> electricity supply highly unreliable. Yet we need to make good use of the
> >>> administrator's time so the cannot sit endlessly pressing the send button.
> >>> There may be 50 student profiles with photos and videos to create and
> >>> upload, and we need to pay them for their time... (gasp! yes, there are
> >>> similarities between me and Scrooge for any not acquainted with my Yorkshire
> >>> ways!)
>
> >>> The users are largely tech un-savvy. We need to reduce maintenance time
> >>> as much as possible, so the code needs to be (or become as we improve it)
> >>> robust with error-handling etc.
>
> >>> Furthermore, it needs to integrate with our Intalio business process.
>
> >>> If you only have time and energy (given the sapping effect of too much
> >>> turkey) then attempt as many parts as you have energy for!
>
> >>> *Proposed solution (after much deliberation and going round in circles):
> >>> *
>
> >>> There are 5 components of the system:
>
> >>> - Intalio workflow/business process engine (Richard setting up with
> >>> free help from a kind company called MphasiS). Interaction via web, web
> >>> services (SOAP) or preferably Apache ActiveMQ for messaging (see below)
> >>> - Switched On FTP server for storing photos
> >>> - YouTube for hosting videos
> >>> - *Switched On Christmas code challenge 1*: PHP running in Apache on
> >>> the local machine (in the Switched On training centre) - to be called from a
> >>> link in an Intalio page that will then receive the file location (video or
> >>> photo) and store it for the PHP Windows service (below)
> >>> - *Switched On Christmas code challenge 2*: PHP Windows service (not
> >>> Linux at this stage, but we'll get em migrated!) to process files and upload
> >>> them reliably in the background, and reliably notify Intalio one completed
>
> >>> The series of steps:
>
> >>> *Code challenge 1*) Comms Admin user selects file to upload and file
> >>> gets put in processing queue:
>
> >>> 1. Web page served to our Communications Administrator by Intalio as
> >>> part of some wider process (Comms Admin enters student profile, messages,
> >>> photos and video in the Switched On training centres. These will be used in
> >>> comms with sponsors.) A given web page will include a section to upload
> >>> photos and/or video. The upload interaction point is simply a hyperlink on
> >>> the page with a get string URL to pass info to the locally hosted apache
> >>> along the lines:
> >>> <a href="
> >>> http://localhost:87/upload_photo?filename=103435345&correlation_id=13..."
> >>> target="NewWin">Click here to upload head and shoulder profile photo</a>
> >>> 2. The Comms Admin clicks this link and a new browser window opens
> >>> that asks the user to browse for the photo using an input field:
> >>> <input type="file" name="somename" size="40">
> >>> The file should be submitted to the local Apache server using the
> >>> submit button (though it is local, this will allow us to host the file
> >>> processing component on another machine in the network in the future).
> >>> 3. The local server renames the file to: photo_id.jpeg/gif/png
> >>> (ending the same as the uploaded file) and saves it in a local folder.
> >>> 4. The local server also appends the correlation_id, file-type and
> >>> photo_id in a local "work-log" file, to be processed by the PHP Windows
> >>> Service (Linux daemon in future!)
> >>> 5. The local server window closes and the user can continue entering
> >>> student profile info in the Intalio service whilst the server is busy
> >>> processing and uploading files in the background. The intalio server will
> >>> be waiting for a notification at some stage that the file has been
> >>> successfully uploaded before sending an email to the sponsor containing
> >>> links to the photos and video (see below)
>
> >>> *Code challenge 2*) PHP Service / Daemon processes and uploads from the
> >>> file queue
> >>> This service/daemon is running in the background to work through
> >>> processing and uploading files
>
> >>> 1. Reads next (top) entry in the work-log file (see above) -
> >>> including filename and correlation_id. If there are no more items in the
> >>> worklog, go to sleep for 2 mins and then check again.
> >>> 2. If is a video
> >>> 1. Compresses the video to flash format .flv (bandwidth can be
> >>> low) to make it significantly smaller. Recommend use the utility ffmpeg (
> >>> www.ffmpeg.org), naming it with the same filename except with the
> >>> .flv extension
> >>> 2. Upload it to YouTube using the Direct Upload API documented at:
>
> >>> http://code.google.com/intl/en/apis/youtube/2.0/developers_guide_php....
> >>> 3. Test that upload is successful. If fails, try again two more
>
> ...
>
> read more »
Richard Newbould
1. I don't know exactly the essentials of ActiveMQ, for example the
persistence with server outages (is the queue stored on disk?). The
advantage of using Zend_Queue is that it doesn't matter. Just one line
of code change (or configuration file) and another type of queue is
used
2. We are using MySQL for SUM. Sqlite is a server/fileformat that can
be accessed with sql-like statements. (sqlite.org). I think that MySQL
is the easiest.