Are persistent EC2 instances needed?

[Ryan Phillips]

I haven't installed a test instance of Panda yet, but this app looks sweet.
Just the other day I was thinking of an app like this.

Does the EC2 app need to run to display from a remote site the encoded
videos? Basically a read only mode.

Is anyone willing to post to the list what their Amazon bill is like using
Panda?

Thanks,
Ryan

[Damien Tanner]

Hey Ryan,

Although the EC2 instance isn't required to view videos (as they come
directly from S3), it is required to serve the upload form and receive
the subsequent upload. Doing this allows us to have a nice upload
progress bar and also means your app doesn't have to handle the
(potentially very large) video file.

The small instance works out at about $72 a month, so not a huge cost
for an easy to integrate solution to video :)

Red (http://github.com/reddavis/panda) has started integrating MySQL
support, and with a little modification the S3 dependancy could also
be made optional. This means that it'll be easy to run Panda on any
host. However getting ffmpeg install with all the codecs isn't
something you want to do every day.

--
Kind Regards,
Damien Tanner
Co-Founder and Director, New Bamboo

Creating fresh, flexible and fast-growing web applications is our passion.

+44 (0)78 6312 7999
+44 (0)20 7099 7486
http://www.new-bamboo.co.uk

[Tys von Gaza]

$72 is still fairly high if your volume of videos is very low. Is
there any options of ad-hoc starting up the instance when needed? Or
some ability to rent time on someone else's Panda EC2 instance that is
running all the time with the video's shipped off to our own S3
storage?

Tys

[Andy Waite]

Might be worth looking at http://heywatch.com if you only need a few
videos encoded.

Andy

[Tys von Gaza]

Neat, thanks!

Tys

[Damien Tanner]

Hi Tys,

One of the big benefits of Panda is that it also provides the upload
form, so your app never has to touch the large video files upload.
This obviously means to upload videos Panda must be available. If
Amazon is looking a bit pricing, it might be worth trying out Red's
MySQL version (hopefully to be pulled into master soon) at
http://github.com/reddavis/panda as this'll allow you to run Panda on
any existing servers you have.

Otherwise HeyWatch isn't a bad choice for the odd encoding job :)

--

[Tys von Gaza]

The issue is then installing all the codec's for ffmpeg, is there a good source for those?

Tys

[Damien Tanner]

Not as such. But Makoto is about to start work on a guide for
'Developing with Panda' which will cover local installation and a
little bit of ffmpeg.

[reddavis]

As we have now decided to use Panda on a VPS instead of EC2 I'm
currently writing a capistrano recipe so getting up and ready will be
just as easy as EC2. The code isn't yet up as I'm currently making it,
but I'll post here when its finished and working(more important ;) )

Cheers, Red

On Sep 16, 11:39 pm, "Damien Tanner" <dam...@new-bamboo.co.uk> wrote:
> Not as such. But Makoto is about to start work on a guide for
> 'Developing with Panda' which will cover local installation and a
> little bit of ffmpeg.
>
> On Tue, Sep 16, 2008 at 11:25 PM, Tys von Gaza <t...@tvg.ca> wrote:
>
>
>
> > The issue is then installing all the codec's for ffmpeg, is there a good
> > source for those?
> > Tys
>
> > On Tue, Sep 16, 2008 at 4:07 PM, Damien Tanner <dam...@new-bamboo.co.uk>
> > wrote:
>
> >> Hi Tys,
>
> >> One of the big benefits of Panda is that it also provides the upload
> >> form, so your app never has to touch the large video files upload.
> >> This obviously means to upload videos Panda must be available. If
> >> Amazon is looking a bit pricing, it might be worth trying out Red's
> >> MySQL version (hopefully to be pulled into master soon) at

> >>http://github.com/reddavis/pandaas this'll allow you to run Panda on

> >> any existing servers you have.
>
> >> Otherwise HeyWatch isn't a bad choice for the odd encoding job :)
>
> >> On Tue, Sep 16, 2008 at 9:07 PM, Tys von Gaza <t...@tvg.ca> wrote:
> >> > Neat, thanks!
> >> > Tys
>

> >> > On Tue, Sep 16, 2008 at 12:24 PM, Andy Waite <andywa...@gmail.com>
> >> > wrote:
>
> >> >> Might be worth looking athttp://heywatch.comif you only need a few

[Makoto Inoue]

Hi.

This is Makoto @ New  Bamboo.

It may take a while to merge Red's Mysql version (http://github.com/reddavis/panda_cap/tree/master) into our master branch.
For the time being, you can still play with Panda using simpledb alternatives.

There are currently 2 simpledb like tools at this moment(as far as I know).

* simplerdb (http://aws.amazon.com/simpledb/) = Ruby implementation.
* simpledb-dev (http://code.google.com/p/simpledb-dev/) = Python implementation

I initially tried simplerdb, but found that the implementation is not complete, therefore not suitable for Panda use. Also it's an in-memory db, so the data will be lost every time you stop simplerdb.

I then tried simplerdb-dev(the Python one). simpledb-dev stores data on file systems, so you don't need to worry about losing data every time you stop simpledb-dev. However, EC2 vapors every time you restart the instance, so you still need to use  Amazon Elastic Block Store ( http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1667 ) to keep your data.

Here are the quick steps to use simpledb-dev at Panda.
Please give a try and let me know if you find any issues. If people find this solution useful, I will add simpledb-dev into next AMI release image, and also update the documentation.

1. Setup simplerdb-dev on your EC2 instance.

# You can skip this part if python version is already 2.5
apt-get update
apt-get install python2.5-dev
rm /usr/bin/python
ln -s /usr/bin/python2.5 /usr/bin/python

# This step is to install easy_install, Python equivalent of Gems.
# If you are installing Panda locally on your machine (I only tried on OSX Leopard),
# you may already have easy_install, so you can skip it.
curl -O http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

# Installing Web module
easy_install web.py

# Downloading simpledb-dev
wget http://simpledb-dev.googlecode.com/files/simpledb-dev-0.1.4.zip
unzip simpledb-dev-0.1.4.zip
cd SimpleDB-Dev/
cd src

# Starting simpledb_dev server
python simpledb_dev.py 8087 (can be any port number.).

2. Open up another terminal, and follow up to "Configure Panda" section of "Getting Started" guide(http://pandastream.com/docs/getting_started)

3. Override simpledb url, before "Create S3 buckets and SimpleDB domains" of "Getting Started" guide(http://pandastream.com/docs/getting_started)
Add the following line at the beginning of config/aws.rb file of Panda

Amazon::SDB::Base::BASE_PATH = 'http://127.0.0.1:8087'

4. Continue "Create S3 buckets and SimpleDB domains" of "Getting Started" guide(http://pandastream.com/docs/getting_started)
Once it's all done, confirm via console that you can connect to simpledb-dev

$cd /var/local/www/panda
$merb -i
~ Loaded DEVELOPMENT Environment...
~ loading gem 'merb-assets' ...
~ loading gem 'merb-mailer' ...
~ loading gem 'merb_helpers' ...
~ loading gem 'uuid' ...
~ loading gem 'to_simple_xml' ...
~ loading gem 'rog' ...
~ loading gem 'amazon_sdb' ...
~ loading gem 'simple_db' ...
~ loading gem 'retryable' ...
~ loading gem 'activesupport' ...
~ loading gem 'rvideo' ...
~ loading gem 'panda' ...
~ loading gem 'gd_resize' ...
~ loading gem 'map_to_hash' ...
~ loading gem 'spec_eql_hash' ...
~ loading gem 'error_sender' ...
~ Compiling routes...
./config/aws.rb:1: warning: already initialized constant BASE_PATH
~ Starting Merb server listening at 0.0.0.0:4000
irb(main):001:0> Amazon::SDB::Base::BASE_PATH
=> "http://127.0.0.1:8087"
irb(main):002:0> User.query
=> [#<User:0xb71fda08 @key="admin", @new_record=false, @attributes=#<Amazon::SDB::Multimap:0xb71fd9e0 @size=nil, @mset={}>>]
irb(main):003:0> Profile.query
=> [#<Profile:0xb71dc1c8 @key="561e38b0-706e-012b-b437-123138002d32", @new_record=false, @attributes=#<Amazon::SDB::Multimap:0xb71dc1a0 @size=nil, @mset={}>>, #<Profile:0xb71dc1b4 @key="562ef420-706e-012b-b437-123138002d32", @new_record=false, @attributes=#<Amazon::SDB::Multimap:0xb71dc164 @size=nil, @mset={}>>, #<Profile:0xb71dc18c @key="56101ea0-706e-012b-b437-123138002d32", @new_record=false, @attributes=#<Amazon::SDB::Multimap:0xb71dc128 @size=nil, @mset={}>>, #<Profile:0xb71dc150 @key="5621fa70-706e-012b-b437-123138002d32", @new_record=false, @attributes=#<Amazon::SDB::Multimap:0xb71dc0ec @size=nil, @mset={}>>, #<Profile:0xb71dc114 @key="56dd8600-706e-012b-b437-123138002d32", @new_record=false, @attributes=#<Amazon::SDB::Multimap:0xb71dc0b0 @size=nil, @mset={}>>]

Enjoy

Makoto

[Jay Adair]

Ah, such a tease, I can't wait to get started with Panda/MySQL.

Still, there are other projects for me to work on in the mean-time.

Jay

2008/9/29 Makoto Inoue <mak...@new-bamboo.co.uk>

--
Yours,

Jay Adair

[greener]

just know that heywatch only offers 1-pass encoding (at least for the
time being), so that is why Panda is a very interesting solution.
their are some ffmpeg profiles that are extremely computational
intensive yet produce fantastic encodes, so they may finally be
reasonable on the cloud... rg

On Sep 16, 3:07 pm, "Damien Tanner" <dam...@new-bamboo.co.uk> wrote:
> Hi Tys,
>
> One of the big benefits of Panda is that it also provides the upload
> form, so your app never has to touch the large video files upload.
> This obviously means to upload videos Panda must be available. If
> Amazon is looking a bit pricing, it might be worth trying out Red's

> MySQL version (hopefully to be pulled into master soon) athttp://github.com/reddavis/pandaas this'll allow you to run Panda on

> any existing servers you have.
>

> OtherwiseHeyWatchisn't a bad choice for the odd encoding job :)

>
> On Tue, Sep 16, 2008 at 9:07 PM, Tys von Gaza <t...@tvg.ca> wrote:
>
>
>
> > Neat, thanks!
> > Tys
>

> > On Tue, Sep 16, 2008 at 12:24 PM, Andy Waite <andywa...@gmail.com> wrote:
>
> >> Might be worth looking athttp://heywatch.comif you only need a few

[Damien Tanner]

Yes, we've tested ffmpeg with 4 threads on a large EC2 instance and
it's very fast :)

[Keith Pitty]

On Sep 17, 8:39 am, "Damien Tanner" <dam...@new-bamboo.co.uk> wrote:
> Not as such. But Makoto is about to start work on a guide for
> 'Developing with Panda' which will cover local installation and a
> little bit of ffmpeg.

I'd love to see that guide! Any indication of when it might be
available?

Regards,
Keith

[Keith Pitty]

On Sep 17, 8:46 pm, reddavis <redda...@gmail.com> wrote:
> As we have now decided to use Panda on a VPS instead of EC2 I'm
> currently writing a capistrano recipe so getting up and ready will be
> just as easy as EC2. The code isn't yet up as I'm currently making it,
> but I'll post here when its finished and working(more important ;) )
>
> Cheers, Red

Red,

Does your fork of Panda just substitute MySQL in place of SimpleDB or
does it obviate the need for S3 as well? In other words, does it rely
on Amazon's services at all?

Regards,
Keith

[Red Davis]

Hey Keith

Yer, it still relies on amazon S3

Cheers, Red

[Martyn Loughran]

Keith,

Just to let you know that I'm currently writing the code to support
file storage locally or on S3. With a bit of luck it will be available
by the end of the day.

You will then be able to run panda without an AWS account by using
simpledb-dev (see
http://groups.google.com/group/pandastream/msg/2f32211a8cc3813e). Also
it should be fairly easy to merge these changes into Red's mysql fork.
Our mysql support is still in the works ;)

Regards,

Martyn

[Martyn Loughran]

FYI I just pushed filesystem support to the master branch on github.

[gr...@g13.tv]

we switch our encoding platform from heywatch to encoding.com due to
reliability problems and latencey issues, (heywatch is in the south of
france :)

On Sep 16, 12:24 pm, Andy Waite <andywa...@gmail.com> wrote:
> Might be worth looking athttp://heywatch.comif you only need a few