How to build multimedia platform backend using App Engine? any recommendation/suggestion

[Hussain Hajjar]

Hello everyone,

 

We are trying to develop a multimedia platform that serves video, image, audio and/or text to end-users (mobile and web). At first, we will launch iOS and Android apps then we will move to the web.

I am thinking to do this using App Engine, but I am not sure especially many people argued that it is expensive. Knowing that Spanchat (which is similar to our use-case) is built on App Engine encouraged us to use App Engine plus all our products are on Google Cloud, so we are more comfortable using Google products.

There are multiple points to take into consideration for our platform, we want it to be able to do the following:

1.       Video/audio/image processing and manipulation.

2.       Video transcoding and transmuxing where it has to create multiple video formats and outputs based on user connection (adaptive streaming).

3.       Video streaming in HLS and MGEP-DASH (future feature) using rtmp, rtmpt, rtmplite, sip or siprtmp.

4.       Resumable uploads and downloads.

5.       Caching and fast media delivery.

I have done a lot of research on how we can do any of these on App Engine but didn't find any helpful forward resources. All I found on official documentation and guides is how you can use/integrate third-party services like wowza, fastly or bitmovin which are paid services.

Here are questions I am trying to have answers to:

1.       Is App Engine the right choice against Compute Engine/Endpoints/Backend? Cost wise, limitations (App Engine has limits comparing to Compute Engine, but it needs to be managed by hand) and Scalability (App Engine scales but Compute Engine doesn't).

2.       What is the best language for the job in terms of cost, performance, maintenance and libraries/samples available? (I am comfortable with java, python and nodejs).

3.       Blobstore or Cloud storage (as App Engine has 32MB request limit)?

4.       Do we need CDN and will it reduce caching and network traffic costs? aside from having Google CDN, memecache and Cloud storage multi-region availability and redundancy.

5.       How's Snapchat's infrastructure is done? What micro-services they have? What does every micro-service do? Language choice (I read java and python, but which one is responsible for the media?)?

It would be great if anyone can share his/her experience, resources, samples or anything related.

Please don't hesitate to ask any questions and I hope that it was clear to you what we are trying to build.

 

Grateful for your help,

Hussain Hajjar

[Jeff Schnitzer]

I’ve done a lot of media work on GAE. Most of your work will be with the Google Cloud Storage API, and you can do that work from pretty much anywhere, including GAE. You won’t be doing video transcoding from GAE Standard, but you probably shouldn’t be doing video transcoding at all. Plenty of services for that, and I would be shocked if you couldn’t find one that directly integrated with GCS.

Don’t use the blobstore; that’s effectively deprecated. GCS is awesome. You’ll create signed urls that your users upload directly to; resuming uploads is part of the API. If you have any traffic whatsoever you’ll want some sort of CDN, but that’s a commodity service. For simple images you can use something like imgix or rethumb to perform resizing + caching on the fly. It’s been a long time since I’ve done video but look around, CDNs exist.

I wouldn’t sweat the language choice. Use what your team is most comfortable with. You aren’t going to write video transcoders yourself (unless you’re insane) so pretty much all you’re doing is gluing the bits together, and that isn’t performance sensitive.

AFAIK Snapchat hasn’t published much about their architecture, but I wouldn’t pay too much attention even if they did. You don’t have their scale problems yet. It’s easier to adapt a small system to large scale than to adapt a large system to changing requirements.

Suerte,

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/4fd4beb6-28f7-4a6c-9d48-2de66baca0a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Katayoon (Cloud Platform Support)]

Hi, I should add that Google Groups are reserved for general product discussions. We can neither provide you any advice on your system’s architecture, nor discuss on Snapchat’s infrastructure.

Here, you can take a look at pricing details on each GCP product and also you can have an estimation on the fees provided by Google Cloud Platform Pricing Calculator to design a cost effective system.

Furthermore, here you may find the solutions offered by Google Cloud and you can also contact Google Cloud Platform sales team and discuss your project with them.