public, dynamic backends are magical (in a good way!!)

[Adam Sah]

hey guys--

If you're using Tasks and haven't tried public, dynamic Backends yet,
they're wonderful:
- for tiny apps, you can offload another $0.72/day for free (they're
dynamic!)
- for medium apps, it's a super easy way to get more RAM
- easy migration from Tasks running on frontends to running them on
public, dynamic backends (incl development, debugging, etc.)
- nearly identical debugging/production methods as frontend Tasks
- since they're public, you can launch Tasks directly on the backend,
removing the frontend from the equation, e.g. you can test out how a
Task will perform on a backend, before committing to the architecture
change.
- cool trick: keep the fast Tasks running on frontends, and use the
(expensive) backends for RAM-intensive jobs-- this is a tiny code
change, and also makes it easier to debug large jobs because the
console isn't flooded with little jobs. For my app, I use Tasks for
various indexing jobs in an online marketplace-- now, I send the
"large" sellers with 1000s of SKUs, to a B4 backend.

GAE team:
- I've noticed that my backends can 500-error but not produce a stack
trace in the logs? are other people seeing this?

my choice of migration: (python)

1. create backends.xml, define *one* backend, e.g.
backends:
- name: backend2647324 # since it's public, give it a hard-to-
guess name
class: B4 # big+fast without blowing the free budget
options: dynamic, public

2. add backend-enabled queued, by copying queues from frontends:

queue:
- name: image-processor
...other options here...

- name: image-processor-backend # note how I append "-backend"
to the name
target: backend2647324
...other options here...

3. in your code, start firing Tasks at the backend

# see bottom for taskname() trick
taskqueue.add(url=..., name=taskname("some-identifier-i-will-
recognize"),
queue_name="image-thumbnailer"+("-backend" if
is_gigantic_image(...) else ""))

4. modify your deploy script to update both the front & backends.
Mine (legacy) is in perl:

# deploy bar ==> deploys to foo-bar and to foo-bar-backend
# deploy bar-backend ==> deploys to foo-bar-backend only
# deploy bar-frontend ==> deploys to foo-bar only
$DEST = pop(@ARGV);
my $push_frontend = 1;
if ($DEST =~ s/-backend//) { $push_frontend = 0; }
my $push_backend = 1;
if ($DEST =~ s/-frontend//) { $push_backend = 0; }
if ($push_frontend) {
open(FH, "echo ".$ENV{"GAEPASSWD"}."|python2.5 ".$ENV{"GAEDIR"}."/
appcfg.py --email=".$ENV{"GAEUSER"}." --passin --application=foo-$DEST
update .|");
while (<FH>) { print; }
close FH;
}
if ($push_backend) {
open(FH, "echo ".$ENV{"GAEPASSWD"}."|python2.5 ".$ENV{"GAEDIR"}."/
appcfg.py --email=".$ENV{"GAEUSER"}." --passin --application=foo-$DEST
backends . update|");
while (<FH>) { print; }
close FH;
}


that's it! now, some jobs will go to the frontends and others to the
backends.

hope this helps,
adam


def taskname(string):
"""safely create Google App Engine Task names."""
# add timestamp and random for ultimate uniqueness, strip disallowed
chars
rnd_salt = str(random.randint(100000, 999999))
return re.sub(r'[^a-zA-Z0-9-]', '-', string +
str(datetime.datetime.now())) + "-" + rnd_salt

[Bart Thate]

I wish i could +1 on postings like this ! Gonna try it out tomorrow ;]

Thnx.

Bart Thate

programming schizofrenic - "till freedom come!"

[!] http://protocol.by/botfather

[!] http://botfather.blogspot.com

[!] http://jsonbot.org

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[pman]

i found a new problem - the cron job run twice (frontend and
backend).

how to specify frontend cron job only?

[Greg Darke (Google)]

This should never occur.

A cron task should only run on one instance. If you have specified the 'target' parameter then it should execute on that version/backend otherwise it will execute on the default version of your application.

What is your app id so I can look into this?

--

[pman]

sorry - that's my mistake.

confused by the log message in admin panel.



On Sep 26, 7:45 pm, "Greg Darke (Google)" <darke+goo...@google.com>
wrote:

> This should never occur.
>
> A cron task should only run on one instance. If you have specified the
> 'target' parameter then it should execute on that version/backend otherwise
> it will execute on the default version of your application.
>
> What is your app id so I can look into this?
>

> On 25 September 2011 17:49, pman <pollk...@gmail.com> wrote:
>
>
>
>
>
>
>
> > i found a new problem - the cron job run twice (frontend and
> > backend).
>
> > how to specify frontend cron job only?
>
> > --

[Adam Sah]

given the happy vibes, I'll see about posting a series of these.  Let's call this post #1 of the series.

cheers,

adam

[Tapir]

why it must be public?
private backends doesn't work?

[Adam Sah]

private backends are wonderful too-- but making them public makes it easier to debug actions because you can connect directly to the backend.

for example, you can use the App Engine hosted console (http://con.appspot.com/) to read the memory of the backend directly.

   (I will post on gae console soon-- it's amazing...)

adam

[Tapir]

I see.
The backends looks great!
I will try it soon.