Daemon service
331 views
Skip to first unread message
SombraX
Oct 31, 2010, 9:54:48 AM10/31/10
to Google App Engine
Hello everyone. I try to build a program that works non-stop, is an
infinite while that collects information from different Web sites and
record the information in the database. The problem is that at startup
is interpreted as a request and get the timeout error.
Is it possible to build a service daemon with python and boot from the
console app enginne?
(my english is terrible , sorry....cago en die).
infinite while that collects information from different Web sites and
record the information in the database. The problem is that at startup
is interpreted as a request and get the timeout error.
Is it possible to build a service daemon with python and boot from the
console app enginne?
(my english is terrible , sorry....cago en die).
djidjadji
Oct 31, 2010, 10:06:33 AM10/31/10
to google-a...@googlegroups.com
Every request should finish within 30 seconds.
You can setup a cron job and/or task queue chain to simulate a
continuous processing.
You have to keep state information in the datastore or part of the
task arguments.
You can setup a cron job and/or task queue chain to simulate a
continuous processing.
You have to keep state information in the datastore or part of the
task arguments.
2010/10/31 SombraX <somb...@gmail.com>:
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
sombra XX
Oct 31, 2010, 12:17:44 PM10/31/10
to google-a...@googlegroups.com
djidjadji: Yes, I thought about the cron, but the intervals are every 1 minute, and I need to be every 5 seconds
2010/10/31 djidjadji <djid...@gmail.com>
Robert Kluin
Oct 31, 2010, 2:51:14 PM10/31/10
to google-a...@googlegroups.com
Then use the other part of djidjadji's suggestion: task-queue.
Robert
sombra XX
Oct 31, 2010, 10:34:57 PM10/31/10
to google-a...@googlegroups.com
Robert Kluin
Nov 1, 2010, 12:13:45 AM11/1/10
to google-a...@googlegroups.com
http://code.google.com/appengine/docs/python/taskqueue/overview.html
First, pop a task to start your crawl. From that task fire another
task to continue it. Keep doing that until you get what you need or
run out of resources. You'll probably want to use either named or
transactional tasks to prevent a fork bomb.
Use cron, or a user request to initiate your 'spider.' Or to ensure
the process keeps running.
Robert
sombra XX
Nov 1, 2010, 12:24:32 PM11/1/10
to google-a...@googlegroups.com
b.r...@web.de
Nov 30, 2010, 10:27:05 AM11/30/10
to Google App Engine
Can somebody please exlpain this more closely. I have more or less the
same issue. I need to get a constant uptodate twitter stream to my
webapp. Dont understand at all how I can substitude the daemon like
behaviour. I understand that the first task hast do to the parsing job
but what about the second? It just calls the first again and again
like a loop?
On Nov 1, 5:13 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> http://code.google.com/appengine/docs/python/taskqueue/overview.html
>
> First, pop a task to start your crawl. From that task fire another
> task to continue it. Keep doing that until you get what you need or
> run out of resources. You'll probably want to use either named or
> transactional tasks to prevent a fork bomb.
>
> Use cron, or a user request to initiate your 'spider.' Or to ensure
> the process keeps running.
>
> Robert
>
> > 2010/10/31 Robert Kluin <robert.kl...@gmail.com>
same issue. I need to get a constant uptodate twitter stream to my
webapp. Dont understand at all how I can substitude the daemon like
behaviour. I understand that the first task hast do to the parsing job
but what about the second? It just calls the first again and again
like a loop?
On Nov 1, 5:13 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> http://code.google.com/appengine/docs/python/taskqueue/overview.html
>
> First, pop a task to start your crawl. From that task fire another
> task to continue it. Keep doing that until you get what you need or
> run out of resources. You'll probably want to use either named or
> transactional tasks to prevent a fork bomb.
>
> Use cron, or a user request to initiate your 'spider.' Or to ensure
> the process keeps running.
>
> Robert
>
>
> >> Then use the other part of djidjadji's suggestion: task-queue.
>
> >> Robert
>
> >> Then use the other part of djidjadji's suggestion: task-queue.
>
> >> Robert
>
> >> On Sun, Oct 31, 2010 at 12:17, sombra XX <sombr...@gmail.com> wrote:
> >> > djidjadji: Yes, I thought about the cron, but the intervals are every 1
> >> > minute, and I need to be every 5 seconds
>
> >> > 2010/10/31 djidjadji <djidja...@gmail.com>
> >> > djidjadji: Yes, I thought about the cron, but the intervals are every 1
> >> > minute, and I need to be every 5 seconds
>
>
> >> >> Every request should finish within 30 seconds.
> >> >> You can setup a cron job and/or task queue chain to simulate a
> >> >> continuous processing.
> >> >> You have to keep state information in the datastore or part of the
> >> >> task arguments.
>
> >> >> 2010/10/31 SombraX <sombr...@gmail.com>:
> >> >> Every request should finish within 30 seconds.
> >> >> You can setup a cron job and/or task queue chain to simulate a
> >> >> continuous processing.
> >> >> You have to keep state information in the datastore or part of the
> >> >> task arguments.
>
> >> >> > Hello everyone. I try to build a program that works non-stop, is an
> >> >> > infinite while that collects information from different Web sites and
> >> >> > record the information in the database. The problem is that at
> >> >> > startup
> >> >> > is interpreted as a request and get the timeout error.
> >> >> > Is it possible to build a servicedaemonwith python and boot from
> >> >> > infinite while that collects information from different Web sites and
> >> >> > record the information in the database. The problem is that at
> >> >> > startup
> >> >> > is interpreted as a request and get the timeout error.
Robert Kluin
Dec 1, 2010, 9:48:12 PM12/1/10
to google-a...@googlegroups.com
Hi,
It sounds like you've got the basic idea. You'll start a task that
will fetch the twitter stream you're interested in, that task will
insert another task to fetch again, and the process continues. You
need to name your tasks to prevent a task-bomb (fork bomb). I think
Nick talks about that in his blog:
http://blog.notdot.net/2010/03/Task-Queue-task-chaining-done-right
It sounds like you've got the basic idea. You'll start a task that
will fetch the twitter stream you're interested in, that task will
insert another task to fetch again, and the process continues. You
need to name your tasks to prevent a task-bomb (fork bomb). I think
Nick talks about that in his blog:
http://blog.notdot.net/2010/03/Task-Queue-task-chaining-done-right
Although if it is something you just need to do over and over every
five minutes I would look into using cron.
Robert
Reply all
Reply to author
Forward
0 new messages