Error starting crawler; another crawl is already in progress.

[stevencrader]

Hello

A while back I sent an email regarding RSS updater not working (see
http://groups.google.com/group/thinkupapp/browse_thread/thread/5b5753c10d3081cd)
which I am still working on, but now the crawler won't run at all.

Error starting crawler; another crawl is already in progress. This
happened about a week ago

Is there something in the software that could cause this or is it my
host?

Just wondering which way I should go to solve the problem.

Thanks

Steven Crader
http://people.oregonstate.edu/~craders/thinkup/?u=stevencrader&n=twitter

[Gina Trapani]

On Wed, Dec 1, 2010 at 9:08 PM, stevencrader <steven...@gmail.com> wrote:
> Error starting crawler; another crawl is already in progress. This
> happened about a week ago
>
> Is there something in the software that could cause this or is it my
> host?

That message means exactly what it says--that the crawl process is
running on your server already. ThinkUp won't start a new crawl if
one's in progress. However, it shouldn't remain in that state for a
week. Do you know for sure RSS isn't kicking off the crawler for you?
Can you look at your server's process list and see it there? If so,
try killing that process.

--
http://ginatrapani.org
http://twitter.com/ginatrapani

[stevencrader]

> That message means exactly what it says--that the crawl process is
> running on your server already. ThinkUp won't start a new crawl if
> one's in progress. However, it shouldn't remain in that state for a
> week. Do you know for sure RSS isn't kicking off the crawler for you?
> Can you look at your server's process list and see it there? If so,
> try killing that process.

How would I check this?

I stopped subscribing to the RSS feed yesterday and am still getting
the error today. How do I go about checking the current processes
running on the server?

[Randy R]

On Fri, Dec 3, 2010 at 8:56 AM, stevencrader <steven...@gmail.com> wrote:
> the error today. How do I go about checking the current processes
> running on the server?

Do you have SSH access to the server?

[stevencrader]

> Do you have SSH access to the server?

Yes

Steven Crader
http://people.oregonstate.edu/~craders/thinkup/?u=stevencrader&n=twitter

[stevencrader]

Does anyone have any ideas about this?

I have SSH access to the server but don't know what to do once I get
there.

Thanks

Steven

[Andy Baio]

If you run "ps aux|grep crawl", it'll show you if the crawler's running.  If it is, you should see something like this:

> ps aux|grep crawl

abaio     32141  4.6  0.9  29224 19264 ?        S    10:10   0:00 /usr/bin/php crawl.php us...@example.com

In the example above, the "32141" is the process number.  You can then try to kill the process by running "kill 32141".  If the process is running as another user, you may need to switch to that user's account or log in as root to kill the process.

-- Andy.

--
You received this message because you are subscribed to the Google
Groups "ThinkUp App" group.
http://groups.google.com/group/thinkupapp?hl=en

Find out more about ThinkUp:
http://thinkupapp.com

[stevencrader]

Hey Andy

Her is what I got:

craders@access:~/public_html$ ps aux|grep crawl
craders 17810 0.0 0.0 1664 556 pts/30 S+ 17:09 0:00 grep
crawl
craders@access:~/public_html$ ps aux|grep crawl
craders 18061 0.0 0.0 1664 560 pts/30 S+ 17:09 0:00 grep
crawl

As you can see, the process number changes each time. Since it changes
so much, I am not able to kill it. Any ideas?

Thanks

Steven

On Dec 9, 8:16 am, Andy Baio <a...@waxy.org> wrote:
> If you run "ps aux|grep crawl", it'll show you if the crawler's running.  If
> it is, you should see something like this:
>
> > ps aux|grep crawl
>
> abaio     32141  4.6  0.9  29224 19264 ?        S    10:10   0:00

> /usr/bin/php crawl.php u...@example.com

>
> In the example above, the "32141" is the process number.  You can then try
> to kill the process by running "kill 32141".  If the process is running as
> another user, you may need to switch to that user's account or log in as
> root to kill the process.
>
> -- Andy.
>

[Andy Baio]

The crawler isn't running, that's just showing your own 'grep' command.  :)

Not sure why it'd say there's another crawl in progress... If it's still saying that, and the crawler isn't running, I can only guess that the lock isn't getting released in MySQL...  Anyone have any ideas for debugging this?

-- Andy.

[Amy]

On 17 Dec, 2010, at 7:53 PM, Andy Baio wrote:

> The crawler isn't running, that's just showing your own 'grep' command. :)
>
> Not sure why it'd say there's another crawl in progress... If it's still
> saying that, and the crawler isn't running, I can only guess that the lock
> isn't getting released in MySQL... Anyone have any ideas for debugging
> this?

A couple of ideas:

Steven- You might just double check that
ps aux | grep php
doesn't show anything of interest running, as depending upon the os sometimes 'ps' is not informative about the process args.

If somehow mysql is holding on to the lock, there is apparently an IS_USED_LOCK() call which can help show what is happening.
http://optimmysql.blogspot.com/2007/11/getlock-family.html
Though, if the process is no longer running and the connection no longer open, mysql is certainly 'supposed' to release the lock.

[Btw, I have noticed that if I kill the crawler unceremoniously (which I often do while testing) it would sometimes take a few minutes for the mysql lock to be released. Though the lock is supposed to be released when the connection closes, it appeared that that was not happening right away for some reason (or, the connection was not closing right away after my ctl-c). To further muddy the waters, this has not happened recently... it could have well been some mysql state on my machine which was fixed by a reboot.]

Since you can ssh to your server, then if you are comfortable on the command line you could try just running the crawler directly and see what it outputs. This could help with further debugging.
cd to
<your_thinkup_install>/webapp/crawler
and at the prompt type:
php crawl.php <your_user_email> <your_password>
(replacing all the stuff between brackets w/ your correct values). You may also need to replace 'php' with the correct path to php on your server, e.g. on dreamhost, you would specify /usr/local/php5/bin/php .

If the crawler can start up normally, it won't output any warnings about 'error starting' to the terminal, and instead will just do its run and write its output to <your_thinkup_install>/logs/crawler.log (or whatever log you have specified in your configuration).
If it encounters a lock, it will output some html to the terminal window, containing the 'Error starting crawler; another crawl is already in progress.' warning, and then terminate.

[stevencrader]

Thanks for your help Amy and Andy. I got it to reset and work by
moving the crawl files to a different directory for a couple of days
and then moving them back. I just ran the crawl 2 times without any
problems.

Thanks again

Steven

On Dec 18, 9:55 am, Amy <amy.j.d...@gmail.com> wrote:
> On 17 Dec, 2010, at 7:53 PM, Andy Baio wrote:
>
> > The crawler isn't running, that's just showing your own 'grep' command.  :)
>
> > Not sure why it'd say there's another crawl in progress... If it's still
> > saying that, and the crawler isn't running, I can only guess that the lock
> > isn't getting released in MySQL...  Anyone have any ideas for debugging
> > this?
>
> A couple of ideas:
>
> Steven- You might just double check that
>    ps aux | grep php
> doesn't show anything of interest running, as depending upon the os sometimes 'ps' is not informative about the process args.
>
> If somehow mysql is holding on to the lock, there is apparently an IS_USED_LOCK() call which can help show what is happening.
>  http://optimmysql.blogspot.com/2007/11/getlock-family.html
> Though, if the process is no longer running and the connection no longer open, mysql is certainly 'supposed' to release the lock.
>
> [Btw, I have noticed that if I kill the crawler unceremoniously (which I often do while testing) it would sometimes take a few minutes for the mysql lock to be released. Though the lock is supposed to be released when the connection closes, it appeared that that was not happening right away for some reason (or, the connection was not closing right away after my ctl-c). To further muddy the waters, this has not happened recently... it could have well been some mysql state on my machine which was fixed by a reboot.]
>
> Since you can ssh to your server, then if you are comfortable on the command line you could try just running the crawler directly and see what it outputs. This could help with further debugging.
> cd to
>   <your_thinkup_install>/webapp/crawler
> and at the prompt type:
>   php crawl.php <your_user_email> <your_password>
> (replacing all the stuff between brackets w/ your correct values). You may also need to replace 'php' with the correct path to php on your server, e.g. on dreamhost, you would specify /usr/local/php5/bin/php .
>
> If the crawler can start up normally, it won't output any warnings about 'error starting' to the terminal, and instead will just do its run and write its output to <your_thinkup_install>/logs/crawler.log (or whatever log you have specified in your configuration).
> If it encounters a lock, it will output some html to the terminal window, containing the 'Error starting crawler; another crawl is already in progress.' warning, and then terminate.
>
>
>
>
>
>
>
>
>
> > -- Andy.
>

[Alper Çuğun]

I'm on shared hosting so I can't restart the database. I tried everything to find the process or pid but they are nowhere to be found. I'm going to try this move around technique, but a better solution would be appreciated.

- Alper

[Alper Çuğun]

I duplicated my database but it still throws this error. -Alper

[Alan Storm]

Alper,

This is a shot in the dark, but if your'e on ThinkUp 2.0 try loading the crawler iframe directly.

    http://stack-exchange-crawl.dev/crawler/run.php

I was running into similar problems with the crawler after upgrading, and I was able to fix it by doing the above.  

- Alan

[Alper Çugun]

Ok, I just did that but it still gives the same result, but then without the surrounding page. I also have no idea why that should change anything of the outcome…

-- 

Alper

--

-- You received this message because you are subscribed to the Google

Groups ThinkUp group.

http://groups.google.com/group/thinkupapp?hl=en
 
Find out more about ThinkUp:
http://thinkupapp.com

---

You received this message because you are subscribed to the Google Groups "ThinkUp" group.

To unsubscribe from this group and stop receiving emails from it, send an email to thinkupapp+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Alper Çugun M.Sc.
T. +49 162 3127744 / +31 6 24 55 33 06

[Mike Kadin]

I too have been experiencing this issue.

I'm new to thinkup, but a reasonably experienced php developer / command line person, so if there's some debugging info I can gather to be helpful, let me know.