Problems using cron to run a python script

[patric...@gmail.com]

I'm trying to use a cron job to run a python script that updates feeds
on an aggregator site I'm building. But I'm getting trackback errors
telling me it can't import the module feedparser. Here's the error:

Traceback (most recent call last):
File "/path/to/script/update_feeds.py", line 10, in ?
import feedparser
ImportError: No module named feedparser

And here is the cron:

export PYTHONPATH=/path/to/python2.3/site-packages:/path/to/django/app
export DJANGO_SETTINGS_MODULE=myproject.settings
1 * * * * /usr/bin/python /path/to/script/update_feeds.py

I'm running the cron on my Dreamhost account BTW.

Any suggestions on what I'm doing wrong?

[Evert Rol]

>
> Traceback (most recent call last):
> File "/path/to/script/update_feeds.py", line 10, in ?
> import feedparser
> ImportError: No module named feedparser
>
> And here is the cron:
>
> export PYTHONPATH=/path/to/python2.3/site-packages:/path/to/django/app
> export DJANGO_SETTINGS_MODULE=myproject.settings
> 1 * * * * /usr/bin/python /path/to/script/update_feeds.py
>
> I'm running the cron on my Dreamhost account BTW.

Did you try printing out the sys.path at the top of your script, so
see if the PYTHONPATH is picked up correctly from the cron environment
settings? I'd guess it should, but it may go wrong there (I'm no cron
expert).

[patric...@gmail.com]

No, I haven't tried that. How would I go about citing that?

[Evert Rol]

> No, I haven't tried that. How would I go about citing that?

Top of script (possibly just below the she-bang):

import sys
print sys.path

[patric...@gmail.com]

Well part of the problem is that the script itself works fine. I can
run from the command line without a hitch. But when I try to run it
via the cron, I get the traceback about the feedparser module.

[Jonas Christian]

I ran into problems like that using cron and instead of trying to
locate the real error in the paths (we were about to go live ...), I
simple changed the cron line to read:

1 * * * * cd /path/to/script; /usr/bin/python update_feeds.py

Why don't you try that and see if it helps...

Jonas


On 4 Mrz., 14:02, "patrickbee...@gmail.com" <patrickbee...@gmail.com>
wrote:

[patric...@gmail.com]

Unfortunately, I get the same traceback:

Traceback (most recent call last):

File "update_feeds.py", line 10, in ?

import feedparser
ImportError: No module named feedparser

Any idea how to import the module within the cron?

[Evert Rol]

> Unfortunately, I get the same traceback:
>
> Traceback (most recent call last):
> File "update_feeds.py", line 10, in ?
> import feedparser
> ImportError: No module named feedparser

Again: did you print the sys.path?

The 'cd' is not going to work, unless your modules are relative to
the /path/to/script/

I may have a solution, but I first want to know if it's a path problem.

[patric...@gmail.com]

I'm not sure where to print the path. Do I add that to the
update_feeds.py? Or does this go in the cron itself?

If I add it to the script, and then run the script from the command
line, it prints what's on my python path and then runs the script as
normal.

But that's never been the issue. The cron is what's throwing the
error.

[Dj Gilcrease]

On Tue, Mar 4, 2008 at 4:55 AM, patric...@gmail.com
<patric...@gmail.com> wrote:
>
> I'm trying to use a cron job to run a python script that updates feeds
> on an aggregator site I'm building. But I'm getting trackback errors
> telling me it can't import the module feedparser. Here's the error:
>
> Traceback (most recent call last):
> File "/path/to/script/update_feeds.py", line 10, in ?
> import feedparser
> ImportError: No module named feedparser

You can try using django-cron, which is a django app that can run
tasks on a specified schedule and since it runs from within the django
web framework it does not have the environment and path issues that
setting up an actual cron job does

[patric...@gmail.com]

Do you have a link to more information about Django-cron? How do I go
about using this?

On Mar 4, 10:34 am, "Dj Gilcrease" <digitalx...@gmail.com> wrote:
> On Tue, Mar 4, 2008 at 4:55 AM, patrickbee...@gmail.com

[Evert Rol]

> I'm not sure where to print the path. Do I add that to the
> update_feeds.py? Or does this go in the cron itself?
>
> If I add it to the script, and then run the script from the command
> line, it prints what's on my python path and then runs the script as
> normal.
>
> But that's never been the issue. The cron is what's throwing the
> error.

You're not reading: I wrote:

"
Top of script (possibly just below the she-bang):
import sys
print sys.path

"

and

"

1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/
script/update_feeds.log 2>&1

"

Then check your update_feeds.log. That gets the output generated from
update_feeds.py when run from cron.

(although I note that the web archive doesn't agree with my sent
emails; hmm...)

[patric...@gmail.com]

I'm sorry, but I still don't understand what this will do here.

The cron can't run the script because it can't import the feedparser
module or the models from my app. So it doesn't matter if I print the
system path. Again, my problem is getting the python script to run
from the cron in the first place.

[dwi...@gmail.com]

Hopefully, it will print the path before the import statement fails,
and also log the error message that cron generates to a log. So it
does matter if you print the sys.path, because that could tell you
that the location of your feedparser module isn't on that path. I
don't know about dreamhost, but is your cron running as a different
user than you? If so, that could explain why you can run the command
from the shell when you login vs. running it as cron.

Derek

On Mar 4, 12:43 pm, "patrickbee...@gmail.com"

[Evert Rol]

> Well part of the problem is that the script itself works fine. I can
> run from the command line without a hitch. But when I try to run it
> via the cron, I get the traceback about the feedparser module.

I believe that, because your PYTHONPATH will be set correctly from the
command line; but possibly not from a default (basic) shell that cron
uses.
So, again, try putting those two statements at the top of your script,
-before- the 'import feedparser' statement, and let cron do its job.
You may need to redirect the output, eg
1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/
script/update_feeds.log 2>&1
to get that output (I'm not sure how you got the error from the cron
job; through email?)

Check the log file: before the traceback, you should see the result of
'print sys.path'. Check if that includes the correct directories.

[patric...@gmail.com]

OK, I did what you instructed and checked the log file, which did
contain the system path and the traceback about the "can't import
module...".

I see the paths specified don't contain the path to the feedparser
module, which explains the error. My question is how do I add this
path in the cron, especially since the script runs from the command
line?

[patric...@gmail.com]

OK, I'm getting a different error now:

Traceback (most recent call last):

File "update_feeds.py", line 58, in ?
update_feeds()
File "update_feeds.py", line 14, in update_feeds
from knoxd.apps.aggregator.models import Feed, FeedItem
ImportError: No module named knoxd.apps.aggregator.models

I had added the following to my cron:

export PYTHONPATH=$PYTHONPATH:$HOME/lib/python2.3/site-packages

I'm assuming this solved the previous traceback issue.(?)


On Mar 4, 1:27 pm, "patrickbee...@gmail.com" <patrickbee...@gmail.com>
wrote:

[Malcolm Tredinnick]

On Tue, 2008-03-04 at 09:43 -0800, patric...@gmail.com wrote:
> I'm sorry, but I still don't understand what this will do here.
>
> The cron can't run the script because it can't import the feedparser
> module or the models from my app. So it doesn't matter if I print the
> system path. Again, my problem is getting the python script to run
> from the cron in the first place.

No, the problem is that the Python path is incorrectly set and so you
script cannot import the feedparser! You have been receiving help on how
to print out what your script thinks the Python path is set to so that
you can see that it is mosty likely incorrect and then go about fixing
it. You are still at the "debugging what is wrong" phase, not at the
"fixing it" phase.

Work out what the Python path is set to (by printing it out, for
example) and then you can set an environment variable to set it
correctly.

Malcolm

--
I don't have a solution, but I admire your problem.
http://www.pointy-stick.com/blog/

[Evert Rol]

> OK, I did what you instructed and checked the log file, which did
> contain the system path and the traceback about the "can't import
> module...".

Your script did run (you thought it completely failed), but -only- up
the 'import feedparser' part. Anything before that ('print sys.path')
actually worked fine.
Printing the sys.path is often quite useful to debug things like this.

> I see the paths specified don't contain the path to the feedparser
> module, which explains the error. My question is how do I add this
> path in the cron, especially since the script runs from the command
> line?

Presumably the environment variables you can set in your cron file are
limited (to, eg HOME and EMAIL), which is why things don't work.
Assuming this cron works the same as other ones I know, you can set
the variables on the command line itself; that may work.

Try this entry in your cron (and remove the top part with the
environment variable settings):

1 * * * * PYTHONPATH=/path/to/python2.3/site-packages:/path/to/django/
app DJANGO_SETTINGS_MODULE=myproject.settings /usr/bin/python /path/
to/script/update_feeds.py > /path/to/script/update_feeds.log 2>1

(in case mailers mess up things: there are no line breaks in that line)
The log file part is still there to check if things go wrong, but if
it works fine, remove that part ('> /path/to/script/update_feeds.log
2>1')

If this seems a bit messy, you can alter the sys.path at the top of
your script, and set the DJANGO_SETTINGS_MODULE from there as well:

import sys
sys.path.extend(['/path/to/python2.3/site-packages', '/path/to/django/
app')
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
.
.
.
import feedparser
<etc>

I prefer the first, since then the script itself is more portable (no
hardcoded paths etc).

I also came across this thread: http://groups.google.com/group/django-users/browse_thread/thread/d77412f3bbbd1d1a/64d567aadbf4b3f9?hl=en&lnk=gst&q=cron#64d567aadbf4b3f9
which uses something slightly different (solution at the bottom). But
since this involves settings everything in your .bashrc (including
possibly silly greetings like 'echo "How are you today"), that may be
a bit overkill.

Let me know if it works.

[Evert Rol]

> OK, I'm getting a different error now:
>
> Traceback (most recent call last):
> File "update_feeds.py", line 58, in ?
> update_feeds()
> File "update_feeds.py", line 14, in update_feeds
> from knoxd.apps.aggregator.models import Feed, FeedItem
> ImportError: No module named knoxd.apps.aggregator.models
>
> I had added the following to my cron:
>
> export PYTHONPATH=$PYTHONPATH:$HOME/lib/python2.3/site-packages
>

(the difference between what the web archive shows and what my emails
shows, makes it somewhat hard to keep track of which messages you've
already seen, and where we exactly are in the problem solving, but
here goes anyway):

You initially had your PYTHONPATH set differently:
export PYTHONPATH=/path/to/python2.3/site-packages:/path/to/django/app

which is not the same as above. So which one should it be? What does
your regular shell have ('printenv PYTHONPATH')?
In the off-chance you've already got my other email, did you try that
(with the correct PATH settings, whatever those may be)? Ie, setting
the variables on the command line, not in the header of the crontab
file.

[Andrew Holt]

Hi,

OK, write a small script that runs you python job.

Try running the job from the shell:

/usr/bin/python -v /path/to/script/update_feeds.py

This will list all the import statements, so find your imports that
fail in the cron job ...

And then your script would be:

#!/bin/sh

PYTHONPATH= ......
export PYTHONPATH

/usr/bin/python /path/to/script/update_feeds.py

=============================
Andrew Holt
Managing Director
4A Solutions Ltd

M: +44(0) 7841 340608
P: +44(0)1257 268351
Email: andre...@4asolutions.co.uk

Illegitimati non su carborundum !
=============================

[Pigletto]

> OK, I'm getting a different error now:
>
> Traceback (most recent call last):
> File "update_feeds.py", line 58, in ?
> update_feeds()
> File "update_feeds.py", line 14, in update_feeds
> from knoxd.apps.aggregator.models import Feed, FeedItem
> ImportError: No module named knoxd.apps.aggregator.models
>
> I had added the following to my cron:
>
> export PYTHONPATH=$PYTHONPATH:$HOME/lib/python2.3/site-packages
>
> I'm assuming this solved the previous traceback issue.(?)

You may see at the solution from webfaction forum:
http://forum.webfaction.com/viewtopic.php?id=1228

It works for me and I don't have to set any environment variables in
cron settings.
My script scheduler.py (that is called by cron) starts with the
following:

# -*- coding: utf-8 -*-
import sys,os
import datetime

sys.path.append('/home/svnSandbox/newforms-admin')
sys.path.append('/home/svnSandbox/blablabla/projects/xyz)
sys.path.append('/home/svnSandbox/blablabla/projects/xyz/apps')
os.environ['DJANGO_SETTINGS_MODULE'] xyz.settings'

from django.core.management import setup_environ
import settings
setup_environ(settings)

(...)

Of course you have to set paths to fit your application layout.

--
Maciej Wisniowski

[patric...@gmail.com]

Thanks for all the help folks!

I solved the issue this morning by running the python script via a
BASH script that source my BASH_PROFILE, which contains exports my
PYTHONPATH.

Here is what the script looked like:

#!/bin/bash
. ~/.bash_profile

/usr/bin/python /home/user/path/to/django/app/update_feeds.py

The file had write permissions set.

And here is what the cron looked like:

MAILTO="patric...@gmail.com"
1 * * * * /home/user/django/django_projects/myproject/feed_updater

One things for sure, I need to get off Dreamhost. Their forum was
crap, and the service ticket I put in didn't produce anything usable.

On Mar 5, 5:35 am, Pigletto <pigle...@gmail.com> wrote:
> > OK, I'm getting a different error now:
>
> > Traceback (most recent call last):
> >  File "update_feeds.py", line 58, in ?
> >    update_feeds()
> >  File "update_feeds.py", line 14, in update_feeds
> >    from knoxd.apps.aggregator.models import Feed, FeedItem
> > ImportError: No module named knoxd.apps.aggregator.models
>
> > I had added the following to mycron:
>
> > export PYTHONPATH=$PYTHONPATH:$HOME/lib/python2.3/site-packages
>
> > I'm assuming this solved the previous traceback issue.(?)
>
> You may see at the solution from webfaction forum:http://forum.webfaction.com/viewtopic.php?id=1228
>
> It works for me and I don't have to set any environment variables incronsettings.

> My script scheduler.py (that is called bycron) starts with the