I'm trying to use a cron job to run a python script that updates feeds
on an aggregator site I'm building. But I'm getting trackback errors
telling me it can't import the module feedparser. Here's the error:
Traceback (most recent call last):
File "/path/to/script/update_feeds.py", line 10, in ?
import feedparser
ImportError: No module named feedparser
> I'm running the cron on my Dreamhost account BTW.
Did you try printing out the sys.path at the top of your script, so see if the PYTHONPATH is picked up correctly from the cron environment settings? I'd guess it should, but it may go wrong there (I'm no cron expert).
> > I'm running the cron on my Dreamhost account BTW.
> Did you try printing out the sys.path at the top of your script, so
> see if the PYTHONPATH is picked up correctly from the cron environment
> settings? I'd guess it should, but it may go wrong there (I'm no cron
> expert).
>>> I'm running the cron on my Dreamhost account BTW.
>> Did you try printing out the sys.path at the top of your script, so >> see if the PYTHONPATH is picked up correctly from the cron >> environment >> settings? I'd guess it should, but it may go wrong there (I'm no cron >> expert).
Well part of the problem is that the script itself works fine. I can
run from the command line without a hitch. But when I try to run it
via the cron, I get the traceback about the feedparser module.
On Mar 4, 7:36 am, Evert Rol <evert....@gmail.com> wrote:
> >>> I'm running the cron on my Dreamhost account BTW.
> >> Did you try printing out the sys.path at the top of your script, so
> >> see if the PYTHONPATH is picked up correctly from the cron
> >> environment
> >> settings? I'd guess it should, but it may go wrong there (I'm no cron
> >> expert).
I ran into problems like that using cron and instead of trying to
locate the real error in the paths (we were about to go live ...), I
simple changed the cron line to read:
1 * * * * cd /path/to/script; /usr/bin/python update_feeds.py
Why don't you try that and see if it helps...
Jonas
On 4 Mrz., 14:02, "patrickbee...@gmail.com" <patrickbee...@gmail.com>
wrote:
> Well part of the problem is that the script itself works fine. I can
> run from the command line without a hitch. But when I try to run it
> via the cron, I get the traceback about the feedparser module.
> I ran into problems like that using cron and instead of trying to
> locate the real error in the paths (we were about to go live ...), I
> simple changed the cron line to read:
> 1 * * * * cd /path/to/script; /usr/bin/python update_feeds.py
> Why don't you try that and see if it helps...
> Jonas
> On 4 Mrz., 14:02, "patrickbee...@gmail.com" <patrickbee...@gmail.com>
> wrote:
> > Well part of the problem is that the script itself works fine. I can
> > run from the command line without a hitch. But when I try to run it
> > via the cron, I get the traceback about the feedparser module.
> Any idea how to import the module within the cron?
> On Mar 4, 9:07 am, Jonas Christian <goo...@vonposer.de> wrote: >> I ran into problems like that using cron and instead of trying to >> locate the real error in the paths (we were about to go live ...), I >> simple changed the cron line to read:
>> 1 * * * * cd /path/to/script; /usr/bin/python update_feeds.py
>> Why don't you try that and see if it helps...
>> Jonas
>> On 4 Mrz., 14:02, "patrickbee...@gmail.com" <patrickbee...@gmail.com> >> wrote:
>>> Well part of the problem is that the script itself works fine. I can >>> run from the command line without a hitch. But when I try to run it >>> via the cron, I get the traceback about the feedparser module.
> > Traceback (most recent call last):
> > File "update_feeds.py", line 10, in ?
> > import feedparser
> > ImportError: No module named feedparser
> Again: did you print the sys.path?
> The 'cd' is not going to work, unless your modules are relative to
> the /path/to/script/
> I may have a solution, but I first want to know if it's a path problem.
> > Any idea how to import the module within the cron?
> > On Mar 4, 9:07 am, Jonas Christian <goo...@vonposer.de> wrote:
> >> I ran into problems like that using cron and instead of trying to
> >> locate the real error in the paths (we were about to go live ...), I
> >> simple changed the cron line to read:
> >>> Well part of the problem is that the script itself works fine. I can
> >>> run from the command line without a hitch. But when I try to run it
> >>> via the cron, I get the traceback about the feedparser module.
> I'm trying to use a cron job to run a python script that updates feeds > on an aggregator site I'm building. But I'm getting trackback errors > telling me it can't import the module feedparser. Here's the error:
> Traceback (most recent call last): > File "/path/to/script/update_feeds.py", line 10, in ? > import feedparser > ImportError: No module named feedparser
You can try using django-cron, which is a django app that can run tasks on a specified schedule and since it runs from within the django web framework it does not have the environment and path issues that setting up an actual cron job does
> > I'm trying to use a cron job to run a python script that updates feeds
> > on an aggregator site I'm building. But I'm getting trackback errors
> > telling me it can't import the module feedparser. Here's the error:
> > Traceback (most recent call last):
> > File "/path/to/script/update_feeds.py", line 10, in ?
> > import feedparser
> > ImportError: No module named feedparser
> You can try using django-cron, which is a django app that can run
> tasks on a specified schedule and since it runs from within the django
> web framework it does not have the environment and path issues that
> setting up an actual cron job does
>>> Traceback (most recent call last): >>> File "update_feeds.py", line 10, in ? >>> import feedparser >>> ImportError: No module named feedparser
>> Again: did you print the sys.path?
>> The 'cd' is not going to work, unless your modules are relative to >> the /path/to/script/
>> I may have a solution, but I first want to know if it's a path >> problem.
>>> Any idea how to import the module within the cron?
>>> On Mar 4, 9:07 am, Jonas Christian <goo...@vonposer.de> wrote: >>>> I ran into problems like that using cron and instead of trying to >>>> locate the real error in the paths (we were about to go >>>> live ...), I >>>> simple changed the cron line to read:
>>>> 1 * * * * cd /path/to/script; /usr/bin/python update_feeds.py
>>>> Why don't you try that and see if it helps...
>>>> Jonas
>>>> On 4 Mrz., 14:02, "patrickbee...@gmail.com" >>>> <patrickbee...@gmail.com> >>>> wrote:
>>>>> Well part of the problem is that the script itself works fine. I >>>>> can >>>>> run from the command line without a hitch. But when I try to run >>>>> it >>>>> via the cron, I get the traceback about the feedparser module.
I'm sorry, but I still don't understand what this will do here.
The cron can't run the script because it can't import the feedparser
module or the models from my app. So it doesn't matter if I print the
system path. Again, my problem is getting the python script to run
from the cron in the first place.
On Mar 4, 11:55 am, Evert Rol <evert....@gmail.com> wrote:
> > I'm not sure where to print the path. Do I add that to the
> > update_feeds.py? Or does this go in the cron itself?
> > If I add it to the script, and then run the script from the command
> > line, it prints what's on my python path and then runs the script as
> > normal.
> > But that's never been the issue. The cron is what's throwing the
> > error.
> You're not reading: I wrote:
> "
> Top of script (possibly just below the she-bang):
> import sys
> print sys.path
> Then check your update_feeds.log. That gets the output generated from
> update_feeds.py when run from cron.
> (although I note that the web archive doesn't agree with my sent
> emails; hmm...)
> >>> Unfortunately, I get the same traceback:
> >>> Traceback (most recent call last):
> >>> File "update_feeds.py", line 10, in ?
> >>> import feedparser
> >>> ImportError: No module named feedparser
> >> Again: did you print the sys.path?
> >> The 'cd' is not going to work, unless your modules are relative to
> >> the /path/to/script/
> >> I may have a solution, but I first want to know if it's a path
> >> problem.
> >>> Any idea how to import the module within the cron?
> >>> On Mar 4, 9:07 am, Jonas Christian <goo...@vonposer.de> wrote:
> >>>> I ran into problems like that using cron and instead of trying to
> >>>> locate the real error in the paths (we were about to go
> >>>> live ...), I
> >>>> simple changed the cron line to read:
> >>>>> Well part of the problem is that the script itself works fine. I
> >>>>> can
> >>>>> run from the command line without a hitch. But when I try to run
> >>>>> it
> >>>>> via the cron, I get the traceback about the feedparser module.
Hopefully, it will print the path before the import statement fails,
and also log the error message that cron generates to a log. So it
does matter if you print the sys.path, because that could tell you
that the location of your feedparser module isn't on that path. I
don't know about dreamhost, but is your cron running as a different
user than you? If so, that could explain why you can run the command
from the shell when you login vs. running it as cron.
<patrickbee...@gmail.com> wrote:
> I'm sorry, but I still don't understand what this will do here.
> The cron can't run the script because it can't import the feedparser
> module or the models from my app. So it doesn't matter if I print the
> system path. Again, my problem is getting the python script to run
> from the cron in the first place.
> On Mar 4, 11:55 am, Evert Rol <evert....@gmail.com> wrote:
> > > I'm not sure where to print the path. Do I add that to the
> > > update_feeds.py? Or does this go in the cron itself?
> > > If I add it to the script, and then run the script from the command
> > > line, it prints what's on my python path and then runs the script as
> > > normal.
> > > But that's never been the issue. The cron is what's throwing the
> > > error.
> > You're not reading: I wrote:
> > "
> > Top of script (possibly just below the she-bang):
> > import sys
> > print sys.path
> > Then check your update_feeds.log. That gets the output generated from
> > update_feeds.py when run from cron.
> > (although I note that the web archive doesn't agree with my sent
> > emails; hmm...)
> > >>> Unfortunately, I get the same traceback:
> > >>> Traceback (most recent call last):
> > >>> File "update_feeds.py", line 10, in ?
> > >>> import feedparser
> > >>> ImportError: No module named feedparser
> > >> Again: did you print the sys.path?
> > >> The 'cd' is not going to work, unless your modules are relative to
> > >> the /path/to/script/
> > >> I may have a solution, but I first want to know if it's a path
> > >> problem.
> > >>> Any idea how to import the module within the cron?
> > >>> On Mar 4, 9:07 am, Jonas Christian <goo...@vonposer.de> wrote:
> > >>>> I ran into problems like that using cron and instead of trying to
> > >>>> locate the real error in the paths (we were about to go
> > >>>> live ...), I
> > >>>> simple changed the cron line to read:
> > >>>>> Well part of the problem is that the script itself works fine. I
> > >>>>> can
> > >>>>> run from the command line without a hitch. But when I try to run
> > >>>>> it
> > >>>>> via the cron, I get the traceback about the feedparser module.
> Well part of the problem is that the script itself works fine. I can > run from the command line without a hitch. But when I try to run it > via the cron, I get the traceback about the feedparser module.
I believe that, because your PYTHONPATH will be set correctly from the command line; but possibly not from a default (basic) shell that cron uses. So, again, try putting those two statements at the top of your script, -before- the 'import feedparser' statement, and let cron do its job. You may need to redirect the output, eg 1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/ script/update_feeds.log 2>&1 to get that output (I'm not sure how you got the error from the cron job; through email?)
Check the log file: before the traceback, you should see the result of 'print sys.path'. Check if that includes the correct directories.
>>>>> I'm running the cron on my Dreamhost account BTW.
>>>> Did you try printing out the sys.path at the top of your script, so >>>> see if the PYTHONPATH is picked up correctly from the cron >>>> environment >>>> settings? I'd guess it should, but it may go wrong there (I'm no >>>> cron >>>> expert).
OK, I did what you instructed and checked the log file, which did
contain the system path and the traceback about the "can't import
module...".
I see the paths specified don't contain the path to the feedparser
module, which explains the error. My question is how do I add this
path in the cron, especially since the script runs from the command
line?
On Mar 4, 9:02 am, Evert Rol <evert....@gmail.com> wrote:
> > Well part of the problem is that the script itself works fine. I can
> > run from the command line without a hitch. But when I try to run it
> > via the cron, I get the traceback about the feedparser module.
> I believe that, because your PYTHONPATH will be set correctly from the
> command line; but possibly not from a default (basic) shell that cron
> uses.
> So, again, try putting those two statements at the top of your script,
> -before- the 'import feedparser' statement, and let cron do its job.
> You may need to redirect the output, eg
> 1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/
> script/update_feeds.log 2>&1
> to get that output (I'm not sure how you got the error from the cron
> job; through email?)
> Check the log file: before the traceback, you should see the result of
> 'print sys.path'. Check if that includes the correct directories.
> >> Top of script (possibly just below the she-bang):
> >> import sys
> >> print sys.path
> >>>>> Traceback (most recent call last):
> >>>>> File "/path/to/script/update_feeds.py", line 10, in ?
> >>>>> import feedparser
> >>>>> ImportError: No module named feedparser
> >>>>> I'm running the cron on my Dreamhost account BTW.
> >>>> Did you try printing out the sys.path at the top of your script, so
> >>>> see if the PYTHONPATH is picked up correctly from the cron
> >>>> environment
> >>>> settings? I'd guess it should, but it may go wrong there (I'm no
> >>>> cron
> >>>> expert).
Traceback (most recent call last):
File "update_feeds.py", line 58, in ?
update_feeds()
File "update_feeds.py", line 14, in update_feeds
from knoxd.apps.aggregator.models import Feed, FeedItem
ImportError: No module named knoxd.apps.aggregator.models
> OK, I did what you instructed and checked the log file, which did
> contain the system path and the traceback about the "can't import
> module...".
> I see the paths specified don't contain the path to the feedparser
> module, which explains the error. My question is how do I add this
> path in the cron, especially since the script runs from the command
> line?
> On Mar 4, 9:02 am, Evert Rol <evert....@gmail.com> wrote:
> > > Well part of the problem is that the script itself works fine. I can
> > > run from the command line without a hitch. But when I try to run it
> > > via the cron, I get the traceback about the feedparser module.
> > I believe that, because your PYTHONPATH will be set correctly from the
> > command line; but possibly not from a default (basic) shell that cron
> > uses.
> > So, again, try putting those two statements at the top of your script,
> > -before- the 'import feedparser' statement, and let cron do its job.
> > You may need to redirect the output, eg
> > 1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/
> > script/update_feeds.log 2>&1
> > to get that output (I'm not sure how you got the error from the cron
> > job; through email?)
> > Check the log file: before the traceback, you should see the result of
> > 'print sys.path'. Check if that includes the correct directories.
> > >> Top of script (possibly just below the she-bang):
> > >> import sys
> > >> print sys.path
> > >>>>> Traceback (most recent call last):
> > >>>>> File "/path/to/script/update_feeds.py", line 10, in ?
> > >>>>> import feedparser
> > >>>>> ImportError: No module named feedparser
> > >>>>> I'm running the cron on my Dreamhost account BTW.
> > >>>> Did you try printing out the sys.path at the top of your script, so
> > >>>> see if the PYTHONPATH is picked up correctly from the cron
> > >>>> environment
> > >>>> settings? I'd guess it should, but it may go wrong there (I'm no
> > >>>> cron
> > >>>> expert).
On Tue, 2008-03-04 at 09:43 -0800, patrickbee...@gmail.com wrote: > I'm sorry, but I still don't understand what this will do here.
> The cron can't run the script because it can't import the feedparser > module or the models from my app. So it doesn't matter if I print the > system path. Again, my problem is getting the python script to run > from the cron in the first place.
No, the problem is that the Python path is incorrectly set and so you script cannot import the feedparser! You have been receiving help on how to print out what your script thinks the Python path is set to so that you can see that it is mosty likely incorrect and then go about fixing it. You are still at the "debugging what is wrong" phase, not at the "fixing it" phase.
Work out what the Python path is set to (by printing it out, for example) and then you can set an environment variable to set it correctly.
> OK, I did what you instructed and checked the log file, which did > contain the system path and the traceback about the "can't import > module...".
Your script did run (you thought it completely failed), but -only- up the 'import feedparser' part. Anything before that ('print sys.path') actually worked fine. Printing the sys.path is often quite useful to debug things like this.
> I see the paths specified don't contain the path to the feedparser > module, which explains the error. My question is how do I add this > path in the cron, especially since the script runs from the command > line?
Presumably the environment variables you can set in your cron file are limited (to, eg HOME and EMAIL), which is why things don't work. Assuming this cron works the same as other ones I know, you can set the variables on the command line itself; that may work.
Try this entry in your cron (and remove the top part with the environment variable settings):
(in case mailers mess up things: there are no line breaks in that line) The log file part is still there to check if things go wrong, but if it works fine, remove that part ('> /path/to/script/update_feeds.log 2>1')
If this seems a bit messy, you can alter the sys.path at the top of your script, and set the DJANGO_SETTINGS_MODULE from there as well:
I prefer the first, since then the script itself is more portable (no hardcoded paths etc).
I also came across this thread: http://groups.google.com/group/django-users/browse_thread/thread/d774... which uses something slightly different (solution at the bottom). But since this involves settings everything in your .bashrc (including possibly silly greetings like 'echo "How are you today"), that may be a bit overkill.
>>> Well part of the problem is that the script itself works fine. I can >>> run from the command line without a hitch. But when I try to run it >>> via the cron, I get the traceback about the feedparser module.
>> I believe that, because your PYTHONPATH will be set correctly from >> the >> command line; but possibly not from a default (basic) shell that cron >> uses. >> So, again, try putting those two statements at the top of your >> script, >> -before- the 'import feedparser' statement, and let cron do its job. >> You may need to redirect the output, eg >> 1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/ >> script/update_feeds.log 2>&1 >> to get that output (I'm not sure how you got the error from the cron >> job; through email?)
>> Check the log file: before the traceback, you should see the result >> of >> 'print sys.path'. Check if that includes the correct directories.
>>>> Top of script (possibly just below the she-bang):
>>>> import sys >>>> print sys.path
>>>>>>> Traceback (most recent call last): >>>>>>> File "/path/to/script/update_feeds.py", line 10, in ? >>>>>>> import feedparser >>>>>>> ImportError: No module named feedparser
>>>>>>> I'm running the cron on my Dreamhost account BTW.
>>>>>> Did you try printing out the sys.path at the top of your >>>>>> script, so >>>>>> see if the PYTHONPATH is picked up correctly from the cron >>>>>> environment >>>>>> settings? I'd guess it should, but it may go wrong there (I'm no >>>>>> cron >>>>>> expert).
> Traceback (most recent call last): > File "update_feeds.py", line 58, in ? > update_feeds() > File "update_feeds.py", line 14, in update_feeds > from knoxd.apps.aggregator.models import Feed, FeedItem > ImportError: No module named knoxd.apps.aggregator.models
(the difference between what the web archive shows and what my emails shows, makes it somewhat hard to keep track of which messages you've already seen, and where we exactly are in the problem solving, but here goes anyway):
You initially had your PYTHONPATH set differently: export PYTHONPATH=/path/to/python2.3/site-packages:/path/to/django/app
which is not the same as above. So which one should it be? What does your regular shell have ('printenv PYTHONPATH')? In the off-chance you've already got my other email, did you try that (with the correct PATH settings, whatever those may be)? Ie, setting the variables on the command line, not in the header of the crontab file.
>> OK, I did what you instructed and checked the log file, which did >> contain the system path and the traceback about the "can't import >> module...".
>> I see the paths specified don't contain the path to the feedparser >> module, which explains the error. My question is how do I add this >> path in the cron, especially since the script runs from the command >> line?
>> On Mar 4, 9:02 am, Evert Rol <evert....@gmail.com> wrote:
>>>> Well part of the problem is that the script itself works fine. I >>>> can >>>> run from the command line without a hitch. But when I try to run it >>>> via the cron, I get the traceback about the feedparser module.
>>> I believe that, because your PYTHONPATH will be set correctly from >>> the >>> command line; but possibly not from a default (basic) shell that >>> cron >>> uses. >>> So, again, try putting those two statements at the top of your >>> script, >>> -before- the 'import feedparser' statement, and let cron do its job. >>> You may need to redirect the output, eg >>> 1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/ >>> to/ >>> script/update_feeds.log 2>&1 >>> to get that output (I'm not sure how you got the error from the cron >>> job; through email?)
>>> Check the log file: before the traceback, you should see the >>> result of >>> 'print sys.path'. Check if that includes the correct directories.
>>>>> Top of script (possibly just below the she-bang):
>>>>> import sys >>>>> print sys.path
>>>>>>>> Traceback (most recent call last): >>>>>>>> File "/path/to/script/update_feeds.py", line 10, in ? >>>>>>>> import feedparser >>>>>>>> ImportError: No module named feedparser
>>>>>>>> I'm running the cron on my Dreamhost account BTW.
>>>>>>> Did you try printing out the sys.path at the top of your >>>>>>> script, so >>>>>>> see if the PYTHONPATH is picked up correctly from the cron >>>>>>> environment >>>>>>> settings? I'd guess it should, but it may go wrong there (I'm no >>>>>>> cron >>>>>>> expert).
> OK, I did what you instructed and checked the log file, which did > contain the system path and the traceback about the "can't import > module...".
> I see the paths specified don't contain the path to the feedparser > module, which explains the error. My question is how do I add this > path in the cron, especially since the script runs from the command > line?
> On Mar 4, 9:02 am, Evert Rol <evert....@gmail.com> wrote: >>> Well part of the problem is that the script itself works fine. I can >>> run from the command line without a hitch. But when I try to run it >>> via the cron, I get the traceback about the feedparser module.
>> I believe that, because your PYTHONPATH will be set correctly from >> the >> command line; but possibly not from a default (basic) shell that cron >> uses. >> So, again, try putting those two statements at the top of your >> script, >> -before- the 'import feedparser' statement, and let cron do its job. >> You may need to redirect the output, eg >> 1 * * * * /usr/bin/python /path/to/script/update_feeds.py > /path/to/ >> script/update_feeds.log 2>&1 >> to get that output (I'm not sure how you got the error from the cron >> job; through email?)
>> Check the log file: before the traceback, you should see the result >> of >> 'print sys.path'. Check if that includes the correct directories.
>>>> Top of script (possibly just below the she-bang):
>>>> import sys >>>> print sys.path
>>>>>>> Traceback (most recent call last): >>>>>>> File "/path/to/script/update_feeds.py", line 10, in ? >>>>>>> import feedparser >>>>>>> ImportError: No module named feedparser
>>>>>>> I'm running the cron on my Dreamhost account BTW.
>>>>>> Did you try printing out the sys.path at the top of your >>>>>> script, so >>>>>> see if the PYTHONPATH is picked up correctly from the cron >>>>>> environment >>>>>> settings? I'd guess it should, but it may go wrong there (I'm no >>>>>> cron >>>>>> expert).
============================= Andrew Holt Managing Director 4A Solutions Ltd
> Traceback (most recent call last):
> File "update_feeds.py", line 58, in ?
> update_feeds()
> File "update_feeds.py", line 14, in update_feeds
> from knoxd.apps.aggregator.models import Feed, FeedItem
> ImportError: No module named knoxd.apps.aggregator.models
It works for me and I don't have to set any environment variables in
cron settings.
My script scheduler.py (that is called by cron) starts with the
following:
> It works for me and I don't have to set any environment variables incronsettings.
> My script scheduler.py (that is called bycron) starts with the
> following:
