scheduler can't normal run my script

[yytr...@gmail.com]

i write a scripy to get some information and use scheduler to auto ,but scheduler is display FAILED,this is my script

def redhat():
    bug_num =''
    try:
        try:
            url = 'https://access.redhat.com/security/vulnerabilities'
            rea = requests.get(url)
            rea.encoding = 'utf-8'
            html = BeautifulSoup(rea.text,"html.parser")
            a= html.find("tbody").find_all("tr")
            try:
                indatabase = db(db.get_data.get_from == "redhat").select()
                if indatabase:
                    indatabase=db(db.get_data.get_from == "redhat").select(
                    db.get_data.title,orderby=~db.get_data.get_time,limitby=(0,5))
                else:
                    indatabase=['1']
            except BaseException as e:
                print e
            for i in range(len(a)):
                for j in indatabase:
                    break_flag ='no'
                    if a[i].find("a").get_text() != j.title:
                        geturl = 'https://access.redhat.com'+ str(a[i].find("a")).split('"')[1]
                        bug_level = a[i].find("div","field-type-list-text").get_text().strip()
                        if re.findall(u'CVE\-[0-9]{4,}\-[0-9]{4,}',str(a[i].find("a").get_text())):
                            bug_num = re.findall(u'CVE\-[0-9]{4,}\-[0-9]{4,}',str(a[i].find("a").get_text()))[0]
                        if bug_level=="Critical":
                            bug_level="High"
                        elif bug_level=="Important":
                            bug_level=" Medium"
                        elif bug_level=="Moderate":
                            bug_level="Low"
                        db.get_data.insert(title=a[i].find("a").get_text(),get_from='redhat',get_url=geturl,get_time=now,
                            body='',data_condition=condition[0],bug_num=bug_num,bug_level=bug_level)
                        print a[i].find("a").get_text()
                        break
                    else:
                        break_flag = "yes"
                        print "redhat no new isuse"
                        break
                if break_flag == "yes":
                    break


        except Exception as e:
            print e
   
    except Exception as e:
        pass
    print "redhat finish in " +str(datetime.datetime.now()) +"\n"


from datetime import timedelta as timed
from gluon.scheduler import Scheduler
scheduler = Scheduler(db, dict(
    mission5=redhat
    )
)

scheduler.queue_task('mission5', 

    start_time=request.now,  # datetime

    stop_time=None,#request.now+timed(seconds=30),  # datetime

    timeout = 90,  # seconds

    period=60,  # seconds

    immediate=True,

    repeats=1

)

my script can normal run in controllers,but  in scheduler can't

and i want to control the quantity of scheduler_task,but it without stop create new task ,please help me

[Anthony]

How are you starting the scheduler workers? Typically the scheduler is run in the context of a web2py app environment, so the database and tasks are defined in a web2py app.

To run the scheduler outside of the web2py context, see https://github.com/web2py/web2py/blob/c9fd0fd71e770c1a367b45ad5def5bd2c6fd7115/gluon/scheduler.py#L1637. You'll need to specify the DB URI, a folder where the DAL model definition metadata files will be stored, and a file where the tasks are defined.

Anthony

[Dave S]

On Wednesday, November 15, 2017 at 7:29:38 AM UTC-8, Anthony wrote:

How are you starting the scheduler workers? Typically the scheduler is run in the context of a web2py app environment, so the database and tasks are defined in a web2py app.

To run the scheduler outside of the web2py context, see https://github.com/web2py/web2py/blob/c9fd0fd71e770c1a367b45ad5def5bd2c6fd7115/gluon/scheduler.py#L1637. You'll need to specify the DB URI, a folder where the DAL model definition metadata files will be stored, and a file where the tasks are defined.

Anthony

And if run in a web2py context, the schedule_run table entry will include the stdout  (and stderr?) for the attempt, up to a maximum byte limit.  Look to see if there are error messages in that field; they may help you diagnose your problem.

/dps
 

[yytr...@gmail.com]

thank for you answer i  use

python G:\\web2py\\web2py.py -K yj:sss -X -a '1' -i 0.0.0.0 -p 8000 &

to run my app,can you  give me a example ? i understand the script,but i don't know how to use it in my app
and when i use scheduler it auto use threading create so many task, i want to control it's quantity,how to do? 

在 2017年11月15日星期三 UTC+8下午11:29:38，Anthony写道：

[yytr...@gmail.com]

thank you for your answer ,it run normal

def cve():

    try:

        try:

            url= 'http://cve.scap.org.cn/cve_list.php?action=recent'

            rea = requests.get(url)

            rea.encoding = 'utf-8'

            html = BeautifulSoup(rea.text,'html.parser')

            a= html.find('div','entry')

            indatabase = db(db.get_data.get_from == 'cve').select()

            if indatabase:

                indatabase=db(db.get_data.get_from == 'cve').select(

                    db.get_data.title,orderby=~db.get_data.get_time,limitby=(0,5))

            else:

                indatabase=['1']

            print indatabase

            for i in a.find_all('a',limit=6):

                print "1"

                for j in indatabase:

                    print "2"

                    break_flag ='no'

                    if str(j) != '1':

                        print "3"

                        if i.get_text() != j.title.decode('utf-8'):

                            print "4"

                            print i.get_text()

                            print i.get('href')

                            db.get_data.update_or_insert(db.get_data.title==i.get_text(),title=i.get_text(),get_from='cve',

                                get_url=i.get('href'),get_time=now,data_condition=condition[0],bug_num=i.get_text())

                            break

                        else:

                            break_flag = "yes"

                            print " cve no new isuse"

                            break

                    else:

                        print "5"

                        print i.get_text()

                        print i.get('href')

                        try:

                            db.get_data.update_or_insert(db.get_data.title==i.get_text(),title=i.get_text(),get_from='cve',

                                get_url=i.get('href'),get_time=now,data_condition=condition[0],bug_num=i.get_text())

                        except Exception as e:

                            print e

                        print "6"

                        break 

        except Exception as e:

            print e

    except Exception as e:

            print e

   

在此输入代码...

return "cve is finish"

the scheduler_run run_out is 

['1']

1

2

5

CVE-2017-6274

http://cve.scap.org.cn/CVE-2017-6274.html

6

no except and 6 is print 

but 

 db.get_data.update_or_insert(db.get_data.title==i.get_text(),title=i.get_text(),get_from='cve',

                                get_url=i.get('href'),get_time=now,data_condition=condition[0],bug_num=i.get_text())

is no run,can you tell me why?



在 2017年11月17日星期五 UTC+8上午4:26:05，Dave S写道：

[Anthony]

the scheduler_run run_out is 

['1']

1

2

5

CVE-2017-6274

http://cve.scap.org.cn/CVE-2017-6274.html

6

no except and 6 is print 

but 

 db.get_data.update_or_insert(db.get_data.title==i.get_text(),title=i.get_text(),get_from='cve',

                                get_url=i.get('href'),get_time=now,data_condition=condition[0],bug_num=i.get_text())

is no run,can you tell me why?

In scheduler tasks, you must call db.commit() to commit transactions to the database. web2py does this automatically on HTTP requests, but the scheduler is not making an HTTP request to web2py.

Anthony

[Anthony]

On Saturday, November 18, 2017 at 1:52:56 AM UTC-5, yytr...@gmail.com wrote:

thank for you answer i  use

python G:\\web2py\\web2py.py -K yj:sss -X -a '1' -i 0.0.0.0 -p 8000 &

to run my app,can you  give me a example ? i understand the script,but i don't know how to use it in my app
and when i use scheduler it auto use threading create so many task, i want to control it's quantity,how to do? 

Where is your script and how are you running it? If you are starting the scheduler worker as above, the scheduler and scheduler task must be defined in the context of a web2py app, not a standalone script. From your script code, the function defining the task should either go in a model file in the app or go in a module and be imported into a model file. The code defining the scheduler should go in a model file in the app. And the code queuing the task should be executed elsewhere -- you can run the queuing code from a web2py shell, define a controller action that queues the task, or even directly add a record to the scheduler_task table (possibly via appadmin) to queue a task.

Anthony