queue_task :: trouble with quoted json strings for pvars and args

[Erwn Ltmann]

Hi,

I'm used to create a new task for example:

scheduler.queue_task(
"a_task",
pvars = dict(csid=row.id, csname=row.name, action=row.action),
...
)

In the past (version 2.10.1), everything was fine. Within the current version (2.14.6) I have some trouble. See this code for example:

for row in db(db.scheduler_task.group_name == "agents").select(
db.scheduler_task.ALL):
...
pvars = json.loads(row.vars)

... will raise this exception:

...
File "/usr/local/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

The reason is a quoted string in column 'VARS' in table 'SCHEDULER_TASK' (no quotes in web2py's old version 2.10.1).

A new value for 'VARS' with quotes for example:

'{"action": "script", "csname": "list", "csid": 121}'

I can skip the quotes with a trick:

pvars = json.loads(row.vars[1:-1])

But, it is not a general solution. The same problem appears for column 'ARGS' in scheduler.py when I process the tasks:

File "/export/home/armadm/dev01/release/web2py/gluon/scheduler.py", line 311, in executor
args = _decode_list(loads(task.args))
File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I use Oracle as database:

SQL> desc scheduler_task
Name Null? Type
----------------------------------------- -------- ----------------------------
ID NOT NULL NUMBER
...
ARGS VARCHAR2(512)
VARS VARCHAR2(512)

Any ideas?

Thx, Erwn

[Dave S]

the vars field is a dict, with the var names as keys and the stuff you want to pass as the values.  Both are strings.

I use a single var in my code, the name is "where" and the value is a path.  So I do

if vars["where"]:
   mywhere = vars["where"]

No stripping of quotes is needed to get the path.

/dps

[Erwn Ltmann]

Hi Dave,

thx for your response. I'm sorry, but it doesn't helps me. My problem concerning the usage of scheduler_task queueing. The web2py scheduler transforms automatically the dictionary parameter 'pvars' into a string represented JSON in order to save this value as VARCHAR2(512) into the vars column of database (oracle) table scheduler_task. So far so good. But IMHO with unnecessary quotes at begin and end. The scheduler part of web2py tried to read scheduler records and fails because this (damn) extra quotes.

E.

[Dave S]

On Thursday, September 8, 2016 at 6:27:15 AM UTC-7, Erwn Ltmann wrote:

Hi Dave,

thx for your response. I'm sorry, but it doesn't helps me. My problem concerning the usage of scheduler_task queueing. The web2py scheduler transforms automatically the dictionary parameter 'pvars' into a string represented JSON in order to save this value as VARCHAR2(512) into the vars column of database (oracle) table scheduler_task. So far so good. But IMHO with unnecessary quotes at begin and end. The scheduler part of web2py tried to read scheduler records and fails because this (damn) extra quotes.

E.

I don't use Oracle and I don't have JSON arguments for tasks (yet), so I can't tell you if the "problem" is with the queue_task() call or with the Oracle driver.  Niphlod is the Scheduler authority, having done the rewrite of it.  I'm not sure who our local Oracle expert is.

Does the error occur when you access vars[0] from your task (launched by the Scheduler?)  Or does it occur when the Scheduler is trying to start up your task?  Also, are you accessing the Scheduler tables directly?  I don't think you should be doing that ... for one thing, that's an implementation detail that's not an official API and is subject to change.  And I don't think it's needed, as your launched task is passed the args.

/dps
(edited for m-fassis)

[Niphlod]

queue_task just does json.dumps(vars)

https://github.com/web2py/web2py/blob/master/gluon/scheduler.py#L1483

the field is defined as 'text'

https://github.com/web2py/web2py/blob/master/gluon/scheduler.py#L813

IMHO it's an issue with the oracle driver.

[Richard P]

I just came across this same issue when using the web2py scheduler for the first time with an Oracle DB. I spent a good amount of time googling  before finding this post which pointed me to where the problem was.

I have implemented a work around  by creating the below trigger in the database that will strip out the single quotes before the data is inserted into the table. 

It's not pretty but it works. Now my tasks are running fine.

create or replace trigger replace_quotes

before insert on SCHEDULER_TASK

for each row

begin

:new.args := replace( :new.args, '''', '' );

:new.vars := replace( :new.vars, '''', '' );

end;

/

[Niphlod]

seems more a problem of the adapter rather than the scheduler itself