DB insert confusion

[Mike Anson]

Greetings.

I have some strange behavior when saving to my DB.

Here is my DB schemer:

db.define_table('messages',
    Field('uid', 'string'),
    Field('message', 'text'),
    Field('status', 'string', default='unsent'),
    Field('added', 'datetime', default=request.now),
    Field('modified', 'datetime', default=request.now))

My Controller:

@service.jsonrpc
def savemessage(message, uid):
    db.messages.insert(message=message, uid=uid)
    db.commit()
    
    message = {"status":"saved"}

    return message

Curl script to send message via jsonrpc:

message=$1

uid=$2

url = someURL.com

curl -v -k -X POST -H "Content-Type: application/json" -d "{\"id\": 1, \"method\": \"savemessage\", \"params\": { \"${message}\": \"mymessage\", \"${uid}\" : \"myemail@localhost\"}}" $url

Command line command:

./send_irc_message.sh "This is my message" so...@email.com

So the first thing to note is that for the params, the value of ${message} is being saved rather than mymessage. Similarly the value for ${uid} is being saved instead of myemail@localhost

Have I got the script syntax correct?

At a push I could live with this except sporadically, even the value of ${message} gets saved as the uid in the DB and visa versa. I can't seem to consistently replicate this behavior. It just does it from time to time.

When I print out $message and $uid from my curl script, the values are correct at this point so it has to be the curl script. Can anyone see something I clearly cannot?

Thanks in advance.

-Mike

[Niphlod]

you are using the wrong order....you have to send something like

{"id": 1, "method": "savemessage", "params": { "message": "variableholdingmessage", "uid" : "variableholdingmail"}}

message and uid are the named parameters to your function def savemessage(message, uid)
and instead you are composing

{"id": 1, "method": "savemessage", "params": { "variableholdingmessage": "mymessage", "variableholdinguid" : "myemail@localhost"}}

PS: you are using probably the overcomplicated quoting style ever :P
try with this

-d '{"id": 1, "method": "savemessage", "params": { "message": "'$message'", "uid" : "'$uid'"}}' $url

PS2: relying on a vanilla bash variable for passing around json parameters is a bad idea. What if your message includes a ' symbol ? e.g. "Let's rock".... The json equivalent is "Let\'s rock"...

[Mike Anson]

Speediest response as ever Niphlod!

Yes I understand your point. The reason it is currently like this is because if I use your suggestion (which I obviously had originally)

{"id": 1, "method": "savemessage", "params": { "message": "variableholdingmessage", "uid" : "variableholdingmail"}}

I get message and uid as the values in the DB. So I switched them.

{"id": 1, "method": "savemessage", "params": { "variableholdingmessage": "mymessage", "variableholdinguid" : "myemail@localhost"}}

The result means that variableholdingmessage is saved as the message and not the expected "mymessage". Exactly the same for uid.

re: PS -- haha yes I know. I did try it with single quotes and it crapped out?? So just kept the doubles. It's not that bad!

re: PS2 -- have you any recommendations to solve this special character potential problem?

--
 
 
 

[Niphlod]

Yes I understand your point. The reason it is currently like this is because if I use your suggestion (which I obviously had originally)

{"id": 1, "method": "savemessage", "params": { "message": "variableholdingmessage", "uid" : "variableholdingmail"}}

I get message and uid as the values in the DB. So I switched them.

{"id": 1, "method": "savemessage", "params": { "variableholdingmessage": "mymessage", "variableholdinguid" : "myemail@localhost"}}

The result means that variableholdingmessage is saved as the message and not the expected "mymessage". Exactly the same for uid.

Unfortunately jsonrpc call method has a bug. in web2py 2.2.1, in gluon/tools.py, line 4231 should be

s = methods[method](**params)

instead of

s = methods[method](*params)

sending a patch to Massimo right now!
 

re: PS -- haha yes I know. I did try it with single quotes and it crapped out?? So just kept the doubles. It's not that bad!

re: PS2 -- have you any recommendations to solve this special character potential problem?

 

Normally with jsonrpc you use something that is not curl, e.g. a programming language that supports json (python?!)
Escaping on bash without awk, sed, etc is always problematic.... but if you're willing to have as only limitation the " character that is less frequent to use within a message, why don't you use one of the methods not requiring a json body to be posted ? e.g. @service.xml, @service.csv or @service.json....

curl -v --get --data-urlencode \"uid=$uid\" --data-urlencode \"message=$message\" $url

here curl takes care of urlencoding the message and the uid parameters.


 

[Mike Anson]

Thanks very much for your help Niphlod.

[Kurt Grutzmacher]

I don't think this is a good JSON-RPC example as the change broke our app that uses simplejsonrpc or jsonrpclib to make API calls.

Based on the jsonrpclib python module @ https://code.google.com/p/jsonrpclib/ requests look like:

>>> import jsonrpclib
>>> server = jsonrpclib.Server('http://localhost:8080')
>>> server.add(5,6)
11
>>> print jsonrpclib.history.request
{"jsonrpc": "2.0", "params": [5, 6], "id": "gb3c9g37", "method": "add"}
>>> print jsonrpclib.history.response
{'jsonrpc': '2.0', 'result': 11, 'id': 'gb3c9g37'}

And the JSON-RPC spec states params should be "An Array of objects to pass as arguments to the method." -- http://json-rpc.org/wiki/specification

However the actual spec doesn't specify array, dict or whatever as it tries to be universal: "A Structured value that holds the parameter values to be used during the invocation of the method. This member MAY be omitted."  http://www.jsonrpc.org/specification#request_object

For simplejsonrpc and jsonrpclib to work we have to undo this change in gluon/tools.py

[Niphlod]

I took some time to watch at the jsonrpc specs. Right now I had experience with jsonrpc only in the "named parameters" format. I thought it was the standard, my bad.
However, I found out that 2.0 introduced "named parameters" that were not supported - explicitely - in 1.0

--> {"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}
<-- {"jsonrpc": "2.0", "result": 19, "id": 1}

--> {"jsonrpc": "2.0", "method": "subtract", "params": {"subtrahend": 23, "minuend": 42}, "id": 3}
<-- {"jsonrpc": "2.0", "result": 19, "id": 3}

are both valid.

Maybe revert this and make a jsonrpc2 decorator to support named parameters explicitely would be a better solution?

[Jonathan Lundell]

On 11 Dec 2012, at 1:09 PM, Niphlod <nip...@gmail.com> wrote:

I took some time to watch at the jsonrpc specs. Right now I had experience with jsonrpc only in the "named parameters" format. I thought it was the standard, my bad.
However, I found out that 2.0 introduced "named parameters" that were not supported - explicitely - in 1.0

--> {"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}
<-- {"jsonrpc": "2.0", "result": 19, "id": 1}

--> {"jsonrpc": "2.0", "method": "subtract", "params": {"subtrahend": 23, "minuend": 42}, "id": 3}
<-- {"jsonrpc": "2.0", "result": 19, "id": 3}

are both valid.

Maybe revert this and make a jsonrpc2 decorator to support named parameters explicitely would be a better solution?

How about both jsonrpc1 and jsonrpc2, and then jsonrpc = jsonrpc2? (I think it'd be better to make v2 the default.)

--
 
 
 

[Niphlod]

uhm. The "problem" is that even though every jsonrpc interface I worked with was parameter-based, for jsonrpc2 position-based and parameter-based are both valid.

[Kurt Grutzmacher]

Any movement on this? I can't seem to get jsonrpclib to force parameterized queries. Replicating this is fairly simple:

1. Create a new app, in default.py add a function. Replace t_test with auth or whatever:

@service.jsonrpc
def list():
data = db(db.t_test.ALL).select()
return dict(data=data)

2. Drop to a python shell:

>>> import jsonrpclib
>>> j = jsonrpclib.Server(uri='http://localhost:8000/appname/default/call/jsonrpc')
>>> j.list()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 276, in __call__
return self.__send(self.__name, kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 225, in _request
check_for_errors(response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 529, in check_for_errors
raise ProtocolError((code, message))
ProtocolError: (100, u'TypeError: list() argument after ** must be a mapping, not str')

Same problem with simplejsonrpc in gluon/contrib...

Niphlod wrote:
> uhm. The "problem" is that even though every jsonrpc interface I
> worked with was parameter-based, for jsonrpc2 position-based and
> parameter-based are both valid.
>
> On Tuesday, December 11, 2012 10:42:06 PM UTC+1, Jonathan Lundell wrote:
>
> On 11 Dec 2012, at 1:09 PM, Niphlod <nip...@gmail.com

> <javascript:>> wrote:
>> I took some time to watch at the jsonrpc specs. Right now I had
>> experience with jsonrpc only in the "named parameters" format. I
>> thought it was the standard, my bad.
>> However, I found out that 2.0 introduced "named parameters" that
>> were not supported - explicitely - in 1.0
>>
>> --> {"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}
>> <-- {"jsonrpc": "2.0", "result": 19, "id": 1}
>>
>> --> {"jsonrpc": "2.0", "method": "subtract", "params": {"subtrahend": 23, "minuend": 42}, "id": 3}
>> <-- {"jsonrpc": "2.0", "result": 19, "id": 3}
>>
>> are both valid.
>>
>> Maybe revert this and make a jsonrpc2 decorator to support named
>> parameters explicitely would be a better solution?
>
> How about both jsonrpc1 and jsonrpc2, and then jsonrpc = jsonrpc2?
> (I think it'd be better to make v2 the default.)
>
>>
>> On Tuesday, December 11, 2012 9:57:50 PM UTC+1, Kurt Grutzmacher
>> wrote:
>>
>> I don't think this is a good JSON-RPC example as the change
>> broke our app that uses simplejsonrpc or jsonrpclib to make
>> API calls.
>>
>> Based on the jsonrpclib python module
>> @ https://code.google.com/p/jsonrpclib/
>> <https://code.google.com/p/jsonrpclib/> requests look like:
>>
>> >>> import jsonrpclib

>> >>> server = jsonrpclib.Server('http://localhost:8080 <http://localhost:8080/>')

>> "*message*": "variableholdingmessage", "*uid*" :

> --
>
>
>

[Niphlod]

ok, let's try to not make any further confusion. Let's agree on what web2py needs to provide and how to call it.
First things first: gluon.contrib.simplejsonrpc supports only 1.1, so only positional arguments sent as lists.
jsonrpclib instead supports 2.0 spec, so both positional arguments sent as lists and keyword arguments sent as dicts.

Second thing: it's hard to follow some example if the example is never going to work :P

data = db(db.t_test.ALL).select()
return dict(data=data)

will go on exception no matter what.

data = db().select(db.t_test.ALL)
return dict(data=data)

works as intended.

Third: decorated functions usually need to have arguments.
can we set on a

@service.jsonrpc
def testlist(arg1, arg2):
    data = db().select(db.t_test.ALL)
    return dict(data=data, arg=arg1, arg2=arg2)

? Right!

Now, I think with a simple patch we can escape the "pass parameters as a mapping or as a list" problem in gluon/tools.py

            if isinstance(params, dict):

                s = methods[method](**params)

            elif isinstance(params, list):

                s = methods[method](*params)

That should restore functionality for who calls functions only with "positional" style, while retaining the "keywords" style feature. This means that using contrib.simplejsonrpc you'll be able to call

s.testlist(1,2)

but not

s.testlist(arg1=1, arg2=2)

NB: This is just because simplejsonrpc adheres only to the 1.1 spec

With jsonrpclib, however, both will work ok (yeah!)

If this is fine, we can further fine-tune responses (e.g. 'version': '2.0' instead of 'version': '1.1') or having 2 separate @service decorators as Jonathan suggested.

Do we have a deal ?

[Kurt Grutzmacher]

Yeah, yeah, yeah.. My on-the-fly testing code wasn't correct. ;)

Checking with isinstance() is good but fails if params is just a string
due to the data.get('params', '') a few lines up.

Changing that to data.get('params', []) will resolve this or just drop
elif isinstance(params, list): and change it to an else: catch

||
ifisinstance(params,dict):
s =methods[method](**params)
else:
s =methods[method](*params)

This way the called method will take defaults or error out if variables
are not defined. Either way seems to work fine on my end.

Niphlod wrote:
> ok, let's try to not make any further confusion. Let's agree on what
> web2py needs to provide and how to call it.
> First things first: gluon.contrib.simplejsonrpc supports only 1.1, so

> *only* positional arguments sent as lists.
> jsonrpclib instead supports 2.0 spec, so *both* positional arguments

> sent as lists and keyword arguments sent as dicts.
>
> Second thing: it's hard to follow some example if the example is never
> going to work :P
>
> ||
> data =db(db.t_test.ALL).select()
> returndict(data=data)
>
> will go on exception no matter what.
> ||
> data =db().select(db.t_test.ALL)
> returndict(data=data)
>
> works as intended.
>
> Third: decorated functions usually need to have arguments.
> can we set on a
> ||
> @service.jsonrpc
> deftestlist(arg1,arg2):
> data =db().select(db.t_test.ALL)
> returndict(data=data,arg=arg1,arg2=arg2)
> ? Right!
>
> Now, I think with a simple patch we can escape the "pass parameters as
> a mapping or as a list" problem in gluon/tools.py
> ||
> ifisinstance(params,dict):
> s =methods[method](**params)
> elifisinstance(params,list):

> s =methods[method](*params)

>
> That should restore functionality for who calls functions only with
> "positional" style, while retaining the "keywords" style feature. This
> means that using contrib.simplejsonrpc you'll be able to call
>
> ||
> s.testlist(1,2)
>
> but not
> ||
>
> s.testlist(arg1=1,arg2=2)
>
>
> NB: This is just because simplejsonrpc adheres only to the 1.1 spec
>
> With jsonrpclib, however, both will work ok (yeah!)
>
> If this is fine, we can further fine-tune responses (e.g. 'version':
> '2.0' instead of 'version': '1.1') or having 2 separate @service
> decorators as Jonathan suggested.
>
> Do we have a deal ?
>

> --
>
>
>

--
- grutz;