Invalid request URL for Fusion Tables insert

[ferenc....@bitongo.com]

Hi,

Trying to use the module for accessing FusionTables (downloaded the most recent google-api-python-client-gae-1.1.zip).

Authentication works fine and tables can be listed however inserting new rows to tables result in InvalidURL exception.

The SQL insert is not sent with POST but with GET and is probably too big.

Why is a GET used? https://www.googleapis.com/discovery/v1/apis/fusiontables/v1/rest suggests that 

POST should be used:

..

  "query": {
   "methods": {
    "sql": {
     "id": "fusiontables.query.sql",
     "path": "query",
     "httpMethod": "POST",

..

Is there a way to force the service client to use the POST method?

https://gist.github.com/f3r3nc/256fa4ab9fb11f12b9be

Ferenc

[Joe Gregorio]

The client library will automatically switch from GET to POST
if the URL length becomes too long:

https://code.google.com/p/google-api-python-client/source/browse/apiclient/http.py#663

> --
> You received this message because you are subscribed to the Google Groups
> "Google API Python Client" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-api-python-...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Ferenc Vehmann]

Hi Joe,

Thanks for your quick response.

This is interesting.

At http.py:633  the URI len is bigger than the max but self.method == 'POST' and so no conversion is made.

So the question is why is the method set to POST when a GET request is constructed.

self.body is empty.

Ferenc

--

Ferenc Vehmann

Co-Founder

Bitongo Ltd.

http://www.bitongo.com

ferenc....@bitongo.com

[Joe Gregorio]

OK, now I understand, the SQL you supply goes in the sql parameter. If the SQL
were in a body= parameter then it would go in the body of the POST request
and wouldn't affect the URL length. You should definitely bring this up in the
FusionTables API user's group as an issue with the API:

https://developers.google.com/fusiontables/groups/fusionTablesApiGroup

You might be able to work around this limitation by putting your request in
a batch:

https://developers.google.com/api-client-library/python/guide/batch

The should let you get by the URL length limit on GAE. Please let me
know if this works for you.

-joe

[Ferenc Vehmann]

Using body= results in an error:

service.query().sql(body = sql_query)

  File "..../apiclient/discovery.py", line 573, in method

    raise TypeError('Got an unexpected keyword argument "%s"' % name)

TypeError: Got an unexpected keyword argument "body"

Batching is not really an options, at least not a straight forward one as the data is one line.

However, I've managed to modify http.py so that it does put the date into the body.

http.py:663

      if len(self.uri) > MAX_URI_LENGTH: # and self.method == 'GET':

        self.method = 'POST'

        # self.headers['x-http-method-override'] = 'GET'

However, this is clearly not nice.

I still don't understand why the query parameters end up in GET as there is POST in the discovery.

..

  "query": {
   "methods": {
    "sql": {
     "id": "fusiontables.query.sql",
     "path": "query",
     "httpMethod": "POST",

..

[Joe Gregorio]

On Tue, Apr 23, 2013 at 10:41 AM, Ferenc Vehmann
<ferenc....@bitongo.com> wrote:
> Using body= results in an error:
>
> service.query().sql(body = sql_query)
>
> File "..../apiclient/discovery.py", line 573, in method
>
> raise TypeError('Got an unexpected keyword argument "%s"' % name)
>
> TypeError: Got an unexpected keyword argument "body"

Sorry, I wasn't suggesting you use body, just explaining that the function
signature would be different if the SQL was sent in the body of the
request.

>
> Batching is not really an options, at least not a straight forward one as
> the data is one line.

Use batching, take the singe request you are making, put it in a batch
request like the code I linked to, and then execute that batch. The request
should work as it did before, but it will not hit the URL limit from App Engine.

>
> However, I've managed to modify http.py so that it does put the date into
> the body.
> http.py:663
>
> if len(self.uri) > MAX_URI_LENGTH: # and self.method == 'GET':
>
> self.method = 'POST'
>
> # self.headers['x-http-method-override'] = 'GET'
>
> However, this is clearly not nice.
>
> I still don't understand why the query parameters end up in GET as there is
> POST in the discovery.
> ..
>
> "query": {
> "methods": {
> "sql": {
> "id": "fusiontables.query.sql",
> "path": "query",
> "httpMethod": "POST",

Because the sql parameter "location" is "query" which means to put it
in the query parameters:

"parameters": {
"hdrs": {
"type": "boolean",
"description": "Should column names be included (in the first
row)?. Default is true.",
"location": "query"
},
"sql": {
"type": "string",
"description": "An SQL
SELECT/SHOW/DESCRIBE/INSERT/UPDATE/DELETE/CREATE statement.",
"required": true,
"location": "query"

[Ferenc Vehmann]

Thanks for the batching suggestion. I don't want to break lines.

I think this is just as nasty as modifying the http.py code so that it does POST  at the end.

Are you suggesting that the signature should be something like this:

      "sql": {
       "type": "string",
       "description": "An SQL
SELECT/SHOW/DESCRIBE/INSERT/UPDATE/DELETE/CREATE statement.",
       "required": true,

       "location": "body"                       <<<<<<<<<<<<< changed

[Joe Gregorio]

No. I am suggesting you take your code:


kml = "<LineString><coordinates>lng,lat lng,lat ..</coordinates></LineString>"
sql = 'INSERT INTO %s (track) VALUES (\'%s\')' % (table_id, kml)
insert_response = service.query().sql(sql = sql).execute()


And change it to:

def do_something(request_id, response, exception):
if exception is not None:
# Do something with the exception
pass
else:
# Do something with the response
pass

kml = "<LineString><coordinates>lng,lat lng,lat ..</coordinates></LineString>"
sql = 'INSERT INTO %s (track) VALUES (\'%s\')' % (table_id, kml)
insert_request = service.query().sql(sql = sql)

batch = BatchHttpRequest()
batch.add(insert_request, callback=do_something)
batch.execute()


On Tue, Apr 23, 2013 at 10:54 AM, Ferenc Vehmann

[Ferenc Vehmann]

This results in: 

HttpError: <HttpError 400 when requesting https://www.googleapis.com/batch returned "Bad Request">

What I didn't understand is what exactly needs to be changed in the Fusion Tables API signature so that the python client will put the SQL parameter in the POST body.

[Joe Gregorio]

There is nothing you can do to change this, it has to be changed
on the server side.

On Tue, Apr 23, 2013 at 11:42 AM, Ferenc Vehmann

[Ferenc Vehmann]

OK.

I am sticking with the http.py modification as long as the client the the fusion api are not working nicely together.

Thanks for your help.

Ferenc

[Tom R]

Has the status of this changed (Google fixed it on the server)? Are so few people using the API client with the FusionTables API?

I'm running into the problem described here and have yet to try either of the workarounds mentioned.