Transactions failing using Python API

[Alex]

Hello.

We're using the latest revision from svn and on Erlang R13B04 and we
are having some trouble writing (not that large) transactions.

Commiting every 5 transaction writes seems to work for our data but
that makes loading all of our data really, really slow. We are
interfacing with Scalaris using the python api and the Transaction
class.

It seems that the writes begin to fail when the transaction log size
hits about 10,000 characters (10kb).

Is this a problem with python's JSON RPC or something we could address
on the server?

You can see the same tlog on both server and client

From the server:
http://pastebin.com/sNBfCT8X

From the client:
http://pastebin.com/Hreiqk1q

Any ideas?

Thanks,
Alex

[Nico Kruber]

unfortunately the problem seems to be on our side in the JSON API in general
(not python-specific) - to be more precise: in the yaws code.
If I see it correctly, the server (or client) is splitting up the request into
several partial data chunks - the handling of this seems to be wrong. We don't
have such a test case yet - I'll add one and have a look on how to handle this
correctly.
If it's not much to do, I can hopefully fix this tomorrow.

If you only need to write the data to a single key, i.e. you don't need
consistency over multiple keys for each data item, prefer using the
TransactionSingleOp class. This does not transfer a tlog back to the client
and should thus be faster than writing multiple keys in a single reqlist.
If it is not faster, we should maybe introduce something like a reqlist for
single operations, too....

Nico

[Alex]

Thanks Nico,

We have started to use the TransactionSingleOp class and have run into
a similar problem using that class.

In this instance we are serializing an object that produces a string
of length : 269708 and receiving the same exception : {badarg,
[{erlang, binart_to_list....

The client log with the value that we tried to pass is here:
http://pastebin.com/k2T7N3NH

Interestingly, when we input this value via the Web Client it adds the
value fine.

Does the Java API use the same web service? i.e. would we have the
same problem if we were using java?

Thanks again,
Alex

>  signature.asc
> < 1KViewDownload

[Nico Kruber]

Hi Alex,
starting with commit r1731, you can configure the maximum size that is
accepted by the JSON API - simply set the yaws_max_post_data config parameter.
I changed it to 1MB in the default config. (default for yaws was 10kb - the
barrier you hit before)

Any requests larger than this configured size will return with a HTTP 413
error (Request Entity Too Large).

The web debug interface does not use the JSON API and was not affected.

The Java-API has not been affected either since it communicates with erlang
directly (without yaws).

Nico

[irina]

Hi Nico,

Thanks for fixing the bug.

We hit a different problem now: we are trying to do 2 small write
requests to the server and they fail with a BadStatusLine("''",)
ConnectionException. We reduced our requests to these 2 and we can't
figure out why the first one succeeds and the second one fails; the
server is not outputing anything (we turned the logging level to
debug).

We got this problem in the context of a big data load and we isolated
the problem to these requests.


The output on the client side is here:

http://pastebin.com/xrrkXV5j


Any ideas what we could be doing wrong?

Thanks,

Irina

>  signature.asc
> < 1KViewDownload

[Nico Kruber]

On Sunday 08 May 2011 19:54:08 irina wrote:
> Hi Nico,
>
> Thanks for fixing the bug.
>
> We hit a different problem now: we are trying to do 2 small write
> requests to the server and they fail with a BadStatusLine("''",)
> ConnectionException. We reduced our requests to these 2 and we can't
> figure out why the first one succeeds and the second one fails; the
> server is not outputing anything (we turned the logging level to
> debug).
>
> We got this problem in the context of a big data load and we isolated
> the problem to these requests.

hmm - can't really see anything suspicious in your logs - is this error
reproducible? what do you mean by "big data load"?
Also: could you produce a minimal test case? Alternatively add debug output to
api_json.erl to see what happens behind the scenes, i.e. on the server.

If you want more output from the client code (maybe this helps understanding
the error), add self._conn.set_debuglevel(1) prior to the call to
self._conn.request in Scalaris.py

Nico

[irina]

It seems the problem was with the connection to the server. It works
now if we reconnect before each TransactionSingleOp write, but it
makes it really slow.

We increased the default timeout in Scalaris.py and the timeouts in
scalaris.cfg for transactions, tcp_send_timeout and
tcp_connect_timeout but it didn't help.
Is there another parameter that we should change in order to run the
transactions without reconnecting each time?

Thank you,

Irina

>  signature.asc
> < 1KViewDownload

[Nico Kruber]

was the problem reproducible and occurred always at the exact same point in
your code?
If yes, I could imagine that the http connection is interrupted somehow due to
a maximum time of being opened or maximum data sent over it - I'd need to
check the parameters, yaws can be configured with - maybe we can change it
then.
The only thing I see right now, may be the keepalive_timeout parameter which
defaults to 30s (you may override it by adding it to the GconfList list in
sup_yaws.erl)
you may have a look at ./contrib/yaws/include/yaws.hrl for further options.

timeouts in scalaris.cfg should not affect this

Nico