JDiameter Performance as a Proxy

[David Palmer]

Hi

I am currently evaluating the jDiameter stack with a view to building a product solution based upon it. Specifically I have been trying to evaluate its performance in terms of requests/sec.

Here is the setup I have been using:

    Stack A  --->   Stack B ---->   Stack C
                 <----               <----

Where

    Stack A - is a load test client
    Stack B - is a proxy (it responds to 5% of the requests it receives with an answer containing an error code sent directly back to A and forwards the remaining 95% to C)
    Stack C - is a simple server that responds automatically with an successful answer back to A (via B)

With this setup I can achieve around 500 requests/sec before things start to go wrong (over 100% CPU and error messages of the form "RequestRoute map size is [10241]. There's probably a leak. Cleaning up after a short wait..."). I have profiled the proxy and it appears the majority of the cpu time is spent in TCPTransportClient.run().

Note that if I remove the proxy and get the load test client to interact directly with the server then I can achieve 5000+ requests per second without any problem.

Is the performance I have observed typical for the JDiameter stack in relay mode? Why is there such a big difference in requests/sec when there is no proxy?

Thanks

David Palmer

P.S. I have tried both the latest version built from the source code and the released version 1.5.10.0-build639.
 

[Alexandre Mendonça]

Hi David,

In theory there should be no reason for such difference in behavior, but it seems the proxy may be retaining the messages in the request route map. I'd try debugging and figuring out why is it growing up to 10240, see if by any chance some of the requests are not being properly discarded... I'd say the 95% for which the answer is not going through it are a good candidate :(

Regards,

Alexandre Mendonça
http://ammendonca.blogspot.com/

--
You received this message because you are subscribed to the Google Groups "mobicents-public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobicents-publ...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[tuan anh do]

Hi brothers,

How do you implement your relay( your relay mechanism). I cannot use pure jdiameter to relay.

How many connection did you use?

We found that with one connection establish between 2 any node ( client, server or relay) we just able to send 1000 messages per second smoothly without lost.

If this is correct we need more connections to get higher performance.

I think performance between client and server is one of the most important thing to discuss.

Thank you.

[David Palmer]

Update:-

On the server (Stack C) I noticed we were getting some buffer overflow exceptions:

server.log.1:26 Jun 2014 10:21:44,224 [TCPReader-21] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - Buffer overflow occured
server.log.1:26 Jun 2014 10:21:44,224 [TCPReader-21] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - Buffer overflow occured
server.log.1:26 Jun 2014 10:21:44,224 [TCPReader-21] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - Buffer overflow occured
server.log.1:26 Jun 2014 10:21:44,224 [TCPReader-21] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - Buffer overflow occured

I added some additional logging  to the TCAPTransportClient.java source file and found that on some occasions the version of the received messages was found to be invalid (not 1 as per RFC 3588).

Here is an extract of the TCAPTransportClient.java file:


  private boolean seekMessage() {
    // make sure there's actual data written on the buffer
    if (storage.position() == 0) {
      return false;
    }

    storage.flip();
    try {
      // get first four bytes for version and message length
      // 0                   1                   2                   3
      // 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      // |    Version    |                 Message Length                |
      // +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      int tmp = storage.getInt();
      // reset position so we can now read whole message
      logger.debug("No data received ?");
      storage.position(0);

      // check that version is 1, as per RFC 3588 - Section 3:
      // This Version field MUST be set to 1 to indicate Diameter Version 1
      byte vers = (byte) (tmp >> 24);
      if (vers != 1) {
        logger.error("Invalid message version detected");
        return false;
      }

Note that I added the line logger line (logger.error("Invalid message version detected");).

When I run the load test I occasionally see this error being reported in the log. When it does happen the storage buffer is left in an invalid state for subsequent reads from the TCP socket. This results in the messages not being sent back to the proxy which then causes a leak in the request route map and the subsequent problem I initially reported in the proxy.

I can reproduce the issue when I am running as a simple client server without the proxy although I need to drive it an a much higher request/sec before the error starts happening.

Is this a bug or could it be caused by incorrect implementation of the server or load test applications?

Thanks

David

[do tuan anh]

Hi,

To avoid that error you need to check function below

private void append(byte[] data)

I think you should add one condition to prevent over buffer size: data.length>storage.remaining()

==> It looks like this:

 if (storage.position() + data.length >= storage.capacity() ||

data.length>storage.remaining()) {

      ByteBuffer tmp = ByteBuffer.allocate(storage.limit() + data.length *

2);

      byte[] tmpData = new byte[storage.position()];

      storage.flip();

     storage.get(tmpData);

      tmp.put(tmpData);

      storage = tmp;

      logger.warn("Increase storage size. Current size is {}",

storage.array().length);

    }

But I think this is just a temporary solution. We need the size of buffer will not be increased.

This is really problem that haunt me.

Not need to setup a proxy, you just send from client to server with high through output ( over 1000 messages per second per connection, each message ~800bytes),you will see this error and other terrible errors like: cannot send any more, lost messages....

We need to pay attention to maximum messages per second, per connection....

Above is my opinion, could you please give more ideas about this, this is very important.

Thank you

[David Palmer]

I think i have found the root cause of the issue.

From TCPTransportClient.java:


  public void sendMessage(ByteBuffer bytes) throws IOException {
    if (logger.isDebugEnabled()) {
      logger.debug("About to send a byte buffer of size [{}] over the TCP nio socket [{}]", bytes.array().length, socketDescription);
    }
    int rc;
    // PCB - removed locking
    // lock.lock();
    try {
      rc = socketChannel.write(bytes);
    }
    catch (Exception e) {
      logger.error("Unable to send message", e);
      throw new IOException("Error while sending message: " + e);
    }
    finally {
    //  lock.unlock();
    }
    if (rc == -1) {
      throw new IOException("Connection closed");
    }
    else if (rc == 0)
    {
      logger.error("socketChannel.write(bytes) - returned zero indicating that perhaps the write buffer is full");
    }
    if (logger.isDebugEnabled()) {
      logger.debug("Sent a byte buffer of size [{}] over the TCP nio socket [{}]", bytes.array().length, socketDescription);
    }
  }

I have added the additional error log information shown in bold text to report when socketChannel.write(bytes); returns zero. When I run my load test client I observe that this error happens:

27 Jun 2014 17:29:03,096 [FSM-SPeer{Uri=aaa://cookie:3868; State=null; con=null; incConnull }_1-0] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - socketChannel.write(bytes) - returned zero indicating that perhaps the write buffer is full
27 Jun 2014 17:29:03,096 [FSM-SPeer{Uri=aaa://cookie:3868; State=null; con=null; incConnull }_1-0] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - socketChannel.write(bytes) - returned zero indicating that perhaps the write buffer is full
27 Jun 2014 17:29:03,096 [FSM-SPeer{Uri=aaa://cookie:3868; State=null; con=null; incConnull }_1-0] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - socketChannel.write(bytes) - returned zero indicating that perhaps the write buffer is full
27 Jun 2014 17:29:03,096 [FSM-SPeer{Uri=aaa://cookie:3868; State=null; con=null; incConnull }_1-0] ERROR org.jdiameter.client.impl.transport.tcp.TCPTransportClient  - socketChannel.write(bytes) - returned zero indicating that perhaps the write buffer is full

SocketChannel.write() returns zero when the send buffer is full (jDiameter uses non blocking IO). If the send buffer is full the message is not sent and this will lead to the server receiving incorrect data (i.e. not recognising the version).

I will raise a bug on this issue. In the meantime I am going to experiement with increasing the send buffer size (on my system the default size in 16kb).

David

[do tuan anh]

Hi Palmer ( Steve Palmer :-D),

Yes, I know the root cause is socket cannot write any more data. But do you know when socket is full an cannot write any more???

This question will solve our problem.

I think increase buffer size is temporary solution not a longterm solution

Vào 23:46:49 UTC+7 Thứ sáu, ngày 27 tháng sáu năm 2014, David Palmer đã viết:

[tuan anh do]

Hi David,
Did you resolve your problem?

>> * else if (rc == 0) { logger.error("socketChannel.write(bytes)
>>
>> - returned zero indicating that perhaps the write buffer is full"); }*

[Alexandre Mendonça]

See https://code.google.com/p/jdiameter/issues/detail?id=41 for a solution. It will be applied soon to the stack codebase to resolve the issue.

​Regards,

Alexandre Mendonça
http://ammendonca.blogspot.com/

[Attila Hajdu]

I use JDiameter as a client with high through output (600 TPS and each message ~1 KB). Usually we get BufferOverflowException from storage.put with this TPS. I think, you are right because ByteBuffer is not ThreadSafe. I wrote a simple program and I got similar output:

toString: java.nio.HeapByteBuffer[pos=0 lim=48 cap=48]
remaining: 48 

 

put(1)
put(2) 

 

toString: java.nio.HeapByteBuffer[pos=8 lim=48 cap=48]
remaining: 40 

 

flip() 

 

toString: java.nio.HeapByteBuffer[pos=0 lim=8 cap=48]
remaining: 8 

 

put(3) 

 

toString: java.nio.HeapByteBuffer[pos=4 lim=8 cap=48]
remaining: 4 

 

put(byte[12]) 

BufferOverflowException

So I think we have to check remaining() before we put data into ByteBuffer.