php nameko adapter

[dar...@yupeek.com]

hello all, we use namko for our python micro-sevices and so far, it work great.

our next move will be to allow rpc call from php scripts.

I am working on a proof of concept, which will probably lead to a open sourced adapter for at least RPC.

(right now, I have a rpc call PHP => python with results.)

but my POC encounter a heavy performance issue.

the setup is a simple consumer-producer.

python consumer call the get method of producer, which return 42.

my benchmark give me from 16s with 5000 async call to 18s with synchronous call.

but the php script which create reply-to queue and mimic python side, run in 1.5sec for  async, to 263s (!!!!) for synchronous call. 

i don't know if this is somehing to do with eventlet or mono-threading... but if you have any tips...

the php service code:

<?php
/**
 * Created by PhpStorm.
 * User: dbernard
 * Date: 09/04/18
 * Time: 17:03
 */


require_once __DIR__ . '/vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
use PhpAmqpLib\Message\AMQPMessage;

class ConsumerRpcClient {
   private $connection;
   private $channel;
   private $callback_queue;
   private $response;
   private $corr_id;
   private $reply_id;

   public function __construct() {
      $this->connection = new AMQPStreamConnection(
         '172.18.0.2', 5672, 'guest', 'guest');
      $this->channel = $this->connection->channel();
      $this->channel->basic_qos(null, 10, null);
      $this->reply_id = uniqid();
      list($this->callback_queue, ,) = $this->channel->queue_declare(
         "rpc.reply-phpconsumer-".$this->reply_id, false, false, true, true);
      $this->channel->queue_bind($this->callback_queue, 'nameko-rpc', $this->reply_id);

      $this->channel->basic_consume(
         $this->callback_queue, '', false, false, false, false,
         array($this, 'on_response'));
   }
   public function on_response($rep) {
      if($rep->get('correlation_id') == $this->corr_id) {
         $this->response = $rep->body;
      }
      $rep->delivery_info['channel']->basic_ack($rep->delivery_info['delivery_tag']);
   }
   public function call_async($n) {
      $this->response = null;
      $this->corr_id = uniqid();

      $msg = new AMQPMessage(
         (string) $n,
         array('correlation_id' => $this->corr_id,
               'reply_to' => $this->reply_id,
               'content_encoding' => 'utf-8',
               'content_type' => 'application/x-myjson'

         )
      );
      $this->channel->basic_publish($msg, 'nameko-rpc', 'producer.get');
   }

   public function call($n) {
      $this->call_async($n);

      while(!$this->response) {
         $this->channel->wait();
      }
      return $this->response;
   }
};

$consumer_rpc = new ConsumerRpcClient();
$start        = microtime(true);

assert( $consumer_rpc->call('{"args": [], "kwargs": {}}') === '{"result": 42, "error": null}');

for ($i = 0; $i < 5000; $i++) {
   $consumer_rpc->call('{"args": [], "kwargs": {}}');
}
$end = microtime(true);
echo 'called '.$i.' times in '.($end - $start).' seconds';

the python producer code

class Producer(object):
    """
    a fake service that produce data in a specified time
    used to test service auto scale
    
    public events
    #############
    
    rcp
    ###
    
    set(rate)
    
    get()
    
    """
    name = 'producer'

    @rpc
    def get(self):
        """
        create a service on the valide scaler
        :param service: 
        :return: 
        """
        # eventlet.patcher.original('time').sleep(0.009)
        return 42

[Matt Yule-Bennett]

Your benchmark should be the standalone RPC client found in nameko.standalone.rpc -- this is a single-threaded, non-eventlet implementation.

Are you creating a new reply queue for every request? That would kill your performance.

Also check that your client is allowing multiple RPC requests to be in flight at the same time, rather than waiting for a reply before dispatching the next one.

[dar...@yupeek.com]

thanks for the reply :)

I wrote a standalone client which work with similar performances as the service oriented one (18s for 5000 calls), so it seem it's not that.

there is only one queue created, and every 5000 calls use it for reply. the for reuse the same connexion.

my python benchmark with problem is synchronous (wait for reply before sending other request). the php implementation do it the same way.

I did a xhprof for the php script, and the time cost is in the fread function. so it seem it's not the php script which is incrimined, but it wait looooong for the reply from rabbitmq.

I will work on it to implement a better request, respecting all header ans property sent by nameko implementation.

is there a specific missing header that can lead the producer to run very slowly ?

[Matt Yule-Bennett]

No header that I'm aware of would cause performance problems if missing.

You may want to check out the issue at https://github.com/nameko/nameko/issues/354 for some detailed info on the RPC implementation.