large rabbitMQ log file

[Bart Vleminckx]

Dear all,

We did a large hazard calculation with OpenQuake and got the problem that there was not enough disk space. It seems that the rabbitMQ log file gets really large (already 10 GB). It writes multiple logs per second (most of them about TCP connections). We tried to change the log levels of rabbitMQ, celery and OpenQuake, but it keeps writing the "TCP connection" logs. Is there a solution for this problem? We also think such massive logging slows down the calculations, no?

Best regards,

Bart Vleminckx

Royal Observatory of Belgium

[Bart Vleminckx]

Dear all,

We found a solution. On the rabbitMQ website you can find the latest version of rabbitMQ (not standard included in Ubuntu 12.04), which supports different log levels. However is it normal that so many TCP connections start and close with OpenQuake?

Best regards and a Happy New Year!

Bart Vleminckx

Royal Observatory of Belgium

Op donderdag 2 januari 2014 14:30:29 UTC+1 schreef Bart Vleminckx:

[Daniele Viganò]

Hi bart,

sorry for the late response.

Yes, that's normal. OpenQuake iterate a TaskSet for results, asking results to RabbitMQ/amqp (see https://github.com/gem/oq-engine/blob/master/openquake/engine/utils/tasks.py#L64)

But we discovered that lots of connections are left in TIME_WAIT. This issue is caused by CELERY_ACKS_LATE and we are investigating about it.

If you upgraded RabbitMQ to a version >= 3.0 the issue is already fixed on the RabbiMQ side.

Regards,
Daniele