Rabbit 3.6.2 crash: no more index entries in atom_tab (max=1048576)

[Kelly Campbell]

I got this crash from a stock RabbitMQ 3.6.2 running on a Ubuntu 16.04 vm.

There was no real traffic going through it at the time. It is not on a production system. I did have a management UI tab left open in a browser for a couple days though, so I think that might possibly be related.

=erl_crash_dump:0.3

Wed Jul 13 00:25:50 2016

Slogan: no more index entries in atom_tab (max=1048576)

System version: Erlang/OTP 18 [erts-7.3] [source] [64-bit] [async-threads:64] [kernel-poll:true]

Compiled: Fri Apr  8 10:11:48 2016

Taints: crypto,asn1rt_nif

Atoms: 1048576

Calling Thread: scheduler:1

At the end of the crash file in the atoms section, it has many lines like this:

=atoms

'2085839021_index'

'2085834928_key_index'

'2085834928_index'

'2085830678_key_index'

'2085830678_index'

'2085826740_key_index'

'2085826740_index'

'2085822635_key_index'

'2085822635_index'

'2085818546_key_index'

'2085818546_index'

'2085814293_key_index'

'2085814293_index'

'2085810346_key_index'

'2085810346_index'

'2085806253_key_index'

'2085806253_index'

'2085802160_key_index'

'2085802160_index'

'2085797910_key_index'

'2085797910_index'

'2085793972_key_index'

'2085793972_index'

'2085789867_key_index'

'2085789867_index'

More info:

$ egrep "[0-9]+_key_index" /var/lib/rabbitmq/erl_crash.dump | wc -l

507988

=hash_table:atom_tab

size: 823117

used: 584084

objs: 1048577

depth: 10

=index_table:atom_tab

size: 1048576

limit: 1048576

entries: 1048576

$ uname -a

Linux ubuntu 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/rabbitmq/enabled_plugins 

[rabbitmq_auth_mechanism_ssl,rabbitmq_management].

$ cat /etc/rabbitmq/rabbitmq.config 

[

  {rabbit, [

    {auth_mechanisms, ['PLAIN','EXTERNAL']},

    {tcp_listeners, [{"0.0.0.0", 5672}]},

    {ssl_cert_login_from, common_name},

    {ssl_listeners, [{"0.0.0.0", 5671}]},

    {ssl_options, [

      {cacertfile, "/etc/rabbitmq/ssl/CA-roots.pem"},

      {certfile, "/etc/rabbitmq/ssl/rabbit_server.crt"},

      {keyfile, "/etc/rabbitmq/ssl/rabbit_server.key"},

      {depth, 2},

      {verify, verify_peer},

      {fail_if_no_peer_cert, true}

      ]}

  ]}

].

[Michael Klishin]

Already fixed: https://github.com/rabbitmq/rabbitmq-management/issues/245 explains the issue

and a workaround.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Kelly Campbell]

Thanks! Sorry, I searched all over for the issue first but didn't find that github project one.

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/Iz-DkVZVtiA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

No worries. 3.6.4 Milestone 1 will likely drop this week.

[aros...@gmail.com]

Hi Michael,

Just wondering, would leaving management UI pages open exacerbate this problem? We are also experience this crash approximately once weekly since upgrading from 3.6.0 to 3.6.2 (OTP 7.2.1 to 7.3), and we tend to have several browsers open to the management page around the clock. Also, we have a service that polls the API periodically to check for partitions.

Would terminating the UI pages mitigate the problem at all (in addition to or as a substitute for the workaround)? We basically need it to just last the work week, and it makes it to Thursday as is, generally. We can institute a scheduled weekend restart if necessary.

Thanks for any insight,

Anthony

[Michael Klishin]

No. Temporary tables are created when outdated stats have to be deleted. It doesn't matter

if you have more or fewer HTTP API requests.

Bump the limit to 3-4 millions. 3.6.4 will be out by then.

[aros...@gmail.com]

Thanks will do.

[aros...@gmail.com]

Sorry, I can just add that, say, after the +P argument in ERLANG_SERVICE_ARGUMENTS definition in rabbitmq-service.bat, and remove/reinstall the service?

[aros...@gmail.com]

Nevermind, read the docs, sorry...

[Michael Klishin]

I'd recommend using RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS

over other options but yes, you can modify the script, too.

[aros...@gmail.com]

Perfect, I did in fact end up using the environment variable.

Is there a rabbitmqctl command or management API call, or any other way, to verify this setting?

[Michael Klishin]

rabbitmqctl eval 'erlang:system_info(port_limit).'

[Michael Klishin]

Sorry, I've mixed up two threads. The above line would output

the effective port limit, not the atom table size one.

[Karl Nilsson]

You could try something like:

rabbitmqctl eval 'string:tokens(binary_to_list(erlang:system_info(info)),"\n").'

This will dump a big list of information and somewhere in there you'll find an entry for "=index_table:atom_tab" and a bit after that a "limit" which I believe is referring to active atom table size limit.

If anyone knows a nicer way to get at this info I'd like to know!

Cheers

Karl

Karl Nilsson

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Try rabbitmqctl eval 'erlang:memory(atom).'

Alternatively you can run `ps aux` or similar on the command line and see

the effective VM flags for beam/beam.smp.

[aros...@gmail.com]

Hmm... I've tried all the commands above, and get "Error: syntax error before:" for each one.

This is on Windows, is there a syntax difference?

[aros...@gmail.com]

Switching to double-quotes seems to have fixed it.

But the command above appears to display the current atom count, not the max - or at least, the number doesn't match the "+t 4190304" I have set in the environment variable.

[aros...@gmail.com]

managed to get this working without the string:tokens (couldn't figure out how to escape the "\n" in Windows).

I do see the limit there, thanks!