vernemq cluster node terminate due to remote_node_not_available

121 views

Skip to first unread message

Roy

unread,

Sep 14, 2020, 3:22:31 PM9/14/20

to vernemq-users

I setup VerneMQ cluster on Kubernetes as StatefulSet, but in pods logs I was seeing following messages

sed: cannot rename /vernemq/etc/sedmHjPDJ: Device or resource busy /usr/sbin/start_vernemq: line 85: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 87: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 104: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 105: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 106: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 107: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 108: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 109: /vernemq/etc/vernemq.conf: Permission denied /usr/sbin/start_vernemq: line 111: /vernemq/etc/vernemq.conf: Permission denied config is OK -config /vernemq/data/generated.configs/app.2020.08.07.13.35.51.config -args_file /vernemq/bin/../etc/vm.args -vm_args /vernemq/bin/../etc/vm.args Exec: /vernemq/bin/../erts-10.7/bin/erlexec -boot /vernemq/bin/../releases/1.10.4.1/vernemq -config /vernemq/data/generated.configs/app.2020.08.07.13.35.51.config -args_file /vernemq/bin/../etc/vm.args -vm_args /vernemq/bin/../etc/vm.args -pa /vernemq/bin/../lib/erlio-patches -- console -noshell -noinput Root: /vernemq/bin/.. 13:35:53.195 [debug] Lager installed handler {lager_file_backend,"./log/error.log"} into lager_event 13:35:53.195 [debug] Lager installed handler error_logger_lager_h into error_logger 13:35:53.345 [info] alarm_handler: {set,{system_memory_high_watermark,[]}} 13:35:53.422 [info] read state from file {[{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[['Ver...@10.100.6.18',{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}]],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}} 13:35:53.423 [info] writing state {[{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[['Ver...@10.100.6.18',{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1}]],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}} to disk <<75,2,131,80,0,0,0,249,120,1,203,96,206,97,96,96,96,204,96,130,82,41,12,172,137,201,37,249,69,185,64,81,17,51,241,237,125,103,220,89,59,210,50,2,110,188,53,74,58,170,36,48,239,65,86,34,99,86,6,103,10,3,75,74,102,114,73,34,99,162,0,16,114,36,6,36,26,100,8,100,161,129,12,70,132,24,216,120,16,193,148,194,32,28,150,90,148,151,234,27,232,96,104,160,103,104,96,160,103,166,103,104,65,154,3,96,22,33,28,194,64,172,67,96,90,1,27,43,76,93>> 13:35:53.438 [info] Datadir ./data/meta/meta/0 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,38549145}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.445 [info] Datadir ./data/meta/meta/1 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,35929523}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.452 [info] Datadir ./data/meta/meta/2 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,31795161}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.459 [info] Datadir ./data/meta/meta/3 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,52387290}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.467 [info] Datadir ./data/meta/meta/4 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,35280518}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.475 [info] Datadir ./data/meta/meta/5 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,60988348}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.483 [info] Datadir ./data/meta/meta/6 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,43175699}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.502 [info] Datadir ./data/meta/meta/7 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,35409215}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.509 [info] Datadir ./data/meta/meta/8 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,51476514}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.516 [info] Datadir ./data/meta/meta/9 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,62879451}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.523 [info] Datadir ./data/meta/meta/10 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,54094689}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.530 [info] Datadir ./data/meta/meta/11 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,6},{use_bloomfilter,true},{write_buffer_size,54172211}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.550 [debug] 0ms mailbox traversal, schedule next lazy broadcast in 10000ms, the min interval is 10000ms 13:35:53.577 [debug] loading modules: [vmq_plumtree,vmq_plumtree_app,vmq_plumtree_sup] 13:35:53.603 [info] Try to start vmq_plumtree: ok 13:35:53.637 [debug] datadir ./data/msgstore/1 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,47750847}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.645 [info] Opening LevelDB database at "./data/msgstore/1" 13:35:53.646 [debug] datadir ./data/msgstore/2 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,50090150}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.653 [info] Opening LevelDB database at "./data/msgstore/2" 13:35:53.653 [debug] datadir ./data/msgstore/3 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,34848743}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.660 [info] Opening LevelDB database at "./data/msgstore/3" 13:35:53.661 [debug] datadir ./data/msgstore/4 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,41593831}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.667 [info] Opening LevelDB database at "./data/msgstore/4" 13:35:53.668 [debug] datadir ./data/msgstore/5 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,45767049}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.674 [info] Opening LevelDB database at "./data/msgstore/5" 13:35:53.675 [debug] datadir ./data/msgstore/6 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,56785000}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.681 [info] Opening LevelDB database at "./data/msgstore/6" 13:35:53.682 [debug] datadir ./data/msgstore/7 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,60588967}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.683 [debug] Lager installed handler lager_backend_throttle into lager_event 13:35:53.688 [info] Opening LevelDB database at "./data/msgstore/7" 13:35:53.689 [debug] datadir ./data/msgstore/8 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,41872335}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.695 [info] Opening LevelDB database at "./data/msgstore/8" 13:35:53.696 [debug] datadir ./data/msgstore/9 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,35898829}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.703 [info] Opening LevelDB database at "./data/msgstore/9" 13:35:53.703 [debug] datadir ./data/msgstore/10 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,47214115}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.710 [info] Opening LevelDB database at "./data/msgstore/10" 13:35:53.711 [debug] datadir ./data/msgstore/11 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,36069724}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.719 [info] Opening LevelDB database at "./data/msgstore/11" 13:35:53.720 [debug] datadir ./data/msgstore/12 options for LevelDB: [{open,[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,1000},{eleveldb_threads,71},{fadvise_willneed,false},{limited_developer_mem,false},{sst_block_size,4096},{tiered_slow_level,0},{total_leveldb_mem_percent,70},{use_bloomfilter,true},{write_buffer_size,56905298}]},{read,[{verify_checksums,true}]},{write,[{sync,false}]},{fold,[{verify_checksums,true},{fill_cache,false}]}] 13:35:53.730 [info] Opening LevelDB database at "./data/msgstore/12" 13:35:53.730 [debug] loading modules: [vmq_generic_msg_store,vmq_generic_msg_store_app,vmq_generic_msg_store_sup,vmq_generic_msg_store_utils,vmq_storage_engine_dets,vmq_storage_engine_ets,vmq_storage_engine_leveldb] 13:35:53.767 [info] Try to start vmq_generic_msg_store: ok 13:35:53.770 [info] rename VerneMQ node from 'Ver...@10.100.6.18' to 'Ver...@10.100.6.34' 13:35:53.773 [info] writing state {[{[{actor,<<54,23,183,142,204,71,5,136,102,104,80,216,237,50,98,197,34,16,158,224>>}],1},{<<105,153,35,109,220,173,228,160,165,82,37,54,41,250,215,10,226,156,138,159>>,1}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[['Ver...@10.100.6.34',{<<105,153,35,109,220,173,228,160,165,82,37,54,41,250,215,10,226,156,138,159>>,1}]],[],[],[],[],[],[],[],[],[],[],[]}}},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}} to disk <<75,2,131,80,0,0,1,6,120,1,203,96,206,97,96,96,96,202,96,2,81,140,25,76,41,12,172,137,201,37,249,69,185,64,174,136,153,248,246,190,51,238,172,29,105,25,1,55,222,26,37,29,85,18,152,247,32,43,17,168,10,44,155,57,83,57,247,206,218,39,11,150,6,169,154,105,254,186,206,245,104,78,215,252,68,198,172,12,206,20,6,150,148,204,228,146,68,198,68,1,32,228,72,12,72,52,200,16,200,66,3,25,140,16,49,176,189,96,55,164,48,8,135,165,22,229,165,250,6,58,24,26,232,25,26,24,232,153,233,25,155,224,183,12,217,80,132,197,12,196,88,140,172,21,0,76,92,86,0>> 13:35:53.868 [info] loaded 0 subscriptions into vmq_reg_trie 13:35:53.872 [info] cluster event handler 'vmq_cluster' registered . . . 15:10:21.166 [debug] connect request from unknown cluster node 'Ver...@vernemq-1.vernemq.dev-ethernet.svc.cluster.local' 15:10:21.166 [warning] terminate due to remote_node_not_available 15:10:21.166 [warning] connection to node 'Ver...@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms 15:10:22.168 [info] successfully connected to cluster node 'Ver...@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' 15:10:22.171 [debug] connect request from unknown cluster node 'Ver...@vernemq-1.vernemq.dev-ethernet.svc.cluster.local' 15:10:22.171 [warning] terminate due to remote_node_not_available 15:10:22.171 [warning] connection to node 'Ver...@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms 15:10:23.174 [info] successfully connected to cluster node 'Ver...@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' 15:10:23.175 [debug] connect request from unknown cluster node 'Ver...@vernemq-1.vernemq.dev-ethernet.svc.cluster.local' 15:10:23.175 [warning] terminate due to remote_node_not_available 15:10:23.175 [warning] connection to node 'Ver...@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' has been closed, reconnect in 1000ms 15:10:24.178 [info] successfully connected to cluster node 'Ver...@vernemq-0.vernemq.dev-ethernet.svc.cluster.local' 15:10:24.181 [debug] connect request from unknown cluster node 'Ver...@vernemq-1.vernemq.dev-ethernet.svc.cluster.local' 15:10:24.181 [warning] terminate due to remote_node_not_available