Getting exception voldemort.store.InsufficientOperationalNodesException in production systems

[Anandh Kumar]

We are getting following exption in production system some time examples out of 84mn call 10000 times we are getting this exception

voldemort.store.InsufficientOperationalNodesException: 1 get alls required, but 0 succeeded. Failing nodes : []

I tried to increase the Voldemort ClientConfig connection_timeout and sockettimeout but till getting this error.

Our config:

stores.xml:

<stores>

<store>

  <name>test</name>

  <persistence>bdb</persistence>

  <description>test</description>

  <owners>test</owners>

  <routing-strategy>consistent-routing</routing-strategy>

  <routing>client</routing>

  <replication-factor>1</replication-factor>

  <required-reads>1</required-reads>

  <required-writes>1</required-writes>

  <key-serializer>

    <type>string</type>

  </key-serializer>

  <value-serializer>

    <type>string</type>

  </value-serializer>

</store>

</stores>

Server.properties

node.id=0

# configs

admin.enable=true

admin.max.threads=40

bdb.cache.evictln=true

bdb.cache.size=12GB

bdb.checkpoint.interval.bytes=2147483648

bdb.checkpointer.off.batch.writes=true

bdb.cleaner.interval.bytes=15728640

bdb.cleaner.lazy.migration=false

bdb.cleaner.min.file.utilization=0

bdb.cleaner.threads=1

bdb.enable=true

bdb.evict.by.level=true

bdb.expose.space.utilization=true

bdb.lock.nLockTables=94

bdb.minimize.scan.impact=true

bdb.one.env.per.store=true

enable.server.routing=false

enable.verbose.logging=false

http.enable=true

nio.connector.selectors=100

num.scan.permits=2

request.format=vp3

restore.data.timeout.sec=1314000

scheduler.threads=24

slop.frequency.ms=300000

socket.enable=true

storage.configs=voldemort.store.bdb.BdbStorageConfiguration, voldemort.store.readonly.ReadOnlyStorageConfiguration

stream.read.byte.per.sec=209715200

stream.write.byte.per.sec=78643200

cluster.xml

<cluster>

        <name>myprodcluster</name>

        <server>

                <id>0</id>

                <host>192.168.1.10</host>

                <http-port>8081</http-port>

                <socket-port>6666</socket-port>

                <admin-port>7777</admin-port>

                <partitions>0, 1</partitions>

        </server>

</cluster>

We have single node cluster

Can you please help me how we can fix this issue

[Arunachalam]

Try increasing the number of connections. From What I infer, there is no failing nodes, so it was just waiting in the client queue and timing out.

Other reason is you are throwing more load than the Client/Server could handle. You should monitor the p90/p99 response times and see if you see a spike and where is the contention happening from.

Thanks,

Arun.

On Wed, Aug 9, 2017 at 2:49 AM, Anandh Kumar <anand...@gmail.com> wrote:

We are getting following exption in production system some time examples out of 84mn call 10000 times we are getting this exception

voldemort.store.InsufficientOperationalNodesException: 1 get alls required, but 0 succeeded. Failing nodes : []

I tried to increase the Voldemort ClientConfig connection_timeout and sockettimeout but till getting this error.

Can you please help me how we can fix this issue

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Anandh Kumar]

Arun,

Still we are getting this exception after increased ConnectionTimedout,SocketTimedout.

Can you please help me here how we can fix this.

Regards,

-Anandh Kumar

On Saturday, August 12, 2017 at 2:56:58 AM UTC+5:30, Arun Thirupathi wrote:

Try increasing the number of connections. From What I infer, there is no failing nodes, so it was just waiting in the client queue and timing out.

Other reason is you are throwing more load than the Client/Server could handle. You should monitor the p90/p99 response times and see if you see a spike and where is the contention happening from.

Thanks,

Arun.

On Wed, Aug 9, 2017 at 2:49 AM, Anandh Kumar <anand...@gmail.com> wrote:

We are getting following exption in production system some time examples out of 84mn call 10000 times we are getting this exception

voldemort.store.InsufficientOperationalNodesException: 1 get alls required, but 0 succeeded. Failing nodes : []

I tried to increase the Voldemort ClientConfig connection_timeout and sockettimeout but till getting this error.

Can you please help me how we can fix this issue

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.

[Félix GV]

Did you try increasing the number of connections like Arun suggested?