Infamous "JedisConnectionException: Unexpected end of stream"

[Tomáš Sedláček]

Hi there,

I use Redis bloom filter component to filter out duplicit elements in a stream of data. To connect to that component from my application I use Jedis client.

Every couple of hours I get an exception "JedisConnectionException: Unexpected end of stream"  when adding a new element from the stream to the Bloom filter component.

Here is the stack trace:

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.

    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202)

    at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:43)

    at redis.clients.jedis.Protocol.process(Protocol.java:155)

    at redis.clients.jedis.Protocol.read(Protocol.java:220)

    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:283)

    at redis.clients.jedis.Connection.getIntegerReply(Connection.java:225)

    at io.rebloom.client.Client.add(Client.java:122)

I searched for possible root causes and find out that:

1) Connection is idle for a long time and therefore disconnected by the server

2) Jedis instance is operated concurrently by multiple threads

Ad 1) There is no timeout configured on my Redis server

> config get timeout "timeout" "0"

Ad 2) I use JedisPool (100 connections in the pool) in my Jedis client which,

as I read, should be thread safe.

Here is the snippet of my code:

class XYZ()

{

private var conn: Client = _

...

private def configureJedisPool(): JedisPool = {
val timeoutMs = 30000 // 30s
val poolSize = 100

val conf = new JedisPoolConfig()
conf.setMaxTotal(poolSize)
conf.setTestOnBorrow(false)
conf.setTestOnReturn(false)
conf.setTestOnCreate(false)
conf.setTestWhileIdle(true)
conf.setMinEvictableIdleTimeMillis(30000)
conf.setTimeBetweenEvictionRunsMillis(15000)
conf.setNumTestsPerEvictionRun(-1)
conf.setFairness(true)

new JedisPool(conf, hostname, port, timeoutMs)
}

def open(): Unit = {

// Create connection to Redis DB
val pool = configureJedisPool()
conn = new Client(pool)

}

def seen(element: ABC): Boolean = {

!conn.add(filterName, MLRowKeySerializer.serialize(element))
}

...

}

Any help would by appreciated. Thanks a lot.

Regards,

Tomas

[Sazzadul Hoque]

Extra large commands can cause this error as well.

Can MLRowKeySerializer.serialize(element) be very large? Say, exceeding 500MB?

--
You received this message because you are subscribed to the Google Groups "Jedis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jedis_redis/6222178e-1944-4d88-9b72-e5ece67fc913n%40googlegroups.com.

[Tomáš Sedláček]

Hi Sazzadul,

Thanks for your reply.

However 'MLRowKeySerializer.serialize(element)'  isn't that big. It is actually pretty small as it is just two key-value pairs where the values are just numbers. 

[Tomáš Sedláček]

However there are plenty of elements in the Bloom filter - tens of millions.

[Oscar Besga]

Should testOnBorrow be put to true, and check it ?
Could be the OS or any proxy cutting the connection in a large operation ?

Just some ideas