Infamous "JedisConnectionException: Unexpected end of stream"

169 views
Skip to first unread message

Tomáš Sedláček

unread,
Jan 21, 2022, 2:48:42 PM1/21/22
to Jedis
Hi there,

I use Redis bloom filter component to filter out duplicit elements in a stream of data. To connect to that component from my application I use Jedis client.
Every couple of hours I get an exception "JedisConnectionException: Unexpected end of stream"  when adding a new element from the stream to the Bloom filter component.
Here is the stack trace:

redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202)
    at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:43)
    at redis.clients.jedis.Protocol.process(Protocol.java:155)
    at redis.clients.jedis.Protocol.read(Protocol.java:220)
    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:283)
    at redis.clients.jedis.Connection.getIntegerReply(Connection.java:225)
    at io.rebloom.client.Client.add(Client.java:122)

I searched for possible root causes and find out that:

1) Connection is idle for a long time and therefore disconnected by the server
2) Jedis instance is operated concurrently by multiple threads

Ad 1) There is no timeout configured on my Redis server

> config get timeout "timeout" "0"

Ad 2) I use JedisPool (100 connections in the pool) in my Jedis client which,
as I read, should be thread safe.

Here is the snippet of my code:

class XYZ()
{
private var conn: Client = _
...
private def configureJedisPool(): JedisPool = {
val timeoutMs = 30000 // 30s
val poolSize = 100

val conf = new JedisPoolConfig()
conf.setMaxTotal(poolSize)
conf.setTestOnBorrow(false)
conf.setTestOnReturn(false)
conf.setTestOnCreate(false)
conf.setTestWhileIdle(true)
conf.setMinEvictableIdleTimeMillis(30000)
conf.setTimeBetweenEvictionRunsMillis(15000)
conf.setNumTestsPerEvictionRun(-1)
conf.setFairness(true)

new JedisPool(conf, hostname, port, timeoutMs)
}

def open(): Unit = {
// Create connection to Redis DB
val pool = configureJedisPool()
conn = new Client(pool)
}

def seen(element: ABC): Boolean = {
!conn.add(filterName, MLRowKeySerializer.serialize(element))
}
...
}

Any help would by appreciated. Thanks a lot.

Regards,

Tomas

Sazzadul Hoque

unread,
Jan 28, 2022, 8:47:27 AM1/28/22
to jedis...@googlegroups.com
Extra large commands can cause this error as well.

Can MLRowKeySerializer.serialize(element) be very large? Say, exceeding 500MB?

--
You received this message because you are subscribed to the Google Groups "Jedis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jedis_redis/6222178e-1944-4d88-9b72-e5ece67fc913n%40googlegroups.com.

Tomáš Sedláček

unread,
Feb 1, 2022, 5:12:30 PM2/1/22
to Jedis
Hi Sazzadul,

Thanks for your reply.

However 'MLRowKeySerializer.serialize(element)'  isn't that big. It is actually pretty small as it is just two key-value pairs where the values are just numbers. 

Tomáš Sedláček

unread,
Feb 1, 2022, 5:17:04 PM2/1/22
to Jedis
However there are plenty of elements in the Bloom filter - tens of millions.

Oscar Besga

unread,
Mar 2, 2022, 5:46:04 AM3/2/22
to Jedis
Should testOnBorrow be put to true, and check it ?
Could be the OS or any proxy cutting the connection in a large operation ?

Just some ideas
Reply all
Reply to author
Forward
0 new messages