redis-cli: hiredis.c:515: redisFormatSdsCommandArgv: Assertion `sdslen(cmd)==totlen' failed.

[Gyanranjan Muduli]

Hi Team,

Please help to fix this issue.

Verison= 6.2.6
1_4333174786_617284272.txt(4GB)
cat 1_4333174786_617284272.txt | redis-cli -h ****** -p 6379 -a **** -n 2 -x restore 1 0
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-cli: hiredis.c:515: redisFormatSdsCommandArgv: Assertion `sdslen(cmd)==totlen' failed.
Aborted

Thanks

Gyan

[Gyanranjan Muduli]

FYI -I have set proto-max-bulk-len to 10GB

Thanks

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/4f199a02-ad52-4634-a1ec-b5e90aefc2adn%40googlegroups.com.

[Gyanranjan Muduli]

Hi Team,

Can anyone help me out on this issue please?

Thanks  in advance.

[Itamar Haber]

Hello Gyan,

I can reproduce this, so I suggest that you open an issue at https://github.com/redis/redis with these details.

Cheers,

Itamar

[sundb]

Hi Gyan,

https://github.com/redis/redis/blob/c7dc17fc0f9341f61be1a1318468409249310316/deps/hiredis/hiredis.c#L605

When hi_sdscatfmt formats ulong parameters it should use "%U", not "%u".
Do you wanna make a pr to fix it?

[Gyanranjan Muduli]

Hi ,

Yes . I have changed   ulong parameter   it works .Again i stuck  with below issue.

we have tried  to restore  dumpdata szie of (4333173058)

dump data pushed size- 4333173058

 ssize_t nwritten = send(c->fd, c->obuf, hi_sdslen(c->obuf), 0);  
But in socket its writing 38205762 after printing value of  unwritten.
I doubt on below function  its setting to int instead of long

static inline size_t hi_sdslen(const hisds s) {
unsigned char flags = s[-1];
switch(flags & HI_SDS_TYPE_MASK) {
case HI_SDS_TYPE_5:
return HI_SDS_TYPE_5_LEN(flags);
case HI_SDS_TYPE_8:
return HI_SDS_HDR(8,s)->len;
case HI_SDS_TYPE_16:
return HI_SDS_HDR(16,s)->len;
case HI_SDS_TYPE_32:
return HI_SDS_HDR(32,s)->len;
case HI_SDS_TYPE_64:
return HI_SDS_HDR(64,s)->len;
}
return 0;
}
unsigned char flags = s[-1]; Since it  is returning 3  it is entering to  case HI_SDS_TYPE_32.
i believe some where flag need to set to 4 (considering it is a long int)

so that it should enter the case HI_SDS_TYPE_64:

Help me to fix issue any data beyond 4GB not able to send redis server ,it always failing it in redis client.

Thanks

Gyan

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/33c1dc20-beb2-41bb-97a5-8d1bc3943bf0n%40googlegroups.com.

[sundb]

Hi Gyan.

The return type of redisFormatSdsCommandArgv is overflow.

You can change the return type of redisFormatSdsCommandArgv to size_t.

[sundb]

Hi Gyan.

Missing something, the type of len in redisAppendCommandArgv also needs to be changed to size_t.

[sundb]

Hi Gyan.

Forgot to tell you again, need to set client-query-buffer-limit 10gb

[Itamar Haber]

Also, since the issue is with redis-cli, you should be able to `RESTORE` the 4GB payload with another client (e.g. Jedis, redis-py,...).

[Gyanranjan Muduli]

we are  able to  restore the files  more than 4GB data into master node, but sync is not  happening properly  among master and slave. getting conncection lost and master is converting  to any one of our slave and removing  particular key is getting deleted from  master node.

we  set  CONFIG SET client-output-buffer-limit "slave 0 0 60"  also.
Same issue not obseved  there is no slave is running.

error logs=
131583:S 20 Dec 2021 17:43:17.519 * MASTER <-> REPLICA sync: Finished with success
131583:M 20 Dec 2021 17:50:58.992 # Connection with master lost.
131583:M 20 Dec 2021 17:50:58.992 * Caching the disconnected master state.
131583:M 20 Dec 2021 17:50:58.992 * Discarding previously cached master state.
131583:M 20 Dec 2021 17:50:58.992 # Setting secondary replication ID to b11f210e866634dfb4e00325096f2aad88e3e16a, valid up to offset: 95202067. New replication ID is c0333623c6b47cbdd93e4f5640ecd5987b419848
131583:M 20 Dec 2021 17:50:58.992 * MASTER MODE enabled (user request from 'id=599589 addr=*.*.*.*:60318 laddr=*.*.*.*:6379 fd=16 name=sentinel-f1ece4e0-cmd age=39 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=188 qbuf-free=40766 argv-mem=4 obl=45 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default redir=-1')
131583:M 20 Dec 2021 17:50:58.998 # CONFIG REWRITE executed with success.
131583:M 20 Dec 2021 17:51:00.091 * Replica *.*.*.*:6379 asks for synchronization
131583:M 20 Dec 2021 17:51:00.091 * Partial resynchronization request from *.*.*.*:6379 accepted. Sending 169 bytes of backlog starting from offset 95202067.
131583:M 20 Dec 2021 17:51:19.665 * Replica *.*.*.*:6379 asks for synchronization
131583:M 20 Dec 2021 17:51:19.665 * Partial resynchronization not accepted: Requested offset for second ID was 4428392207, but I can reply up to 95202067
131583:M 20 Dec 2021 17:51:19.665 * Starting BGSAVE for SYNC with target: disk
131583:M 20 Dec 2021 17:51:19.786 * Background saving started by pid 123067
123067:C 20 Dec 2021 17:51:59.512 * DB saved on disk
123067:C 20 Dec 2021 17:51:59.611 * RDB: 4133 MB of memory used by copy-on-write
131583:M 20 Dec 2021 17:52:00.090 * Background saving terminated with success
131583:M 20 Dec 2021 17:52:07.981 * Synchronization with replica *.*.*.*:6379 succeeded
131583:M 20 Dec 2021 18:07:01.059 * 1 changes in 900 seconds. Saving...
131583:M 20 Dec 2021 18:07:01.194 * Background saving started by pid 137790
137790:C 20 Dec 2021 18:07:56.116 * DB saved on disk
137790:C 20 Dec 2021 18:07:56.275 * RDB: 4146 MB of memory used by copy-on-write
131583:M 20 Dec 2021 18:07:56.697 * Background saving terminated with success

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/79f5a391-712f-4bcd-b05d-093e1287c46bn%40googlegroups.com.

[Gyanranjan Muduli]

Hii Team,

Can someone  please  help me to fix this issue?

Thanks

 Gyan 

[sundb]

I am not familiar with slave, you can open an issue at  https://github.com/redis/redis/issues for help.

[sundb]

Hi  Gyan.

Do you have one key size over 4GB?

[Gyanranjan Muduli]

Yes single dump bloom file key size is 4gb.

Thanks

Gyan

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/19ac8ba1-ad19-4840-8752-00d2e3b07788n%40googlegroups.com.

[Itamar Haber]

@sundb - yes, I've spoken with Gyan's colleague Munmendu and he explained that they are storing mpgs (IIRC) in keys.

I suggest we continue the discussion over a new issue (perhaps two) on in the repo - lets wait until the guys open it.

[sundb]

@Gyan Does repl have failed logs? Are there any lzf related errors?