Segmentation Fault with multithread module and redis-benchmark

[Simone Mosciatti]

Hi all,

I am developing a simple modules that embed SQLitein Redis.
The single thread modules works, I won't say that it is ready but it works. github.com/siscia/rediSQL

Since SQLite support multithread I am trying to use that capability on the module.

However I found that while executing at the same time both redis-benchmark and some SQL statements via my own modules everything breaks and I get a segmentation fault.
If I execute only the SQL statements, everything works just fine.
If I execute the SQL statements and I play around with the redis-cli everything works.
Only the combination of SQL statements and redis-benchmark break the multithread module.

Usually the segmentation fault is generated by some free of memory done too early or by some access out of bound.
If it was some out of bound array I believe it would affect also the scenario where I execute only the SQL statements.
What I am left with is some too early free of memory, however I simply never use the keyword RedisModule_Free or free in all my code.

What else can cause a segmentation fault?

What the module does is quite simple, it receive a query, block the client and spawn a new pthread where the query it is executed.
If the query returns some data (think a select against a delete or an update) it memorize all the data in another data structure; when the query is over it unblock the client, loops through the data structure and return a nested array.
Then I do not free this data structure but I just keep it in memory.

The SQL statements are generated by a python script in a single thread way.

Where should I look?

I can definitily share the source of the module if it helps.

Cheers,

Simone

[Dvir Volk]

Hi, welcome to the group!

1. Can you provide a stack trace of the crash?

2. Did you handle client disconnects?

--
You received this message because you are subscribed to the Google Groups "redis-module-devs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-module-d...@googlegroups.com.
To post to this group, send email to redis-mo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-module-devs/5f1227ff-d4d1-4906-890e-0641c9f10181%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simone Mosciatti]

Hi Dvir :)

thanks for your suggestion over reddit :)

1. Sure, here it is: https://gist.github.com/siscia/35ea937ada05a9a2e70a67454a29b2d2

2. No I didn't, reading http://redis-module-redoc.readthedocs.io/en/latest/ I wasn't able to understand clearly what I should do. Can you point me on this?

[Dvir Volk]

Sorry, my bad, there is no such API, but it seems like it's unrelated anyway. 

Your stack trace doesn't provide enough info, and it seems like the the version you are running is not on github, am I right?

The problem happens when creating a string. So it's either an OOM situation, or an invalid argument (maybe the char * you're creating from is null? or was freed by another thread?)

Anyway without more insight into your code it's impossible to tell what's going on. 

Try running redis in gdb so in case of a crash you'll have much more verbose info. Also valgrind might help in this situation.

[Simone Mosciatti]

Yes you are right, the version I am running was not on github, anyway it is this one: https://github.com/siscia/rediSQL/tree/non_blocking

I will investigate further and hopefully come up with something :)

[Simone Mosciatti]

It is possible to use RedisModule_AutoMemory inside a blocking function that spawn a pthread?

My module is still giving problem, but the valgrind is showing different output.

[Dvir Volk]

Do not use any redis related function in a thread that is not the main redis thread. The can certainly fuck things up!

[ncha...@mux.com]

My suggestion is that you get things working *without* RedisModule_AutoMemory enabled, free'ing all memory you create manually.

Then, make sure that nothing you're freeing is being passed to your thread, and also that anything you pass to your thread is eventually getting free'd in a callback.

If you'd like to look at a working example, the module I wrote using the blocking api and passes data to it. You can take a look here:

https://github.com/chadnickbok/redis-zstd-module/blob/master/zstd_vals/zstd_vals.cpp#L80

My current code uses a threadpool and task-queue approach to concurrency, but in a previous revision I was spawning pthreads explicitly: https://github.com/chadnickbok/RedisModulesSDK/blob/983283f2c91c2f9cb6bb61791de0914beeeb624a/zstd_vals/zstd_vals.cpp#L93

[Dvir Volk]

Until we get thread safe module contexts (they are planned for redis 4.2 probably), just don't use any function that needs RedisModuleCtx * or RedisModuleKey * etc in a separate thread. 

--

You received this message because you are subscribed to the Google Groups "redis-module-devs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-module-d...@googlegroups.com.
To post to this group, send email to redis-mo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-module-devs/79f90fe4-305d-4e0f-a02f-57ced9a0fb57%40googlegroups.com.

[Simone Mosciatti]

I was just going to ask if I wasn't suppose to use RedisModuleCtx in separated thread.

Great :)

Maybe the documentation should mention it somewhere...

[Dvir Volk]

agreed. 

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-module-devs/f20c3f64-1b6a-42b7-9c85-bebe3c627424%40googlegroups.com.