Hi Assad,
Thanks a lot for your help and response. I know the following:
1. This error does not come up with other clients that do not alter the application. So this is definitely my doing, I was just trying to find out which part of my doing :)
2. This error and that BB both happen inside a `pthread_create` call, I know because I wrapped it and printed before and after it
3. I heavily modify the application's memory in the following way(s), for the purpose of "splitting" one application across multiple machines:
i. Using drwrap_replace, I override all calls to `malloc`, `mmap`, and any other memory allocation/freeing function. I use my own fake allocators which give out fake addresses.
ii. Using drwrap_wrap, I wrap all calls to `pthread_create` and the like, so I can decide whether I'd send them to another machine or let them through.
iii. Using BB instrumentation, I modify every memory access so I can translate the fake addresses from before to my real memory address. So far it has been successful, but it breaks inside pthread_create as I mentioned earlier.
4. Inside `pthread_create` I get one of these calls to `mmap` for memory allocation, which I also override.
Here's my theory, based on all the above plus your input:
Inside the `pthread_create` call, I got a call to `mmap`, which I have of course overridden. If the syscall (or DR, after taking over the syscall) is trying to change the memory protection for the fake memory address returned by my mmap, it will definitely fail. If that's the case, then I also need to drwrap_replace whatever function that calls this syscall (mprotect?).