How to store the address of C struct on Go side in an array (cgo)

[Nitish Saboo]

Hi,

I have the following function 'LoadPatternDB' where I am allocating memory for a C struct.My C struct contains the callback function.

1)I cannot deference the struct in the same func 'LoadPatternDB' because I get the following error when my callback function is called from C code:

signal SIGSEGV: segmentation violation code=0x1 addr=0x7f1e00415341 pc=0x7f1e00415341]

2)The same code is running in multiple go routines.And because of that if I don't deallocate the memory I get the following error:

*** Error in `fatal error: bin/main': double free or corruption (fasttop)unexpected signal during runtime execution: 0x00007fd2b8588060 ***

My question is:

How can I store the address of the C struct (when memory is allocated using calloc) on Go side in an array so that I can deallocate the memory using 'defer C.free(unsafe.Pointer(InitStruct))' when the callback is done.

'InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))'  >>>> I want to store this address on Go side.

Can some please guide me here ?

syslogparser.go

=============

func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {

patterndbpath := C.CString(opts.Patterndb)

defer C.free(unsafe.Pointer(patterndbpath))

InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))

//defer C.free(unsafe.Pointer(InitStruct)) . <<<<<<<<<<<<<<<<<<<<<<I cannot do this here

InitStruct.callback = (C.key_value_cb)(C.callback)

InitStruct.data = C.int(workerId)

C.load_pattern_db(patterndbpath, (*C.Accumulatedparams)(unsafe.Pointer(InitStruct)), C.int(workerId))

}

//export Result

func Result(key *C.char, value *C.char, value_len C.size_t, mapid C.int) {

fmt.Println("Map Id: " + strconv.Itoa(int(mapid)))

value_field := C.GoStringN(value, C.int(value_len))

key_field := C.GoString(key)

remap, ok := constants.FIELD_MAPPINGS[key_field]

if ok {

Check.result[int(mapid)][remap] = value_field

//check.result[remap] = value_field

} else {

//check.result[key_field] = value_field

Check.result[int(mapid)][key_field] = value_field

}

}

cfuncs.go

-========

package lib

/*

#include <stdio.h>

// The gateway function

void callback(char *key, char *value, size_t value_len, int mapid)

{

//printf("C.callOnMeGo_cgo(): called");

void Result(const char *key, const char *value, size_t value_len, int mapid);

Result(key, value, value_len, mapid);

}

*/

import "C"

syslog-node.h

============

#ifndef TEST_H_INCLUDED

#define TEST_H_INCLUDED

#include <stdlib.h>

typedef void (*key_value_cb)(const char* key, const char* value, size_t value_len, int data);

typedef struct Accumulatedparams{

    key_value_cb callback;

    int data;

}Accumulatedparams;

int initialize_engine(const char* filename, const char* module_path);

int reload_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);

int load_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);

#endif

[Ian Lance Taylor]

I don't see anything obviously wrong in your Go code, but it's
impossible to answer your question without knowing what the C code
does. What you describe is consistent with C code that expects to
retain the memory passed to load_pattern_db, or alternatively is also
consistent with C code that does not expect to be called concurrently
from multiple threads of execution.

Ian

[Nitish Saboo]

Hi Ian,

Following is my C Code:

PatternDB *patterndb[WORKERIDS];

int load_pattern_db(const gchar* filename, key_value_cb cb, int workerId)
{
if(patterndb[workerId] != NULL){
pattern_db_free(patterndb[workerId]);
}
patterndb[workerId] = pattern_db_new();
pattern_db_reload_ruleset(patterndb[workerId], configuration, filename);
pattern_db_set_emit_func(patterndb[workerId], pdbtool_pdb_emit_accumulate, cb);
return 0;
}

This is just the C wrapper that internally call syslog-ng methods.

Since I am using an array of patterndbs, I am calling the C code with multiple Go routines.

Single iteration of the code is working fine.

The issue is seen only with multiple go routines. Can we store the address of the C struct (when memory is allocated using calloc) on Go side in an array so that I can deallocate the memory using 'defer C.free(unsafe.Pointer(InitStruct))' when the callback is done?This should make multiple go routines to work...Am I right ?

Thanks,

Nitish

[Ian Lance Taylor]

On Thu, Jul 11, 2019 at 9:27 AM Nitish Saboo <nitish....@gmail.com> wrote:
>
> Following is my C Code:
>
> PatternDB *patterndb[WORKERIDS];
>
> int load_pattern_db(const gchar* filename, key_value_cb cb, int workerId)
> {
> if(patterndb[workerId] != NULL){
> pattern_db_free(patterndb[workerId]);
> }
> patterndb[workerId] = pattern_db_new();
> pattern_db_reload_ruleset(patterndb[workerId], configuration, filename);
> pattern_db_set_emit_func(patterndb[workerId], pdbtool_pdb_emit_accumulate, cb);
> return 0;
> }
>
> This is just the C wrapper that internally call syslog-ng methods.
>
> Since I am using an array of patterndbs, I am calling the C code with multiple Go routines.
>
> Single iteration of the code is working fine.
>
> The issue is seen only with multiple go routines. Can we store the address of the C struct (when memory is allocated using calloc) on Go side in an array so that I can deallocate the memory using 'defer C.free(unsafe.Pointer(InitStruct))' when the callback is done?This should make multiple go routines to work...Am I right ?

I don't know what this code does, but the line

pattern_db_set_emit_func(patterndb[workerId], pdbtool_pdb_emit_accumulate, cb);

suggests that cb is being saved away for future use. If that is
indeed the case then freeing the memory in Go will cause a dangling
pointer, leading to crashes as you see.

You should also double-check that each separate goroutine is using a
different "workerId", as duplication of workerId values could
certainly cause trouble when running concurrently.

Yes, it's fine to store the address of the C struct on the Go side, if
that helps.

Ian

[Nitish Saboo]

pattern_db_set_emit_func(patterndb[workerId], pdbtool_pdb_emit_accumulate, cb);
suggests that cb is being saved away for future use.

>>>>yes cb is being saved away and is being used for future use for callbacks to Go code.

Yes, it's fine to store the address of the C struct on the Go side, if
that helps.

syslog-node.h
 ============

 #ifndef TEST_H_INCLUDED
 #define TEST_H_INCLUDED

 #include <stdlib.h>
 typedef void (*key_value_cb)(const char* key, const char* value, size_t value_len, int data);
 typedef struct Accumulatedparams{
     key_value_cb callback;
    int data;
 }Accumulatedparams;
 int initialize_engine(const char* filename, const char* module_path);
 int reload_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);
int load_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);

>>>>>If I declare the array in the following manner:

result := make([]*C.struct_Accumulatedparams)

store[0] = (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))

Here also the code gets compiled successfully, just that the GoLand IDE shows store[0].callback and store[0].data in red color saying Unresolved reference 'callback' and Unresolved reference 'data' respectively.Not sure how the code compiles if the IDE shows it in red color. Binary(./main) works fine

Though this works,

store[0] = (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))
InitStruct := store[0]
InitStruct.callback = (C.key_value_cb)(C.callOnMeGo_cgo)
InitStruct.data = C.int(1)

Here the code compiles successfully and there is no complain from GoLand IDE as well.Binary (./main) too works fine

So what i thought to do :

type Store struct{
result []*C.struct_Accumulatedparams
data []C.int
callback []C.key_value_cb
}

var store = Store{result: make([]*C.struct_Accumulatedparams, 2), data: make([]C.int ,2), callback:make([]C.key_value_cb, 2)}

func (obj Syslogparser) LoadPatternDB(opts Syslog) {

patterndb_path := C.CString(opts.patterndb)
store.result[0] = (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams)) <<<<<< 
store.callback[0] = (C.key_value_cb)(C.callOnMeGo_cgo) <<<<< 
store.data[0] = C.int(1)
C.load_pattern_db(patterndb_path, (*C.Accumulatedparams)(unsafe.Pointer(store.result[0])))

}

Here the code gets compiled successfully. But I am getting the following  error when I run the binary './main'

1In reload PatternDb: 
PatternDb :/home/nsaboo/Documents/goworkspace/src/poc/reload.xml
ModulePath :usr/local/lib/syslog-ng
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw(0x4fa203, 0x2a)
/usr/local/go/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:374 +0x4a9

goroutine 1 [syscall]:
runtime.cgocall(0x4b8fb0, 0xc000042cc8, 0x4d9280)
/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000042c98 sp=0xc000042c60 pc=0x404f6b
main._Cfunc_execute_percustomer_db(0x7fb0fb5a5fa0, 0x1d5, 0x7fb0fb8259b0, 0x8)
_cgo_gotypes.go:121 +0x45 fp=0xc000042cc8 sp=0xc000042c98 pc=0x4b5ca5
main.Syslogparser.ParseMessagePerCustomer(0xc0000a2000, 0x36, 0x4f6b0c, 0x18, 0x4f3fc4, 0x8, 0xc0000d0000, 0x1d5)
/home/nsaboo/Documents/goworkspace/src/poc/main.go:132 +0x109 fp=0xc000042d38 sp=0xc000042cc8 pc=0x4b6f09
main.perCustomerParsing(...)
/home/nsaboo/Documents/goworkspace/src/poc/main.go:249
main.main()
/home/nsaboo/Documents/goworkspace/src/poc/main.go:232 +0x6e1 fp=0xc000042f98 sp=0xc000042d38 pc=0x4b78e1
runtime.main()
/usr/local/go/src/runtime/proc.go:200 +0x20c fp=0xc000042fe0 sp=0xc000042f98 pc=0x42cd9c
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x4548c1
}

Where am i going wrong here ?

Thanks,

Nitish

[Ian Lance Taylor]

On Thu, Jul 11, 2019 at 10:58 AM Nitish Saboo <nitish....@gmail.com> wrote:
>
> pattern_db_set_emit_func(patterndb[workerId], pdbtool_pdb_emit_accumulate, cb);
> suggests that cb is being saved away for future use.
>
> >>>>yes cb is being saved away and is being used for future use for callbacks to Go code.

Then clearly the Go code must not immediately free the C memory. That
is no different from how C code would work.

> syslog-node.h
> ============
>
> #ifndef TEST_H_INCLUDED
> #define TEST_H_INCLUDED
>
> #include <stdlib.h>
> typedef void (*key_value_cb)(const char* key, const char* value, size_t value_len, int data);
> typedef struct Accumulatedparams{
> key_value_cb callback;
> int data;
> }Accumulatedparams;
> int initialize_engine(const char* filename, const char* module_path);
> int reload_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);
> int load_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);
>
> >>>>>If I declare the array in the following manner:
>
> result := make([]*C.struct_Accumulatedparams)
> store[0] = (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))
>
> Here also the code gets compiled successfully, just that the GoLand IDE shows store[0].callback and store[0].data in red color saying Unresolved reference 'callback' and Unresolved reference 'data' respectively.Not sure how the code compiles if the IDE shows it in red color. Binary(./main) works fine

I don't know what the GoLand IDE does. I don't see any
store[0].callback or store[0].data in that code snippet.

Somewhere your C code is getting a NULL pointer dereference.

I don't see much in these questions that seems specific to Go.

Ian

[Nitish Saboo]

This is the code snippet I was talking about:

store := make([]*C.struct_Accumulatedparams, 2)

func (obj Syslogparser) LoadPatternDB(opts Syslog) {

>

> patterndb_path := C.CString(opts.patterndb)

> store[0] = (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams)) 

> store[0].callback = (C.key_value_cb)(C.callOnMeGo_cgo) 

> store[0].data = C.int(1)

> C.load_pattern_db(patterndb_path, (*C.Accumulatedparams)(unsafe.Pointer(store[0])))

> }

On Fri, 12 Jul 2019, 00:22 Nitish Saboo, <nitish....@gmail.com> wrote:

Hi Ian,

Can you please guide me with the correct way  to store the address of a C struct in a Go array so that I can use it later for deallocating the memory?

Thanks,

Nitish

[Nitish Saboo]

Hi,

I have the following function 'LoadPatternDB' where I am allocating memory for a C struct.My C struct contains the callback function.

I have the function 'LoadPatternDB' that  is running in multiple go routines.I have written a C wrapper that internally uses syslog-ng methods.

While running the method 'LoadPatternDB' in syslogparser.go with multiple go  routines, I get the following error.

*** Error in `fatal error: bin/main': double free or corruption (fasttop)unexpected signal during runtime execution: 0x00007fd2b8588060 ***

My queries are:

1)When do we get this error ?

2)Is it because of not deallocating the struct memory, I am getting this error ?But at the same time I am allocating separate memories for each of the go routines.

3)Is it because of the memory crunch on the system that can lead to this error ?

4)The issues is seen with only multiple go routines.Single iteration of the code is working fine.

5)Can I not call Go code and C code using multiple threads ?

Can some one please guide me here ?

syslog-node.c

===========

PatternDB *patterndb[WORKERIDS];

int load_pattern_db(const gchar* filename, key_value_cb cb, int workerId)
{
if(patterndb[workerId] != NULL){
pattern_db_free(patterndb[workerId]);
}
patterndb[workerId] = pattern_db_new();
pattern_db_reload_ruleset(patterndb[workerId], configuration, filename);

pattern_db_set_emit_func(patterndb[workerId], pdbtool_pdb_emit_accumulate, cb);

return 0;
}

gboolean pdbtool_accumulate_fields(NVHandle handle, const gchar *name, const gchar *value, gssize length, gpointer user_data)

{
struct Accumulatedparams *params = user_data;
  key_value_cb cb = params->callback;
  cb(name, value, length, params->data);
  return FALSE;
}

void pdbtool_pdb_emit_accumulate(LogMessage *msg, gboolean synthetic, gpointer user_data)
{
nv_table_foreach(msg->payload, logmsg_registry, pdbtool_accumulate_fields, user_data);
}

syslogparser.go

=============

func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {

patterndbpath := C.CString(opts.Patterndb)

defer C.free(unsafe.Pointer(patterndbpath))

InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))

syslog-node.h

============

#ifndef TEST_H_INCLUDED

#define TEST_H_INCLUDED

#include <stdlib.h>

typedef void (*key_value_cb)(const char* key, const char* value, size_t value_len, int data);

typedef struct Accumulatedparams{

    key_value_cb callback;

    int data;

}Accumulatedparams;

int load_pattern_db(const char* filename, Accumulatedparams *cb, int workerId);

#endif

[Ian Lance Taylor]

On Thu, Jul 11, 2019 at 8:45 PM Nitish Saboo <nitish....@gmail.com> wrote:
>
> I have the following function 'LoadPatternDB' where I am allocating memory for a C struct.My C struct contains the callback function.
> I have the function 'LoadPatternDB' that is running in multiple go routines.I have written a C wrapper that internally uses syslog-ng methods.
> While running the method 'LoadPatternDB' in syslogparser.go with multiple go routines, I get the following error.
>
> *** Error in `fatal error: bin/main': double free or corruption (fasttop)unexpected signal during runtime execution: 0x00007fd2b8588060 ***
>
> My queries are:
>
> 1)When do we get this error ?

That error is from the C memory allocator. It does not come from Go.
It means that you called a C function such as malloc, calloc, or free,
and the call detected memory corruption. It may be calling free twice
on the same pointer, or it may be using a dangling pointer, or it may
be some other memory corruption. We don't know.

> 2)Is it because of not deallocating the struct memory, I am getting this error ?But at the same time I am allocating separate memories for each of the go routines.

No.

> 3)Is it because of the memory crunch on the system that can lead to this error ?

Unlikely. It is conceivable that malloc or calloc is returning NULL,
and you are not checking for NULL, and that is leading to memory
corruption. But that is unlikely to happen, and it is unlikely to
cause this kind of error.

> 4)The issues is seen with only multiple go routines.Single iteration of the code is working fine.

That sounds like a good place to start looking.

> 5)Can I not call Go code and C code using multiple threads ?

You can call C code from multiple goroutines, but of course the C code
has to be prepared to handle it. It's no different from a C program
that calls the C function from multiple threads.

If you show us a small, complete, self-contained example, that does
not call any other code, then we may be able to see the problem.

Ian

[Nitish Saboo]

Hi Ian,

syslogparser.go

=============

func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {

patterndbpath := C.CString(opts.Patterndb) .  <<<<<<<<<<<<<<< . STEP 1

defer C.free(unsafe.Pointer(patterndbpath)) .  <<<<<<<<<<<<<<<< STEP 2

InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))

InitStruct.callback = (C.key_value_cb)(C.callback)

InitStruct.data = C.int(workerId)

C.load_pattern_db(patterndbpath, (*C.Accumulatedparams)(unsafe.Pointer(InitStruct)), C.int(workerId))

}

Since I am running this program in multiple go routines, when we do above STEP 1 :

1)Will the C memory allocate different addresses for  'patterndbpath' in both the routines ...Am I right ?

2)You said 'It may be calling free twice on the same pointer', but I am assigning a different pointer in STEP 1 everytime for each routine...Am I right ?

3)Is it possible that one Go routine has already freed up the address that the other Go routine is trying to free  and that is causing this 'fatal error: bin/main': double free or corruption (fasttop)'

Is this because of the race condition between two routines that this error is seen ?

4)If you show us a small, complete, self-contained example, that does not call any other code, then we may be able to see the problem.

>> As you have mentioned problem is happening when we call 'C.free'.Do you mean by not calling internal syslog-ng methods and just calling a normal C program with multiple go routines ?

Please correct me here if I am wrong

Thanks,

Nitish

[Ian Lance Taylor]

On Thu, Jul 11, 2019 at 11:41 PM Nitish Saboo <nitish....@gmail.com> wrote:
>
> syslogparser.go
> =============
>
> func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {
> patterndbpath := C.CString(opts.Patterndb) . <<<<<<<<<<<<<<< . STEP 1
> defer C.free(unsafe.Pointer(patterndbpath)) . <<<<<<<<<<<<<<<< STEP 2
> InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))
> InitStruct.callback = (C.key_value_cb)(C.callback)
> InitStruct.data = C.int(workerId)
> C.load_pattern_db(patterndbpath, (*C.Accumulatedparams)(unsafe.Pointer(InitStruct)), C.int(workerId))
> }
>
> Since I am running this program in multiple go routines, when we do above STEP 1 :
>
> 1)Will the C memory allocate different addresses for 'patterndbpath' in both the routines ...Am I right ?

Yes.

> 2)You said 'It may be calling free twice on the same pointer', but I am assigning a different pointer in STEP 1 everytime for each routine...Am I right ?

Yes, you will see a different pointer each time you call C.calloc, and
in the code that I see that pointer is only freed once, but I have no
idea what the rest of your code is doing.

> 3)Is it possible that one Go routine has already freed up the address that the other Go routine is trying to free and that is causing this 'fatal error: bin/main': double free or corruption (fasttop)'

Yes, that is possible. It won't happen with the code I see above but
I have no idea what the rest of your code is doing.

> Is this because of the race condition between two routines that this error is seen ?

It is possible.

> 4)If you show us a small, complete, self-contained example, that does not call any other code, then we may be able to see the problem.
> >> As you have mentioned problem is happening when we call 'C.free'.Do you mean by not calling internal syslog-ng methods and just calling a normal C program with multiple go routines ?
> Please correct me here if I am wrong

I mean any small, self-contained example where we can see all the code.

You seem to be looking at a memory corruption or double free problem.
These are normal, ordinary, problems when writing C code. We can't
debug your problem for you.

Ian

[Nitish Saboo]

Hi Ian,

> syslogparser.go
> =============
>
> func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {

   log.Info("Loading pattern db with path:", opts.Patterndb)

> patterndbpath := C.CString(opts.Patterndb) .  <<<<<<<<<<<<<<< . STEP 1
> defer C.free(unsafe.Pointer(patterndbpath)) .  <<<<<<<<<<<<<<<< STEP 2
> InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))
> InitStruct.callback = (C.key_value_cb)(C.callback)
> InitStruct.data = C.int(workerId)
> C.load_pattern_db(patterndbpath, (*C.Accumulatedparams)(unsafe.Pointer(InitStruct)), C.int(workerId))
> }
>
> Since I am running this program in multiple go routines, when we do above STEP 1 :
>
> 1)Will the C memory allocate different addresses for  'patterndbpath' in both the routines ...Am I right ?

Yes.

> 2)You said 'It may be calling free twice on the same pointer', but I am assigning a different pointer in STEP 1 everytime for each routine...Am I right ?

Yes, you will see a different pointer each time you call C.calloc, and
in the code that I see that pointer is only freed once, but I have no
idea what the rest of your code is doing.

I am calling free only once  and that is for 'patterndbpath' in this particular method.In the entire codebase I am NOT calling free for InitStruct (Memory allocated for the struct)

I am pointing out this particular method 'LoadPatternDB' of Go code base  because I had put a 'Info' log in this method at the starting and that got printed before the corruption error came up.

And my testing  failed at the third iteration:

1)WorkerID0 completed the  work

2)WorkerID1 completed the work

3)WorkerID1 failed in this particular method with 'fatal error: bin/main': double free or corruption (fasttop)'

Following is one of the methods called before this method :

func (obj SyslogParser) InitializeEngine() {
patterndb := C.CString(obj.Patterndb)
defer C.free(unsafe.Pointer(patterndb))
modulepath := C.CString(obj.Modulepath)
defer C.free(unsafe.Pointer(modulepath))
C.initialize_engine(patterndb, modulepath)

}

Not deallocating the memory for the struct can be  the issue for  'fatal error: bin/main': double free or corruption (fasttop)' ?

> 3)Is it possible that one Go routine has already freed up the address that the other Go routine is trying to free  and that is causing this 'fatal error: bin/main': double free or corruption (fasttop)'

Yes, that is possible.  It won't happen with the code I see above but
I have no idea what the rest of your code is doing.

How can it not happen with this code ?I did not understand this point.

Can you please clarify this point ?

> Is this because of the race condition between two routines that this error is seen ?

It is possible.

> 4)If you show us a small, complete, self-contained example, that does not call any other code, then we may be able to see the problem.
> >> As you have mentioned problem is happening when we call 'C.free'.Do you mean by not calling internal syslog-ng methods and just calling a normal C program with multiple go routines ?
> Please correct me here if I am wrong

I mean any small, self-contained example where we can see all the code.

>>>I will try to send a very small, basic code but not sure how to reproduce the issue with that.

You seem to be looking at a memory corruption or double free problem.
These are normal, ordinary, problems when writing C code.  We can't
debug your problem for you.

[Michael Jones]

It is really impossible to help you debug what is to the thousands of readers here an invisible unseen program. You have a logic error. You don't know where it is (of course, you are busy trying to find it) but we can't know either as we don't see the code. General advice, as always, is to "trap" the bug in the smallest stand-alone code that you can. That will focus your search.

You just shared this:

Following is one of the methods called before this method :

func (obj SyslogParser) InitializeEngine() {
patterndb := C.CString(obj.Patterndb)
defer C.free(unsafe.Pointer(patterndb))
modulepath := C.CString(obj.Modulepath)
defer C.free(unsafe.Pointer(modulepath))
C.initialize_engine(patterndb, modulepath)

}

Not deallocating the memory for the struct can be  the issue for  'fatal error: bin/main': double free or corruption (fasttop)' ?

Is this right? Is it correct to call C.initialize_engine(a,b) then immediately call C.free(a) and C.free(b)? It might be fine if your initialize engine clones/duplicates/or otherwise uses a & b just once. But if not, then of course A & B will both instantly become death traps for your C code during runtime of the engine, for any allocations that are where A & B were located, and potentially when your "engine" is deallocated later on. 

Just one example of might be fine, might be disaster, no way for us to tell. It seems that you have a logic problem in the program, not a C-to-Go or Go-to-C interfacing problem. You're going to need to study your code (unseen by us) yourself to see what that is. I think so anyway. Maybe Ian is so smart that he can debug this for you by ESP. ;-)

Michael

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CALjMrq6eM1_ykJYmNVYqnFnjxqRvRCJF8Bqgh6k2N7kBV%2B_3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael T. Jones
michae...@gmail.com

[Nitish Saboo]

Hi Michael,

Thanks for pointing it out.I am a beginner with Go and  CGO and have never worked on corruption issues with C.

I am familiar with python and Java.So just seeking help from the experts to find the pain points that should be looked into.

Hence, asking questions to Ian as in where all I should look for the corruption issues.

Thanks,

Nitish

[Ian Lance Taylor]

On Fri, Jul 12, 2019 at 10:53 AM Nitish Saboo <nitish....@gmail.com> wrote:
>
> Thanks for pointing it out.I am a beginner with Go and CGO and have never worked on corruption issues with C.
> I am familiar with python and Java.So just seeking help from the experts to find the pain points that should be looked into.
> Hence, asking questions to Ian as in where all I should look for the corruption issues.

From my perspective, it's kind of like you have a broken clock, and
you are showing us a photo of the clock face, and asking how to fix
the clock. We don't know. The clock face looks OK. The problem is
most likely inside the clock, but we can't see that. Only you can see
that.

Ian

[Kurtis Rader]

On Fri, Jul 12, 2019 at 10:53 AM Nitish Saboo <nitish....@gmail.com> wrote:

Thanks for pointing it out.I am a beginner with Go and  CGO and have never worked on corruption issues with C.

I am familiar with python and Java.So just seeking help from the experts to find the pain points that should be looked into.

Hence, asking questions to Ian as in where all I should look for the corruption issues.

Nitish, I would start by taking the error message at face value. That is, assume you're freeing a dynamically allocated buffer more than once. Then review your code with that assumption in mind to see how that might occur.

There are also tools that are invaluable when debugging this type of issue. Starting with code linters such as cppcheck, oclint, and enabling all warnings your compiler supports and paying attention to them. Next, take advantage of your platform debug malloc library which is enabled by setting environment variables such as "MallocGuardEdges" (the actual env vars is specific to your platform so "man 3 malloc"). There are more sophisticated tool such as Valgrind and ASAN (address sanitizer). ASAN is an incredibly helpful tool for this type of issue and I always use it if possible for debugging memory management bugs in C/C++ code. Unfortunately I don't know how to enable its use with cgo or even if it is possible.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Nitish Saboo]

Hi Ian,

Apologies for not being very descriptive with the issue.I will try to be more clear from next time.

Can you please clarify the following that you had mentioned in the previous conversation of this email:

> 3)Is it possible that one Go routine has already freed up the address that the other Go routine is trying to free  and that is causing this 'fatal error: bin/main': double free or corruption (fasttop)'

Yes, that is possible.  It won't happen with the code I see above but
I have no idea what the rest of your code is doing.

How can it not happen with this particular code mentioned ?I did not understand this point.

Can you please clarify this point ?

Thans,

Nitish

[fge...@gmail.com]

> --

Hi Nitish,
you won't be able to get help, unless you show the code with the
reported behavior.

Please work on creating a small (<100 lines) self-contained,
executable reproduction case. That's most probably a significant
amount of work.
If this type of error happened to anybody else, he or she would have
to do this work him or herself as well. To be clear: that work must be
done. Without that there is no step 2 or better description or any
help.

The good news is that during that work, you might find the cause of
the error and even if not, you'll learn a lot about the environment
you are trying to create a program in.

When you are ready please share the result. Also please note that for
sharing code, links to gist.github.com or play.golang.org are
preferred.

Code snippets are not enough. Better descriptions are not enough.

Please don't ignore Ian's clock analogy, it's very good.

cheers

[Ian Lance Taylor]

On Sat, Jul 13, 2019, 1:20 AM Nitish Saboo <nitish....@gmail.com> wrote:

Hi Ian,

Apologies for not being very descriptive with the issue.I will try to be more clear from next time.

Can you please clarify the following that you had mentioned in the previous conversation of this email:

> 3)Is it possible that one Go routine has already freed up the address that the other Go routine is trying to free  and that is causing this 'fatal error: bin/main': double free or corruption (fasttop)'

Yes, that is possible.  It won't happen with the code I see above but
I have no idea what the rest of your code is doing.

How can it not happen with this particular code mentioned ?I did not understand this point.

Can you please clarify this point ?

In the code you showed us, there is one calloc call producing a pointer and one free call releasing that pointer.  So in that code there is no way for one goroutine to free code allocated by another goroutine.  I'm not sure how else to say it.  Perhaps you could clarify the code path you are concerned about.

Iam

[Nitish Saboo]

HI Ian,

func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {

   log.Info("Loading pattern db with path:", opts.Patterndb)
> patterndbpath := C.CString(opts.Patterndb) .  <<<<<<<<<<<<<<< . STEP 1
> defer C.free(unsafe.Pointer(patterndbpath)) .  <<<<<<<<<<<<<<<< STEP 2
> InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))
> InitStruct.callback = (C.key_value_cb)(C.callback)
> InitStruct.data = C.int(workerId)
> C.load_pattern_db(patterndbpath, (*C.Accumulatedparams)(unsafe.Pointer(InitStruct)), C.int(workerId))
> }
>

In my code  there is one calloc call producing a pointer but I am not freeing up calloc pointer.I am freeing up 'patterndbpath' pointer.

Thanks,

Nitish

[Nitish Saboo]

Hi,

Apologies to the experts who have been kind enough to help me on this issue.

I will try to come up with a small, complete reproducible test case for this issue.

Thanks,

Nitish

[Ian Lance Taylor]

On Sun, Jul 14, 2019 at 6:06 AM Nitish Saboo <nitish....@gmail.com> wrote:
>
> func (obj SyslogParser) LoadPatternDB(opts Syslog, workerId int) {
> log.Info("Loading pattern db with path:", opts.Patterndb)
> > patterndbpath := C.CString(opts.Patterndb) . <<<<<<<<<<<<<<< . STEP 1
> > defer C.free(unsafe.Pointer(patterndbpath)) . <<<<<<<<<<<<<<<< STEP 2
> > InitStruct := (*C.Accumulatedparams)(C.calloc(1, C.sizeof_struct_Accumulatedparams))
> > InitStruct.callback = (C.key_value_cb)(C.callback)
> > InitStruct.data = C.int(workerId)
> > C.load_pattern_db(patterndbpath, (*C.Accumulatedparams)(unsafe.Pointer(InitStruct)), C.int(workerId))
> > }
> >
>
> In my code there is one calloc call producing a pointer but I am not freeing up calloc pointer.I am freeing up 'patterndbpath' pointer.

Ah, my apologies.

Ian