mixing onion & libgc....

[Basile Starynkevitch]

Hello all,

I'm using libonion on Linux/Debian/AMD64
in https://github.com/bstarynk/monimelt

monimelt is multi-threaded and using Boehm GC http://hboehm.info/gc

It would be nice if Onion becomes more Boehm GC friendly.

This mostly would mean two things.

First, provide a way, very early at Onion initialization time,
to give some custom allocation routines. In principle,
that should be easy: define functions onion_malloc, onion_free, ....
and use them everywhere inside onion. These functions would call
by default the standard malloc & free (etc...) or else a previously
registered malloc-like function. As a bonus, it would be great
if Onion distinguished the malloc which allocate a memory zone
containing pointers (for GC_MALLOC) with the malloc allocating
only a memory zone without pointers (like character buffers, for GC_MALLOC_ATOMIC).

Second, provide some wrapping around thread creation,
perhaps by providing some hookable onion_thread_create or
giving the ability to use GC_register_my_thread & GC_unregister_my_thread &
GC_call_with_stack_base. Notice that
Boehm's GC file gc/gc.h is #define pthread_create to make things easier.


Any clues or advices for that?

Regards!
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mines, sont seulement les miennes} ***

[Basile Starynkevitch]

On Monday, May 12, 2014 4:23:27 PM UTC+2, Basile Starynkevitch wrote:

Hello all,

I'm using libonion on Linux/Debian/AMD64
in https://github.com/bstarynk/monimelt

monimelt is multi-threaded and using Boehm GC http://hboehm.info/gc

The attached patch is not bug proof and is not even fully compiling, but is it towards the right direction?
 

[David Moreno Montero]

Hi Basile,

I'm thinking on if and how this should be merged, but the Heartbleed bug keeps resonating in my head, and the cause was custom allocation. Jsut a coupleof questions to wrap my head up

* Onion it self takes big care about not leaking resources, isnt it possible to use gc managed code with non-managed code? Maybe it needs some kind of hook to know that to free an onion object it needs to call onion_free, but I guess this happens anyway.

If it really need to be bohem friendly:

* I seem to miss some files, maybe some header (specifially: low_prim.h).

* If there is a new memory subsystem, all functions about memory should be onion_memory_* (onion_memory_alloc, onion_memory_calloc, onion_memory_free ...), or similar (onion_mem_* sounds good too).

* Also for performance reasons, does it make sense to make this memory functions inlined in the header? I guess that normally is just to call another función (normal malloc or free, on non-gc version).

* Finally it would be great to check also how can it be made compatible with other allocationschemes, as talloc (http://talloc.samba.org/talloc/doc/html/index.html) or halloc (http://swapped.cc/#!/halloc).But no need for the first versión

Finally, I really love github pull requests, as I can just clone them, test them, the great TravisCI takes care to check it compiles... Do you mind create a bohem branch in a onion github fork?

--
You received this message because you are subscribed to the Google Groups "onion-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to onion-dev+...@coralbits.com.
To post to this group, send email to onio...@coralbits.com.
Visit this group at http://groups.google.com/a/coralbits.com/group/onion-dev/.
For more options, visit https://groups.google.com/a/coralbits.com/d/optout.

--

David Moreno Montero 

dmo...@coralbits.com
+34 658 18 77 17

http://www.coralbits.com

[Basile Starynkevitch]

On Tue, May 13, 2014 at 09:45:51AM +0200, David Moreno Montero wrote:
> Hi Basile,
>
> I'm thinking on if and how this should be merged,

I forked onion on github. Will commit code in a few hours.
It is a bit cleaner than what I have sent previously.

> but the Heartbleed bug
> keeps resonating in my head, and the cause was custom allocation.

To that, I have a simple answer. The custom allocator is entirely optional.
By default, onion will continue to use malloc etc....


Also, a lot of code in current onion is not friendly w.r.t. malloc failure.
For example (amongst many others) in src/onion/onion.c line 461

server->listen_points=malloc(sizeof(onion_listen_point*)*2);
server->listen_points[0]=protocol;
server->listen_points[1]=NULL;

If the malloc is failing and returning NULL that code will surely crash or exhibit the
dreaded undefined behavior....

> Just a coupleof questions to wrap my head up

>
> * Onion it self takes big care about not leaking resources, isnt it
> possible to use gc managed code with non-managed code? Maybe it needs some
> kind of hook to know that to free an onion object it needs to call
> onion_free, but I guess this happens anyway.

You can use GC_FREE on a pointer returned by a call
to GC_MALLOC or GC_MALLOC_ATOMIC or GC_CALLOC or GC_STRDUP or GC_REALLOC

>
> If it really need to be bohem friendly:

Typo, Boehm, not bohem... (Hans Boehm is a nice person that I did met once, so use capitals ....)

>
> * I seem to miss some files, maybe some header (specifially: low_prim.h).
> * If there is a new memory subsystem, all functions about memory should be
> onion_memory_* (onion_memory_alloc, onion_memory_calloc, onion_memory_free
> ...), or similar (onion_mem_* sounds good too).

I'm calling them onionlow_ because Boehm GC also needs to wrap pthread_create
(with <gc/gc.h> it is #define-d to call GC_pthread_create....), so I'll
have to provide a onionlow_pthread_create etc....

> * Also for performance reasons, does it make sense to make this memory
> functions inlined in the header? I guess that normally is just to call
> another función (normal malloc or free, on non-gc version).

I did inline them yesterday, but today I believe that I should not.
First, malloc is not that quick so that inlining matters. Most
importantly, inlining would require access a global data directly (holding
the user provided routine pointers), and that is not friendly
with shared libraries on Linux/x86 (lots of relocation happens).
See www.akkadia.org/drepper/dsohowto.pdf (Drepper's paper:
how to write shared libraries) for explanations.

> * Finally it would be great to check also how can it be made compatible
> with other allocationschemes, as talloc (
> http://talloc.samba.org/talloc/doc/html/index.html) or halloc (
> http://swapped.cc/#!/halloc).But no need for the first versión

I forked onion on http://github.com/bstarynk/onion and I will commit code soon,
in a few hours (and probably less)... In the mean time, I am attaching my latest
low_util.h with comments. Please give feedback quickly if that does not fit you.


I also have a great favor to ask. I know almost nothing about cmake.
Could you document more (perhaps on a wiki, on even in the README.rst file)
how to build a debugging variant of libonion (i.e. compiled with
gcc -O0 -g3 -Wall -Wextra), with ONION_DEBUG0 working!

Regards.

[David Moreno Montero]

Just a quick debugging tutorial:

Create the Makefiles with cmake -DCMAKE_BUILD_TYPE=Debug. This will output all ONION_DEBUG messages. To output the ONION_DEBUG0 messages, export the ONION_DEBUG0=file.c environment variable, for example:

export ONION_DEBUG0=request.c

About the malloc checks, please if you see more of there, fix them too. With coverity I'm fixing all these, but I wish I had more time. I'm still working on the onion_free with pending websockets, but it will take more than I thought.

About the inlining. Makes sense. Anyway I would like, maybe not for the first version, to benchmark it so we know if there is a sensible difference, or really none at all.

About the name, maybe just call it os (onion_os_alloc...) instead of low_prim. Somehow I dont like too much the name.

Regards,

[Basile Starynkevitch]

On Tue, May 13, 2014 at 01:01:59PM +0200, Basile Starynkevitch wrote:
> On Tue, May 13, 2014 at 09:45:51AM +0200, David Moreno Montero wrote:
> > Hi Basile,
> >
> > I'm thinking on if and how this should be merged,
>
> I forked onion on github. Will commit code in a few hours.
> It is a bit cleaner than what I have sent previously.

Just committed on https://github.com/bstarynk/onion but it compiles but does not link
(low_util.o is not linked in but low_util.c is compiled).
As I said, I am a cmake newbie. I am stuck. Please help!

FWIW, I'm doing the usual:

mkdir build
cd build
cmake ..
make

This is all I know about cmake.

David or others, could you explain me how to add a new file into the build.
I edited src/onion/CMakeLists.txt to add low_util.c but apparently that is not enough.

Please help!

Cheers.

[Basile Starynkevitch]

On Tuesday, May 13, 2014 1:20:20 PM UTC+2, David Moreno Montero wrote:

Just a quick debugging tutorial:

Create the Makefiles with cmake -DCMAKE_BUILD_TYPE=Debug.

I commit a patch (on my branch  https://github.com/bstarynk/onion to README.rst explaining that.

But this does not solve my issue of getting my new file src/onion/low_util.c compiled (it is compiled) and linked (it does not seems to be linked).

About the malloc checks, please if you see more of there, fix them too.

There are lots of them. Just grep for malloc inside your code. In many occurrences,   it is not checked.

And my branch (which I hope you'll merge back into the onion trunk when it is working...) is avoiding that. Calls to onionlow_malloc will abort after an ONION_ERROR if the malloc fails. The user can install a memory failure handler if he wishes.

 

About the name, maybe just call it os (onion_os_alloc...) instead of low_prim. Somehow I dont like too much the name.

Well, I don't like much your onion_free name (it makes me think of a routine like my onionlow_free) . But it is there and we are stuck. The worst is of course the many occurrences of petition instead of web requests! I would prefer a systematic naming for creation and destroying. So for a data of type onion I would have preferred onion_new and onion_delete or onion_destroy (not onion_free).

There are two namings involved in my patch: the name of the added files low_util.c and low_util.h , do you prefer os_wrap.c instead. And most importantly the prefix to

the newly added function. Currently I am using onionlow_. Sadly, using just onion_ is not possible because onion_free is unfortunately already used and has an entirely different meaning. Would you prefer the longer prefix onion_wrap_  so replacing every call to free (in my patch they are replaced with onionlow_free ...) with onion_wrap_free for instance? Please choose a prefix which cannot be onion_ 

But I am stuck with cmake. Please help me to get my current file low_util.c be linked in the libonion library!

Regard. Basile Starynkevitch 

[Basile Starynkevitch]

On Tuesday, May 13, 2014 2:48:51 PM UTC+2, Basile Starynkevitch wrote:

On Tuesday, May 13, 2014 1:20:20 PM UTC+2, David Moreno Montero wrote:

Just a quick debugging tutorial:

Create the Makefiles with cmake -DCMAKE_BUILD_TYPE=Debug.

I commit a patch (on my branch  https://github.com/bstarynk/onion to README.rst explaining that.

But this does not solve my issue of getting my new file src/onion/low_util.c compiled (it is compiled) and linked (it does not seems to be linked).

But I am stuck with cmake. Please help me to get my current file low_util.c be linked in the libonion library!

I temporarily added a #warning at the end of my new src/onion/low_util.c here is the compilation output:

make -j 2

Scanning dependencies of target onion

Scanning dependencies of target onion_static

[  1%] [  1%] Building C object src/onion/CMakeFiles/onion_static.dir/low_util.c.o

Building C object src/onion/CMakeFiles/onion.dir/low_util.c.o

In file included from /usr/src/Libs/onion/src/onion/low_util.c:39:0:

/usr/src/Libs/onion/src/onion/low_util.h:110:5: warning: ‘noreturn’ attribute ignored [-Wattributes]

     ;

     ^

In file included from /usr/src/Libs/onion/src/onion/low_util.c:39:0:

/usr/src/Libs/onion/src/onion/low_util.h:110:5: warning: ‘noreturn’ attribute ignored [-Wattributes]

     ;

     ^

/usr/src/Libs/onion/src/onion/low_util.c:289:2: warning: #warning this is the end of low_util.c [-Wcpp]

 #warning this is the end of low_util.c

  ^

/usr/src/Libs/onion/src/onion/low_util.c:289:2: warning: #warning this is the end of low_util.c [-Wcpp]

 #warning this is the end of low_util.c

  ^

Linking C shared library libonion.so

Linking C static library libonion_static.a

[ 13%] Built target onion_static

[ 17%] Built target onion_handlers_static

[ 30%] Built target onion

[ 31%] Built target onion_extras

[ 34%] Built target onioncpp

[ 38%] Built target onioncpp_static

Linking C executable opack

[ 38%] Built target crl

Linking C executable otemplate

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_new':

dict.c:(.text+0x1bc): undefined reference to `onionlow_calloc'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_data_free':

dict.c:(.text+0x2ec): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_add':

dict.c:(.text+0x3c9): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_remove':

dict.c:(.text+0x63d): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_add':

dict.c:(.text+0x782): undefined reference to `onionlow_malloc'

dict.c:(.text+0x801): undefined reference to `onionlow_strdup'

dict.c:(.text+0x814): undefined reference to `onionlow_strdup'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_json_preorder':

dict.c:(.text+0xc83): undefined reference to `onionlow_free'

dict.c:(.text+0xcbc): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_free':

dict.c:(.text+0x29e): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_data_free':

dict.c:(.text+0x2cb): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_free':

dict.c:(.text+0x32d): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/block.c.o: In function `onion_block_new':

block.c:(.text+0x7): undefined reference to `onionlow_malloc'

block.c:(.text+0x14): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/opack.dir/__/__/src/onion/block.c.o: In function `onion_block_free':

block.c:(.text+0x38): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/block.c.o: In function `onion_block_min_maxsize':

block.c:(.text+0x77): undefined reference to `onionlow_realloc'

CMakeFiles/opack.dir/__/__/src/onion/block.c.o: In function `onion_block_add_char':

block.c:(.text+0xef): undefined reference to `onionlow_realloc'

CMakeFiles/opack.dir/__/__/src/onion/block.c.o: In function `onion_block_add_data':

block.c:(.text+0x18d): undefined reference to `onionlow_scalar_malloc'

block.c:(.text+0x1c1): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/block.c.o: In function `onion_block_free':

block.c:(.text+0x41): undefined reference to `onionlow_free'

CMakeFiles/opack.dir/__/__/src/onion/codecs.c.o: In function `onion_base64_decode':

codecs.c:(.text+0xb3): undefined reference to `onionlow_scalar_malloc'

codecs.c:(.text+0xeb): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/opack.dir/__/__/src/onion/codecs.c.o: In function `onion_base64_encode':

codecs.c:(.text+0x291): undefined reference to `onionlow_malloc'

CMakeFiles/opack.dir/__/__/src/onion/codecs.c.o: In function `onion_quote_new':

codecs.c:(.text+0x788): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/opack.dir/__/__/src/onion/codecs.c.o: In function `onion_c_quote_new':

codecs.c:(.text+0x941): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/opack.dir/__/__/src/onion/codecs.c.o: In function `onion_html_quote':

codecs.c:(.text+0xaa4): undefined reference to `onionlow_scalar_malloc'

collect2: error: ld returned 1 exit status

make[2]: *** [tools/opack/opack] Error 1

make[1]: *** [tools/opack/CMakeFiles/opack.dir/all] Error 2

make[1]: *** Waiting for unfinished jobs....

CMakeFiles/otemplate.dir/__/__/src/onion/block.c.o: In function `onion_block_new':

block.c:(.text+0x7): undefined reference to `onionlow_malloc'

block.c:(.text+0x14): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/otemplate.dir/__/__/src/onion/block.c.o: In function `onion_block_free':

block.c:(.text+0x38): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/block.c.o: In function `onion_block_min_maxsize':

block.c:(.text+0x77): undefined reference to `onionlow_realloc'

CMakeFiles/otemplate.dir/__/__/src/onion/block.c.o: In function `onion_block_add_char':

block.c:(.text+0xef): undefined reference to `onionlow_realloc'

CMakeFiles/otemplate.dir/__/__/src/onion/block.c.o: In function `onion_block_add_data':

block.c:(.text+0x18d): undefined reference to `onionlow_scalar_malloc'

block.c:(.text+0x1c1): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/block.c.o: In function `onion_block_free':

block.c:(.text+0x41): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/codecs.c.o: In function `onion_base64_decode':

codecs.c:(.text+0xb3): undefined reference to `onionlow_scalar_malloc'

codecs.c:(.text+0xeb): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/otemplate.dir/__/__/src/onion/codecs.c.o: In function `onion_base64_encode':

codecs.c:(.text+0x291): undefined reference to `onionlow_malloc'

CMakeFiles/otemplate.dir/__/__/src/onion/codecs.c.o: In function `onion_quote_new':

codecs.c:(.text+0x788): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/otemplate.dir/__/__/src/onion/codecs.c.o: In function `onion_c_quote_new':

codecs.c:(.text+0x941): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/otemplate.dir/__/__/src/onion/codecs.c.o: In function `onion_html_quote':

codecs.c:(.text+0xac4): undefined reference to `onionlow_scalar_malloc'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_new':

dict.c:(.text+0x1bf): undefined reference to `onionlow_calloc'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_data_free':

dict.c:(.text+0x27c): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_add':

dict.c:(.text+0x359): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_remove':

dict.c:(.text+0x5cd): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_add':

dict.c:(.text+0x712): undefined reference to `onionlow_malloc'

dict.c:(.text+0x791): undefined reference to `onionlow_strdup'

dict.c:(.text+0x7a4): undefined reference to `onionlow_strdup'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_json_preorder':

dict.c:(.text+0xc13): undefined reference to `onionlow_free'

dict.c:(.text+0xc4c): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_free':

dict.c:(.text+0x231): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_data_free':

dict.c:(.text+0x25b): undefined reference to `onionlow_free'

CMakeFiles/otemplate.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_free':

dict.c:(.text+0x2bd): undefined reference to `onionlow_free'

collect2: error: ld returned 1 exit status

make[2]: *** [tools/otemplate/otemplate] Error 1

make[1]: *** [tools/otemplate/CMakeFiles/otemplate.dir/all] Error 2

make: *** [all] Error 2

Please help about cmake..... Regards.

[Basile Starynkevitch]

On Tue, May 13, 2014 at 03:26:44PM +0200, David Moreno Montero wrote:
> I don't think I will be able to check it out myself what is exactly wrong
> with cmake, although everything looks ok. It might be, maybe that the
> linking to libgc, or whatever is the name, is missing.

I am not linking any library yet.
I just added the function which could enable me to register libgc allocation routines.
I won't call that function from onion, just from my application.


> Can you try to create again the build directory, maybe it's some cmake
> cache problem. Also with
>
> $ VERBOSE=1 make

Will try.

(I'm considering giving up libonion because of that issue).

[Basile Starynkevitch]

Hello All,

I just made a github pull request for https://github.com/bstarynk/onion
There are some cmake related issues previously described on the onion-dev list.
I'm sure David knows how to add 2 new files for cmake: low_util.c & low_util.h
... but I don't.

It does not add anything specifically related to Boehm GC, but it should help.

If that patch is accepted in libonion, it won't change much (except that failing
malloc won't crash, but more nicely abort the library).

If that patch is accepted in libonion, I would be able to use Boehm GC
http://hboehm.info/gc in my http://github.com/bstarynk/monimelt/ project.
(otherwise, I'll stop using libonion in it since this is a blocking bug in my case:
Boehm's GC require threads to be initialized with GC_pthread_create, and libonion
is creating threads without that).

If that patch is accepted, I would just call in my initialization (after GC_INIT and
before any call to onion)

static void panic (const char* msg); // my panic routine, which aborts.

onionlow_initialize_memory_allocation
(GC_malloc,
GC_malloc_atomic,
GC_calloc,
GC_realloc,
GC_strdup,
GC_free,
panic);

onionlow_initialize_threads
(GC_pthread_create,
GC_pthread_join,
GC_pthread_cancel,
GC_pthread_detach,
GC_pthread_exit,
GC_pthread_sigmask);

and then I would use onion as usual. Nothing new is linked to libonion
(in particular, libgc is not linked to it, but libgc is linked
to my application).


Of course, all the GC_* functions above are provided by Boehm GC and are equivalent
in behavior to their standard counterpart, except for GC_malloc_atomic
which does a garbage collected malloc of a zone guaranteed by the application
to not contain any pointer, in other words GC_malloc_atomic(sz) is allocating
like GC_malloc but in addition giving the promise to Boehm collector that the
llocated zone does not contain any pointer, like e.g. most char buffers...!.


I'm considering giving up libonion if such a small patch is not accepted
(or maybe, make a fork of libonion inside my own application monimelt).

Again, this is a blocking deficiency in libonion. The issue is that Boehm's GC
obviously requires some initialization in every thread (which is done by its
GC_pthread_create....) but onion is creating threads without that.


There are other usecases for my patch (completely independent of libgc
or my monimelt program):

O. better handling of malloc out-of-memory failure: with the patch you got a message
before aborting or terminating (without it, you could get a SIGSEGV)

1. using some other alternative malloc implementation

2. providing its own malloc & free which clear data before freeing it
(to be sure that e.g. sensitive passwords don't leak back into the heap)

3. initializing threads created by libonion, e.g. by having them call
pthread_setname_np or pthread_setschedprio ... In a strongly multithreaded
application both are definitely useful (and cannot be done on the
libonion trunk today!)


I believe that these usescases justify libonion to be improved
(independently of my libgc needs).

I'm ready to improve that patch (perhaps renaming files, changing prefixes, ....)
if I was able to get cmake compile and link my low_util.c file (which gets compiled
but not linked).

Regards.

[Basile Starynkevitch]

I don't know cmake at all. I asked on stackoverflow, and got this nice answer. David, does that answer makes some sense for you? It looks like its author is knowledgeable about cmake. Unfortunately for me, I am not. (and after spending an hour reading its documentation, I can't help more).

David, could you help me adding a single C file (low_util.c) and its corresponding header (low_util.h) to libonion. I'm sure that if you edited some README describing in a few sentences how to add a new file to libonion, it will help some future contributors.

It is not as simple as I thought.

Regards.

[Basile Starynkevitch]

On Tue, 2014-05-13 at 19:05 +0200, David Moreno Montero wrote:
> Can you make a full 'make clean', 'VERBOSE=1 make' and post the output
> to a pastebin?

> http://pastebin.com/aVkaGf8h
>
I'm also attaching the output.

The onion from https://github.com/bstarynk/onion is in onion-basile

I did:

mkdir build-onion-basile


cmake ../onion-basile |& tee /tmp/onion-basile-build.txt

make VERBOSE=1 |& tee /tmp/onion-basile-build.txt

and I am attaching the onion-basile-build.txt

Thanks for your attention.

Regards.
--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France

*** opinions {are only mine, sont seulement les miennes} ***

[Basile Starynkevitch]

On Wed, 2014-05-14 at 17:15 +0200, David Moreno Montero wrote:
> Hi Basile,
>
> I sent yesterday a pull request to your onion, from my bstarynk-master
> branch.
>
> Does it compile properly?

Yes it does, thanks for the help on cmake....

What should I do now to have chances that this patch gets into your
onion trunk?

Is it simply a matter of renaming, perhaps

low_util.c -> os_wrap.c
low_util.h -> os_wrap.h

and onionlow_ prefix -> onionos_ prefix?

Or what do you suggest?

I'm busily experimenting on monimelt an in-house patched nanohttp (from
csoap-1.1) as a replacement to onion.... But I would be much more happy
to use onion again (but for that, I absolutely need the ability to
customize the thread starting and the memory allocator, to make Boehm's
GC happy, as explained in previous emails).

[Basile Starynkevitch]

On Wed, 2014-05-14 at 21:38 +0200, Basile Starynkevitch wrote:
> On Wed, 2014-05-14 at 17:15 +0200, David Moreno Montero wrote
>
> and onionlow_ prefix -> onion_os_ prefix?

This renaming is done in
https://github.com/bstarynk/onion/commit/319f9cbf87dd8e27bac32619c9c185733187ac95

I'm waiting for your wish about file renaming. Still want os.c & os.h
for two files which are not operating system related? I believe
os_wrap.c & os_wrap.h might be better (and actually I feel low_util.c &
low_util.h better even, but you don't like that name).

BTW, there is a very simple way to test this. Just code a web server in
onion which interfaces to GUILE 2 http://www.gnu.org/software/guile/
(Guile is using Boehm GC). Probably that example would have to be under
GPL licence (not LGPL) since Guile is...


I probably could code such an example, but I am so afraid of the cmake
mess that I need your step by step guidance. How should I patch the
cmake things to add a new file which needs to link libguile using
pkg-config. i.e. add the output of
pkg-config --cflags guile-2.0
for the compilation flags, and the output of
pkg-config --libs guile-2.0
for the link flags.

I probably could make a tests/10-guile/ directory (it probably would
contain some html file, perhaps some javascript, and of course some C
file using libguile), but I need your guidance for the cmake things.

[David Moreno Montero]

low_util sounds ok. low.c sounds even better :) (all the others are just request.c, no request_xxx.c, but its ok).

About guile it would be awesome to have guile bindings. I have half working python bindings, and there are also php bindings around. But to program a nice web server in guile sounds really awesome. Anyway for this test, just use the minimum to serve something, no need for full bindings, of course :)

To avoid cmake, you can create the example at tests/10-guile/ using standard make, and at merge I will do the translation to cmake.

So, after this onion_low renaming, just send me a pull request and I merge ASAP.

Regards,
David.

[Basile Starynkevitch]

On Wed, 2014-05-14 at 22:35 +0200, David Moreno Montero wrote:
> low_util sounds ok. low.c sounds even better :) (all the others are
> just request.c, no request_xxx.c, but its ok).
>
>

> So, after this onion_low renaming, just send me a pull request and I
> merge ASAP.

Done. I also added something in the README.rst (and replaced the
confusing petition word with web-request).

I hope this will be merged quickly in onion.

Will work soon on a test case using GUILE which is not at all the same
as providing GUILE bindings to onion...

Cheers.

[David Moreno Montero]

Merged!

I fixed the GPLv2 stuff, and changed current coments to be more Doxygen friendly.

as I said I have no time now, but I would like to experiment on performance and different (API/ABI compatible) implementations.

About the exampe no bwing bindings, you are right, but with the proper example that wraps some onion calls, it might be the begining, although of course more work is needed just to call it bindings.

Thank you very much for the effort.

Regards,
David.

[Basile Starynkevitch]

On Thu, May 15, 2014 at 07:32:54AM +0200, Basile Starynkevitch wrote:
> On Wed, 2014-05-14 at 22:35 +0200, David Moreno Montero wrote:
> > low_util sounds ok. low.c sounds even better :) (all the others are
> > just request.c, no request_xxx.c, but its ok).
> >
> >
> > So, after this onion_low renaming, just send me a pull request and I
> > merge ASAP.
>
> Done. I also added something in the README.rst (and replaced the
> confusing petition word with web-request).
>
> I hope this will be merged quickly in onion.

There is a small issue which is very probably cmake related.

The current public header onion/low.h has to have:

#ifdef HAVE_PTHREADS
void onion_low_initialize_threads
(onion_low_pthread_create_sigt * thrcreator,
onion_low_pthread_join_sigt * thrjoiner,
onion_low_pthread_cancel_sigt * thrcanceler,
onion_low_pthread_detach_sigt * thrdetacher,
onion_low_pthread_exit_sigt * threxiter,
onion_low_pthread_sigmask_sigt * thrsigmasker);
//// etc...
#endif

So the user of onion has to #define HAVE_PTHREADS 1
before #include <onion/low.h> which is an annoyance.


Ordinarily (e.g. in autoconf-ed & automake-ed GNU software) some onion/onion-config.h header
would have been generated by the build process (with all the HAVE_* things appropriately
defined) and onion/low.h would #include that onion-config.h header.

From what I have read on cmake (the more I read on cmake, the more I hate it! but it
is a matter of taste. autoconf is also a nightmare, but I am just less scared of it...)
there is a possibility to have cmake generate these things (e.g. some
onion/onion-config.h generated header, to be installed...). But I don't know how to do that.

So again please, when you have time David, have a look on that issue.

I'm beginning to test my onion branch on my monimelt software.

Regards.

--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France

*** opinions {are only mines, sont seulement les miennes} ***

[Basile Starynkevitch]

On Thu, May 15, 2014 at 07:32:54AM +0200, Basile Starynkevitch wrote:

> On Wed, 2014-05-14 at 22:35 +0200, David Moreno Montero wrote:
> > low_util sounds ok. low.c sounds even better :) (all the others are
> > just request.c, no request_xxx.c, but its ok).

> I hope this will be merged quickly in onion.

Thanks for having merged it.

I'm doing some very preliminary experiments inside my monimelt. It works much better.
Without the onion_low_* things, my monimelt program crashed immediately.
Now it is able to run for 10 minutes....

Cheers.

PS. I will look this evening on the guile test case.

--
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basile<at>starynkevitch<dot>net mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France

*** opinions {are only mines, sont seulement les miennes} ***