Should the op 'pin' create a C-string ?
Bernhard Schmalhofer
Hi,
I am trying to pass a string, containing YAML, from Parrot to the shared
library 'libsyck'.
'libsyck' provides two functions:
void syck_parser_str( SyckParser *, char *, long, SyckIoStrRead ); (
v_ptip )
SYMID syck_parse( SyckParser * ); ( i_p
)
'syck_parser_str' tells libsyck about the string to be parsed.
'syck_parse' does the parsing.
This means that the string buffer has to stay around, until the parsing is
done.
Currently, when calling 'syck_parser_str', the string buffer is copied into
newly allocated system memory. In order to avoid memory leaks, this
allocated system memory is freed immediately after 'syck_parser_str'. Not
surprisingly, bad things are happening when 'syck_parse' is called later.
>From strings.pod I gather that the op 'pin' is meant to make strings fit
for
passing strings to external libraries. For me it would be convenient, if
'pin' would put a trailing '\0' at the end of the used string buffer. Also a
pinned string shouldn't have to be copied into newly allocated memory.
Is that the right way to go?
CU, Bernhard
--
/* Bernhard.S...@biomax.de */
Geschenkt: 3 Monate GMX ProMail + 3 Top-Spielfilme auf DVD
++ Jetzt kostenlos testen http://www.gmx.net/de/go/mail ++
--
Geschenkt: 3 Monate GMX ProMail + 3 Top-Spielfilme auf DVD
++ Jetzt kostenlos testen http://www.gmx.net/de/go/mail ++
Dan Sugalski
>
>>From strings.pod I gather that the op 'pin' is meant to make strings fit
>for
>passing strings to external libraries. For me it would be convenient, if
>'pin' would put a trailing '\0' at the end of the used string buffer. Also a
>pinned string shouldn't have to be copied into newly allocated memory.
Sort of. The point is to make the string not move. Generally for
external libraries, but not universally.
It shouldn't add the trailing null, because that would be modifying
the buffer. If you want to make sure there's a null on the end of the
string, then the best thing to do is put one there yourself. (Which,
since it's your data, would be a safe thing to do)
--
Dan
--------------------------------------it's like this-------------------
Dan Sugalski even samurai
d...@sidhe.org have teddy bears and even
teddy bears get drunk
Bernhard Schmalhofer
>> From strings.pod I gather that the op 'pin' is meant to
>> make strings fit for passing strings to external libraries.
> Sort of. The point is to make the string not move. Generally for
> external libraries, but not universally.
>
> It shouldn't add the trailing null, because that would be modifying the
> buffer. If you want to make sure there's a null on the end of the
> string, then the best thing to do is put one there yourself. (Which,
> since it's your data, would be a safe thing to do)
Yes, I have no problem with pushing a '\0' before pinning a string, that
I want to pass to an external library.
The real problem is that pinning is useless, because the NCI copies the
string buffer to new location and then passes the new location to the
external library.
Would things break when the NCI would generally just pass the string
buffer to the external library?
Not copying the string buffer would propably prevent memory checker like
'efence' from doing their work. The external libs could also modify the
string, which is good and bad.
CU, Bernhard
--
**************************************************
Dipl.-Physiker Bernhard Schmalhofer
Senior Developer
Biomax Informatics AG
Lochhamer Str. 11
82152 Martinsried, Germany
Tel: +49 89 895574-839
Fax: +49 89 895574-825
eMail: Bernhard.S...@biomax.com
Website: www.biomax.com
**************************************************
Leopold Toetsch
> Hi,
> I am trying to pass a string, containing YAML, from Parrot to the shared
> library 'libsyck'.
> void syck_parser_str( SyckParser *, char *, long, SyckIoStrRead ); (
> v_ptip )
^^^^^^
> SYMID syck_parse( SyckParser * ); ( i_p
> )
> 'syck_parser_str' tells libsyck about the string to be parsed.
> 'syck_parse' does the parsing.
> This means that the string buffer has to stay around, until the parsing is
> done.
This is what the "t" signature char is doing anyway - Oops or better,
what it should to. While it's using string_to_cstring the created string
isn't freed (unless the library would free it, which is unlikely).
So what we need for the "t" signature is a string_pin with NUL append.
Patches welcome,
leo
Bernhard Schmalhofer
>
>>This means that the string buffer has to stay around, until the
parsing is
>>done.
>
>
> This is what the "t" signature char is doing anyway - Oops or better,
> what it should to. While it's using string_to_cstring the created string
> isn't freed (unless the library would free it, which is unlikely).
Actually it is freed right after the external function call.
Which is bad when the freed memory is accessed later.
>
> So what we need for the "t" signature is a string_pin with NUL append.
Sounds good to me. I'm just wondering how string pinning can be tested.
I guess I have to look into the garbage collection tests.
Leopold Toetsch
> Leopold Toetsch wrote:
> >
> > This is what the "t" signature char is doing anyway - Oops or better,
> > what it should to. While it's using string_to_cstring the created string
> > isn't freed (unless the library would free it, which is unlikely).
> Actually it is freed right after the external function call.
in nci.c yes in jit/i386 no.
> Which is bad when the freed memory is accessed later.
Yep.
> > So what we need for the "t" signature is a string_pin with NUL append.
> Sounds good to me. I'm just wondering how string pinning can be tested.
> I guess I have to look into the garbage collection tests.
Dead strings with the PObj_sysmem_FLAG set, do the right thing. You
just have to take care that you hold a ref to that string as long as you
need it (ot the lib of course).
> CU, Bernhard
leo
Dan Sugalski
>Leopold Toetsch wrote:
>>
> >>This means that the string buffer has to stay around, until the parsing is
>>>done.
>>
>>
>> This is what the "t" signature char is doing anyway - Oops or better,
>> what it should to. While it's using string_to_cstring the created string
>> isn't freed (unless the library would free it, which is unlikely).
>
>Actually it is freed right after the external function call.
>Which is bad when the freed memory is accessed later.
No, that's not bad.
The problem is that you're using the wrong signature type here. 't'
is for plain c strings passed into functions for temporary usage. It
is *not* for passing in of long-lived buffers. You really want the
'b' type if it's a long-lived thing. That pulls the buffer pointer
out of the string structure and passes it in, so it's suitable for
mutable stuff. (That's what it's there for, actually)
I think maybe the docs for this need some updating. I'll go do that
while I'm waiting for tests to churn.
Leopold Toetsch
> ... You really want the
> 'b' type if it's a long-lived thing. That pulls the buffer pointer
> out of the string structure and passes it in, so it's suitable for
> mutable stuff. (That's what it's there for, actually)
Using bufstart in string code is almost always wrong, e. g. a COWed
substring has strstart pointing to the string while bufstart is the
start of the whole buffer. But that's only a small implementation flaw.
Anyway, yes 'b' seems the right thing here. If the library code messes
with the string this might not be enough, though.
leo
Dan Sugalski
>Dan Sugalski <d...@sidhe.org> wrote:
>
>> ... You really want the
>> 'b' type if it's a long-lived thing. That pulls the buffer pointer
>> out of the string structure and passes it in, so it's suitable for
>> mutable stuff. (That's what it's there for, actually)
>
>Using bufstart in string code is almost always wrong, e. g. a COWed
>substring has strstart pointing to the string while bufstart is the
>start of the whole buffer. But that's only a small implementation flaw.
Fair 'nuff. It ought to be strstart.
>Anyway, yes 'b' seems the right thing here. If the library code messes
>with the string this might not be enough, though.
I think it should be OK. There are a few cases where it's still not
quite right, but with proper documentation we ought to be fine.
Bernhard Schmalhofer
>
> The problem is that you're using the wrong signature type here. 't' is
> for plain c strings passed into functions for temporary usage. It is
> *not* for passing in of long-lived buffers. You really want the 'b' type
> if it's a long-lived thing. That pulls the buffer pointer out of the
> string structure and passes it in, so it's suitable for mutable stuff.
> (That's what it's there for, actually)
>
Yes, the 'b' signature is what I want and it works fine. For
'syck_parser_str()' I don't need the trailing NUL, as I pass in the
string length.
A writeup of my experiences with 'libsyck' for Parrat is on
http://yaml.kwiki.org/index.cgi?ParrotDev .