NCI 't' parameter type removed; no replacement?
Jonathan Worthington
The deprecated 't' parameter in NCI broke Zavolaj (the Perl 6 native
call interface). This in turn broke various modules, including MiniDBI.
Looking at:
It seems that the signatures for e.g. mysql database connection were
removed, not updated! The deprecation notice doesn't provide anything
that seems to help with our use case. Replacing t with S seems to lead
to segfaults; a further look shows that seems not to be passing a C
string or even the string buffer, but the STRING * pointer. None of the
examples that use the 't' parameter have been updated.
Seriously, what the hell? I'm all for improvements. I'm fine with
deprecations if I'm given a clue what to do to get things wroking again.
But a removal of a feature used by *Perl 6 database access* with no
instructions or updated examples that I can find, or maybe not even an
actual replacement? Please tell me I'm missing something obvious
somewhere.... Or just revert the patch that removed it.
Thanks,
/jnthn
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
<jona...@jnthn.net> wrote:
> Hi,
>
> The deprecated 't' parameter in NCI broke Zavolaj (the Perl 6 native call
> interface). This in turn broke various modules, including MiniDBI. Looking
> at:
>
> https://github.com/parrot/parrot/commit/75e2b4c94d602d3d7e8af4b679c325c8ec8d3fd9#src/nci/extra_thunks.nci
>
> It seems that the signatures for e.g. mysql database connection were
> removed, not updated! The deprecation notice doesn't provide anything that
> seems to help with our use case. Replacing t with S seems to lead to
> segfaults; a further look shows that seems not to be passing a C string or
> even the string buffer, but the STRING * pointer. None of the examples that
> use the 't' parameter have been updated.
The signatures were removed with the intent that they be replaced
latter with their equivalents as needed. All code that was exercised
by tests (pcre, pg) was updated. It is regrettable that we have so
much untested code being used as examples of NCI.
To update your code, I would recommend making use of
Parrot_str_to_cstring/Parrot_str_free_cstring. These can be accessed
from HLL code using the NCI interface.
> Seriously, what the hell? I'm all for improvements. I'm fine with
> deprecations if I'm given a clue what to do to get things wroking again. But
> a removal of a feature used by *Perl 6 database access* with no instructions
> or updated examples that I can find, or maybe not even an actual
> replacement? Please tell me I'm missing something obvious somewhere.... Or
> just revert the patch that removed it.
Some of the code that was updated did make use of the 't' signature
type. These updates usually followed the advice given above. pcre, for
example, would be useless if not for its ability to operate on
strings.
> Thanks,
>
> /jnthn
>
>
>
>
>
>
> _______________________________________________
> http://lists.parrot.org/mailman/listinfo/parrot-dev
>
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Carl Mäsak
>> It seems that the signatures for e.g. mysql database connection were
>> removed, not updated! The deprecation notice doesn't provide anything that
>> seems to help with our use case. Replacing t with S seems to lead to
>> segfaults; a further look shows that seems not to be passing a C string or
>> even the string buffer, but the STRING * pointer. None of the examples that
>> use the 't' parameter have been updated.
>
> The signatures were removed with the intent that they be replaced
> latter with their equivalents as needed.
...but they *were* needed. The bottom line is that before those
removals, Rakudo had a mysql database connection, and after, it
doesn't.
It seems to me that some kind of fundamental upstream-downstream
relationship isn't being done very carefully in this case. Is there
any way Rakudo Perl 6 application/module developers like me can
advertise our dependencies on Parrot interfaces in order to get
notified when they disappear?
// Carl
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Jonathan Worthington
> Jonathan (>>), Peter (>):
>
>>> It seems that the signatures for e.g. mysql database connection were
>>> removed, not updated! The deprecation notice doesn't provide anything that
>>> seems to help with our use case. Replacing t with S seems to lead to
>>> segfaults; a further look shows that seems not to be passing a C string or
>>> even the string buffer, but the STRING * pointer. None of the examples that
>>> use the 't' parameter have been updated.
>>>
>> The signatures were removed with the intent that they be replaced
>> latter with their equivalents as needed.
>>
> ...but they *were* needed. The bottom line is that before those
> removals, Rakudo had a mysql database connection, and after, it
> doesn't.
>
>
is that the NCI signatures were just removed for *parts* of the mysql
jlibrary access too. As if leaving the others was going to be of any
use, when you can't even call mysql_connect any more...
/jnthn
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
Parrot provides the option of warning the user when they use
deprecated and/or experimental features
(PARROT_WARNINGS_DEPRECATED_FLAG). Unfortunately, this is relatively
obscure, infrequently if ever run, and has fallen into disuse.
> // Carl
> _______________________________________________
> http://lists.parrot.org/mailman/listinfo/parrot-dev
>
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Will Coleda
> On Fri, May 13, 2011 at 7:24 AM, Jonathan Worthington
> <jona...@jnthn.net> wrote:
>> Hi,
>>
>> The deprecated 't' parameter in NCI broke Zavolaj (the Perl 6 native call
>> interface). This in turn broke various modules, including MiniDBI. Looking
>> at:
>>
>> https://github.com/parrot/parrot/commit/75e2b4c94d602d3d7e8af4b679c325c8ec8d3fd9#src/nci/extra_thunks.nci
>>
>> It seems that the signatures for e.g. mysql database connection were
>> removed, not updated! The deprecation notice doesn't provide anything that
>> seems to help with our use case. Replacing t with S seems to lead to
>> segfaults; a further look shows that seems not to be passing a C string or
>> even the string buffer, but the STRING * pointer. None of the examples that
>> use the 't' parameter have been updated.
>
> The signatures were removed with the intent that they be replaced
> latter with their equivalents as needed. All code that was exercised
Added in ebf2de7f (Christoph Otto 2010-07-28 02:39:31 +0000 85) -
> HLLs, libraries and other users of Parrot will inevitably find that we
> deprecate some core features which they depend on. In order to minimize the
> pain caused by this deprecation and the subsequent "upgrade tax" (i.e. the time
> users must spend to keep their code working after an upgrade), we now forbid
> the removal of any deprecated feature until an appropriate upgrade path has
> been documented. Any deprecated features that are removed without an
> appropriate notification are subject to reversion until such notifications have
> been added. The single (rare) exception to this rule is any core feature which
> is broken or incomplete to the point that it is deemed unusable by any user.
This seems to fall under this clause, and I heartily agree.
Let's bring this back, fix Zavolaj, document the upgrade path,
IMPLEMENT IT, and then we can worry about deleting the old stuff.
> by tests (pcre, pg) was updated. It is regrettable that we have so
> much untested code being used as examples of NCI.
>
> To update your code, I would recommend making use of
> Parrot_str_to_cstring/Parrot_str_free_cstring. These can be accessed
> from HLL code using the NCI interface.
>
>> Seriously, what the hell? I'm all for improvements. I'm fine with
>> deprecations if I'm given a clue what to do to get things wroking again. But
>> a removal of a feature used by *Perl 6 database access* with no instructions
>> or updated examples that I can find, or maybe not even an actual
>> replacement? Please tell me I'm missing something obvious somewhere.... Or
>> just revert the patch that removed it.
>
> Some of the code that was updated did make use of the 't' signature
> type. These updates usually followed the advice given above. pcre, for
> example, would be useless if not for its ability to operate on
> strings.
>
>> Thanks,
>>
>> /jnthn
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> http://lists.parrot.org/mailman/listinfo/parrot-dev
>>
> _______________________________________________
> http://lists.parrot.org/mailman/listinfo/parrot-dev
>
--
Will "Coke" Coleda
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Patrick R. Michaud
> On Fri, May 13, 2011 at 7:24 AM, Jonathan Worthington
> Parrot_str_to_cstring/Parrot_str_free_cstring. These can be accessed
> from HLL code using the NCI interface.
> Some of the code that was updated did make use of the 't' signature
> type. These updates usually followed the advice given above. pcre, for
> example, would be useless if not for its ability to operate on
> strings.
Passing C-strings to NCI functions seems like it would be such a
common operation that I'm surprised that Parrot isn't offering native
support for it.
Pm
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Jim Keenan
May 13, 2011 01:53:53 PM, wi...@coleda.com wrote:
> This seems to fall under this clause, and I heartily agree.
> Let's bring this back, fix Zavolaj, document the upgrade path,
> IMPLEMENT IT, and then we can worry about deleting the old stuff.
I agree as well.
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Patrick R. Michaud
> [...]
> This seems to fall under this clause, and I heartily agree.
>
> Let's bring this back, fix Zavolaj, document the upgrade path,
> IMPLEMENT IT, and then we can worry about deleting the old stuff.
+1.
I also notice that according to api.yaml the UnManagedStruct type
is deprecated and eligible for removal in 3.4 (i.e., "now").
Zavolaj uses that also, so in order to avoid another "surprise"
we need details on the upgrade path for that as well.
Or perhaps the eligibility dates should be pushed back if the
new features / upgrade path aren't available yet.
Pm
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
The upgrade path exists, as evidenced by the libraries and tests that
were upgraded. The wiki page contains a description of this, and it
has been stated in this email thread as well: Call
Parrot_str_to_cstring and Parrot_str_free_cstring as is appropriate
for the lifetime of the buffer you are working with.
> Let's bring this back, fix Zavolaj, document the upgrade path,
> IMPLEMENT IT, and then we can worry about deleting the old stuff.
I am opposed to this course of action. The deprecation policy was
followed in this update. The deprecation policy does not guarrantee
that nothing will ever break. It aims to mitigate the associated
problems through notification and documentation, both of which have
been provided in this instance.
Peter Lobsinger
> I also notice that according to api.yaml the UnManagedStruct type
> is deprecated and eligible for removal in 3.4 (i.e., "now").
> Zavolaj uses that also, so in order to avoid another "surprise"
> we need details on the upgrade path for that as well.
TT #2035 already gives an upgrade path.
Will Coleda
No, the policy was not followed. Please reread the section I quoted above.
>> --
>> Will "Coke" Coleda
Peter Lobsinger
I have reviewed the above mentioned paragraph. I draw three main points:
1) "Upgrade tax" cannot be eliminated completely but can be
controlled. This situation is a case of a user having to pay this tax.
2) Feature removals require an upgrade strategy be provided. This has been done.
3) All deprecations require notice. This was done.
Is there something I'm missing?
Patrick R. Michaud
> On Fri, May 13, 2011 at 3:58 PM, Patrick R. Michaud <pmic...@pobox.com> wrote:
> > I also notice that according to api.yaml the UnManagedStruct type
> > is deprecated and eligible for removal in 3.4 (i.e., "now").
> > Zavolaj uses that also, so in order to avoid another "surprise"
> > we need details on the upgrade path for that as well.
>
> TT #2035 already gives an upgrade path.
I vehemently disagree with the notion of "upgrade path" implied by this
response and the ticket.
To me, an "upgrade path" is more than saying "try using one of these
other undocumented classes over here". An "upgrade path" gives me
(a user) sufficient information that I can take my existing code and
make it work in the new environment without having to ask basic questions
or search lots of Parrot C source code to try to figure things out.
I should be able to find examples of how the code is to be used.
Based on reading the #2035 ticket, I have no clue what a Ptr, PtrBuf,
PtrObj, or StructView PMC might be, what they can do, or how to use them.
It provides me no links or suggestions where I might look for this
information.
Doing a search through the Parrot sources reveals little documentation
on any of these classes, unless I want to try to decipher the C code that
implements them. AFAICT there is exactly one test that makes use of
StructView (in t/pmc/nci.t), and in that test I have no clue what
the constants .DATATYPE_STRUCT or .DATATYPE_PIR are intended to
represent. Unlike UnManagedStruct, there are no existing examples,
PDDs, or library code that I can look to for an idea of how these new
classes might be used. (UnManagedStruct has all of these.)
A search for "StructView" in trac.parrot.org only brings up a few
tickets; the #2035 ticket and a few others that describe problems
with StructView. A search for "PtrObj" only brings up the #2035 ticket.
I'm sorry, but this notion of "upgrade path" is not at all the sense
I get from reading the relevant Parrot support policy documents.
If I'm wrong about this, please tell me now so Rakudo can decide to
simply pick a current version of Parrot and stick with it while other
Parrot users pay the associated upgrade penalties.
Pm
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Patrick R. Michaud
> It is clear you want a level of stability not afforded by following
> HEAD rather than the supported releases. Perhaps Rakudo would be best
> to revise its decision to do so (and implicit associated agreement to
> increased risk of breakage as a result).
I firmly believe that having Rakudo follow the Parrot supported
releases simply means that we'll encounter these same problems in
July instead of May. And, unfortunately, it will then take until
October to have them "fixed" such that we can use them, or require
additional supported releases in July where the broken features
get backported/backpatched into Parrot.
It has been generally agreed in *many* previous Parrot discussions that
Rakudo is one of Parrot's key canaries in the coal mine for finding these
sorts of issues, and that it's also in Parrot's vital interest for
Rakudo to report problems early rather than later. If you believe
that Rakudo should not be serving in this capacity, and if the
rest of the Parrot leadership agrees, then Rakudo will figure out
how to deal with that.
Pm
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Andrew Whitworth
>> TT #2035 already gives an upgrade path.
>
> I vehemently disagree with the notion of "upgrade path" implied by this
> response and the ticket.
We're obviously not going to get anywhere arguing yuh-huh/nuh-uh over
and over again. For the sake of everybody's sanity, can we agree to
disagree on this point? Information was provided, the user would have
liked more of it. Let's make a note of this going forward and make a
conscious effort not to fall into this situation again.
> Based on reading the #2035 ticket, I have no clue what a Ptr, PtrBuf,
> PtrObj, or StructView PMC might be, what they can do, or how to use them.
> It provides me no links or suggestions where I might look for this
> information.
>
> Doing a search through the Parrot sources reveals little documentation
> on any of these classes, unless I want to try to decipher the C code that
> implements them. AFAICT there is exactly one test that makes use of
> StructView (in t/pmc/nci.t), and in that test I have no clue what
> the constants .DATATYPE_STRUCT or .DATATYPE_PIR are intended to
> represent. Unlike UnManagedStruct, there are no existing examples,
> PDDs, or library code that I can look to for an idea of how these new
> classes might be used. (UnManagedStruct has all of these.)
>
> A search for "StructView" in trac.parrot.org only brings up a few
> tickets; the #2035 ticket and a few others that describe problems
> with StructView. A search for "PtrObj" only brings up the #2035 ticket.
We clearly need more/better documentation about these things. I'll
open a ticket to make sure we do that. Also, it seems like we need to
take some effort to improve the quality and accuracy of PDD16 and
eventually move that out of draft so that everybody is on the same
page about what Parrot is trying to achieve with recent and future
changes to that system.
What we need to do right now is find a satisfactory way forward. We
need to figure out how much of a headache it is going to be for Parrot
to add back the 't' signature (or something very very similar to it),
vs how much headache is going to be required to work around it in
Zavolaj. We need to do this keeping in mind that Parrot has a release
coming up in only a few days, and we very likely will have a code
freeze in place (depending on what the release manager has to say)
during part of the remaining time.
I'm going to focus as much of my attention to this issue as possible.
We should aim to have some kind of a solution agreed upon by tomorrow.
--Andrew Whitworth
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Donald Hunter
binding has been updated to the new API and makes a good reference, in
my opinion."
Peter Lobsinger
> On Fri, May 13, 2011 at 09:53:10PM -0400, Peter Lobsinger wrote:
>> On Fri, May 13, 2011 at 3:58 PM, Patrick R. Michaud <pmic...@pobox.com> wrote:
>> > I also notice that according to api.yaml the UnManagedStruct type
>> > is deprecated and eligible for removal in 3.4 (i.e., "now").
>> > Zavolaj uses that also, so in order to avoid another "surprise"
>> > we need details on the upgrade path for that as well.
>>
>> TT #2035 already gives an upgrade path.
>
> I vehemently disagree with the notion of "upgrade path" implied by this
> response and the ticket.
I should have used the word "suggests" as opposed to "gives". It is
the starting of an upgrade path. The policy does not require a full
upgrade path to exist before something is listed as deprecated, only
that it exist before removal. Since the feature in question has yet to
be removed, this is not a problem.
> To me, an "upgrade path" is more than saying "try using one of these
> other undocumented classes over here". An "upgrade path" gives me
> (a user) sufficient information that I can take my existing code and
> make it work in the new environment without having to ask basic questions
> or search lots of Parrot C source code to try to figure things out.
> I should be able to find examples of how the code is to be used.
>
> Based on reading the #2035 ticket, I have no clue what a Ptr, PtrBuf,
> PtrObj, or StructView PMC might be, what they can do, or how to use them.
> It provides me no links or suggestions where I might look for this
> information.
>
> Doing a search through the Parrot sources reveals little documentation
> on any of these classes, unless I want to try to decipher the C code that
> implements them. AFAICT there is exactly one test that makes use of
> StructView (in t/pmc/nci.t), and in that test I have no clue what
> the constants .DATATYPE_STRUCT or .DATATYPE_PIR are intended to
> represent. Unlike UnManagedStruct, there are no existing examples,
> PDDs, or library code that I can look to for an idea of how these new
> classes might be used. (UnManagedStruct has all of these.)
All of these PMCs provide POD documentation about their usage. If you
feel this is insufficient, I welcome constructive criticism and advice
on how this could be made better.
As for the constants, they mean the same thing they do for UnManagedStruct.
Examples of use, as part of the body of code exercised by the test
suite, would be ported over before or during the removal.
> A search for "StructView" in trac.parrot.org only brings up a few
> tickets; the #2035 ticket and a few others that describe problems
> with StructView. A search for "PtrObj" only brings up the #2035 ticket.
>
> I'm sorry, but this notion of "upgrade path" is not at all the sense
> I get from reading the relevant Parrot support policy documents.
> If I'm wrong about this, please tell me now so Rakudo can decide to
> simply pick a current version of Parrot and stick with it while other
> Parrot users pay the associated upgrade penalties.
It is clear you want a level of stability not afforded by following
HEAD rather than the supported releases. Perhaps Rakudo would be best
to revise its decision to do so (and implicit associated agreement to
increased risk of breakage as a result).
> Pm
>
_______________________________________________
http://lists.parrot.org/mailman/listinfo/parrot-dev
Kevin Tew
I think some of Jonathan and Patrick extreme surprise is that we are making the extremely common use of C byte strings more laborious to use than they were in the past.
The whole point of a ffi interface is to remove the repetitive nature of the mundane cases and make the arcane cases possible.
We shouldn't force NCI/ffi/(whatever we are calling it these days) users to become responsible for manually marshaling such a primitive datatype such as C bytes stings to and form parrot Strings.
I understand for some extravagant use cases manual marshaling will be necessary.
However, we should make the extremely common case of 't' easy. ( Provide a NCI type that automatically marshals)
And make the uncommon cases possible. (Via manual marshaling.)
Kevin
_______________________________________________ http://lists.parrot.org/mailman/listinfo/parrot-dev
Peter Lobsinger
> Byte strings, the former 't' NCI type, are as primitive a data type as int
> is for libraries written in C.
> I think some of Jonathan and Patrick extreme surprise is that we are making
> the extremely common use of C byte strings more laborious to use than they
> were in the past.
>
> The whole point of a ffi interface is to remove the repetitive nature of the
> mundane cases and make the arcane cases possible.
I would put more emphasis on make possible than on make easy, at least
at the opcode-level. A simple native call interface operating on
primitives and doing minimal marshalling facilitates alternate
implementations. Of particular interest would be an alternate
implementation that tied in with just-in-time native code generation
functionality.
We already provide a runtime library, NCI::Utils, whose aim is to make
the common cases of NCI easy. I suggest we provide the sugar layer
there.
> We shouldn't force NCI/ffi/(whatever we are calling it these days) users to
> become responsible for manually marshaling such a primitive datatype such as
> C bytes stings to and form parrot Strings.
> I understand for some extravagant use cases manual marshaling will be
> necessary.
>
> However, we should make the extremely common case of 't' easy. ( Provide a
> NCI type that automatically marshals)
> And make the uncommon cases possible. (Via manual marshaling.)
I can understand the argument that char pointer as string convention
has become much more cumbersome. However, I don't think that embedding
such a convenience functionality directly into the NCI is the best
solution to this problem. Look at the other NCI types. These are all C
or Parrot primitives and their translation is fairly simple. By
comparison, the translation of Parrot strings to asciiz strings is a
much more complex operation. It is a composite operation that can be
built out of the primitives.
Due to the fact that this form of marshalling is so common, I propose
the functionality be made available in the NCI::Utils helper library
that exists solely to make common NCI situations easier.
> Kevin
>
>
> On 05/15/2011 02:16 PM, Donald Hunter wrote:
>
> Hi,
> I raised my concern about the removal of NCI signatures breaking Zavolaj on
> 27 April. I received what I felt was a very curt response from Peter
> suggesting that:
> " PCRE's pcre_compile function also makes use of pass-by-reference. This
> binding has been updated to the new API and makes a good reference, in
> my opinion."
> It was hard to find the example and it wasn't directly relevant.
> Unfortunately the response I got was enough to dissuade me from contributing
> the scarce time I had available to work on Zavolaj.
> I think it would have been more helpful to have backed off removing features
> that I'd just said were in use, rather than pressing ahead and leaving me to
> figure it out.
> Cheers,
> Donald.
>
>
> _______________________________________________
> http://lists.parrot.org/mailman/listinfo/parrot-dev
>
>