Typecasting Pointers on a 64 bit System

[Quentin Pope]

What is the best way to handle this warning:

warning: cast from pointer to integer of different size

I am casting in and out of a function that requires a pointer type. I
am casting an integer as a pointer, but the pointer is 8 bytes while
the integer is only 4 bytes.

Here's an example function:

pthread_create(&tid, NULL, readit, (void *)(long)connfd)

I need to pass the file descriptor as a void * for pthread_create. But
I need the file descriptor as an integer to read and write to it. Is the
above the best approach to turn off the warning? I am considering just
typecasting to size_t before typecasting to the pointer. Is that a
better approach than typecasting to long since size_t should remain equal
to the size of pointers?

Of course, I run into the reverse problem when I have to close the
file descriptor.

close((int) arg);

I don't even know what to do in the above case. I shouldn't be
truncating anything, so I should be okay.

I originally wrote the code using a 32 bit version, so I am receiving
these errors when I ported it over to a 64 bit version.

[Lauri Alanko]

In article <j9hbnp$13g$1...@speranza.aioe.org>,

Quentin Pope <qp1...@hotmail.NOSPAM.com> wrote:
> pthread_create(&tid, NULL, readit, (void *)(long)connfd)
>
> I need to pass the file descriptor as a void * for pthread_create. But
> I need the file descriptor as an integer to read and write to it. Is the
> above the best approach to turn off the warning?

No.

Just pass a pointer to the fd:

pthread_create(&tid, NULL, readit, &connfd);

Then dereference the pointer in the readit function to get the
descriptor. If connfd is a local variable, you need to make sure that
connfd doesn't exit the scope before the new thread has read it. If
this is difficult, you can allocate a new copy for the thread:

int* fdp = malloc(sizeof(int));
*fdp = connfd;
pthread_create(&tid, NULL, readit, fdp);

void readit(void* fdp) {
int connfd = *(int*)fdp;
free(fdp);
...;
}


Lauri

[Ben Pfaff]

Quentin Pope <qp1...@hotmail.NOSPAM.com> writes:

> What is the best way to handle this warning:
>
> warning: cast from pointer to integer of different size

The best way may be to avoid casting between a pointer and an
integer.

Another way is to use (u)intptr_t from <stdint.h>, which is
ordinarily defined as an integer type that is the same width as a
pointer.
--
Ben Pfaff
http://benpfaff.org

[James Kuyper]

On 11/10/2011 03:32 PM, Ben Pfaff wrote:
...

> Another way is to use (u)intptr_t from <stdint.h>, which is
> ordinarily defined as an integer type that is the same width as a
> pointer.

More accurately, it's defined to be big enough for the conversion to be
reversible. In practice, it's likely to be the same size, but that's not
part of the definition. It could easily be larger. If there's any
redundancy in the pointer representation, and the conversion removes
that redundancy, it could even be smaller.

[Kaz Kylheku]

On 2011-11-10, Quentin Pope <qp1...@hotmail.NOSPAM.com> wrote:
> What is the best way to handle this warning:
>
> warning: cast from pointer to integer of different size

Use an integer of a suitable size.

> I am casting in and out of a function that requires a pointer type. I
> am casting an integer as a pointer, but the pointer is 8 bytes while
> the integer is only 4 bytes.

In the C99 language, there is the intptr_t type for this sort of thing.

For older dialects of C, you have to cob together some kind of compile
time configuration. Define the type as a typedef and change it based
on target compiler and platform.

I've written a configure script which detects what integral type is
suitable for holding pointers (based on size) and throws the info
into a "config.h" header.

This is BSD licensed so you can borrow it.

http://www.kylheku.com/cgit/txr/tree/configure

The approach taken in this script is to compile a sample translation unit and
then inspect the symbols (using the POSIX-standard nm command).

This creates an int_ptr_t typedef in "config.h" along with macros
giving the minimum and maximum value.

It's only been tested on various GNU/Linux architectures, Cygwin and MinGW.

It supports cross-compiling because the test programs are only compiled, never
executed.

[Kaz Kylheku]

On 2011-11-10, Lauri Alanko <l...@iki.fi> wrote:
> In article <j9hbnp$13g$1...@speranza.aioe.org>,
> Quentin Pope <qp1...@hotmail.NOSPAM.com> wrote:
>> pthread_create(&tid, NULL, readit, (void *)(long)connfd)
>>
>> I need to pass the file descriptor as a void * for pthread_create. But
>> I need the file descriptor as an integer to read and write to it. Is the
>> above the best approach to turn off the warning?
>
> No.
>
> Just pass a pointer to the fd:
>
> pthread_create(&tid, NULL, readit, &connfd);

Ths is an inferior approach because now the object &connfd has
to remain stable for as long as the created thread wants to touch
that object.

> Then dereference the pointer in the readit function to get the
> descriptor. If connfd is a local variable, you need to make sure that
> connfd doesn't exit the scope before the new thread has read it.

You need to ensure that AND that the object is not overwritten with a different
value (e.g. in a loop that launches more than one thread).

In this case, why use threads? Just call "readit" directly from
this thread.

If the parent thread has to be confined while a thread executes, there is no
point. Just thread creation and context switching overhead.

What you can do is implement some little "mousetrap" protocol
wherby the thread indicates "I have consumed the argument, you can
trash it now". E.g condition variable, etc.

Or you could just cast the darn value to a pointer!

> this is difficult, you can allocate a new copy for the thread:
>
> int* fdp = malloc(sizeof(int));
> *fdp = connfd;
> pthread_create(&tid, NULL, readit, fdp);

Or just cast the darn value to a pointer. Save this trick for when the argument
list grows to include more than one value.

[Keith Thompson]

But (u)intptr_t won't exist on systems where no integer type is
wide enough to hold a converted pointer value without loss of
information, so code that depends on it is non-portable. But if
you insist on passing an integer value in a pointer argument,
uintptr_t or intptr_t is the way to go.

In this case, the intent of the "arg" argument to pthread_create()
is that it should be a pointer to data needed by the start routine.

--
Keith Thompson (The_Other_Keith) ks...@mib.org <http://www.ghoti.net/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Kaz Kylheku]

P.S. what is more important: having an allergic reaction to
pointer-integer casts? Or checking the return value of malloc.

:)

[Kaz Kylheku]

On 2011-11-10, Kaz Kylheku <k...@kylheku.com> wrote:

> On 2011-11-10, Quentin Pope <qp1...@hotmail.NOSPAM.com> wrote:
>> What is the best way to handle this warning:
>>
>> warning: cast from pointer to integer of different size
>
> Use an integer of a suitable size.
>
>> I am casting in and out of a function that requires a pointer type. I
>> am casting an integer as a pointer, but the pointer is 8 bytes while
>> the integer is only 4 bytes.
>
> In the C99 language, there is the intptr_t type for this sort of thing.
>
> For older dialects of C, you have to cob together some kind of compile
> time configuration. Define the type as a typedef and change it based
> on target compiler and platform.
>
> I've written a configure script which detects what integral type is
> suitable for holding pointers (based on size) and throws the info
> into a "config.h" header.

But, by the way, I would not bother with this for something so trivial.

Here, your pointer type is larger than the integer you are trying to
pass through, so it doesn't matter.

Just kill the warning for that source file.

[James Kuyper]

On 11/10/2011 03:51 PM, Kaz Kylheku wrote:
> On 2011-11-10, Lauri Alanko <l...@iki.fi> wrote:
>> In article <j9hbnp$13g$1...@speranza.aioe.org>,
>> Quentin Pope <qp1...@hotmail.NOSPAM.com> wrote:
>>> pthread_create(&tid, NULL, readit, (void *)(long)connfd)
>>>
>>> I need to pass the file descriptor as a void * for pthread_create. But
>>> I need the file descriptor as an integer to read and write to it. Is the
>>> above the best approach to turn off the warning?
>>
>> No.
>>
>> Just pass a pointer to the fd:
>>
>> pthread_create(&tid, NULL, readit, &connfd);
>
> Ths is an inferior approach because now the object &connfd has
> to remain stable for as long as the created thread wants to touch
> that object.
>
>> Then dereference the pointer in the readit function to get the
>> descriptor. If connfd is a local variable, you need to make sure that
>> connfd doesn't exit the scope before the new thread has read it.

...

> Or you could just cast the darn value to a pointer!

Unless you're worried about the possibility that the result of the
conversion is a trap representation, as is explicitly allowed by the C
standard. Does POSIX say something to guarantee that this is not a
possibility?

[Kaz Kylheku]

This type of concern has no economic consequence.

[Joe Pfeiffer]

Why do you need to pass the file descriptor as a void*? Why not pass a
pointer to the file descriptor?

[James Kuyper]

On 11/10/2011 04:35 PM, Kaz Kylheku wrote:
> On 2011-11-10, James Kuyper <james...@verizon.net> wrote:
...

>> Unless you're worried about the possibility that the result of the
>> conversion is a trap representation, as is explicitly allowed by the C
>> standard. Does POSIX say something to guarantee that this is not a
>> possibility?
>
> This type of concern has no economic consequence.

If trap representations are in fact possible, programs can radically
malfunction (if they can't malfunction, it isn't really a trap
representation). That often can have economic consequences, depending
upon the nature of the malfunction. Therefore, your response, if
accurate, would seem to be a pointlessly complicated way of saying
"Yes". If you meant something else, it needs more (and clearer) explanation.

[Ian Collins]

On 11/11/11 09:21 AM, Quentin Pope wrote:
> What is the best way to handle this warning:
>
> warning: cast from pointer to integer of different size
>
> I am casting in and out of a function that requires a pointer type. I
> am casting an integer as a pointer, but the pointer is 8 bytes while
> the integer is only 4 bytes.
>
> Here's an example function:
>
> pthread_create(&tid, NULL, readit, (void *)(long)connfd)
>
> I need to pass the file descriptor as a void * for pthread_create. But
> I need the file descriptor as an integer to read and write to it. Is the
> above the best approach to turn off the warning?

No. just pass the address of the descriptor.

> I am considering just
> typecasting to size_t before typecasting to the pointer. Is that a
> better approach than typecasting to long since size_t should remain equal
> to the size of pointers?

Pointers aren't actors.

> Of course, I run into the reverse problem when I have to close the
> file descriptor.
>
> close((int) arg);
>
> I don't even know what to do in the above case. I shouldn't be
> truncating anything, so I should be okay.
>
> I originally wrote the code using a 32 bit version, so I am receiving
> these errors when I ported it over to a 64 bit version.

Hacks invariably come back and bite you in the arse!

--
Ian Collins

[Kaz Kylheku]

Tankan --- Effective Kanji Memorization http://kylheku.com/tankan

[Kaz Kylheku]

On 2011-11-10, James Kuyper <james...@verizon.net> wrote:
> On 11/10/2011 04:35 PM, Kaz Kylheku wrote:
>> On 2011-11-10, James Kuyper <james...@verizon.net> wrote:
> ...
>>> Unless you're worried about the possibility that the result of the
>>> conversion is a trap representation, as is explicitly allowed by the C
>>> standard. Does POSIX say something to guarantee that this is not a
>>> possibility?
>>
>> This type of concern has no economic consequence.
>
> If trap representations are in fact possible, programs can radically
> malfunction (if they can't malfunction, it isn't really a trap
> representation). That often can have economic consequences, depending
> upon the nature of the malfunction.

Before you can discover that this POSIX-threaded program malfunctions in this
way, you have to get a POSIX implementation running on a chip which has these
trap representations.

The economic consequence is that there will be so many problems with this,
leading to so many delays, that the platform will be left without customers,
and soon without investors.

Where can order an evaluation board for a chip with trap representations, and a
POSIX runtime and toolchain?

[christian.bau]

On Nov 10, 8:21 pm, Quentin Pope <qp19...@hotmail.NOSPAM.com> wrote:
> What is the best way to handle this warning:
>
> warning: cast from pointer to integer of different size

On MacOS X 64 bit, an int or unsigned int cast to a pointer will
_never_ produce a valid pointer. So the warning is very much
justified. Now imagine you wanted to pass a long double. Or a long
long on a system with 32 bit pointers. Your approach just cannot work
in these cases.

The safe thing to do is to allocate some memory with malloc, store
whatever you want to pass in the allocated memory, and pass the
pointer, so someone else can then safely pick up the data.

[Kaz Kylheku]

On 2011-11-10, christian.bau <christ...@cbau.wanadoo.co.uk> wrote:
> On Nov 10, 8:21 pm, Quentin Pope <qp19...@hotmail.NOSPAM.com> wrote:
>> What is the best way to handle this warning:
>>
>> warning: cast from pointer to integer of different size
>
> On MacOS X 64 bit, an int or unsigned int cast to a pointer will
> _never_ produce a valid pointer. So the warning is very much
> justified.

The warning is about a different size, not about never producing
a valid pointer.

> Now imagine you wanted to pass a long double.

Then you'd be writing a different program, requiring a different
approach.

Conversions between pointer and long double are not even a standard feature of
the C language; a diagnostic is required.

> Or a long
> long on a system with 32 bit pointers.

In what situation would you need a POSIX file descriptor to be held in a long
long? Descriptor values are allocated into the smallest available position, so
if your program is ever to achieve a file descriptor value of N, it must at
least momentarily create a situation where there N+1 descriptors simultaneously
open.

Single Unix Spec Issue 6, on the subject of the open function:

``Upon successful completion, the function shall open the file and return a
non-negative integer representing the lowest numbered unused file descriptor.
Otherwise, -1 shall be returned and errno set to indicate the error. No files
shall be created or modified if the function returns -1.''

> Your approach just cannot work
> in these cases.

Casting a small long long value to a 32 bit pointer and back to recover
that value is something that is /widely/ portable.

> The safe thing to do is to allocate some memory with malloc, store
> whatever you want to pass in the allocated memory, and pass the
> pointer, so someone else can then safely pick up the data.

There is nothing unsafe about the conversions when they work.

They're only unsafe on some fictitious machine where you get a trap
representation, etc.

"Less than maximally portable" and "unsafe" are totally different concepts.

Getting it running on the weird machine is the porter's responsibility. He
hopes to make X dollars by porting the program to a weird platform Y, guessing
that he can do so with an investment of Z dollars in an amount of time T and
takes his chances accordingly.

[Richard Damon]

Look at the 16 bit protected mode x86 architecture, admittedly a bit old
now, but was a mainstay architecture, and I do believe did have POSIX
available (and in fact was probably at one time a dominate architecture
for it). In this model, far pointers (which were the default for some
memory models) consisted of a 16 bit selector and a 16 bit offset. The
typical instruction to manipulate a data pointer would load the pointer
into the registers in a way that caused the processor to check the
validity of the selector. (An all 0 selector would always be valid, so a
NULL pointer didn't flag, so small integral values would be safe).
Converting an arbitrary 32 bit value gave you a very good chance of
generating a Segment Fault when you accessed the pointer.

You actually can run into this same issue with 32 bit x86 processors, it
is just that "far" pointers are so much rarer that you aren't apt to
misuse it this way, and my guess the current x64 mode would do the same
if you used a far pointer.

[Keith Thompson]

Kaz Kylheku <k...@kylheku.com> writes:
> On 2011-11-10, christian.bau <christ...@cbau.wanadoo.co.uk> wrote:
>> On Nov 10, 8:21 pm, Quentin Pope <qp19...@hotmail.NOSPAM.com> wrote:
>>> What is the best way to handle this warning:
>>>
>>> warning: cast from pointer to integer of different size
>>
>> On MacOS X 64 bit, an int or unsigned int cast to a pointer will
>> _never_ produce a valid pointer. So the warning is very much
>> justified.
>
> The warning is about a different size, not about never producing
> a valid pointer.

Ok, but casting to a different size could well be a problem *because*
it produces an invalid pointer (among other reasons).

>> Now imagine you wanted to pass a long double.
>
> Then you'd be writing a different program, requiring a different
> approach.
>
> Conversions between pointer and long double are not even a standard feature of
> the C language; a diagnostic is required.

Are you sure about that? What constraint does such a conversion
(which must be written as a cast) violate?

C99 6.5.4 "Cast operators" says that the operand and target type must
both have scalar types (which covers pointers and long double).

6.3.2.3 (pointer conversions) doesn't define the behavior of converting
between pointers and floating-point types.

I believe the behavior of a conversion from long double to void* is
undefined by omission.

(Data point: gcc rejects such a cast with an error message; this is a
valid response to undefined behavior according to the note in 3.4.3).

>> Or a long
>> long on a system with 32 bit pointers.
>
> In what situation would you need a POSIX file descriptor to be held in a long
> long? Descriptor values are allocated into the smallest available position, so
> if your program is ever to achieve a file descriptor value of N, it must at
> least momentarily create a situation where there N+1 descriptors simultaneously
> open.

Obviously that would be in a situation where you're passing something
other than a POSIX file descriptor.

[...]

[christian.bau]

On Nov 10, 11:41 pm, Kaz Kylheku <k...@kylheku.com> wrote:

> Casting a small long long value to a 32 bit pointer and back to recover
> that value is something that is /widely/ portable.

Casting an integer that doesn't contain any address information to a
pointer is always a hack. The OP is worried because the compiler gave
a warning about the hack. I explained why this warning is a very
sensible warning indeed, and there have been various posts explaining
how the poster can solve this problem with completely clean and
unhacked code.

Now an optimizing compiler could easily conclude that producing a data
pointer value that is not a null pointer or a pointer to an object or
past the last byte of an object is always undefined behavior, and on
MacOS X 64 bit and probably on other 64 bit systems casting an int to
a pointer will _always_ invoke undefined behavior unless the int has a
value of 0, and that therefore the result is always a null pointer.
gcc tends to do that kind of optimizations and break "widely portable"
undefined behavior.

[Lauri Alanko]

In article <4eeab940-418e-4fd9...@j10g2000vbe.googlegroups.com>,

christian.bau <christ...@cbau.wanadoo.co.uk> wrote:
> on
> MacOS X 64 bit and probably on other 64 bit systems casting an int to
> a pointer will _always_ invoke undefined behavior unless the int has a
> value of 0, and that therefore the result is always a null pointer.

On MacOS X the result may indeed be a null pointer, but, as discussed
previously (the thread of <ijo6qc$kql$1...@oravannahka.helsinki.fi>),
this behavior does not seem to be guaranteed by the standard.


Lauri

[James Kuyper]

It is not.
--
James Kuyper

[William Ahern]

Ian Collins <ian-...@hotmail.com> wrote:
> On 11/11/11 09:21 AM, Quentin Pope wrote:

<snip>

> > pthread_create(&tid, NULL, readit, (void *)(long)connfd)
> >
> > I need to pass the file descriptor as a void * for pthread_create. But
> > I need the file descriptor as an integer to read and write to it. Is the
> > above the best approach to turn off the warning?

> No. just pass the address of the descriptor.

Why do people keep suggesting this? Pass-by-reference and threads aren't
known to play well. IMHO the OP had sensible intuition to pass the
descriptor by value. It may not be pretty in C, but neither is threaded
programming.

[Ian Collins]

On 11/15/11 08:10 AM, William Ahern wrote:
> Ian Collins<ian-...@hotmail.com> wrote:
>> On 11/11/11 09:21 AM, Quentin Pope wrote:
> <snip>
>>> pthread_create(&tid, NULL, readit, (void *)(long)connfd)
>>>
>>> I need to pass the file descriptor as a void * for pthread_create. But
>>> I need the file descriptor as an integer to read and write to it. Is the
>>> above the best approach to turn off the warning?
>
>> No. just pass the address of the descriptor.
>
> Why do people keep suggesting this?

If OP had, this thread would never have been born.

> Pass-by-reference and threads aren't
> known to play well. IMHO the OP had sensible intuition to pass the
> descriptor by value. It may not be pretty in C, but neither is threaded
> programming.

As long as the address is still valid when a thread dereferences it,
what's the problem?

--
Ian Collins

[James Kuyper]

On 11/14/2011 02:28 PM, Ian Collins wrote:
> On 11/15/11 08:10 AM, William Ahern wrote:

...

>> Pass-by-reference and threads aren't
>> known to play well. IMHO the OP had sensible intuition to pass the
>> descriptor by value. It may not be pretty in C, but neither is threaded
>> programming.
>
> As long as the address is still valid when a thread dereferences it,
> what's the problem?

The possibility that the address might NOT still be valid at the time of
use. This has already been pointed out, and is explicitly considered
(but somehow ignored as unimportant) in your own response.
A relevant response would give reasons justifying your treatment of that
possibility as unimportant.

[Keith Thompson]

"christian.bau" <christ...@cbau.wanadoo.co.uk> writes:
[...]

> Now an optimizing compiler could easily conclude that producing a data
> pointer value that is not a null pointer or a pointer to an object or
> past the last byte of an object is always undefined behavior, and on
> MacOS X 64 bit and probably on other 64 bit systems casting an int to
> a pointer will _always_ invoke undefined behavior unless the int has a
> value of 0, and that therefore the result is always a null pointer.
> gcc tends to do that kind of optimizations and break "widely portable"
> undefined behavior.

This:

void *ptr = 0;

is guaranteed to set ptr to a null pointer value, because the constant
0 is a null pointer constant. This:

int var = 0;
void *ptr = (int*)var;

is not (though it's likely to do so under most implementations).

[Ian Collins]

On 11/15/11 08:48 AM, James Kuyper wrote:
> On 11/14/2011 02:28 PM, Ian Collins wrote:
>> On 11/15/11 08:10 AM, William Ahern wrote:

> ....

>>> Pass-by-reference and threads aren't
>>> known to play well. IMHO the OP had sensible intuition to pass the
>>> descriptor by value. It may not be pretty in C, but neither is threaded
>>> programming.
>>
>> As long as the address is still valid when a thread dereferences it,
>> what's the problem?
>
> The possibility that the address might NOT still be valid at the time of
> use. This has already been pointed out, and is explicitly considered
> (but somehow ignored as unimportant) in your own response.

I had assumed a competent programmer would be aware of the issue. I had
further assumed a programmer writing threaded applications to be
competent. If either of those assumptions are untrue, the programmer's
foray into threaded programming will be brief and painful.

> A relevant response would give reasons justifying your treatment of that
> possibility as unimportant.

Where did I say it was unimportant? Do you expect me to qualify every
mention of passing an address?

--
Ian Collins

[jacob navia]

Le 14/11/11 22:15, Ian Collins a écrit :

> On 11/15/11 08:48 AM, James Kuyper wrote:
>> On 11/14/2011 02:28 PM, Ian Collins wrote:
>>> On 11/15/11 08:10 AM, William Ahern wrote:
>> ....
>>>> Pass-by-reference and threads aren't
>>>> known to play well. IMHO the OP had sensible intuition to pass the
>>>> descriptor by value. It may not be pretty in C, but neither is threaded
>>>> programming.
>>>
>>> As long as the address is still valid when a thread dereferences it,
>>> what's the problem?
>>
>> The possibility that the address might NOT still be valid at the time of
>> use. This has already been pointed out, and is explicitly considered
>> (but somehow ignored as unimportant) in your own response.
>
> I had assumed a competent programmer would be aware of the issue. I had
> further assumed a programmer writing threaded applications to be
> competent. If either of those assumptions are untrue, the programmer's
> foray into threaded programming will be brief and painful.
>

Well, it is not a matter of competence Ian, calm down. I think Mr Kuyper
wants to point out that a solution passing some pointer instead of a
value provokes further complexities that obviously *could* be handled
by a competent programmer but that complexify the interface of that
function...

If I understood him correctly that is.

[Ian Collins]

On 11/15/11 10:18 AM, jacob navia wrote:
> Le 14/11/11 22:15, Ian Collins a �crit :

:)

I'm quite calm. I just had a strong urge to be pompous...

As an aside, the problem (a dangling pointer) tends not to happen with
pthreads because the default behaviour of pthread_create is to create a
joinable thread. So I'd also assumed there would be pthread_join
further on in the code.

--
Ian Collins

[James Kuyper]

On 11/14/2011 04:15 PM, Ian Collins wrote:
> On 11/15/11 08:48 AM, James Kuyper wrote:
>> On 11/14/2011 02:28 PM, Ian Collins wrote:
>>> On 11/15/11 08:10 AM, William Ahern wrote:
>> ....
>>>> Pass-by-reference and threads aren't
>>>> known to play well. IMHO the OP had sensible intuition to pass the
>>>> descriptor by value. It may not be pretty in C, but neither is threaded
>>>> programming.
>>>
>>> As long as the address is still valid when a thread dereferences it,
>>> what's the problem?
>>
>> The possibility that the address might NOT still be valid at the time of
>> use. This has already been pointed out, and is explicitly considered
>> (but somehow ignored as unimportant) in your own response.
>
> I had assumed a competent programmer would be aware of the issue. I had
> further assumed a programmer writing threaded applications to be
> competent. If either of those assumptions are untrue, the programmer's
> foray into threaded programming will be brief and painful.

That second assumption seems unjustified to me, and I would say the same
about your assumption that the painful foray will be short. I've seen an
incompetent programmer spend several long painful months trying to deal
with problems caused by a much simpler misunderstanding (and I'm not
talking about Bill Cunningham). If the particular programmer I'm
thinking of had bothered spending 5 minutes asking me questions, that
waste could have been avoided, but I had no idea that he was wasting his
time that way. He was under orders to ask me questions if he ran into
trouble, but never did so - he never gave a coherent explanation of why
he didn't.

However, neither assumption seems to be relevant here. As long as this
is an issue that the programmer needs to be aware of, the need for such
awareness IS the problem that you're asking about.

>> A relevant response would give reasons justifying your treatment of that
>> possibility as unimportant.
>
> Where did I say it was unimportant?

You asked "what's the problem?" in the same sentence where you conceded
both the existence and relevance of the problem you were asking about.
The simplest explanation I could come up with for such a seemingly
bizarre combination was that you considered the problem you mentioned to
be unimportant, and were challenging people to come up with a more
important one, with the expectation that they would not succeed. I'll
concede that there might be other reasons for writing a sentence with
that structure, but the variety of those other possible reasons is
great, and the probability that any particular one of them is justified
is small. It would improve my understanding of the point of your message
if you could explain your reason for posting such a question.

[Ian Collins]

On 11/15/11 10:46 AM, James Kuyper wrote:
> On 11/14/2011 04:15 PM, Ian Collins wrote:
>> On 11/15/11 08:48 AM, James Kuyper wrote:
>
>>> A relevant response would give reasons justifying your treatment of that
>>> possibility as unimportant.
>>
>> Where did I say it was unimportant?
>
> You asked "what's the problem?" in the same sentence where you conceded
> both the existence and relevance of the problem you were asking about.

In the same vein as "as long as you hold the knife by the handle, what's
the problem?".

> The simplest explanation I could come up with for such a seemingly
> bizarre combination was that you considered the problem you mentioned to
> be unimportant, and were challenging people to come up with a more
> important one, with the expectation that they would not succeed. I'll
> concede that there might be other reasons for writing a sentence with
> that structure, but the variety of those other possible reasons is
> great, and the probability that any particular one of them is justified
> is small. It would improve my understanding of the point of your message
> if you could explain your reason for posting such a question.

Wow, I consider my pomposity well and truly trumped!

--
Ian Collins

[James Kuyper]

On 11/14/2011 04:58 PM, Ian Collins wrote:
> On 11/15/11 10:46 AM, James Kuyper wrote:
>> On 11/14/2011 04:15 PM, Ian Collins wrote:
>>> On 11/15/11 08:48 AM, James Kuyper wrote:
>>
>>>> A relevant response would give reasons justifying your treatment of that
>>>> possibility as unimportant.
>>>
>>> Where did I say it was unimportant?
>>
>> You asked "what's the problem?" in the same sentence where you conceded
>> both the existence and relevance of the problem you were asking about.
>
> In the same vein as "as long as you hold the knife by the handle, what's
> the problem?".

I would interpret a question such as that as one which dismisses the
importance of having to hold the knife by the handle, because it's easy
to arrange to do so. I had assumed you were doing the same: implicitly
asserting that it's trivial to arrange that the address remains valid; I
was rather surprised when you vehemently denied that inference. I had
expected an explanation of why it is trivial.

[Ben Pfaff]

Sure it's trivial. In the old thread, malloc(sizeof(int)), put
the file descriptor there and pass the malloc'd pointer. In the
new thread, dereference the pointer to get the file descriptor,
then free it.
--
char a[]="\n .CJacehknorstu";int putchar(int);int main(void){unsigned long b[]
={0x67dffdff,0x9aa9aa6a,0xa77ffda9,0x7da6aa6a,0xa67f6aaa,0xaa9aa9f6,0x11f6},*p
=b,i=24;for(;p+=!*p;*p/=4)switch(0[p]&3)case 0:{return 0;for(p--;i--;i--)case+
2:{i++;if(i)break;else default:continue;if(0)case 1:putchar(a[i&15]);break;}}}

[Ian Collins]

I think our opinions on what constitutes common sense differ somewhat.
I see not passing a pointer to something that has gone out of scope in
the same way as not holding a knife by the blade. In both cases, doing
the consequences of doing the wrong thing tend to deter a repeat.

Or maybe I've just been doing this stuff for way to long...

--
Ian Collins

[Joe Pfeiffer]

Ian Collins <ian-...@hotmail.com> writes:

> On 11/15/11 08:48 AM, James Kuyper wrote:
>> On 11/14/2011 02:28 PM, Ian Collins wrote:
>>> On 11/15/11 08:10 AM, William Ahern wrote:
>> ....
>>>> Pass-by-reference and threads aren't
>>>> known to play well. IMHO the OP had sensible intuition to pass the
>>>> descriptor by value. It may not be pretty in C, but neither is threaded
>>>> programming.
>>>
>>> As long as the address is still valid when a thread dereferences it,
>>> what's the problem?
>>
>> The possibility that the address might NOT still be valid at the time of
>> use. This has already been pointed out, and is explicitly considered
>> (but somehow ignored as unimportant) in your own response.
>
> I had assumed a competent programmer would be aware of the issue. I
> had further assumed a programmer writing threaded applications to be
> competent. If either of those assumptions are untrue, the
> programmer's foray into threaded programming will be brief and
> painful.

Painful, anyway :)

[James Kuyper]

On 11/14/2011 05:29 PM, Ian Collins wrote:
...

> I think our opinions on what constitutes common sense differ somewhat.

Probably; I think that the existence of "common sense" is a popular
delusion, based upon people projecting onto others their own
understanding of things. People suffering from that delusion often hold
those others to blame for the failure of that projection to match reality.

> I see not passing a pointer to something that has gone out of scope in
> the same way as not holding a knife by the blade. In both cases, doing
> the consequences of doing the wrong thing tend to deter a repeat.
>
> Or maybe I've just been doing this stuff for way to long...

Attempting to dereference a pointer to memory that has already been
deallocated is one of the most common nasty mistakes made by C
programmers. There are many other mistakes that are more common, but
most of them are easier to diagnose than that one. I'm no expert in
multi-threaded code, but it seems to me that the problem can only be
worse if the dynamically allocated memory is shared between threads.

Several participants on this thread suggested that this was a serious
problem, you've suggested that its trivial. I think that they may be
exaggerating the difficulty, while you may be doing the opposite; I was
hoping to provoke a more detailed discussion that would clarify the
discrepancy. Instead I seem to have only provoked a discussion about my
provocation.

[Seebs]

On 2011-11-14, James Kuyper <james...@verizon.net> wrote:
> Probably; I think that the existence of "common sense" is a popular
> delusion, based upon people projecting onto others their own
> understanding of things. People suffering from that delusion often hold
> those others to blame for the failure of that projection to match reality.

I tend towards this view, myself.

If nothing else, I do not consider it safe to assume that generic programmers
not previously verified to understand the issues have any awareness of the
issues of passing the address of "some object somewhere" into a thread
mechanism or the like.

Heck.

Consider:

struct foo *f = malloc(sizeof(*f));
free(f);
f->x = 1;

I do not consider it a reasonable assumption that an arbitrary person
with 3-5 years or more of experience programming C will see anything
wrong here. I've seen someone who had >10 years claimed experience and
got promoted to lead engineer for a project who insisted that this was
not only permissible but the only way to avoid memory leaks in C.

... And yes, the project failed utterly and everyone involved lost their
jobs. Funny only because it was over a decade ago. :)

It is very easy for people who have finally been taught that it's not okay
to return the address of a local variable to conclude that it's fine to
pass the address of a local variable to a function you call. And you're
*calling* pthread_create(), see. So that's fine.

Worse yet if they have done a lot of work in a language with closures. :)

-s
--
Copyright 2011, all wrongs reversed. Peter Seebach / usenet...@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.

[Ian Collins]

On 11/15/11 12:14 PM, James Kuyper wrote:
> On 11/14/2011 05:29 PM, Ian Collins wrote:

> ....

>> I think our opinions on what constitutes common sense differ somewhat.
>
> Probably; I think that the existence of "common sense" is a popular
> delusion, based upon people projecting onto others their own
> understanding of things. People suffering from that delusion often hold
> those others to blame for the failure of that projection to match reality.

You're probably right there!

>> I see not passing a pointer to something that has gone out of scope in
>> the same way as not holding a knife by the blade. In both cases, doing
>> the consequences of doing the wrong thing tend to deter a repeat.
>>
>> Or maybe I've just been doing this stuff for way to long...
>
> Attempting to dereference a pointer to memory that has already been
> deallocated is one of the most common nasty mistakes made by C
> programmers. There are many other mistakes that are more common, but
> most of them are easier to diagnose than that one. I'm no expert in
> multi-threaded code, but it seems to me that the problem can only be
> worse if the dynamically allocated memory is shared between threads.

Sharing dynamically allocation memory between threads isn't really any
mode dangerous than using it in single threaded applications. The same
care has to be taken in determining where the memory is released. It
goes without saying the problem largely evaporates in C++ with smart
pointers.

> Several participants on this thread suggested that this was a serious
> problem, you've suggested that its trivial. I think that they may be
> exaggerating the difficulty, while you may be doing the opposite; I was
> hoping to provoke a more detailed discussion that would clarify the
> discrepancy. Instead I seem to have only provoked a discussion about my
> provocation.

I didn't so much suggest it was trivial, only that is was no worse than
any other situation with pointers. If you understand threading that is!

--
Ian Collins

[Phil Carmody]

James Kuyper <james...@verizon.net> writes:
> On 11/14/2011 04:58 PM, Ian Collins wrote:
> > On 11/15/11 10:46 AM, James Kuyper wrote:
> >> On 11/14/2011 04:15 PM, Ian Collins wrote:
> >>> On 11/15/11 08:48 AM, James Kuyper wrote:
> >>
> >>>> A relevant response would give reasons justifying your treatment of that
> >>>> possibility as unimportant.
> >>>
> >>> Where did I say it was unimportant?
> >>
> >> You asked "what's the problem?" in the same sentence where you conceded
> >> both the existence and relevance of the problem you were asking about.
> >
> > In the same vein as "as long as you hold the knife by the handle, what's
> > the problem?".
>
> I would interpret a question such as that as one which dismisses the
> importance of having to hold the knife by the handle, because it's easy
> to arrange to do so. I had assumed you were doing the same: implicitly
> asserting that it's trivial to arrange that the address remains valid;

By default, addresses remain valid unless you explicitly do something
that makes them invalid.

Phil
--
Unix is simple. It just takes a genius to understand its simplicity
-- Dennis Ritchie (1941-2011), Unix Co-Creator

[Phil Carmody]

Seebs <usenet...@seebs.net> writes:
> Consider:
>
> struct foo *f = malloc(sizeof(*f));
> free(f);
> f->x = 1;
>
> I do not consider it a reasonable assumption that an arbitrary person
> with 3-5 years or more of experience programming C will see anything
> wrong here.

Astounding. And I say that as someone who's spent a long time in maintenance
fixing major fuck-ups by people who just didn't get it. But not one of them
(even the "we've read /Design Patterns/, we don't have to care about O(n^2)
getting huge" crowd, and even the "shift by N bits by shifting by 1 bit N-1
times" dude) would have done *that*.

I hope you're exagerating. I guess I should get a nice warm feeling of
comfort that ${DAYJOB} really is about as great a bunch of coders as
you could ever hope to work with.

[James Kuyper]

On 11/16/2011 03:23 AM, Phil Carmody wrote:
> James Kuyper <james...@verizon.net> writes:
>> On 11/14/2011 04:58 PM, Ian Collins wrote:

...

>>> In the same vein as "as long as you hold the knife by the handle, what's
>>> the problem?".
>>
>> I would interpret a question such as that as one which dismisses the
>> importance of having to hold the knife by the handle, because it's easy
>> to arrange to do so. I had assumed you were doing the same: implicitly
>> asserting that it's trivial to arrange that the address remains valid;
>
> By default, addresses remain valid unless you explicitly do something
> that makes them invalid.

Off-hand I can think of three ways by which previously valid addresses
can be come invalid, and three of them are quite explicit: free(),
realloc(), and fclose().
However, the addresses of variables with automatic storage duration
become invalid as soon as execution of the block in which they are
defined ends; that seems pretty implicit to me. YMMV

I hope you're not suggesting that the fact that free(), realloc() and
fclose() can only occur explicitly somehow makes it trivial to ensure
that they have not yet been called for a given pointer? If that were the
case, garbage collection wouldn't be as popular an extension as it is.
Even in single-threaded code it takes a certain amount of discipline to
ensure that a pointer is not used after the memory it points to has been
free()d.

When possible, I try to make sure that pointers into dynamically
allocated memory are stored only in automatic objects whose lifetime
ends immediately after the call to free(), rendering it impossible for
such problems to occur. However, it's not always possible to impose such
restrictions. When I can't do that, I insert code setting the pointer to
NULL immediately after the free(). However, to be helpful, that requires
that I protect all problematic uses of such a pointer with tests to
determine whether it's null.

When the expressions p=malloc(), *p=3, x=*p, and free(p) occur in two or
more different threads executing asynchronously, it seems to me that it
would be all that much harder to ensure that they occur only in the
proper order. I'm not saying it's impossible - just that it's harder.
Though Ian has assured me that this is not the case, I don't quite see
how that could be.
--
James Kuyper

[James Kuyper]

On 11/16/2011 06:53 AM, James Kuyper wrote:
...

> Off-hand I can think of three ways by which previously valid addresses

Yes, I can count. However, when I decided to add realloc() to the list,
I forgot to update that to "four".

[Ben Bacarisse]

For even greater generality you could replace "variables" with "objects"
and you'd be including compound literals that appear inside a function.

Oh, reading on I see "defined" which does fit for CLs. "Created" maybe?

--
Ben.

[Seebs]

On 2011-11-16, Phil Carmody <thefatphi...@yahoo.co.uk> wrote:
> Seebs <usenet...@seebs.net> writes:
>> Consider:
>>
>> struct foo *f = malloc(sizeof(*f));
>> free(f);
>> f->x = 1;
>>
>> I do not consider it a reasonable assumption that an arbitrary person
>> with 3-5 years or more of experience programming C will see anything
>> wrong here.

> Astounding. And I say that as someone who's spent a long time in maintenance
> fixing major fuck-ups by people who just didn't get it. But not one of them
> (even the "we've read /Design Patterns/, we don't have to care about O(n^2)
> getting huge" crowd, and even the "shift by N bits by shifting by 1 bit N-1
> times" dude) would have done *that*.

You would be amazed.

> I hope you're exagerating. I guess I should get a nice warm feeling of
> comfort that ${DAYJOB} really is about as great a bunch of coders as
> you could ever hope to work with.

Nope, not exaggerating. Couple jobs back, there was this guy who had
on paper a decade of C experience, who absolutely *insisted* that this
had to work, and was the only reasonable way to avoid memory leaks.

And he got promoted to being in charge of stuff.

But yeah. It is sort of astounding, but then... I read a lot of code,
and it is full of nonsense. Today's:

*p++ = toupper((unsigned char) *p);

This is in code that's widely used, and gcc even warns about it... If
you compile at -O0. Which most people don't.

In another fairly famous piece of software, we came across:

unsigned char x;

x = NULL;

which, again, "had worked most of the time".

I guess that's the thing. Most code is not subject to serious technical
review, and most programmers are sorta shoddy. Heck, I do horrible stuff,
it's just that I usually get it caught in review, or I catch it myself
if I'm lucky.

[Fritz Wuehler]

> Seebs <usenet...@seebs.net> writes:
> > Consider:
> >
> > struct foo *f = malloc(sizeof(*f));
> > free(f);
> > f->x = 1;
> >
> > I do not consider it a reasonable assumption that an arbitrary person
> > with 3-5 years or more of experience programming C will see anything
> > wrong here.

Let's try something else. I have a lot of experience in programming but I
have written about 10 lines of C (and no Java, C++ or anything related) and
not had to work with it or close to it ever.

Does this code fragment mean

1. allocate some storage
2. free it
3. assign a value to the area pointed to by the pointer to the storage you just
freed

If so that won't work forever, especially not in a multiuser or
multiprocessing environment. It might work for a long time on a single user
DOS though. Where is this code supposed to run?

If I got it right you don't need 3-5 years of programming in C to realize
the problem you just need to have a clue. If I am wrong then I will probably
need 3-5 years of programming in C to understand the issue!

Btw I don't understand the nuances of the first line, does it mean define a
structure called foo that contains a pointer to storage enough for one
pointer to a structure named foo that contains only a pointer?

[pete]

Fritz Wuehler wrote:
>
> > Seebs <usenet...@seebs.net> writes:
> > > Consider:
> > >
> > > struct foo *f = malloc(sizeof(*f));

> Btw I don't understand the nuances of the first line,
> does it mean define a
> structure called foo that contains a pointer to storage enough for one
> pointer to a structure named foo that contains only a pointer?

Close.

It means that struct foo has been previously defined,

and that (f) is a pointer to that type
and that the malloc call will
either
allocate enough memory for one structure of that type
and initialize (f) with the address of that memory
or
malloc will not allocate memory
and will initialize (f) with a null pointer value.

--
pete

[Seebs]

On 2011-11-17, Fritz Wuehler <fr...@spamexpire-201111.rodent.frell.theremailer.net> wrote:
>> Seebs <usenet...@seebs.net> writes:
>> > Consider:

>> > struct foo *f = malloc(sizeof(*f));
>> > free(f);
>> > f->x = 1;

>> > I do not consider it a reasonable assumption that an arbitrary person
>> > with 3-5 years or more of experience programming C will see anything
>> > wrong here.

> Let's try something else. I have a lot of experience in programming but I
> have written about 10 lines of C (and no Java, C++ or anything related) and
> not had to work with it or close to it ever.

> Does this code fragment mean

> 1. allocate some storage
> 2. free it
> 3. assign a value to the area pointed to by the pointer to the storage you just
> freed

Yup.

> If so that won't work forever, especially not in a multiuser or
> multiprocessing environment. It might work for a long time on a single user
> DOS though. Where is this code supposed to run?

It was running originally on SunOS, which is Unixy, and in which the
default behavior (everything involved was single-threaded) was basically
that the pointer would probably stay valid for a while until something else
got allocated using the same space.

> If I got it right you don't need 3-5 years of programming in C to realize
> the problem you just need to have a clue. If I am wrong then I will probably
> need 3-5 years of programming in C to understand the issue!

Well, yes. And that's my point; it is not reasonable to infer that someone
who has been professionally programming for several years in a language has
any particular clue.

Which is why I think pointing out that there are risks involved in, say,
handing a pointer to allocated memory to another thread in a multithreaded
program is probably reasonable. Yes, all the experienced-and-competent
programmers know that, but lots of people are only one or the other of
those, and lots are neither.

> Btw I don't understand the nuances of the first line, does it mean define a
> structure called foo that contains a pointer to storage enough for one
> pointer to a structure named foo that contains only a pointer?

No, it means declare a pointer-to-struct-foo named 'f', and set the pointer
to the return of malloc(N), where N is "the number of bytes you need to
hold a thing of the same type as *f".

[Kaz Kylheku]

On 2011-11-17, Fritz Wuehler <fr...@spamexpire-201111.rodent.frell.theremailer.net> wrote:

>> Seebs <usenet...@seebs.net> writes:
>> > Consider:
>> >
>> > struct foo *f = malloc(sizeof(*f));
>> > free(f);
>> > f->x = 1;
>> >
>> > I do not consider it a reasonable assumption that an arbitrary person
>> > with 3-5 years or more of experience programming C will see anything
>> > wrong here.
>
> Let's try something else. I have a lot of experience in programming but I
> have written about 10 lines of C (and no Java, C++ or anything related) and
> not had to work with it or close to it ever.
>
> Does this code fragment mean
>
> 1. allocate some storage
> 2. free it
> 3. assign a value to the area pointed to by the pointer to the storage you just
> freed
>
> If so that won't work forever, especially not in a multiuser or
> multiprocessing environment.

In practical terms (how this typically works): the freed object's memory area
is no longer owned by the program, but by the memory allocator. The memory
allocator can put bits there, such as link pointers that put the object on a
free list. If that is how the allocator works, it will typically put those
bits there right away, before the free function returns.

In many memory allocators, in the interests of saving space, allocated objects
do not have any headers. This allows objects to be adjacently allocated with
no wasted space. Only free objects are kept on free lists, and the bookkeeping
pieces such as pointer and flag fields for keeping free lists go inside
the objects, in the same memory where the application's data used to be
when those objects were allocated.

After the program did the above, it might continue executing fine until the
next time it makes any kind of call to the memory allocator. This is true even
if there is no concurrency. The memory allocator might walk its free lists
and hit a bad pointer due to the clobbered memory location.

In ``ISO C language lawyer'' terms, as soon sa free(f) is called, the pointer f
(and all copies of f that the program may have elsewhere) become
``indeterminate'' values. The use of an indeterminately-valued object results
in undefined behavior.

I.e undefined behavior occurs in the expression f->x itself, regardless of the
assignment to that location f->x = 1. The expression f->x uses the pointer f
(to access the place x, or to designate it as an lvalue). This use is undefined
since f is indeterminate.

> It might work for a long time on a single user
> DOS though.

Perhaps, if the program does not ever make another call to functions like
malloc, free, realloc, calloc again.

If the DOS memory allocator is being directly used by malloc, this might screw
up the operating system.

> If I got it right you don't need 3-5 years of programming in C to realize
> the problem you just need to have a clue.
> If I am wrong then I will probably
> need 3-5 years of programming in C to understand the issue!
>
> Btw I don't understand the nuances of the first line, does it mean define a

The snippet is incomplete. The first line assumes that there is a 'struct foo'
type that has been previously defined, which has a member x of integer type (or
some arithmetic type to which 1 can be assigned). The first line could define
the type by including the body of the structure:

struct foo { int x; } *f = malloc(sizeof *f);

This is a declaration with an initializer. The 'struct foo .. { ... }'
is the list of declaration specifiers (containing one specifier: that
for a structure type). The *f part is a declarator, declaring a name f.
The * type construction operator means that f is declared as a pointer
type (a pointer to what? The type produced by the declaration specifier list).

C declarations are split into specifiers and declarators, which allows
multiple declarators to hang off a shared "stem" of specifiers, and
yet declare different kinds of things:

/* x is of type int; y is a pointer to int;
z is a function of no arguments returning int. */

int x, *y, (*z)(void);

Our pointer f is initialized from the return value of a call to the principal C
memory allocation function, malloc. sizeof is an operator for computing the
sizes of types based on object-designating expressions or type expressions, and
*f is an object-designating expression which means "the object obtained by
dereferencing the pointer f". f is not yet initialized, so the object does not
exist, but that doesn't matter because sizeof doesn't use an expression's
value, only its type. The type of *f is "struct foo". So we are asking malloc
for enough bytes to cover the structure.

If you write some_type *p = malloc(sizeof *p), then the size automatically
adjusts itself if you edit something to other_type: you have not
mentioned the type name in two places, as in:
some_type *p = malloc(sizeof (some_type)); /* more error-prone */

[io_x]

"Seebs" <usenet...@seebs.net> ha scritto nel messaggio
news:slrnjc38ub.r7s...@guild.seebs.net...

> On 2011-11-14, James Kuyper <james...@verizon.net> wrote:
>> Probably; I think that the existence of "common sense" is a popular
>> delusion, based upon people projecting onto others their own
>> understanding of things. People suffering from that delusion often hold
>> those others to blame for the failure of that projection to match reality.
>
> I tend towards this view, myself.
>
> If nothing else, I do not consider it safe to assume that generic programmers
> not previously verified to understand the issues have any awareness of the
> issues of passing the address of "some object somewhere" into a thread
> mechanism or the like.
>
> Heck.
>
> Consider:
>
> struct foo *f = malloc(sizeof(*f));
> free(f);
> f->x = 1;
>
> I do not consider it a reasonable assumption that an arbitrary person
> with 3-5 years or more of experience programming C will see anything
> wrong here. I've seen someone who had >10 years claimed experience and
> got promoted to lead engineer for a project who insisted that this was
> not only permissible but the only way to avoid memory leaks in C.

problems of above code in my vew:
0) i cannot write memory just free()
1) when free() is possible that, afther some time some piece of heap
can be free() to the sys: so that code here has segfault
2) when free() the data is put in the free list, in a multitread program
it is possible some other thread has the memory that thread write
3) the same for malloc() in a multithread program
that return &f + 20 if the size of that pointer is in &f->x
in the header of the pointer in the size of pointer area;
so this can make wrong all heap if that heap not check for size too
of each pointer return from malloc()

[Joe Pfeiffer]

Fritz Wuehler <fr...@spamexpire-201111.rodent.frell.theremailer.net>
writes:

>> Seebs <usenet...@seebs.net> writes:
>> > Consider:
>> >
>> > struct foo *f = malloc(sizeof(*f));
>> > free(f);
>> > f->x = 1;
>> >
>> > I do not consider it a reasonable assumption that an arbitrary person
>> > with 3-5 years or more of experience programming C will see anything
>> > wrong here.
>
> Let's try something else. I have a lot of experience in programming but I
> have written about 10 lines of C (and no Java, C++ or anything related) and
> not had to work with it or close to it ever.
>
> Does this code fragment mean
>
> 1. allocate some storage
> 2. free it
> 3. assign a value to the area pointed to by the pointer to the storage you just
> freed
>
> If so that won't work forever, especially not in a multiuser or
> multiprocessing environment. It might work for a long time on a single user
> DOS though. Where is this code supposed to run?

You're right that it won't work forever -- it's going to fail as soon as
that space is malloc'ed again, and written over. Whenever that is.

This has absolutely nothing to do with multiuser or multiprocessing
environments, as processes all have their own address space (subject to
some very specific sharing, which won't have anything to do with the
dynamic memory heap).

> If I got it right you don't need 3-5 years of programming in C to realize
> the problem you just need to have a clue. If I am wrong then I will probably
> need 3-5 years of programming in C to understand the issue!
>
> Btw I don't understand the nuances of the first line, does it mean define a
> structure called foo that contains a pointer to storage enough for one
> pointer to a structure named foo that contains only a pointer?

It means declare a pointer to an object of type 'struct foo' (that's
what the 'struct foo *f' part means, allocate a block of storage
large enough to hold an object of that size, and assign the pointer to
point at the space (that's what the '= malloc(sizeof(*f));' part
means).

'struct foo' must have been defined elsewhere for this to work.

[blmblm.m...@gmail.com]

In article <slrnjc38ub.r7s...@guild.seebs.net>,

Seebs <usenet...@seebs.net> wrote:
> On 2011-11-14, James Kuyper <james...@verizon.net> wrote:
> > Probably; I think that the existence of "common sense" is a popular
> > delusion, based upon people projecting onto others their own
> > understanding of things. People suffering from that delusion often hold
> > those others to blame for the failure of that projection to match reality.
>
> I tend towards this view, myself.
>
> If nothing else, I do not consider it safe to assume that generic programmers
> not previously verified to understand the issues have any awareness of the
> issues of passing the address of "some object somewhere" into a thread
> mechanism or the like.
>
> Heck.
>
> Consider:
>
> struct foo *f = malloc(sizeof(*f));
> free(f);
> f->x = 1;
>
> I do not consider it a reasonable assumption that an arbitrary person
> with 3-5 years or more of experience programming C will see anything
> wrong here. I've seen someone who had >10 years claimed experience and
> got promoted to lead engineer for a project who insisted that this was
> not only permissible but the only way to avoid memory leaks in C.

Remarkable. Just .... remarkable.

For some reason lately the following quotation, attributed to Charles
Babbage, has been coming to mind a lot:

"On two occasions, I have been asked [by members of Parliament],
'Pray, Mr. Babbage, if you put into the machine wrong figures,
will the right answers come out?' I am not able to rightly
apprehend the kind of confusion of ideas that could provoke such
a question." -- Charles Babbage (1791-1871)

Something about that "I am not able to rightly apprehend the kind
of confusion of ideas ...." :-)

> ... And yes, the project failed utterly and everyone involved lost their
> jobs. Funny only because it was over a decade ago. :)
>
> It is very easy for people who have finally been taught that it's not okay
> to return the address of a local variable to conclude that it's fine to
> pass the address of a local variable to a function you call. And you're
> *calling* pthread_create(), see. So that's fine.

Well, now *I*'m the one who's confused .... Are you saying
it's not fine? I suppose if the function you call creates an
independent thread, and that thread persists longer than the one
that created it, yes, there could be a problem .... Was that
your point?

> Worse yet if they have done a lot of work in a language with closures. :)

--
B. L. Massingill
ObDisclaimer: I don't speak for my employers; they return the favor.

[blmblm.m...@gmail.com]

In article <ln8vnn8...@nuthaus.mib.org>,
Keith Thompson <ks...@mib.org> wrote:
> Ben Pfaff <b...@cs.stanford.edu> writes:
> > Quentin Pope <qp1...@hotmail.NOSPAM.com> writes:
> >> What is the best way to handle this warning:
> >>
> >> warning: cast from pointer to integer of different size
> >
> > The best way may be to avoid casting between a pointer and an
> > integer.
> >
> > Another way is to use (u)intptr_t from <stdint.h>, which is
> > ordinarily defined as an integer type that is the same width as a
> > pointer.

[ snip ]
>
> In this case, the intent of the "arg" argument to pthread_create()
> is that it should be a pointer to data needed by the start routine.
>

Which in my thinking is a strong argument for *not* using creative
casting to pass some other kind of data.

Granted that it may take a bit of thought to be sure that both
the pointer and the pointed-to data are stable enough to be
passed to another thread without creating the potential for race
conditions, and maybe that argues against using the parameter as
it was clearly meant to be used. But still, if anyone else is
ever going to read or use this code, well ....

[Seebs]

On 2011-11-17, blmblm myrealbox.com <blmblm.m...@gmail.com> wrote:
> In article <slrnjc38ub.r7s...@guild.seebs.net>,
> Seebs <usenet...@seebs.net> wrote:
>> It is very easy for people who have finally been taught that it's not okay
>> to return the address of a local variable to conclude that it's fine to
>> pass the address of a local variable to a function you call. And you're
>> *calling* pthread_create(), see. So that's fine.

> Well, now *I*'m the one who's confused .... Are you saying
> it's not fine?

Yes.

> I suppose if the function you call creates an
> independent thread, and that thread persists longer than the one
> that created it, yes, there could be a problem .... Was that
> your point?

Yes.

But it's "fine" according to the simple rule people learn if they don't
really *get* threaded programming. In the absence of threads, passing the
address of a local variable to a function you call is generally safe.
(Not, of course, if that function stashes the pointer for later use...)

[Phil Carmody]

James Kuyper <james...@verizon.net> writes:
> On 11/16/2011 03:23 AM, Phil Carmody wrote:
> > James Kuyper <james...@verizon.net> writes:
> >> On 11/14/2011 04:58 PM, Ian Collins wrote:
> ...
> >>> In the same vein as "as long as you hold the knife by the handle, what's
> >>> the problem?".
> >>
> >> I would interpret a question such as that as one which dismisses the
> >> importance of having to hold the knife by the handle, because it's easy
> >> to arrange to do so. I had assumed you were doing the same: implicitly
> >> asserting that it's trivial to arrange that the address remains valid;
> >
> > By default, addresses remain valid unless you explicitly do something
> > that makes them invalid.
>
> Off-hand I can think of three ways by which previously valid addresses
> can be come invalid, and three of them are quite explicit: free(),
> realloc(), and fclose().
> However, the addresses of variables with automatic storage duration
> become invalid as soon as execution of the block in which they are
> defined ends; that seems pretty implicit to me. YMMV

I was considering the ending of a scope to be quite explicit.

> I hope you're not suggesting that the fact that free(), realloc() and
> fclose() can only occur explicitly somehow makes it trivial to ensure
> that they have not yet been called for a given pointer? If that were the
> case, garbage collection wouldn't be as popular an extension as it is.
> Even in single-threaded code it takes a certain amount of discipline to
> ensure that a pointer is not used after the memory it points to has been
> free()d.

That's the fallacy of the hasty generalisation. Whilst /in general/ it's
hard to ensure all pointers are treated safely, that doesn't mean for every
single pointer it's hard.

> When possible, I try to make sure that pointers into dynamically
> allocated memory are stored only in automatic objects whose lifetime
> ends immediately after the call to free(), rendering it impossible for
> such problems to occur. However, it's not always possible to impose such
> restrictions. When I can't do that, I insert code setting the pointer to
> NULL immediately after the free(). However, to be helpful, that requires
> that I protect all problematic uses of such a pointer with tests to
> determine whether it's null.

That started off sounding like good defensive programming, but as the
issue in question was having multiple pointers to the same block of
memory, resetting and checking the pointer is not a solution.

> When the expressions p=malloc(), *p=3, x=*p, and free(p) occur in two or
> more different threads executing asynchronously, it seems to me that it
> would be all that much harder to ensure that they occur only in the
> proper order. I'm not saying it's impossible - just that it's harder.
> Though Ian has assured me that this is not the case, I don't quite see
> how that could be.

If the creator is tasked with freeing the memory, then clearly some
synchronisation is required. However, it the thread is tasked with the
free, then none is required (and the creator may defensively NULL the
pointer it no longer may assume is valid).

[Phil Carmody]

Seebs <usenet...@seebs.net> writes:
> On 2011-11-17, Fritz Wuehler <fr...@spamexpire-201111.rodent.frell.theremailer.net> wrote:
> >> Seebs <usenet...@seebs.net> writes:
> >> > Consider:
>
> >> > struct foo *f = malloc(sizeof(*f));
> >> > free(f);
> >> > f->x = 1;
>
> >> > I do not consider it a reasonable assumption that an arbitrary person
> >> > with 3-5 years or more of experience programming C will see anything
> >> > wrong here.
>
> > Let's try something else. I have a lot of experience in programming but I
> > have written about 10 lines of C (and no Java, C++ or anything related) and
> > not had to work with it or close to it ever.
>
> > Does this code fragment mean
>
> > 1. allocate some storage
> > 2. free it
> > 3. assign a value to the area pointed to by the pointer to the storage you just
> > freed
>
> Yup.

I wonder if the guy had learnt just enough Unixy stuff to make that knowledge
dangerous. Compare:

open temp file
unlink it
use it

[Phil Carmody]

James Kuyper <james...@verizon.net> writes:
> On 11/16/2011 06:53 AM, James Kuyper wrote:
> ...
> > Off-hand I can think of three ways by which previously valid addresses
>
> Yes, I can count. However, when I decided to add realloc() to the list,
> I forgot to update that to "four".

It's OK, when in a discussion for mutual edification, I read for meaning,
not typos.

[blmblm.m...@gmail.com]

In article <slrnjcakol.11i...@guild.seebs.net>,

Seebs <usenet...@seebs.net> wrote:
> On 2011-11-17, blmblm myrealbox.com <blmblm.m...@gmail.com> wrote:
> > In article <slrnjc38ub.r7s...@guild.seebs.net>,
> > Seebs <usenet...@seebs.net> wrote:
> >> It is very easy for people who have finally been taught that it's not okay
> >> to return the address of a local variable to conclude that it's fine to
> >> pass the address of a local variable to a function you call. And you're
> >> *calling* pthread_create(), see. So that's fine.
>
> > Well, now *I*'m the one who's confused .... Are you saying
> > it's not fine?
>
> Yes.
>
> > I suppose if the function you call creates an
> > independent thread, and that thread persists longer than the one
> > that created it, yes, there could be a problem .... Was that
> > your point?
>
> Yes.
>
> But it's "fine" according to the simple rule people learn if they don't
> really *get* threaded programming. In the absence of threads, passing the
> address of a local variable to a function you call is generally safe.
> (Not, of course, if that function stashes the pointer for later use...)

Good heavens. That way lies madness .... though maybe not, since
is it paranoia if they really *are* out to get you?

I'd have put myself in the group that claims to "get" threaded
programming, but I'll admit I never thought about this particular
potential pitfall. Then again, I tend to write the kind of
threaded programs in which there's a "master" thread that creates
the other threads and doesn't terminate until all the others do,
and in that case surely passing addresses of local variables in the
master thread to the other threads is okay, no? (At least in the
sense that the memory will still be there for the other threads,
though if multiple threads have access to the same location, well,
there are the usual potential problems with that.)

[James Kuyper]

On 11/18/2011 07:24 AM, Phil Carmody wrote:
> James Kuyper <james...@verizon.net> writes:

...

>> I hope you're not suggesting that the fact that free(), realloc() and
>> fclose() can only occur explicitly somehow makes it trivial to ensure
>> that they have not yet been called for a given pointer? If that were the
>> case, garbage collection wouldn't be as popular an extension as it is.
>> Even in single-threaded code it takes a certain amount of discipline to
>> ensure that a pointer is not used after the memory it points to has been
>> free()d.
>
> That's the fallacy of the hasty generalisation. Whilst /in general/ it's
> hard to ensure all pointers are treated safely, that doesn't mean for every
> single pointer it's hard.

I never meant to suggest that it's always hard. The origin of this
discussion was a general comment that "Pass-by-reference and threads
aren't known to play well" - I don't think it would appropriate to
interpret a comment so casually worded, as having been meant in an
absolute sense. That comment, in turn, was in response to proposals to
solve Quentin's problem by dynamically allocating memory to store the
file descriptor in. I don't think Quentin's description of the problem
was ever sufficiently specific to allow us to conclude that this would
have been an easy solution.

>> When possible, I try to make sure that pointers into dynamically
>> allocated memory are stored only in automatic objects whose lifetime
>> ends immediately after the call to free(), rendering it impossible for
>> such problems to occur. However, it's not always possible to impose such
>> restrictions. When I can't do that, I insert code setting the pointer to
>> NULL immediately after the free(). However, to be helpful, that requires
>> that I protect all problematic uses of such a pointer with tests to
>> determine whether it's null.
>
> That started off sounding like good defensive programming, but as the
> issue in question was having multiple pointers to the same block of
> memory, resetting and checking the pointer is not a solution.

I don't think multiple pointers to the same block of memory has ever
been brought up in the messages leading to this one. You are, of
course, perfectly correct that the solution is not sufficient for that
case. Options include nulling all of the saved pointers, or designating
one particular pointer (most reasonably, the one that free() will be
applied to) which must be checked to make sure it's not null before
using any of the others. There's other, more complicated solutions
possible, depending upon the nature of the problem. The key point is
that, one way or another, if a stored pointer has a possibility of being
de-referenced when invalid, something must be done to remove that
possibility. That constrains designs in ways that make them harder to
write, harder to read, possibly less efficient, and more prone to error.

>> When the expressions p=malloc(), *p=3, x=*p, and free(p) occur in two or
>> more different threads executing asynchronously, it seems to me that it
>> would be all that much harder to ensure that they occur only in the
>> proper order. I'm not saying it's impossible - just that it's harder.
>> Though Ian has assured me that this is not the case, I don't quite see
>> how that could be.
>
> If the creator is tasked with freeing the memory, then clearly some
> synchronisation is required. However, it the thread is tasked with the
> free, then none is required (and the creator may defensively NULL the
> pointer it no longer may assume is valid).

I was talking about pointers to memory that are shared between threads;
you're talking about the simpler case where a pointer to dynamically
allocated memory is passed between threads, but is never in use by two
of them at the same time. That's certainly a safer approach, but I doubt
that it's always acceptable to impose such constraints on how the
pointer is used.

[Phil Carmody]

James Kuyper <james...@verizon.net> writes:
> On 11/18/2011 07:24 AM, Phil Carmody wrote:
> > James Kuyper <james...@verizon.net> writes:
> ...

> >> When the expressions p=malloc(), *p=3, x=*p, and free(p) occur in two or
> >> more different threads executing asynchronously, it seems to me that it
> >> would be all that much harder to ensure that they occur only in the
> >> proper order. I'm not saying it's impossible - just that it's harder.
> >> Though Ian has assured me that this is not the case, I don't quite see
> >> how that could be.
> >
> > If the creator is tasked with freeing the memory, then clearly some
> > synchronisation is required. However, it the thread is tasked with the
> > free, then none is required (and the creator may defensively NULL the
> > pointer it no longer may assume is valid).
>
> I was talking about pointers to memory that are shared between threads;
> you're talking about the simpler case where a pointer to dynamically
> allocated memory is passed between threads, but is never in use by two
> of them at the same time. That's certainly a safer approach, but I doubt
> that it's always acceptable to impose such constraints on how the
> pointer is used.

There's certainly been some drift in the thread, and I'm quite possibly
guilty of blindly steering that.

The particular (pthread) scenario is impossible in the C codebase I
maintain, but I know that were something equivalent to crop up in a
pull request without adequate synchronisation primitives, I'd reject
it immediately. I.e. I don't just think it's acceptable, I think it's
obligatory in the context I'm familiar with. However, my perspective
is admittedly narrow.