reading from stdin via pipe, buffering?

[Rudy Gevaert]

Hi,

I have written a perl program that reads from stdin:
while(<STDIN>)
{
chomp
do_it($_);
}
Data is fed to it via a pipe:

cat myfile | ./myprogram

When running the program, the program doesn't always read the whole
line. I'm guessing this has something to do with the stdin buffer.

I would like to know how I can make that perl gets the whole line. As
it is clearly failing from time to time.

Thanks in advance,

Rudy

[John W. Krahn]

It is not your program's STDIN that is the problem, it is the STDOUT
through the pipe that is the problem. If possible, try opening 'myfile'
in your program instead, possibly using sysopen instead of open.

perldoc -f open
perldoc perlopentut
perldoc -f sysopen

John
--
Perl isn't a toolbox, but a small machine shop where you
can special-order certain sorts of tools at low cost and
in short order. -- Larry Wall

[Rudy Gevaert]

John W. Krahn wrote:
> Rudy Gevaert wrote:
>>
>> I have written a perl program that reads from stdin:
>> while(<STDIN>)
>> {
>> chomp
>> do_it($_);
>> }
>> Data is fed to it via a pipe:
>>
>> cat myfile | ./myprogram
>>
>> When running the program, the program doesn't always read the whole
>> line. I'm guessing this has something to do with the stdin buffer.
>>
>> I would like to know how I can make that perl gets the whole line. As
>> it is clearly failing from time to time.
>
> It is not your program's STDIN that is the problem, it is the STDOUT
> through the pipe that is the problem. If possible, try opening 'myfile'
> in your program instead, possibly using sysopen instead of open.

Unfortunately it needs to work with a pipe. What is the 'perl way' to
handle input coming from a pipe?

Thanks in advance,

Rudy

[xho...@gmail.com]

Rudy Gevaert <Rudy.Ge...@spamugent.be> wrote:
> Hi,
>
> I have written a perl program that reads from stdin:
> while(<STDIN>)
> {
> chomp
> do_it($_);
> }
> Data is fed to it via a pipe:
>
> cat myfile | ./myprogram
>
> When running the program, the program doesn't always read the whole
> line. I'm guessing this has something to do with the stdin buffer.

My guess is that you are misinterpreting something. Maybe you have
cross OS line ending problems.

>
> I would like to know how I can make that perl gets the whole line. As
> it is clearly failing from time to time.

That is far from clear to me. Can you produce an example we can
run to show this problem?

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

[Rudy Gevaert]

xho...@gmail.com wrote:
> Rudy Gevaert <Rudy.Ge...@spamugent.be> wrote:
>> Hi,
>>
>> I have written a perl program that reads from stdin:
>> while(<STDIN>)
>> {
>> chomp
>> do_it($_);
>> }
>> Data is fed to it via a pipe:
>>
>> cat myfile | ./myprogram
>>
>> When running the program, the program doesn't always read the whole
>> line. I'm guessing this has something to do with the stdin buffer.
>
> My guess is that you are misinterpreting something. Maybe you have
> cross OS line ending problems.

I have checked the file and it only has \n on the end of all the lines.

>> I would like to know how I can make that perl gets the whole line. As
>> it is clearly failing from time to time.
>
> That is far from clear to me. Can you produce an example we can
> run to show this problem?

I have looked at my problem a bit more and changed the program so I call
open(FILE, "-") || die("Can't read from stdin: $!");
while(<FILE>){ ...}

The following happens:

1) I run my command: cat bigfile | ./myprogram
2) I do an strace on 'myprogram' and on 'cat'. I observe:
a) cat is reading the whole file!
b) cat finishes reading the whole and 'myprogram' happily carries on
c) cat reads from/ writes to the file in blocks of 4096 bytes:
read(3, "\n...@email.com\not...@mail.com"..., 4096) = 4096
write(1, "\n...@email.com\not...@mail.com"..., 4096) = 4096
d) 'myprogram' reads from stdin in blocks of 4096 bytes:
read(0, "e\nmariek...@ugent.be\nsofie."..., 4096) = 4096
e) But then, 'myprogram' reads:
read(0, "", 4096) = 0
But that is not at the end of the file!

I'll try to come up with a program. But reading a lot of lines and just
printing them doesn't trigger the error.

Thanks in advance,

Rudy