"read line" ignoring last line

[Anoop]

Hi All,

I am facing a wierd problem. I have about a 100 shell scripts all coded
in a similar fashion. All of them read from some file(s) and process
the data.

But there is this one script which reads from a small file which has
only 2 lines of data. What happens (and happens only for this script)
is that only the first line is read, and the last line is ignored
completely. I do not get any errors.

What we do to workaround this problem is to just open the file (vi)..
add a new line to the end and just remove that line. Then when we run
the same script it works. But we do not want to keep doing that.

Here is my rather simple script:
*****************************************************
cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
do
echo $line
done
*****************************************************

And here is the data file:
*****************************************************
$ more data/BACSTEL_APPLICATION_DATA.txt
BACSTEL~account type~Business User~~0~
BACSTEL~account type~Admin User~~0~
*****************************************************

When I run the script. This is the output:
*****************************************************
$ ./populate_bacstel.ksh
BACSTEL~account type~Business User~~0~
*****************************************************

All other scripts are coded in a very similar fashion... they all run
fine, except this. Can it be that this is too small a file .....?? Coz
that is the only diff.. the next shortest data file is 4 lines long....
but that is soo wierd!

Thanks,
Anoop

[Bill Marcum]

On 18 Aug 2006 07:49:37 -0700, Anoop

<anoop...@gmail.com> wrote:
> Hi All,
>
> I am facing a wierd problem. I have about a 100 shell scripts all coded
> in a similar fashion. All of them read from some file(s) and process
> the data.
>
> But there is this one script which reads from a small file which has
> only 2 lines of data. What happens (and happens only for this script)
> is that only the first line is read, and the last line is ignored
> completely. I do not get any errors.
>
> What we do to workaround this problem is to just open the file (vi)..
> add a new line to the end and just remove that line. Then when we run
> the same script it works. But we do not want to keep doing that.
>
> Here is my rather simple script:
> *****************************************************
> cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
> do
> echo $line
> done
> *****************************************************

Perhaps the file does not end with a newline. A workaround would be
{ cat ./data/BACSTEL_APPLICATION_DATA.txt; echo; } | while IFS= read line

or fix the program that creates the file.

--
Try to be the best of whatever you are, even if what you are is no good.

[Stephane Chazelas]

On Fri, 18 Aug 2006 11:21:25 -0400, Bill Marcum wrote:
[...]

>> cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
>> do
>> echo $line
>> done
>> *****************************************************
>
> Perhaps the file does not end with a newline. A workaround would be
> { cat ./data/BACSTEL_APPLICATION_DATA.txt; echo; } | while IFS= read line

[...]

I don't think any shell implementation would discard such a
line. I don't know of any that does.

POSIX doesn't seem very clear on that matter. It doesn't seem to
cover the case where eof is reached while reading a line. Reading
http://www.opengroup.org/onlinepubs/009695399/utilities/read.html,
one could tell that not a single shell is conformant.

Apparently, if $IFS doesn't contain the newline character, then
in

read var

$var should contain the entire line read, that is including the
trailing newline character. All the shells I know strip the line
terminator, which makes sense.

--
Stephane

[Anoop]

The files are created manually. And I assure you they were created in
exactly the same way as the others... The only thing (which is also
common for every file) is that we get it into windows - into CVS and
then we ftp the files onto the SunOS boxes. there we run dos2unix
command on every file we ftp-ed.

Let me try your workaround - i will have to fix this one shell script
with that. If that works - i will stick to it.

Thanks,
Anoop

[William]

"Anoop" <anoop...@gmail.com> wrote in message
news:1155912577.3...@75g2000cwc.googlegroups.com...

> Hi All,
>
> I am facing a wierd problem. I have about a 100 shell scripts all coded
> in a similar fashion. All of them read from some file(s) and process
> the data.
>
> But there is this one script which reads from a small file which has
> only 2 lines of data. What happens (and happens only for this script)
> is that only the first line is read, and the last line is ignored
> completely. I do not get any errors.

As Bill Marcum suggested, the problem is in the file, not the
script. I can reproduce your problem by creating a file whose
last line does not have a newline character at the end (read
is looking for this character, cat doesn't care). A newline
tells read it has a full line in the buffer and it's time to
copy it to your variable, but it's hitting the end of the file
before that happens, so...

-Wm

[Anoop]

I agree - but these files were handcoded and come from the client side
- we cannot control what comes from them..

But the workaround posted by Bill Marcum works fine. I think I am going
ahead with modifying just that one script...

Thanks alot to all,
Anoop

[William]

"Anoop" <anoop...@gmail.com> wrote in message

news:1155919273.1...@p79g2000cwp.googlegroups.com...

>
>
> I agree - but these files were handcoded and come from the client side
> - we cannot control what comes from them..
>
> But the workaround posted by Bill Marcum works fine. I think I am going
> ahead with modifying just that one script...
>
> Thanks alot to all,
> Anoop

Watch out for other problems with these files - some things will
handle them fine, but many versions of common tools will drop the
last line. Running the files through sed, for example, may chop
off the last line. (We experienced that with some configuration
files - some of the developers decided to make them "neater" by
removing the last newline. That broke our build and was very
hard to track down.)

-Wm

[Bruce Barnett]

Stephane Chazelas <stephane...@yahoo.fr> writes:

> I don't think any shell implementation would discard such a
> line. I don't know of any that does.

I do know of cases where other utilities will not see the last line if
it's not terminated by a '\n'. I think I had problems with sed and/or
grep a decade ago (was probably an old Solaris system).

I remember it because I had to write a work-around for it.

--
Sending unsolicited commercial e-mail to this account incurs a fee of
$500 per message, and acknowledges the legality of this contract.

[Jon LaBadie]

Stephane Chazelas wrote:
> On Fri, 18 Aug 2006 11:21:25 -0400, Bill Marcum wrote:
> [...]
>>> cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
>>> do
>>> echo $line
>>> done
>>> *****************************************************
>> Perhaps the file does not end with a newline. A workaround would be
>> { cat ./data/BACSTEL_APPLICATION_DATA.txt; echo; } | while IFS= read line
> [...]
>
> I don't think any shell implementation would discard such a
> line. I don't know of any that does.
>

I ran that command line on 2 OS's, Sol 9 x86 and Fedora Core 4
using on each bash, ksh, and zsh, plus Sun's Bourne shell.

None outputted the final line of the input if it was not newline terminated.

[Stephane CHAZELAS]

2006-08-18, 15:13(-04), Jon LaBadie:

Actually, read doesn't discard that line, but returns with a
non-zero exit status if the end of file is reached.

So:

printf foo | sh -c 'read; echo "$? $REPLY"'

would output "1 foo".

So, the OP could do:

{
while IFS= read -r line; do
printf '%s\n' "$line"
done
printf %s "$line"
} < ./data/BACSTEL_APPLICATION_DATA.txt | od -tc

which should be equivalent to

od -tc < ./data/BACSTEL_APPLICATION_DATA.txt

unless the file contains NUL characters.

--
Stéphane

[Janis]

William wrote:
> "Anoop" <anoop...@gmail.com> wrote in message
> news:1155919273.1...@p79g2000cwp.googlegroups.com...
> >
> > I agree - but these files were handcoded and come from the client side
> > - we cannot control what comes from them..

Your company should really inform your customer about any inherent
flaws in their files.

The second best approach is to pre-process *any* files that comes
from your customer to add any missing line terminator before you
start the real task.

> > But the workaround posted by Bill Marcum works fine. I think I am going
> > ahead with modifying just that one script...

Better fix the process.

> > Thanks alot to all,
> > Anoop
>
> Watch out for other problems with these files - some things will
> handle them fine, but many versions of common tools will drop the
> last line.

Or it may catenate the last (incomplete) line in the first file with
the first line in the next file; yet more surprises to come.

> Running the files through sed, for example, may chop
> off the last line. (We experienced that with some configuration
> files - some of the developers decided to make them "neater" by
> removing the last newline. That broke our build and was very
> hard to track down.)
>
> -Wm

Janis

[Anoop]

Janis wrote:
> Or it may catenate the last (incomplete) line in the first file with
> the first line in the next file; yet more surprises to come.
>

> Janis

Thanks Janis.. thanks for the heads up.. I too was contemplating the
retro-effects of having one script unlike the 100 others.. I dont think
I have thoroughly tested the approach yet..

To have a fool proof, always workable way - I think the suggestion to
pre-process the files will be best...

We cannot fix the process as these files are handcoded by 100's of
system admins for various applications and they come in every week....
They are all created on varying platforms etc. hence I start with
cleaning of the files first.. like dos2unix etc... - I think I need
another process which will just add a new line if there is none
present... (not yet sure how I will identify that... but I am on top of
it)

Thanks..
Anoop

[Chris F.A. Johnson]

The make sure that a file has a newline at the end:

printf "%s\n" "$(cat "$FILE")" > tempfile && mv tempfile "$FILE"

(That will compress multiple newlines to a single newline.)

--
Chris F.A. Johnson, author <http://cfaj.freeshell.org>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence

[John W. Krahn]

Chris F.A. Johnson wrote:
> On 2006-08-19, Anoop wrote:
>>Janis wrote:
>>>Or it may catenate the last (incomplete) line in the first file with
>>>the first line in the next file; yet more surprises to come.
>>>
>>>Janis
>>Thanks Janis.. thanks for the heads up.. I too was contemplating the
>>retro-effects of having one script unlike the 100 others.. I dont think
>>I have thoroughly tested the approach yet..
>>
>>To have a fool proof, always workable way - I think the suggestion to
>>pre-process the files will be best...
>>
>>We cannot fix the process as these files are handcoded by 100's of
>>system admins for various applications and they come in every week....
>>They are all created on varying platforms etc. hence I start with
>>cleaning of the files first.. like dos2unix etc... - I think I need
>>another process which will just add a new line if there is none
>>present... (not yet sure how I will identify that... but I am on top of
>>it)
>
> The make sure that a file has a newline at the end:
>
> printf "%s\n" "$(cat "$FILE")" > tempfile && mv tempfile "$FILE"
>
> (That will compress multiple newlines to a single newline.)

Or:

perl -i -lpe1 "$FILE"

John
--
use Perl;
program
fulfillment