Quick Script needed to strip line wrap
Seth Able
I admit, I can't program my way out of a wet paper bag. :(
I have installed a program where some files within the program have the
partial line wrap character at the end of each line. Many of these files
are CGI scripts and this breaks them.
I've found that opening the file in "Pico" (maybe others?) then
immidiately saving it again strips these characters off. The trick is
though there are a LOT of files to go through in many sub directories.
Could anyone recommend a script or app that I could run recursively on
a directory to strip this line wrap character where ever it may exist? I
can't simply rip off the last character in each file though because many
files are okay (like the ones I've already fixed).
Thank!!
Seth
Bruce Burhans
"Seth Able" <r...@alteeve.com> wrote in message
news:gY0A9.5454$md7.5...@news20.bellglobal.com...
Got them from a windows machine?
If this seems likely, cp a few of the files
to an isolated directory (no subdirectories) and
run this to make sure it's what you need--
$ for file in $ ( find . -type f ) ; do dos2unix
-ad $file ; done
All one line
If it works like you want, run it from the toplevel
directory. It removes the dos/windows CR/LF
non-printing characters and nothing else.
HTH
Bruce
--
Bellingham Washington USA
bburhan1 AT earthlink DOT net
Paul Lutus
> Hi all,
>
> I admit, I can't program my way out of a wet paper bag. :(
>
> I have installed a program where some files within the program have the
> partial line wrap character at the end of each line. Many of these files
> are CGI scripts and this breaks them.
What "partial line wrap character" is that? You must find out what
character it is for us to be able to help you.
> I've found that opening the file in "Pico" (maybe others?) then
> immidiately saving it again strips these characters off. The trick is
> though there are a LOT of files to go through in many sub directories.
>
> Could anyone recommend a script or app that I could run recursively on
> a directory to strip this line wrap character where ever it may exist?
For a given directory tree, and for known file suffixes, and for a known
search target, the following will search recursively and edit all matching
files:
*************************************
#!/bin/sh
LIST=$(find /temp2 -name '*[.html|.txt|.doc]')
for fn in $LIST;
do
sed -e 's/a/aaa/g' $fn > $fn~
mv -f $fn~ $fn
let 'n=n+1'
done
echo processed $n files.
*************************************
Notice this: 's/a/aaa/g'. This is the heart of the routine. Right now the
example finds 'a' and replaces it with 'aaa', globally. But once you find
out what the magic character is, just plug it in:
's/magic-char//g'
This replaces 'magic-char' with nothing, which is what you want.
--
Paul Lutus
www.arachnoid.com
Mina Naguib
Hash: SHA1
perl -ni -e 's/\r+//g' filename.extension
or
perl -ni -e 's/\r+//g' cgi-bin/*.cgi
(The above is a very crude dos2unix utility)
Best of luck. (Make sure you have a backup copy of the data first. Also
do NOT run the above on binary files such as images/etc...)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQE90PUSeS99pGMif6wRAuZfAKDAAVtmJzGbgzJAr1TX9R8cw8aoBQCgh91z
4TY7W8WEkuPwhAqWX0+CNLQ=
=NsB/
-----END PGP SIGNATURE-----
Seth Able
PERL script? Yep, I'm that much of a n00b. BTW, yeah, it's "Ikonboard"
which works on Linux but was developed on Windows (it appears). It's a
really nice Board, so I guess it's worth the hassle.
Seth
Bruce Burhans
"Seth Able" <r...@alteeve.com> wrote in message
news:6xkA9.5696$Vr1.9...@news20.bellglobal.com...
> Hey Bruce, thanks! Trick is, how do I run it? Do I simply make a simple
> PERL script? Yep, I'm that much of a n00b. BTW, yeah, it's "Ikonboard"
> which works on Linux but was developed on Windows (it appears). It's a
> really nice Board, so I guess it's worth the hassle.
>
> Seth
Just like it says, from the command prompt.
IN the topmost directory where the files are.. That dot after "find"
means "this directory"
$ for file in $ ( find . -type f ) ; do dos2unix
-ad $file ; done
all one line.
PLEASE TEST THIS LIKE I SUGGESTED
If you want to put it in a script for later use, then
---------------------------------------------------------------------
#!/bin/bash
read NM
#quote names with spaces
$ for file in $ ( find $NM -type f ) ; do dos2unix
-ad $file ; done
# all one line
exit 0
#This variation will let you enter the name of
# the top directory at a prompt it provides. Full #path necessary
---------------------------------------------------------------------
Do
$ chmod +rx scriptname
and put it in a directory in your $PATH
$ echo $PATH
Instead of, or before, worrying about PERL, learn
the Bash shell.
"Learning the Bash Shell" 2nd Edition by Newham & Rosenblatt OREILLY
Chris F.A. Johnson
>>>
>>>I have installed a program where some files within the program have the
>>>partial line wrap character at the end of each line. Many of these files
>>>are CGI scripts and this breaks them.
>>>
>>>I've found that opening the file in "Pico" (maybe others?) then
>>>immidiately saving it again strips these characters off. The trick is
>>>though there are a LOT of files to go through in many sub directories.
>>>
>>>Could anyone recommend a script or app that I could run recursively on
>>>a directory to strip this line wrap character where ever it may exist? I
>>>can't simply rip off the last character in each file though because many
>>>files are okay (like the ones I've already fixed).
>>
>>
>> If this seems likely, cp a few of the files
>> to an isolated directory (no subdirectories) and
>> run this to make sure it's what you need--
>>
>> $ for file in $ ( find . -type f ) ; do dos2unix
>> -ad $file ; done
>>
>> All one line
>>
>> If it works like you want, run it from the toplevel
>> directory. It removes the dos/windows CR/LF
>> non-printing characters and nothing else.
[please don't top post]
> Hey Bruce, thanks! Trick is, how do I run it? Do I simply make a simple
> PERL script? Yep, I'm that much of a n00b. BTW, yeah, it's "Ikonboard"
> which works on Linux but was developed on Windows (it appears). It's a
> really nice Board, so I guess it's worth the hassle.
If you have the dos2unix command, you can just type Bruce's script
at a shell prompt. Unfortunately, not all distros include dos2unix.
And not all versions of dos2unix have the -a and -d options. The
version I found has neither option, but I expect (and hope) the -a
option tells it only to convert ASCII files, or all your executable
files (applications, etc.) will be toast (as in buttered side down
on the floor). One version I've seen uses the -ascii option to
convert only ascii files; another uses -b to skip binary files.
You can determine whether you have dos2unix by typing this at a
command prompt:
type dos2unix
If it tells you something like "dos2unix is /usr/local/bin/dos2unix"
you have it. Type "man dos2unix" to see what options it takes.
If, instead, it says: "-bash: type: dos2unix: not found"
you don't have it.
Luckily, it is easy to accomplish the same thing with standard
commands, the only trick being to determine whether a file is a
text file. For that we can use the "file" command; in fact it will
tell us (on Linux and FreeBSD systems, but not on all Unix-type
systems) if the file contains CRLF line terminators.
You can type (better, copy and paste) this script at the shell
prompt, or put it into a file and execute it:
tmpfile=tmp$$
find . -type f -print0 | xargs -0 file |
while IFS=': ' read file type
do
case $type in
*text*with\ CRLF*)
tr -d "\r" < $file > $tmpfile
mv $tmpfile $file
;;
esac
done
Follow Bruce's advice and copy some of the files in question to a
new directory, cd to that directory, and test the script there
before using it on the real directory.
--
Chris F.A. Johnson http://cfaj.freeshell.org
===================================================================
My code (if any) in this post is copyright 2002, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License
Bruce Burhans
"Seth Able" <r...@alteeve.com> wrote in message news:6xkA9.5696
Chris FAJ is a REAL pro. Pay attention to what
he says.
Bruce Burhans
"Seth Able" <r...@alteeve.com> wrote in message news:gY0A9.5454
I just thought of something else that might be
relevant.
There is no man page for dos2unix on my distro,
(Someone just told me I had it and what it was for)
$ dos2unix --help
(or -h, if that won't work )
Seth Able
between the "$ (" that needed the be killed... I ran it verbatim though
after that and it killed the program... Have a backup though, of course!
:). I'm going to try it again with the "-d" switch... Thanks!
Seth
Seth Able
where you have "*text*with\ CRLF*)", how would I actually enter the
erroneous cd\lf? In text editors like gEdit it shows as a dotted box,
which I can't seem to copy/past. Is there an ASCII value I could type in
for that character? Many thanks again, to both! I'm a n00b but I am
slowly learning :D
Seth
PS - I installed dos2unix from the rpmfind.net listing (for RH7.3)
Bruce Burhans
"Seth Able" <r...@alteeve.com> wrote in message
news:YmtA9.6443$Vr1.1...@news20.bellglobal.com...
> Hi, I figured out the problem with you script, there was a white space
> between the "$ (" that needed the be killed... I ran it verbatim though
> after that and it killed the program... Have a backup though, of course!
> :). I'm going to try it again with the "-d" switch... Thanks!
>
> Seth
Formatting often does not survive the trip through
Usenet. Can be a *big* problem with scripts!
"$" means many things to the shell and various
programs, depending on context. But never dollars to my knowledge!
$ ascii -o
will give you the octal numbers for control characters like CR
$ help echo
[ jusr $ help will give you a list of all the bash builtins, and $ help
help will tell you how to use
the help program ]
will give you the below and more
\n newline (Enter/Return)
\r CR carriage return
\f formfeed (which I THINK is the same as line
feed LF )
\nnn nnn being the octal numbers for the various
characters, printing and not.