while loop taking input from file via iconv

[Java Jive]

I have the following lines in a shell script ...

while [ -n "${LINE}" ]
do
if [ -n "${LINE} ]
then
# Do processing
fi
done < "${DATA}"

... and this works fine for all but two lines in the data file, which
contain accented characters. A file erroneously named with an e acute
needs to be renamed to have an e grave, and a filename containing an e
umlaut needs to be moved to a new location and given a new name.

I had the same problem in the Windows BATch version of the script, which
I fixed by changing the codepage to ANSI 1252 before doing the
processing in the script, and then changing it back when it was
finished. The Windows BATch program now works perfectly.

I now need something similar for the Linux shell script.

If I replace the 'done' line with ...

done < iconv -f 'CP1252' -t 'UTF-8' "${DATA}"

... I get the following error:

<script name>: line ###: syntax error near unexpected token '-f'
<script name>: line ###: done < iconv -f 'CP1252' -t 'UTF-8' "${DATA}"

So next I tried ...

done < $(iconv -f 'CP1252' -t 'UTF-8' "${DATA}")

... but this gives ...

<script name>: line ###: done < iconv -f 'CP1252' -t 'UTF-8'
"${DATA}": ambiguous redirect

What is is the correct incantational magic to achieve what I want?

If it's of any interest, the script renames directories and files
between versions 1 & 2 of a family archive, so that when v2 is unzipped
over v1, there won't be any old versions of files left lying around to
cause confusion.

About this, I have another question. Some of my family use Macs, not
Windows PCs. Am I right in thinking that, since recent versions of
MacOS are Linux based, they should be able to run this shell script to
achieve what those using Windows will be able to do with the BATch file?
Do I have to give them any special instructions, for example how to
get into a console?

--

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

[J.O. Aho]

On 13/08/2021 21.28, Java Jive wrote:

Bash not my specialty, I would go with python... so I let someone else
answer that part.

> About this, I have another question.  Some of my family use Macs, not
> Windows PCs.  Am I right in thinking that, since recent versions of
> MacOS are Linux based,

No, macOS ain't based on Linux, it's based on BSD.

> they should be able to run this shell script to
> achieve what those using Windows will be able to do with the BATch file?

Depends on the version of macOS, newer one uses zsh wile older ones use
bash and then you of course can have users that have installed another
shell in the same way as a Linux user could have a different shell than
bash.

>  Do I have to give them any special instructions, for example how to
> get into a console?

That depends on their skill level in the same way as for Linux users,
sure it's more common that a macOS user never used a terminal, but there
are quite many Linux users who never used a terminal.

--

//Aho

[Jasen Betts]

On 2021-08-13, Java Jive <ja...@evij.com.invalid> wrote:

> done < "${DATA}"
>
> ... and this works fine for all but two lines in the data file, which
> contain accented characters. A file erroneously named with an e acute
> needs to be renamed to have an e grave, and a filename containing an e
> umlaut needs to be moved to a new location and given a new name.

> If I replace the 'done' line with ...
>
> done < iconv -f 'CP1252' -t 'UTF-8' "${DATA}"
>

you want this:

done <( iconv -f 'CP1252' -t 'UTF-8' "${DATA}" )


--
Jasen.

[Mike Easter]

J.O. Aho wrote:
> there are quite many Linux users who never used a terminal.

Many linux users who are *almost* exclusively graphical often find
support in the form of a command someone gives them which they then
paste into the terminal, because communicating a solution by a problem
solver is very frequently much easier by way of a command even if there
is a graphical route to the same solution.

Because of that, I would say that virtually all linux users have had
occasion to use a command, even those who 'never' use the terminal 'on
their own'.

--
Mike Easter

[Paul]

And sometimes, the little syntax details like this,
place the line contents in a sub-shell. So it is more
than just "brackets are pretty" -- there is a functional
side effect too.

Paul

[Spiros Bousbouras]

On Fri, 13 Aug 2021 20:28:07 +0100
Java Jive <ja...@evij.com.invalid> wrote:
> I have the following lines in a shell script ...
>
> while [ -n "${LINE}" ]
> do
> if [ -n "${LINE} ]

There is a double quote missing in the previous line.

> then
> # Do processing
> fi
> done < "${DATA}"
>
> ... and this works fine for all but two lines in the data file, which
> contain accented characters. A file erroneously named with an e acute
> needs to be renamed to have an e grave, and a filename containing an e
> umlaut needs to be moved to a new location and given a new name.

Just to be clear , the above loop will exit at the first empty line read
from $DATA even though the file may have more (possibly non empty) lines
after that. Is this what you want ? It seems doubtful because you also have
the test if [ -n "${LINE} ] .

> I had the same problem in the Windows BATch version of the script, which
> I fixed by changing the codepage to ANSI 1252 before doing the
> processing in the script, and then changing it back when it was
> finished. The Windows BATch program now works perfectly.
>
> I now need something similar for the Linux shell script.
>
> If I replace the 'done' line with ...
>
> done < iconv -f 'CP1252' -t 'UTF-8' "${DATA}"
>
> ... I get the following error:
>
> <script name>: line ###: syntax error near unexpected token '-f'
> <script name>: line ###: done < iconv -f 'CP1252' -t 'UTF-8' "${DATA}"
>
> So next I tried ...
>
> done < $(iconv -f 'CP1252' -t 'UTF-8' "${DATA}")
>
> ... but this gives ...
>
> <script name>: line ###: done < iconv -f 'CP1252' -t 'UTF-8'
> "${DATA}": ambiguous redirect
>
> What is is the correct incantational magic to achieve what I want?

Assuming that the example code above is really what you want then

iconv -f 'CP1252' -t 'UTF-8' "${DATA}" | while read LINE ; do
if [ -z "${LINE}" ] ; then break ; fi
# Do processing
done

[Spiros Bousbouras]

On Fri, 13 Aug 2021 23:30:33 -0000 (UTC)
Spiros Bousbouras <spi...@gmail.com> wrote:
[...]

> Assuming that the example code above is really what you want then
>
> iconv -f 'CP1252' -t 'UTF-8' "${DATA}" | while read LINE ; do
> if [ -z "${LINE}" ] ; then break ; fi
> # Do processing
> done

Forgot to say , this won't work if any of the filenames contain newline
characters.

For shell questions there is also comp.unix.shell .

[Spiros Bousbouras]

I think it should be

[Java Jive]

Thanks, and thanks to the others who have put forward an alternative
solution to the one below. I'll investigate to-morrow morning and
report back. For the rest, see below ...

On 14/08/2021 00:30, Spiros Bousbouras wrote:
> On Fri, 13 Aug 2021 20:28:07 +0100
> Java Jive <ja...@evij.com.invalid> wrote:
>> I have the following lines in a shell script ...
>>
>> while [ -n "${LINE}" ]
>> do
>> if [ -n "${LINE} ]
>
> There is a double quote missing in the previous line.

Correct, a typo while I was transposing from the screen of the Linux PC,
the original script file is correct.

>> then
>> # Do processing
>> fi
>> done < "${DATA}"
>>
>> ... and this works fine for all but two lines in the data file, which
>> contain accented characters. A file erroneously named with an e acute
>> needs to be renamed to have an e grave, and a filename containing an e
>> umlaut needs to be moved to a new location and given a new name.
>
> Just to be clear , the above loop will exit at the first empty line read
> from $DATA even though the file may have more (possibly non empty) lines
> after that. Is this what you want ? It seems doubtful because you also have
> the test if [ -n "${LINE} ] .

Yes, but no, the data file is the same one as is used by the Windows
BATch file, and was created on a Windows machine, so, as far as Linux is
concerned, every line, even the blank ones - yes, there are some -
actually contains at least one character, a carriage return, which is
removed by the first statement within the 'if' statement.

>> What is is the correct incantational magic to achieve what I want?
>
> Assuming that the example code above is really what you want then
>
> iconv -f 'CP1252' -t 'UTF-8' "${DATA}" | while read LINE ; do
> if [ -z "${LINE}" ] ; then break ; fi
> # Do processing
> done

Thanks again, and thanks again to others. I'll report back soon.

[Jasen Betts]

You're right!

--
Jasen.

[Michael Glasser]

A persistent, fervid, posting itch, regardless of content - basically gunky
Mac-using prostitutes, and satisfactorily-waxed fish, deactivated with wit
for his own secret yearnings. Snit bamboozled Apd some, big fricking deal.
Why be a blubbering ass about it? Of course I quoted specific examples of
Apd stating quotable lies, trolling multiple groups, etc. His response:
to change the topic. Lines of text containing hints and signs.

--
Top Ten Ways Apd Trolls!
https://www.bing.com/search?q=Dustin+Cook%3A+functional+illiterate+fraud
<https://findwhocallsyou.com/4234911448?CallerInfo>
<http://web.archive.org/web/20200909135108/https://www.truepeoplesearch.com/details?name=dustin+james+cook&citystatezip=Kingston%2C+TN&rid=0xl&Diesel=&Gremlin=&Raid=>
Dustin Cook the Fraud

[J.O. Aho]

On 14/08/2021 00.07, Mike Easter wrote:
> J.O. Aho wrote:
>> there are quite many Linux users who never used a terminal.
>
> Many linux users who are *almost* exclusively graphical often find
> support in the form of a command someone gives them which they then
> paste into the terminal, because communicating a solution by a problem
> solver is very frequently much easier by way of a command even if there
> is a graphical route to the same solution.

You are assuming that people always has problem with their Linux
installation, I do personally know people who never used the terminal at
all. Linux has evolved quite a lot since the 1990's.

--

//Aho

[Martin Gregorie]

On Sat, 14 Aug 2021 02:35:18 +0100, Java Jive wrote:

>
> Correct, a typo while I was transposing from the screen of the Linux PC,
> the original script file is correct.
>

Copy & Paste:

- highlight text in a window on the desk top
- hit then Ctrl-C to copy
- hit Ctrl-V to paste into (another) window open on the desktop

avoids typos (works for XFCE and, I assume, for all other Linux graphical
desktops.


--
--
Martin | martin at
Gregorie | gregorie dot org

[Java Jive]

Actually, neither of those worked either, similar messages though I
didn't note them down at the time. What did work was this ...

done <<< $(iconv -f 'CP1252' -t 'UTF-8' "${DATA}")

... but thanks to all for the suggestions which put me on the right track.

[Java Jive]

On 14/08/2021 10:07, Martin Gregorie wrote:
> On Sat, 14 Aug 2021 02:35:18 +0100, Java Jive wrote:
>
>>
>> Correct, a typo while I was transposing from the screen of the Linux PC,
>> the original script file is correct.
>>
> Copy & Paste:
>
> - highlight text in a window on the desk top
> - hit then Ctrl-C to copy
> - hit Ctrl-V to paste into (another) window open on the desktop
>
> avoids typos (works for XFCE and, I assume, for all other Linux graphical
> desktops.

But doesn't work across the physical space between two different PCs!

[Stéphane CARPENTIER]

Le 14-08-2021, Java Jive <ja...@evij.com.invalid> a écrit :
> and was created on a Windows machine,

Have you any control on the creation of the file? If you do, try to
avoid cp1252 and use exclusively utf8.

--
Si vous avez du temps à perdre :
https://scarpet42.gitlab.io

[Mike Easter]

I can't deny that you know people who have never used the terminal; I'm
just asserting that graphically-oriented linux users are frequently
advised to accomplish a *necessary* task using the command line.

For example, there have been discussions here about how the most
standard way of instructing people how to authenticate their linux .iso
download is by way of a command.

I'm not trying to infer that linux is full of problems that need
commands to solve; I'm asserting that the easiest way to instruct
someone how to DO something, problem or not, is often conveyed by way of
a command *because* conveying instructions by way of a command is very
VERY often the most efficient and clear way to convey the information.

Sometimes one would practically need a vid to show a graphical route to
do something relatively simple which can be expressed as a command in a
line or two. Lord knows we don't need more vid/s on the planet :-)

--
Mike Easter

[Mike Easter]

Mike Easter wrote:
> I'm not trying to infer that linux is full of problems that need
> commands to solve;

... nor am I trying to imply that :-)

--
Mike Easter

[Aragorn]

On 14.08.2021 at 08:57, Mike Easter scribbled:

> J.O. Aho wrote:
>
> > You are assuming that people always has problem with their Linux
> > installation, I do personally know people who never used the
> > terminal at all. Linux has evolved quite a lot since the 1990's.
>
> I can't deny that you know people who have never used the terminal;
> I'm just asserting that graphically-oriented linux users are
> frequently advised to accomplish a *necessary* task using the command
> line.
>
> For example, there have been discussions here about how the most
> standard way of instructing people how to authenticate their linux
> .iso download is by way of a command.
>
> I'm not trying to infer that linux is full of problems that need
> commands to solve; I'm asserting that the easiest way to instruct
> someone how to DO something, problem or not, is often conveyed by way
> of a command *because* conveying instructions by way of a command is
> very VERY often the most efficient and clear way to convey the
> information.
>
> Sometimes one would practically need a vid to show a graphical route
> to do something relatively simple which can be expressed as a command
> in a line or two. Lord knows we don't need more vid/s on the planet
> :-)

I agree. Over at the Manjaro forum, we often help people -- mostly
n00bs -- by letting them issue one or a whole set of commands, andwe
also often have to ask them for information about their system, for
which we always request the output of an inxi invocation.

And, we also frequently have to ask them to copy and paste the output,
because many n00bs simply post screenshots of their terminals, rather
than paste the output in an editable and forum-optimized formatted
form.

Sometimes, a shell command can speak a thousand mouse clicks. :p

--
With respect,
= Aragorn =

[Martin Gregorie]

On Sat, 14 Aug 2021 08:57:18 -0700, Mike Easter wrote:

> I can't deny that you know people who have never used the terminal; I'm
> just asserting that graphically-oriented linux users are frequently
> advised to accomplish a *necessary* task using the command line.
>
> For example, there have been discussions here about how the most
> standard way of instructing people how to authenticate their linux .iso
> download is by way of a command.
>

Makes sense, and while you're sorting out their immediate problem, I hope
you'd also tell them how to use 'apropos' and 'man', if only because that
can short circuit a lot of future hand-holding.

[Steve Petruzzellis - fretwizer]

Why does ~BD~ focus on his own pride so much? The entire comedy is full
of unbelievable gaps and counts heavily on ~BD~'s 'riding Rod Speed's
ass' gag.

The virus is real, its presence at any point in time being on your Linux
installer is a false negative. Rod Speed has been over this, in excruciating
detail previously, ~BD~. ~BD~ says I'm not 'allowed' to call out their
crap, and they aren't going to pay the price for any prior crap they
caused. To the contrary, I'm expected to forgive it all, and let them
start from scratch so they can just replay the nonsense all over again,
while they accept no responsibility for what they say and do online towards
myself?

Linux is a false advocate's hero. I am working on an idea which will
be more than anything ~BD~ can do!

--
Puppy Videos!
https://search.givewater.com/serp?q=%22narcissistic%20bigot%22
<http://web.archive.org/web/20200911090855/https://www.usphonebook.com/423-
491-1448?Dustin-Cook=&Diesel=&Gremlin=>
Steve Petruzzellis the Narcissistic Bigot

[Mike Easter]

Martin Gregorie wrote:
> while you're sorting out their immediate problem, I hope
> you'd also tell them how to use 'apropos' and 'man', if only because that
> can short circuit a lot of future hand-holding.

Those two commands are excellent examples of how much there is for the
inexperienced to learn about commands, namely how to use the options for
commands.

That 'problem' (actually *strength*) is one of the intimidating factors
for the less experienced in terms of commands, namely options and syntax.

The commands were designed to be very powerful; the greater the power
the greater the 'responsibility'. Sometimes when one sees a man result
he feels like he is drowning.

Of course, being thrown into the water and needing to swim back out is
one way to learn to swim :-)

--
Mike Easter

[Martin Gregorie]

On Sat, 14 Aug 2021 12:57:09 -0700, Mike Easter wrote:

> Martin Gregorie wrote:
>> while you're sorting out their immediate problem, I hope you'd also
>> tell them how to use 'apropos' and 'man', if only because that can
>> short circuit a lot of future hand-holding.
>
> Those two commands are excellent examples of how much there is for the
> inexperienced to learn about commands, namely how to use the options for
> commands.
>
> That 'problem' (actually *strength*) is one of the intimidating factors
> for the less experienced in terms of commands, namely options and
> syntax.
>
> The commands were designed to be very powerful; the greater the power
> the greater the 'responsibility'. Sometimes when one sees a man result
> he feels like he is drowning.
>

The other thing that needs to be taught is that each command is designed
to do one thing and to do it well, which is why there are so *many*
commands. hence the need to understand 'apropos', or better,

apropos 'action name' | less

as an aid to finding the command they want and than

man commandname

to see how to use it.

> Of course, being thrown into the water and needing to swim back out is
> one way to learn to swim :-)
>

.. and sadly there isn't a lot else. If the student is capable of using
command lines on another system, then something like 'Linux in a
Nutshell' may be helpful, but that's about the only decent book I know
unless there's a 'Linux for Dummies' available and they're not put off by
the title.

I head for books every time, but then my background is not altogether
usual. I arrived at Linux via:

- being taught to run an ICL 1900 mainframe using the control teletype
and to program it in Algol 60, assembler and COBOL, running under both
George2 and George 3

- another job with formal training got me up to speed on ICL 2966
mainframes running VME/B, COBOL (again) and IDMSX database

- after that I worked for software houses, were the style was very much
"What, you've heard of it[*]? Just the man: here are the manuals and
you'll be on the project team in a week."

[*] 'it' might be a new programming language, database, operating
system, hardware or some combination of all of four. Fun times!

[Mike Easter]

Martin Gregorie wrote:
> If the student is capable of using command lines on another system,
> then something like 'Linux in a Nutshell' may be helpful, but that's
> about the only decent book I know unless there's a 'Linux for
> Dummies' available and they're not put off by the title.

In my graphically-oriented opinion as a 'longtime intermediate' (I used
to call myself an advanced intermediate until I came to realize that
advanced /time/ as an intermediate doesn't actually make one
'advanced'), part of the 'problem' (in one sense) is the aspect that JOA
expressed earlier, that modern linux is sufficiently graphically
oriented nowadays that it isn't 'imperative' that the inexperienced
become very knowledgeable about 'everything' (euphemistically) about the
infrastructure and commands including their options and syntax.

They can 'get by' w/o knowing much of that; except sometimes (I say/IMO).

But, it is a LOT better the more one knows, so it is worth staying on
the learning curve. The experienced have a *lot* easier time of it; one
of my favorite examples of command line insights is the business of
tab-completion. The noob doesn't know that; but even the 'early'
intermediate has already found that out.

--
Mike Easter

[William Unruh]

On 2021-08-14, Martin Gregorie <mar...@mydomain.invalid> wrote:
> On Sat, 14 Aug 2021 12:57:09 -0700, Mike Easter wrote:
>
>> Martin Gregorie wrote:
>>> while you're sorting out their immediate problem, I hope you'd also
>>> tell them how to use 'apropos' and 'man', if only because that can
>>> short circuit a lot of future hand-holding.
>>
>> Those two commands are excellent examples of how much there is for the
>> inexperienced to learn about commands, namely how to use the options for
>> commands.
>>
>> That 'problem' (actually *strength*) is one of the intimidating factors
>> for the less experienced in terms of commands, namely options and
>> syntax.
>>
>> The commands were designed to be very powerful; the greater the power
>> the greater the 'responsibility'. Sometimes when one sees a man result
>> he feels like he is drowning.
>>
> The other thing that needs to be taught is that each command is designed
> to do one thing and to do it well, which is why there are so *many*

Says someone, apparently, who has never looked at the command "find", or
may other commands. billions of different option combinations, some of
which work, others of which do not.

> commands. hence the need to understand 'apropos', or better,
>
> apropos 'action name' | less
>
> as an aid to finding the command they want and than
>
> man commandname

Unfortunately man is really good for reminding someone what various
things mean, but is pretty bad at teaching anyone how to use the
command. A much much larger section for each man page with explicity
examples would go a long way to making "man" useful for newbies (and
oldies as well).

[Mike Easter]

William Unruh wrote:
> Unfortunately man is really good for reminding someone what various
> things mean, but is pretty bad at teaching anyone how to use the
> command. A much much larger section for each man page with explicity
> examples would go a long way to making "man" useful for newbies (and
> oldies as well).

For me, when I 'man' something, i just flip thru' the pages to get an
'overview' w/o trying to /really/ learn how to use a command there. I
don't consider the 'man' construct to be a good teacher, more like an
'unpleasant' reference book.

I go looking around on the internet for examples of something I'm
actually trying to do. Once I get a better sense of a command and its
uses w/ options, then I can use 'man' again as a reference source for
such as those options.

Something like the difference between a huge dictionary like the OED vs
a site which explains something about English usage.

It is one thing to learn something about a language with just vocabulary
flash cards and another to be either conversational or able to read and
write in it.

--
Mike Easter

[Martin Gregorie]

On Sat, 14 Aug 2021 22:52:08 +0000, William Unruh wrote:

> Says someone, apparently, who has never looked at the command "find", or
> may other commands. billions of different option combinations, some of
> which work, others of which do not.
>

Quite right: I don't use it because 'locate' is *much* faster and easier
to use, especially if updatedb is run overnight by a cronjob

Similarly, 'apropos' is nearly as fast 'locate' since it only has to scan
the contents of /usr/share/man/* - and in addition, because its scanning
the first line of each manpage, it also matches words of phrases
describing what a program does, so often searching on a word or phrase
describing what a program does means you a suitable program without
knowing its name:

$ apropos 'free space'
e2freefrag (8) - report free space fragmentation information
xfs_spaceman (8) - show free space information about an XFS filesystem

$ apropos 'space used'
space used: nothing appropriate.
$ apropos 'space usage'
df (1) - report file system disk space usage
du (1) - estimate file space usage
du (1p) - estimate file space usage

... and its no use complaining about how well or badly a manpage is
written: often the only way to fit would be to submit a manpage patch.

Yes, I know some Linux manpages are pretty bad. However, others (those
for bash, sort and awk to name but a few) are excellent and most are
usable.

[Steve Petruzzellis - fretwizer]

Yup. It seems this is what we have to face. Jerks who obviously have no reason
for being here other than to upset me. Do not get too full of hot air, Wolffan,
sometimes "your unquotable lies" are just that.

What did you hope for from the lying imbecile? That Wolffan crackhead has
nothing to blow but time. He has nothing else. Especially not a caring family.
Your system will crawl while finding a way to stop Wolffan. And it takes
a long time. Just look at the effort -hh once again had to spend to get Wolffan
able to pull messages from an IRC server via CLI. -hh once again, took abuse
from Wolffan, instead of a great thanks, much appreciated, response - that
I think -hh was owed for the trouble and time he spent on the matter.

--
Get Rich Slow!
https://www.google.com/search?q=steve+carroll+the+narcissistic+bigot
https://www.youtube.com/watch?v=prZeTJKpc3Y
<https://youtu.be/xRvaRLlb3b8>
Narcissistic Bigot Steve Carroll

[Spiros Bousbouras]

On Sat, 14 Aug 2021 20:53:08 -0000 (UTC)
Martin Gregorie <mar...@mydomain.invalid> wrote:
> On Sat, 14 Aug 2021 12:57:09 -0700, Mike Easter wrote:
> > The commands were designed to be very powerful; the greater the power
> > the greater the 'responsibility'. Sometimes when one sees a man result
> > he feels like he is drowning.
> >
> The other thing that needs to be taught is that each command is designed
> to do one thing and to do it well, which is why there are so *many*
> commands. hence the need to understand 'apropos', or better,
>
> apropos 'action name' | less
>
> as an aid to finding the command they want and than
>
> man commandname
>
> to see how to use it.
>
> > Of course, being thrown into the water and needing to swim back out is
> > one way to learn to swim :-)
> >
> .. and sadly there isn't a lot else. If the student is capable of using
> command lines on another system, then something like 'Linux in a
> Nutshell' may be helpful, but that's about the only decent book I know
> unless there's a 'Linux for Dummies' available and they're not put off by
> the title.

If I go on amazon and search for "Linux command line" I see many books and
they tend to have high rating average. Either you don't consider them decent
or you haven't performed any such search in a long time even just for
curiosity.

--
vlaho.ninja/prog

[Richard Kettlewell]

Martin Gregorie <mar...@mydomain.invalid> writes:
> William Unruh wrote:
>
>> Says someone, apparently, who has never looked at the command "find", or
>> may other commands. billions of different option combinations, some of
>> which work, others of which do not.
>
> Quite right: I don't use it because 'locate' is *much* faster and easier
> to use, especially if updatedb is run overnight by a cronjob

Bad comparison, since it doesn’t do the same thing.

--
https://www.greenend.org.uk/rjk/

[Paul]

This is why you keep a "notes" file.

find /media/somedisk -type d -exec ls -al -1 -d {} + > directories.txt
find /media/somedisk -type f -exec ls -al -1 {} + > filelist.txt

Any time you put some effort into crafting one, you
record it for the future.

./ffmpeg -hwaccel nvdec -i "fedora.mkv" -y -acodec aac -vcodec h264_nvenc -crf 23 "output2.mp4"

16.3x speed, 488FPS

Part of the fun is making them cryptic, so you can't understand them later.

Paul

[Java Jive]

On 13/08/2021 20:28, Java Jive wrote:
> I have the following lines in a shell script ...
>
> while [ -n "${LINE}" ]
>     do
>         if [ -n "${LINE} ]

>             then
>                 # Do processing
>         fi
>     done < "${DATA}"
>

> .... and this works fine for all but two lines in the data file, which

> contain accented characters. A file erroneously named with an e acute
> needs to be renamed to have an e grave, and a filename containing an e
> umlaut needs to be moved to a new location and given a new name.

Uggghhh! The reason for this disgust will become clear shortly!

This is a follow up question about character encodings ...

Previously I have released to my family two versions of the same archive
of family documents going back to the reign of Queen Anne, some items
possibly a little earlier. These documents were scanned (1o for
original scan) and then put through four possible stages of post-processing:
2n Contrast 'normalised' using pnnorm
3t Textcleaned
4nt n followed by 3
5tn t followed by n

For each document, the best result was copied into the main archive,
while the above preprocessing stages were left in an '_all'
sub-directory structure, with five subdirectories named as above, each
of which having beneath it a directory tree mirroring the main archive.

The main version of the archive, which most family members seem to have
downloaded, only included the main archive and didn't include the _all
subdirectory with all the pre-processing results, the full version
included this directory. IIRC, the former was compressed by WinZip from
the archive as it existed on a Windows PC at the time, but WinZip threw
a wobbly over the size of the full archive, so for that I had to use 7zip.

Now the crunch, when I unzip these on a Linux machine, I see different
bastardisations of accented characters. So, for example where the full
7zip archive when extracted shows an e acute correctly in both a console
and a file manager listing ...
"Chat Botté, Le" [e is correctly acute]
... (if you're wondering, a French children's picture book version of
apparently 'Puss In Boots'), while with the WinZip main archive a
console listing shows a very odd character sequence instead of the e
acute ...
"Chat Bott'$'\302\202'', Le"
... and a file manager listing has a graphic character resembling a 2x2
matrix, concerning which note that while \302 octal = \xC2 hex, and
\202 octal = \x82 hex, only the second of these and not the first
appears in the symbol:
|00|
|82|

My problem is that I can't find a search term to trap this strange
character to correct it, for example the following, and a few similar
that I've tried, don't work because they don't find the directory:
mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"

I could use a glob wildcard character such as '?', but currently all the
filenames are within quotes, where globbing doesn't seem to work, and it
would be a hell of a business removing the quotes, because many names in
the archive use many characters that would each need to be anticipated
and escaped for in an unquoted filename, such as spaces, ampersands,
brackets, etc.

Can anyone suggest a sequence that will find the file, when put inside
quotes as the filename in the controlling data file mentioned previously
in the thread, so that it can just be treated like all the other lines?
As someone here suggested the data file is now stored as UTF-8 rather
than ANSI as it was formerly, and some example lines are given below in
a form for easier readability in a ng - in reality the fields are tab
separated but here are separated by double spacing and have been further
abbreviated to keep them from wrapping; leading symbols such as '+' and
'=' have special meanings for the program doing the work; and, yes, the
commands are basically DOS commands which for Linux are translated to
their bash equivalents:

=ATTRIB -R "./F H/Close/Sts Mary & John Churchyard Monuments.pdf"
=RD "./F H /_all/1o/Blessig & Heyder"
REN "./Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
MOVE "./Photo - D & M Close.png" "./Photos/D & M Close.png"
[etc]

[Martin Gregorie]

Both look for filenames, but 'find' can be restricted to a directory
structure - thats about the difference I can see in a quick manpage scan.

[Martin Gregorie]

Back in 2003 when I was moving my home systems from OS9/68000 to Linux I
got a copy of 'UNIX in Nutshell', which contained most of what I needed'
except for sysadmin stuff - and I couldn't find anything useful for
system administration and hor the kernel is organised until I settled on
'Debian Reference':

https://www.debian.org/doc/manuals/debian-reference/

Since then I've added Fedora documentation:
https://fedoraproject.org/wiki/Category:Documentation?rd=Docs

and stuff on Systemd

https://www.freedesktop.org/software/systemd/man/systemd.html#

[Richard Kettlewell]

Martin Gregorie <mar...@mydomain.invalid> writes:
> Richard Kettlewell wrote:
>> Martin Gregorie <mar...@mydomain.invalid> writes:
>>> William Unruh wrote:

>>>> Says someone, apparently, who has never looked at the command
>>>> "find", or may other commands. billions of different option
>>>> combinations, some of which work, others of which do not.
>>>
>>> Quite right: I don't use it because 'locate' is *much* faster and
>>> easier to use, especially if updatedb is run overnight by a cronjob
>>
>> Bad comparison, since it doesn’t do the same thing.
>
> Both look for filenames,

No. I mean, that’s one thing find can do, but it’s nowhere near all of
it.

> but 'find' can be restricted to a directory structure - thats about
> the difference I can see in a quick manpage scan.

find has numerous ways of selecting which files to act on and what to do
with them. It also differs more fundamentally in that it traverses the
real directory structure, rather than an incomplete snapshot made at
some point in the past.

--
https://www.greenend.org.uk/rjk/

[Spiros Bousbouras]

On Sun, 15 Aug 2021 12:57:24 +0100
Java Jive <ja...@evij.com.invalid> wrote:
> Now the crunch, when I unzip these on a Linux machine, I see different
> bastardisations of accented characters. So, for example where the full
> 7zip archive when extracted shows an e acute correctly in both a console
> and a file manager listing ...
> "Chat Botté, Le" [e is correctly acute]
> ... (if you're wondering, a French children's picture book version of
> apparently 'Puss In Boots'), while with the WinZip main archive a
> console listing shows a very odd character sequence instead of the e
> acute ...
> "Chat Bott'$'\302\202'', Le"
> ... and a file manager listing has a graphic character resembling a 2x2
> matrix, concerning which note that while \302 octal = \xC2 hex, and
> \202 octal = \x82 hex, only the second of these and not the first
> appears in the symbol:
> |00|
> |82|

You aren't going to get anywhere with using high level tools for this. You
need to go low level and see the values of the actual bytes in the filenames.
So for example something like

ls *Chat* | od -A n -t x1

which will show the bytes in hexadecimal.

> My problem is that I can't find a search term to trap this strange
> character to correct it, for example the following, and a few similar
> that I've tried, don't work because they don't find the directory:
> mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
> mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"

What directory ? Your post says that some files have strange names. Do also
some directories have strange names ? In any case , the commands above do not
show a directory separator.

--
Who is the poster boy for posters ?

[Aragorn]

On 14.08.2021 at 22:52, William Unruh scribbled:

> On 2021-08-14, Martin Gregorie <mar...@mydomain.invalid> wrote:
>
> > apropos 'action name' | less
> >
> > as an aid to finding the command they want and than
> >
> > man commandname
>
> Unfortunately man is really good for reminding someone what various
> things mean, but is pretty bad at teaching anyone how to use the
> command. A much much larger section for each man page with explicity
> examples would go a long way to making "man" useful for newbies (and
> oldies as well).

True, but this is what the GNU "info" command was (supposedly) intended
for. Well, that, and a more markup-style browsing experience, with
embedded links to other locally installed pages.

But then again, not all distributions install "info", and conversely,
I have already come across a distribution -- I seem to remember that it
was an earlier release of either PCLinuxOS or Mageia, but I'm not sure
anymore -- that installed the "info" pages by default but not the "man"
pages. And that was odd, because every UNIX system should at the very
least have "man" installed.

On my Manjaro system here, I have both "man" and "info" installed. The
latter did not come installed by default, but I either way find myself
looking far more at "man" pages than at "info" pages, as the "info"
system is actually quite bulky and a bit overkill -- sort of like the
difference between the simplicity of how the original GRUB worked and
the complexity of how GRUB2 works.

[Java Jive]

On 15/08/2021 13:58, Spiros Bousbouras wrote:
>
> On Sun, 15 Aug 2021 12:57:24 +0100
> Java Jive <ja...@evij.com.invalid> wrote:
>>
>> Now the crunch, when I unzip these on a Linux machine, I see different
>> bastardisations of accented characters. So, for example where the full
>> 7zip archive when extracted shows an e acute correctly in both a console
>> and a file manager listing ...
>> "Chat Botté, Le" [e is correctly acute]
>> ... (if you're wondering, a French children's picture book version of
>> apparently 'Puss In Boots'), while with the WinZip main archive a
>> console listing shows a very odd character sequence instead of the e
>> acute ...
>> "Chat Bott'$'\302\202'', Le"
>> ... and a file manager listing has a graphic character resembling a 2x2
>> matrix, concerning which note that while \302 octal = \xC2 hex, and
>> \202 octal = \x82 hex, only the second of these and not the first
>> appears in the symbol:
>> |00|
>> |82|
>
> You aren't going to get anywhere with using high level tools for this. You
> need to go low level and see the values of the actual bytes in the filenames.
> So for example something like
>
> ls *Chat* | od -A n -t x1
>
> which will show the bytes in hexadecimal.

Thanks again, will look into that.

>> My problem is that I can't find a search term to trap this strange
>> character to correct it, for example the following, and a few similar
>> that I've tried, don't work because they don't find the directory:
>> mv "Chat Bott'$'\302\202'', Le" "Chat Botté, Le"
>> mv Chat\ Bott\'$\'\\302\\202\'\',\ Le "Chat Botté, Le"
>
> What directory ? Your post says that some files have strange names. Do also
> some directories have strange names ? In any case , the commands above do not
> show a directory separator.

As part of my manual investigations of the problem, I changed to the
directory of which the problem directory is a direct sub-directory, to
allow experimentation without having to type tediously extended pathnames.

[Chris Elvidge]

Why not:
find /media/somedisk -type f -ls > filelist.txt

Saves a process (or two)


--
Chris Elvidge
England

[Java Jive]

On 15/08/2021 16:00, Java Jive wrote:
>
> On 15/08/2021 13:58, Spiros Bousbouras wrote:
>>
>> You aren't going to get anywhere with using high level tools for this.
>> You
>> need to go low level and see the values of the actual bytes in the
>> filenames.
>> So for example something like
>>
>>      ls *Chat* | od -A n -t x1
>>
>> which will show the bytes in hexadecimal.
>
> Thanks again, will look into that.

As I suspected from the octal, it's C2 82 ...

For example the first line is (though really I need a fixed font to show
this):
43 68 61 74 20 42 6f 74 74 c2 82 2c 20 4c 65 20
C h a t B o t t , L e

[Stéphane CARPENTIER]

Your search looks very quick. I don't use locate but I found its
manpage on Internet. From it, I don't see how to use locate to know
which files have been created recently. For example, I see a lot of
things find can do and not locate.

You can use find to look for files in an usb stick too. I guess running
updatedb on an usb stick may become ugly.

[Paul]

https://stackoverflow.com/questions/4177783/xc3-xa9-and-other-codes/4177813#4177813

It looks like perhaps this "text string" for the filename,
went through some web encoding at some point. With a hex
editor, I can change C3 A9 to E9 hex, and the character in
the hex editor (on the right hand side) looks visually correct.

https://i.postimg.cc/TP57bLD9/C3-A9-to-E9.gif

You could do such an operation, in Perl, right on the
file system.

*********************** rename2.ps *************************
printf("this is a test\n");

$start = "Chat Bott";
$finish = ", Le";
$naughty1 = <\x{C3}\x{A9}> ;
$naughty2 = <\x{E9}> ;

$x = $start.$finish ;
$y = $start.$naughty1.$finish ;
$z = $start.$naughty2.$finish ;

open(OUT, ">>$x") || die("Cannot create X");
close(OUT);

open(OUT, ">>$y") || die("Cannot create Y");
close(OUT);

open(OUT, ">>$z") || die("Cannot create Z");
close(OUT);

use Cwd;

$c = getcwd ;

printf("Making a mess in %s\n", $c );

#rename( $y , $z );

exit(0);
*********************** end of rename2.ps *************************

I ran this in Windows 11, by double-clicking the file. I
could not run it using one of their terminals. I just thought
it was mildly amusing as to what the filenames looked like.

The idea of the script above, is you run it multiple times,
commenting out a line here or there, while you do your tests.
For example, comment out the creation of file $z and
enable the rename(y,z) command near the bottom, to see
if the created $y can be renamed to the presumed operational $z value.

https://i.postimg.cc/gksLyGFL/rename2-output.gif [Picture]

So far, I only tested it as copy/pasted above. I haven't
tested the rename.

Then, you'd need to pick up a recursive tree ("find-next-file")
type pattern, and look for a filename with $naughty1 in it,
and rename it somehow. Maybe something like one of the
examples here. You would probably need to look for a
substring of $naughty1, in the filenames returned.

https://stackoverflow.com/questions/5089680/how-to-find-files-folders-recursively-in-perl-script

File renaming, is the only thing I've done with Perl :-)
I'll never be a Perl person I guess.

Paul

[Paul]

At the time, this was a speed optimization.

I have no idea today, whether it's a good idea or not,
but I still use that.

It's also possible it might blow up on a pathological
file tree of some sort ("line too long").

Paul

[Paul]

Paul wrote:

>
> I ran this in Windows 11

Now, before everyone gets on my case about where I ran it,
I needed to be able to see what the users see when they
unpack their 7zip in Windows, and whether the filename
looks as intended.

If I did the test purely in Linux, against an NTFS file
system, who knows whether the text string display would
look just like it does on Windows. I'm not a character
set expert and cannot predict what those look like on
the Linux side. It's unlikely at the moment, that
Linux will even mount that file system (MFTMIRR) :-/ Thanks
to Microsoft. Only Fedora could mount it without whining.

It's hardly easy to do anything in a heterogenous
environment now. Like pulling teeth with dull pliers.

Paul

[Jasen Betts]

find can search accoding to user, group, permission, age, size, type ...
find can launch other commands, and delete files.

locate only looks at filenames.


--
Jasen.

[J.O. Aho]

On 15/08/2021 20.33, Paul wrote:

> If I did the test purely in Linux, against an NTFS file
> system, who knows whether the text string display would
> look just like it does on Windows. I'm not a character
> set expert and cannot predict what those look like on
> the Linux side.

As long the systems has the same charset, then there shouldn't be any
differences, this do not just apply to Linux but other operating systems
as microsoft windows.

> It's unlikely at the moment, that
> Linux will even mount that file system (MFTMIRR) :-/ Thanks
> to Microsoft. Only Fedora could mount it without whining.

Much depends on the ntfs module loaded, the current in kernel ntfs
support is crappy and still used by some distributions, but most do have
support for the ntfs-3g driver, just you may install it manually.
The good news is that this driver will be in the kernel in a near future.

Mounting BitLock encrypted file systems can also be done on the Linux,
just in case you need to access files from your work computers harddrive.


Had been nice to see an in kernel exFat support too, but I doubt
microsoft has need of that in their Linux distributions, so I doubt they
will provide a driver.

--

//Aho

[jak]

Hi,
you could use the find command looking for filenames as a regular
expression, then use the command you need on them.
In this example I search for files with the extension ".o", display the
name with the command 'echo' and display it again converted to
uppercase:

find . -iregex ".*\.o$" -exec bash -c "echo -n original: {} && echo \"
modified: {}\" | tr [a-z] [A-Z]}" \;

There should be everything you need.

cheers

[Java Jive]

Thanks but no, that doesn't work. I had considered, before the script
works through the data file, of running a pre-process to find and rename
all these characters, but neither find nor ls will actually find the
erroneous characters *DIRECTLY*. The best either can do is find the
characters either side, but that means I have to know in advance where
all the problems are, and I'm not sure yet that I do. Really, if I'm
going to go down that road, I need a way of searching the entire archive
structure directly for affected files and renaming them, as a separate
process from working through the data file.

So, for example, this works because I'm specifying and finding the
neighbouring characters of one known instance, not because ls is finding
the oddball characters directly ...
ls Chat\ Bott?,\ Le | sed 's~\xc2\x82~é~g'
... whereas these don't, with neither single nor double backslashes nor
various other combinations that I've tried, because neither find nor ls
seem able to find the oddball characters directly:
find . -regex ".*\\xc2\\x82.*"
ls -R *\\xc2\\x82*
ls -R *'$'\\302\\202''*

[Spiros Bousbouras]

On Mon, 16 Aug 2021 10:23:23 +0100
Java Jive <ja...@evij.com.invalid> wrote:
> So, for example, this works because I'm specifying and finding the
> neighbouring characters of one known instance, not because ls is finding
> the oddball characters directly ...
> ls Chat\ Bott?,\ Le | sed 's~\xc2\x82~é~g'
> ... whereas these don't, with neither single nor double backslashes nor
> various other combinations that I've tried, because neither find nor ls
> seem able to find the oddball characters directly:
> find . -regex ".*\\xc2\\x82.*"
> ls -R *\\xc2\\x82*
> ls -R *'$'\\302\\202''*

Try ls -R *$'\302\202'*

[Java Jive]

No luck with that either ...
ls: cannot access '*'$'\302\202''*': No such file or directory

I think the trouble with all these methods is that they are specifying a
succession of two characters, where as far as unicode is concerned the
oddballs are single characters, so I fear they will never match, no
matter what magic incantation is used.

So I've been looking at putting in the following as a hack around. It's
designed to search for wildcards in the file name coming from the data
file, and if one is found, escape all the other 'dodgy' characters in it
and use it without quotes, but, although everything *LOOKS* as though it
should work, it gives an error message at the final file testing if
statement:

Before, works except for filenames containing wildcard characters:

if [ -n "${Debug}" ]
then
echo "CE3 = ${CE3}"
fi
if [ "${CE3/./}" != "${CE3}" ]
then
# Is file spec
if [ ! -f "${CE3}" ] # Note quotes
then
if [ -n "${Debug}" ]
then
echo "WARNING - File '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
else
# Is path spec
if [ ! -d "${CE3}" ] # Note quotes
then
if [ -n "${Debug}" ]
then
echo "WARNING - Directory '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
fi


After, try to escape wildcard containing filenames:

# Need to remove any enclosing quotes at this stage
while [ "${CE3:0:1}" == "'" ] && [ "${CE3: -1:1}" == "'" ]
do
CE3="${CE3:1:${#CE3}-2}"
done
while [ "${CE3:0:1}" == "\"" ] && [ "${CE3: -1:1}" == "\"" ]
do
CE3="${CE3:1:${#CE3}-2}"
done
# Check for wildcard chars
if [ "${CE3/\?/}" != "${CE3}" ] || [ "${CE3/\*/}" != "${CE3}" ]
then
# Wildcards, cannot quote, so escape difficult characters
CE3=$(echo "${CE3}" | sed "s~\([ #&'(),;-]}\)~\\\\\1~g")
else
# No wildcards, enclose in quotes
CE3="\"${CE3}\""
fi
if [ -n "${Debug}" ]
then
echo "CE3 = ${CE3}"
# Example output here seems correct, for example ...
# CE3 = ... /Newscuttings\ \-\ Wedding\ Of\ Zo?\ <Surname>.png
fi
if [ "${CE3/./}" != "${CE3}" ]
then
# Is file spec
if [ ! -f ${CE3} ] # Note no quotes
# ... but errors here: <scriptname>: line <num>: [: too many arguments
then
if [ -n "${Debug}" ]
then
echo "WARNING - File '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
else
# Is path spec
if [ ! -d ${CE3} ] # Note no quotes
# ... and here: <scriptname>: line <num>: [: too many arguments
then
if [ -n "${Debug}" ]
then
echo "WARNING - Directory '${CE3}' does not exist!"
fi
Result=1
CE2=""
fi
fi

[Martin Gregorie]

On Mon, 16 Aug 2021 15:33:15 +0100, Java Jive wrote:

> No luck with that either ...
> ls: cannot access '*'$'\302\202''*': No such file or directory
>

Might be worth writing a noddy Java program to see if it can resolve your
problem character codes.

The Java 'char' primitive can hold multibyte character values. and the
Character() class provides methods to recognise character types, lengths,
and non-Unicode characters.

[Java Jive]

On 16/08/2021 16:58, Martin Gregorie wrote:
> On Mon, 16 Aug 2021 15:33:15 +0100, Java Jive wrote:
>
>> No luck with that either ...
>> ls: cannot access '*'$'\302\202''*': No such file or directory
>>
> Might be worth writing a noddy Java program to see if it can resolve your
> problem character codes.
>
> The Java 'char' primitive can hold multibyte character values. and the
> Character() class provides methods to recognise character types, lengths,
> and non-Unicode characters.

But I can't be sure that any of the target machines will have Java,
Perl, or Python installed. This has to be achieved with what will
normally be installed on a Linux or MacOS box.

[Andy Burns]

Java Jive wrote:

> console listing shows a very odd character sequence instead of the e
> acute ...
>     "Chat Bott'$'\302\202'', Le"

Are you sure the filename is exactly as you say/think? What does

ls -b

show?

[jak]

Ok. I finally understood your problem (late age?). I tried to reproduce
your problem and in my opinion you could use this way:

add the -b option to the ls command; this will translate the bad
characters into octal sequence of text then this:

-rw-r--r-- 1 jak NONE 0 Aug 16 16:57 'foo'$'\302\202

$ ls -1 foo*
'foo' $ '\ 302 \ 202'

will become:
$ ls -1b foo*
foo\302\202

now you can search for it as if it were text. For example with the grep
command:

$ ls -1b foo* | grep -F "\\302"
foo\302\202

$ ls -1b | grep -F "foo\\302\\202"
foo\302\202

I hope it helps you
cheers

[Paul]

Using a Perl script, I created some examples.
File "Y" is the php-failure induced problem name the OP has.
File "Z" is the visually-correct one.

https://i.postimg.cc/gksLyGFL/rename2-output.gif

So you can create your own for a test.

*********************** rename2.ps *************************
printf("this is a test\n");

$start = "Chat Bott";
$finish = ", Le";
$naughty1 = <\x{C3}\x{A9}> ;
$naughty2 = <\x{E9}> ;

$x = $start.$finish ;
$y = $start.$naughty1.$finish ;
$z = $start.$naughty2.$finish ;

open(OUT, ">>$x") || die("Cannot create X");
close(OUT);

open(OUT, ">>$y") || die("Cannot create Y");
close(OUT);

open(OUT, ">>$z") || die("Cannot create Z");
close(OUT);

use Cwd;

$c = getcwd ;

printf("Making a mess in %s\n", $c );

#rename( $y , $z );

exit(0);
*********************** end of rename2.ps *************************

Paul

[Martin Gregorie]

On Mon, 16 Aug 2021 17:28:06 +0100, Java Jive wrote:

> On 16/08/2021 16:58, Martin Gregorie wrote:
>> On Mon, 16 Aug 2021 15:33:15 +0100, Java Jive wrote:
>>
>>> No luck with that either ...
>>> ls: cannot access '*'$'\302\202''*': No such file or directory
>>>
>> Might be worth writing a noddy Java program to see if it can resolve
>> your problem character codes.
>>
>> The Java 'char' primitive can hold multibyte character values. and the
>> Character() class provides methods to recognise character types,
>> lengths,
>> and non-Unicode characters.
>
> But I can't be sure that any of the target machines will have Java,
> Perl, or Python installed. This has to be achieved with what will
> normally be installed on a Linux or MacOS box.

Does thet matter? I thought you were treating this archived article name
sanitization as either a one-off activity of something that doesn't
happen regularly and, anyway that it was something that you did on your
system before distributing the results round your family group.

As it happens I've just knocked up a bit of Java to see just what it can
do in the way of automated character translation, so if you'd care to
send me, mar...@gregorie.org, a short file (100-500 chars max) containing
a mix of readable and non-readable example text, I'll run it through my
code.

Attaching it as a gzipped file should get it here without further
mangling.

[Java Jive]

On 16/08/2021 19:46, Paul wrote:
> Andy Burns wrote:
>> Java Jive wrote:
>>
>>> console listing shows a very odd character sequence instead of the e
>>> acute ...
>>>      "Chat Bott'$'\302\202'', Le"
>>
>> Are you sure the filename is exactly as you say/think? What does
>>
>> ls -b
>>
>> show?

Thanks to you and 'jak' for suggesting this!

While still none of the following work ...
ls -R -b | grep '\xc2\x82'
ls -R -b | grep -E '\xc2\x82'
ls -R -b | grep '\uc282'
ls -R -b | grep -E '\uc282'
ls -R -b | grep '\u82c2'
ls -R -b | grep -E '\u82c2'
ls -R -b | grep '\uc282'
ls -R -b | grep -E '\uc282'
ls -R -b | grep '\u82c2'
ls -R -b | grep -E '\u82c2'
... this at least finds all the files that I'm already aware of,
suggesting that I may know about all of them ...
ls -R -b | grep -E '\\[0-7]{3}'

There are 35 files or directories at fault, nearly all are e acute, but
there a couple of e umlaut and 6 files with both an e grave and an e
acute :-(

Now I have to devise a method of renaming them, in other words of
ensuring that the mv command will find them. I've just tried the
following manual command to see what happens (it'll wrap, but originally
it was all one command-line):

OLDIFS=${IFS}; IFS=$'\n'; for A in $(ls -1Rb * | grep -E
'(:|\\302\\202)'; do if [ "S{A: -1}" == ":" ]; then export
LASTDIR="${A/:/}; else pushd "${LASTDIR}"; mv ${A/\\302\\202/?}
${A/\\302\\202/é}; popd; fi; done; unset LASTDIR; IFS=${OLDIFS}

Guess what now! The files were renamed, but the slashes that were
supposed to escape the spaces were included in the name! FFS, HOW
INCONSISTENT IS THAT???!!! Why are the slashes successful in escaping
the spaces in the source name but getting included as part of the target
name? Alright, so I can programme around that, but I shouldn't have to,
the illogicality of it all is just maddening!

> Using a Perl script, I created some examples.
> File "Y" is the php-failure induced problem name the OP has.
> File "Z" is the visually-correct one.

PHP was not involved, it was WinZip that created the problem, whereas 7z
did not, but for one thing, I didn't notice at the time, and for
another, people would have had to install software to handle *.7z files,
whereas the ability to handle *.zip files is native to many/most/all
modern OSs.

> https://i.postimg.cc/gksLyGFL/rename2-output.gif
>
> So you can create your own for a test.
>
> *********************** rename2.ps *************************
> printf("this is a test\n");
>
> $start = "Chat Bott";
> $finish = ", Le";
> $naughty1 = <\x{C3}\x{A9}> ;
> $naughty2 = <\x{E9}> ;

I think this is suffering from the same problem that all the other
approaches have had, that you're creating two characters not one. BTW,
it's hex C2, followed by hex 82.

After some further thought, I remembered about the \u regular expression
syntax. Being unsure of the correct byte order, I tried both, but
neither of the following work either, whereas logically I would have
thought that one of them should:
find . -regex ".*\uc282.*"
find . -regex ".*\u82c2.*"

But at least now there's hope, see above.

Tx again to all.

[STALKING_TARGET_41]

The ScreenFlow CD Snit sock Sockboy mentioned is read-only media. It is
not possible to burn new information to it. Mike Easter must have read
Snit sock Sockboy urls and became awfully resentful. I suspect this is
not our first manifestation of the improved Mike Easter.

Mike Easter's unedited statement stands true and correct. Why would you
want to limit any applications on open source tools to what can be done
on Macs? It is a well know fact when Snit sock Sockboy automatically uses
the word 'attack' the way Fox does, to the point where the word stops
sounding like a word. Snit sock Sockboy asked to be rated according to
direct statistical evidence, which I obliged him with.

--
This broke the Internet
https://duckduckgo.com/?q=%22NARCISSISTIC+BIGOT%22
<https://www.truepeoplesearch.com/results?name=4234911448&Diesel_Gremlin_Dustin_James_Cook>
<https://alt.computer.workshop.narkive.com/dCDisEHZ/dustin-cook-aka-diesel-
aka-gremlin-i-fucked-you-over-with-your-bank-account>
Dustin Cook is a functionally illiterate fraud

[Java Jive]

On 16/08/2021 21:47, Martin Gregorie wrote:
> On Mon, 16 Aug 2021 17:28:06 +0100, Java Jive wrote:
>
>> On 16/08/2021 16:58, Martin Gregorie wrote:
>>> On Mon, 16 Aug 2021 15:33:15 +0100, Java Jive wrote:
>>>
>>>> No luck with that either ...
>>>> ls: cannot access '*'$'\302\202''*': No such file or directory
>>>>
>>> Might be worth writing a noddy Java program to see if it can resolve
>>> your problem character codes.
>>>
>>> The Java 'char' primitive can hold multibyte character values. and the
>>> Character() class provides methods to recognise character types,
>>> lengths,
>>> and non-Unicode characters.
>>
>> But I can't be sure that any of the target machines will have Java,
>> Perl, or Python installed. This has to be achieved with what will
>> normally be installed on a Linux or MacOS box.
>
> Does thet matter? I thought you were treating this archived article name
> sanitization as either a one-off activity of something that doesn't
> happen regularly and, anyway that it was something that you did on your
> system before distributing the results round your family group.

No, I have to have the run one or other of the programs on the machine
of any family member who has already downloaded the first and, as I've
now discovered, faulty version of the archive.

> As it happens I've just knocked up a bit of Java to see just what it can
> do in the way of automated character translation, so if you'd care to
> send me, mar...@gregorie.org, a short file (100-500 chars max) containing
> a mix of readable and non-readable example text, I'll run it through my
> code.
>
> Attaching it as a gzipped file should get it here without further
> mangling.

Thanks, but I'm busy writing my own solution based on what I've already
posted.

[jak]

mmmumble...
... winzip probably got it wrong when saving/restoring files between
systems that have different code pages. the "é" (e-acute), in fact,
corresponds to the position 0x82 in the table cp863 (french codepage)
which is probably not the default in your system. To work around this
problem it is necessary to enable "Store Unicode filenames in Zip files"
in the "Advanced options" of WinZip. This can also be done on systems
that have WinZip integrated.

[Andy Burns]

Java Jive wrote:

> Thanks to you and 'jak' for suggesting this!
>
> While still none of the following work ...

You could show us the "ls -b" output for your previous Chatt Botte
filename ...

[jak]

Il 16/08/2021 23:59, Java Jive ha scritto:

try this way to rename your file with the strange name:

$ find . -iname `echo -e "foo\0302\0202"` -exec mv {} new_name \;

[Java Jive]

On 15/08/2021 12:57, Java Jive wrote:
>
> Can anyone suggest a sequence that will find the file, when put inside
> quotes as the filename in the controlling data file mentioned previously
> in the thread, so that it can just be treated like all the other lines?
> As someone here suggested the data file is now stored as UTF-8 rather
> than ANSI as it was formerly, and some example lines are given below in
> a form for easier readability in a ng  -  in reality the fields are tab
> separated but here are separated by double spacing and have been further
> abbreviated to keep them from wrapping; leading symbols such as '+' and
> '=' have special meanings for the program doing the work; and, yes, the
> commands are basically DOS commands which for Linux are translated to
> their bash equivalents:
>
> =ATTRIB -R  "./F H/Close/Sts Mary & John Churchyard Monuments.pdf"
> =RD  "./F H /_all/1o/Blessig & Heyder"
> REN  "./Chat Bott'$'\302\202'', Le"  "Chat Botté, Le"
> MOVE  "./Photo - D & M Close.png" "./Photos/D & M Close.png"
> [etc]

I've completely fixed the problem with the following code inserted
before processing the data file. Thanks for all the help here that
enabled me to do this. It'll wrap of course, sorry can't help that,
beyond reducing the tabs to two spaces:

# Search for WinZip's botched accented characters
# in the main download of v1: MacFarlane-Main.zip
# 35 pathnames affected, botched characters are:
# Intended Stored incorrectly as
# Char Octal Hex
# é (acute) \302\202 \xC2\x82
# ë (diaeresis) \302\211 \xC2\x89
# è (grave) \302\212 \xC2\x8A
# Á (acute) µ

OLDIFS=${IFS} # Normally IFS=$' \t\n'
IFS=$'\n'
LASTREN=""
for A in $(ls -1bR | grep -E '(:|µ|\\[0-7]{3}\\[0-7]{3})')
do

if [ -n "${Debug}" ]
then

echo "A = \"${A}\""
fi
if [ "${A: -1}" == ":" ]
then
THISDIR="${A/:/}"
if [ "${THISDIR}" == "${LASTREN/ -> .*/}" ]
then
THISDIR="${LASTREN/.* -> /}"

fi
if [ -n "${Debug}" ]
then

echo "THISDIR = \"${THISDIR}\""
fi
else
SC="${A}"
DS="${A}"
while [ -n "$(echo \"${SC}\" | grep -E
'(µ|\\[0-7]{3}\\[0-7]{3})')" ]
do
case $(echo "${SC}" | sed -E
's~^.*(µ|\\[0-7]{3}\\[0-7]{3}).*$~\1~') in
"µ") # A acute
SC="${SC//µ/?}"
DS="${DS//µ/Á}"
;;
"\302\202") # e acute
SC="${SC//\\302\\202/?}"
DS="${DS//\\302\\202/é}"
;;
"\302\211") # e diaeresis
SC="${SC//\\302\\211/?}"
DS="${DS//\\302\\211/ë}"
;;
"\302\212") # e grave
SC="${SC//\\302\\212/?}"
DS="${DS//\\302\\212/è}"
;;
esac
done

DS="${DS//\\/}"
pushd "${THISDIR}"
echo "mv ${SC} \"${DS}\""
if [ -z "${Dummy}" ]
then
mv ${SC} "${DS}"
fi
popd

# Remember rename in case it's a directory containing others
LASTREN="${THISDIR}/${A//\\ / } -> ${THISDIR}/${DS}"

if [ -n "${Debug}" ]
then

echo "LASTREN = \"${LASTREN}\""
fi

fi
done
IFS=${OLDIFS}

[Jasen Betts]

On 2021-08-16, Java Jive <ja...@evij.com.invalid> wrote:
> On 16/08/2021 19:46, Paul wrote:
>> Andy Burns wrote:
>>> Java Jive wrote:
>>>
>>>> console listing shows a very odd character sequence instead of the e
>>>> acute ...
>>>>      "Chat Bott'$'\302\202'', Le"

That's a control character \u0082 "break permitted here"

>>> Are you sure the filename is exactly as you say/think? What does
>>>
>>> ls -b
>>>
>>> show?
>
> Thanks to you and 'jak' for suggesting this!
>
> While still none of the following work ...
> ls -R -b | grep '\xc2\x82'
> ls -R -b | grep -E '\xc2\x82'

There's no chance of that working try fgrep instead, or double up
the backslashes.

what does "ls -b" show?





--
Jasen.

[Steve Carroll]

https://secure.tennesseetrustee.org/index.php?main=Y

Select Sullivan County
Select Pay/Search Property Taxes
Enter 108 Warrior Drive
On second line click View

Why does Dustin Cook's family owe $4,742.45 in property taxes on a home
worth only $115,500

Rodney Wood has been a bad boy it seems!

--
One Smart Penny
https://www.google.com/search?q=Dustin+Cook%3A+functional+illiterate+fraud
https://www.bing.com/search?q=%22narcissistic%20bigot%22
https://www.bing.com/search?q=dustin%20cook%20functionally%20illiterate%20fraud
Dustin Cook: Functionally Illiterate Fraud

[jak]

Just because I had also tried to write a version of the script shell:

These are the files I created for testing:

$ ls -1 jak/foo*
'jak/foo'$'\302\202'
'jak/foo'$'\302\202\302\202'
'jak/foo'$'\302\202\302\211'
'jak/foo'$'\302\212\302\202'
'jak/foo'$'\302\212\302\202''foo'

This is the result of the script:

$ ./renbadch
mv "./jak/foo\302\202" "./jak/fooé"
mv "./jak/foo\302\202\302\202" "./jak/fooéé"
mv "./jak/foo\302\202\302\211" "./jak/fooéë"
mv "./jak/foo\302\212\302\202" "./jak/fooèé"
mv "./jak/foo\302\212\302\202foo" "./jak/fooèéfoo"

This is the code:

#! /usr/bin/bash

regex='([^\\]*[^0-7]*)(\\[0-7]{3})(\\[0-7]{3})'

while read -r ll
do
orig=$ll
transl=""
while [[ $ll =~ $regex ]]
do
start=${BASH_REMATCH[1]}
goodch=$(printf %d ${BASH_REMATCH[3]:1})
newch=$(echo -e "\0${goodch}" | iconv -f 'CP863' -t
'UTF-8')
transl="${transl}${start}${newch}"
m=${BASH_REMATCH[0]}
ll=${ll##*"$m"}
done
echo "mv \"${orig}\" \"${transl}${ll}\""
done < <(find . -type f -exec ls -1b {} + | egrep '\\[0-7]{3}\\[0-7]{3}')

[HHI]

Snit sock Digger Thomnson doesn't have any idea what he is crying about.
Many advocates continue replying to Snit sock Digger Thomnson. I do not
chide Gremlin for being pissed but, truly, I don't get why he writes here
other than to mess with Snit sock Digger Thomnson. Gremlin is far more
interested in conversations as is common in a BBS environment and trolling
environments simply are not it. My uptime is almost fifty-five weeks and
Gremlin is a better man than you will even be!

Sophos was initially coded and transcoded using a bootleg copy of Consulo,
in assembly.

--
I Left My Husband & Daughter At Home And THIS happened!!
https://duckduckgo.com/?q=steve%20carroll%20narcissistic%20bigot
https://duckduckgo.com/?q=%22functionally+illiterate+fraud%22
Steve 'Narcissistic Bigot' Petruzzellis

[STALKING_TARGET_55]

On Tuesday, August 17, 2021 at 7:00:46 AM UTC-7, Jasen Betts wrote:

The one talent Dustin Cook the functional illiterate fraud learned well
is to attempt to humiliate Troll Killer Snit into submission and if that
fails to work, troll-splain or dishonesty change the argument.

LOL! Out-and-out hokum by an uneducated, fabricating, Machiavellian,
tag-teaming laughingstock who would not tell the truth if his life depended
on it.

--
Do not click this link!
https://duckduckgo.com/?q=%22functionally+illiterate+fraud%22
<https://www.truepeoplesearch.com/results?name=4234911448&Diesel_Gremlin_Dustin_James_Cook>
Dustin Cook is a functional illiterate fraud

[Steven]

Why would you want to limit all scripts on open source tools to what can
be done on Windows?

His "troubleshooting competence" blinded him to a plugin he needs that
Rbowman found in under a minute for this software.

Lying sack of shit does it every time. Then the deluge begins. Because
the sissy just has to run to other groups. I nonetheless remain unpersuaded
that these posts are unequivocally from a bot.

That is exactly what's taken place. Rbowman likes to leave out such irksome
little particulars, though. Who else do we know who pulls the same crap?
Oh, that's right, Bilby. Our second Bilby. LOL.

Rbowman gave back what he gets from Bilby as much as he deserves, big
fricking deal. No need to be a howling jerk about it?


--
Get Rich Slow!
https://search.givewater.com/serp?q=Dustin+Cook+%22functional+illiterate+fraud%22
Dustin Cook the Fraud

[Steve - fretwizer]

Commander Kinsey was once again a charlatan by having the pretense to not
debate Snit (a real advocate) via blatantly screaming 'Into the bin with
you!' and then bombarding him again via piggybacking. You _do_ realize
that the massive floods ending up in multiple groups started out as a joke
about hacking Google. With no support at all, as is usual for Commander
Kinsey. Snit (a real advocate) must have read Commander Kinsey urls and
became awfully envious. I suspect this is our first demonstration of the
improved Snit (a real advocate). Did Commander Kinsey think that was clever?



-
What Every Entrepreneur Must Know!
https://duckduckgo.com/?q=Dustin%20Cook%20functional%20illiterate%20fraud
Dustin Cook the Fraud

[STALKING_TARGET_27]

It was Shadow who stated that he and his work friends used to lie to
strangers all the time and it was free entertainment, not even a small
deal.

Sheer drivel by a delusional, tall-tale telling, conniving, colluding
mouth-breather who would not be honest no matter what his socks/shills
say.

Carroll's original statement stands literal and correct. Just look at
Shadow's posts and look at Carroll's, there is nothing for us to learn
from an idiot like Shadow. But so be it, let him keep making a moron
of himself.

Gremlin Dustin Cook:
<https://www.google.com/search?q=gremlin+dustin+cook>
<https://www.bing.com/search?q=gremlin+dustin+cook>
<https://duckduckgo.com/?q=gremlin+dustin+cook>

Functionally Illiterate Fraud:
<https://www.google.com/search?q=functionally+illiterate+fraud>
<https://www.bing.com/search?q=functionally+illiterate+fraud>
<https://duckduckgo.com/?q=functionally+illiterate+fraud>

Steve Carroll Petruzzellis:
<https://www.google.com/search?q=steve+carroll+petruzzellis>
<https://www.bing.com/search?q=steve+carroll+petruzzellis>
<https://duckduckgo.com/?q=steve+carroll+petruzzellis>

Narcissistic Bigot:
<https://www.google.com/search?q=narcissistic+bigot>
<https://www.bing.com/search?q=narcissistic+bigot>
<https://duckduckgo.com/?q=narcissistic+bigot>
Generally, I don't call a comment like Shadow's posts a fantasy right
up until you challenge it (as you are here) and Shadow responds with
the gibberish knowing he can't prove it.

-
What Every Entrepreneur Must Know!

https://gibiru.com/results.html?q=%22functional%20illiterate%20fraud%22
https://www.bing.com/search?q=steve+carroll+the+narcissistic+bigot
https://duckduckgo.com/?q=Steve+Petruzzellis+the+narcissistic+bigot
Dustin Cook the Fraud

[Steve Carroll]

He is a pinhead -- advocates have kill filtered his bot posts. What he
really needs to figure out is a script that keeps switching names, then
bot posts have a shot at being half as bothersome as he is.

Just Wondering is trying again to slime Trend Micro. I'm not irate, in
fact I'm laughing at him because Just Wondering's lying is so clear. He
purposefully didn't list all the tasks that he would soon dismiss... and
we all know why. At least he has trolls rushing to his rescue ;) You won't
go into a rave, piss in all the hard stuff, screw all the boys, snatch
the bananas, and throw up in the foyer without being treated with contempt.

--
I Left My Husband & Daughter At Home And THIS happened

<https://groups.google.com/g/comp.sys.mac.apps/c/VMCw29DnV84>
https://duckduckgo.com/?q=%22NARCISSISTIC+BIGOT%22
https://swisscows.com/web?query=%22narcissistic%20bigot%22

[Steve Carroll]

Allah is the only person I know who had a 'blog' intended to embarrass me
that quickly turned into JavaScript errors with placeholders where jpgs/multimedia
used to be, and if you clicked it, you would be reading about a news article
that was removed for copyright infringement. It's a fact when Allah automatically
uses the word 'troll' the way Trump does, to the point where the word stops
sounding like a word. This is something you know nothing about. All joking
aside, what lie? For most I'd simply suspect it is worth looking into. However,
given that it's Allah I would skip that step and go straight to 'drug-induced
delusion' because that's most of what Allah does. IOW, there is no reason
to respect presumption of innocence.

That is the problem today and American educators don't know what they are
doing; people from older generations are the only ones who know better than
to fall for the New World agenda.

-
This broke the Internet

https://www.bing.com/search?q=dustin%20cook%20functionally%20illiterate%20fraud
Automate Google Groups https://www.youtube.com/watch?v=hYQ4Tg0r0g0&feature=youtu.be

[Steve Carroll]

It was Prescott Computer Guy who flooded David's site millions of times
and denied it. The one talent Prescott Computer Guy learned fully is to
work to humiliate David into back peddling and if that fails to work, shout
him down or completely change the target. Everyone is Prescott Computer
Guy -- the oldest gag in the book.

--
One Smart Penny
https://www.google.com/search?q=dustin%20cook%20functionally%20illiterate%20fraud
https://duckduckgo.com/?q=Dustin%20Cook%20functional%20illiterate%20fraud
https://duckduckgo.com/?q=Steve+Petruzzellis+the+narcissistic+bigot
Steve 'Narcissistic Bigot' Carroll

[STALKING_TARGET_03]

What Native irc clients looks like is what it is and of no concern, especially
if it's better than the competition.

Just look at your videos and look at Diesel's, there is nothing for Diesel
to learn from a small time player like F 'The Flooder' Russell. But so
be it, let him keep making an ignoramus of himself. I am sure one of his
stooges will come to the rescue. Idiot.


--
Best CMS Solution of 2017!
https://www.bing.com/search?q=%22narcissistic%20bigot%22
Steve 'Narcissistic Bigot' Carroll