Again this problem come to my mind: indent heredoc to match code block's indentation.

[Hongyi Zhao]

Hi,

Once more, the problem come to my mind: indent heredoc to match code block's indentation.

Based on the comments here: https://unix.stackexchange.com/questions/76481/cant-indent-heredoc-to-match-code-blocks-indentation. It seems the following one is preferable:


cat <<-EOF | awk 'NR==1 && match($0, /^ +/){n=RLENGTH} {print substr($0, n+1)}' > ~/Desktop/text.txt
Load time-out reached and nothing to resume.
$(date +%T) - Transmission-daemon exiting.
EOF


Any good ideas for this problem?

Best regards,
HY

[Lew Pitcher]

On August 27, 2020 19:17, Hongyi Zhao wrote:

> Hi,
>
> Once more, the problem come to my mind: indent heredoc to match code
> block's indentation.
>
> Based on the comments here:
> https://unix.stackexchange.com

The few times I've checked out stackexchange, I've found the answers to be,
shall we say, less than optimal. For the most part, I wouldn't blindly trust
any answer I get on stackexchange.

> It seems the following one is preferable:

Preferrable? Not really. It was /one/ suggestion, and seems to be a bit
over-architected for my taste. It certainly wins a UUOC ("Useless Use of
Cat") award, and (given the capabilities of awk(1)) is overly complex for
the task.

> cat <<-EOF | awk 'NR==1 && match($0, /^ +/){n=RLENGTH} {print substr($0,
> n+1)}' > ~/Desktop/text.txt
> Load time-out reached and nothing to resume.
> $(date +%T) - Transmission-daemon exiting.
> EOF
>
> Any good ideas for this problem?

Of the answers in that stackexchange thread, yy preference is for the sed(1)
answer
sed 's/^ *//' >> ~/Desktop/text.txt << EOF

Load time-out reached and nothing to resume.
$(date +%T) - Transmission-daemon exiting.
EOF

It is simple, flexible, and does not involve cat(1).

--
Lew Pitcher
"In Skills, We Trust"

[Lew Pitcher]

A bit better:
sed 's/^[[:blank:]]*//' >> ~/Desktop/text.txt << EOF

Load time-out reached and nothing to resume.
$(date +%T) - Transmission-daemon exiting.
EOF

[Janis Papanagnou]

As done with sed, the awk way also doesn't need to use cat

awk '...' <<-EOF
...
EOF

And the awk code seems to do something different than the sed code.

Awk code: determine the number of leading spaces in the first line
and remove that amount of characters from any line.

Sed code: remove all leading spaces on all lines (and independent
of the number of spaces in the first line).

Janis

[Hongyi Zhao]

This is the wonderful feature which will facilitate the writing of the here-doc block in specific layouts and still can keep it in the out file and meanwhile let me indent the here-doc block according to the locations of the code block where it appears.

[Kaz Kylheku]

On 2020-08-28, Janis Papanagnou <janis_pa...@hotmail.com> wrote:
> As done with sed, the awk way also doesn't need to use cat
>
> awk '...' <<-EOF
> ...
> EOF

I would just do this:
command <<EOF
blah
blah
EOF

That is idiomatic code.

Any coding convention or tool which doesn't like it can just bugger off.

Adding superfluous complexity and execution cycles just to make the
source code look prettier is a poor idea.

I will give you another reason for avoiding this. A here doc's
body is supposed to be a template, with perhaps some variant
pieces such as substituted variables.

The non-variant parts are verbatim.

If you indent it, it is no longer verbatim. A here doc may be derived
from some original text somewhere; if whitespace is added, it doesn't
match that text any more. Someone needing to keep the two in sync has
to deal with the annoying indentation.

If I had to write a script that contains a lot of here doc templates
that have to be maintained, I would break them out into template
files, processed with a bit of gluing and eval:

$ cat > template
Dear $USER,

your home directory is $HOME.
$ eval "$(echo "cat <<EOFMARKER" ; cat template; echo EOFMARKER)"
Dear kaz,

your home directory is /home/kaz.

Now there is no problem; the script is nicely indented, and the
texts are broken out into nice template files which require
no indentation, since they are not nested in anything.
Of course, these are now rather "there documents", than
"here documents".

Another way is to prepare the here doc texts in variables,
in a dedicates section of the script, outside of any
function. Then just refer to the variables

$ template=$(cat <<'EOF'
Dear $USER,

Your home directory is $HOME.
EOF
)

$ eval "$(echo "cat <<EOFMARKER" ; echo "$template"; echo EOFMARKER)"
Dear kaz,

Your home directory is /home/kaz.

[Janis Papanagnou]

On 28.08.2020 17:58, Kaz Kylheku wrote:
> On 2020-08-28, Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>> As done with sed, the awk way also doesn't need to use cat
>>
>> awk '...' <<-EOF
>> ...
>> EOF
>
> I would just do this:
> command <<EOF
> blah
> blah
> EOF
>
> That is idiomatic code.

Both versions are idiomatic, IMO, and standard; not sure why you think
the other idiom would not be one. But, frankly, I don't see much gain
to discuss that; I am using both versions as they seem to best fit to
the given program context. (Above I've just copied the original code.)

> Any coding convention or tool which doesn't like it can just bugger off.

It's the '-' version's property usually makes the code structure clearer
and thus increases readability and thus maintainability. The downside is
that at the same time it also decreases maintainability; invisible white
space differences in code are harder to maintain. (The OP's request is
one way to tackle that issue.)

>
> Adding superfluous complexity and execution cycles just to make the
> source code look prettier is a poor idea.

Opinions obviously differ. But I see where you're coming from.

There are two potential complexities here; runtime and code. For such
small pieces of template code the first one is typically negligible.
The second one (if one uses that idiom regularly) can be addrdessed by
using a function, say 'aligned_indent', so that the not really complex
but probably annoying awk/sed/whatever code is hidden.

aligned_indent <<EOT >outfile
...
EOT

> I will give you another reason for avoiding this. A here doc's
> body is supposed to be a template, with perhaps some variant
> pieces such as substituted variables.
>
> The non-variant parts are verbatim.
>
> If you indent it, it is no longer verbatim. A here doc may be derived
> from some original text somewhere; if whitespace is added, it doesn't
> match that text any more. Someone needing to keep the two in sync has
> to deal with the annoying indentation.

I've never had this case. I've done a lot with heredoc-templates; what
they had in common was that *I* was the one that defined the contents,
be it html-code, source code, data, whatever.

As mentioned already, I take that issue very pragmatically and don't
mind innovative code patterns like the one the OP quoted from SE.

One final remark for Kornshell users - the shell that I regularly use;
I think it's not widely known that ksh has that feature as a built-in.
With the syntax

cat <<# EOT
...
EOT

you get exactly what the OP asked for, without need for other code that
complicates the shell program's source code structure.

Janis

> [...]

[Lew Pitcher]

Agreed. My UUOC comment wasn't about /any/ awk solution; it was about the
specific awk solution provided on stackexchange and copied here by Hongyi
Zhao.

> And the awk code seems to do something different than the sed code.
>
> Awk code: determine the number of leading spaces in the first line
> and remove that amount of characters from any line.

Yes. I noticed that. Hence my comment about it being "a bit over-architected
for my taste". That particular awk code enforces a (IMHO unnecessary)
/requirement/ that each line be prefaced with exactly the same indentation
as the /first/ line. Should that first line's indentation be altered (by
intent or by accident), all the remaining lines /must/ be altered in the
same way. That's something that needs to be documented.

> Sed code: remove all leading spaces on all lines (and independent
> of the number of spaces in the first line).

Agreed, I took a simpler approach, and just discard all indentation. A more
sophisticated sed might be
sed 's/^ *|//' >> ~/Desktop/text.txt << EOF

| Load time-out reached and nothing to resume.
| $(date +%T) - Transmission-daemon exiting.
EOF

but, once we go there, we are almost at the awk solution.

[Janis Papanagnou]

On 28.08.2020 19:47, Lew Pitcher wrote:
> [...]

> Agreed, I took a simpler approach, and just discard all indentation.

...which missed the original point to keep the second level indent intact.

> A more sophisticated sed might be
> sed 's/^ *|//' >> ~/Desktop/text.txt << EOF
> | Load time-out reached and nothing to resume.
> | $(date +%T) - Transmission-daemon exiting.
> EOF
> but, once we go there, we are almost at the awk solution.

Not that bad, IMO. With that small adjustment it got effective and is now
also much simpler than the upthread posted awk code. With that dodge we
can now even switch to yet simpler tools, like cut(1), and also avoid sed.

Janis

[Hongyi Zhao]

I don't this method is graceful. It will need to type a leading | for each line.

[Kaz Kylheku]

The method is very graceful. It makes plainly readable what
you are generating. There is no guess-work about which part of the
whitespace is stripped and which is part of the output.

It allows the first line to have leading whitespace, like the | Load
line in the example.

Unobtrusive explicitness is often better than implicitness.

> It will need to type a leading | for each line.

There is an excellent solution which requires you to type nothing
extra for each line, namely:

cat >> ~/Desktop/text.txt << EOF

Load time-out reached and nothing to resume.
$(date +%T) - Transmission-daemon exiting.
EOF

The identation support you're asking for requires extra typing at the
start of each line.

[Lew Pitcher]

Grace, like beauty, is in the eye of the beholder.

> It will need to type a leading | for each line.

Each alternative presented, including the one you suggested from
stackexchange, has it's bad points. You asked for suggestions, and you got
them.

While we probably haven't exhausted the alternatives yet, we have come to
the end of my part in this discussion.

Luck be with you on your quest.

[Hongyi Zhao]

This one will always give the following warning:

line 4: warning: here-document at line 1 delimited by end-of-file (wanted `-EOF')


Then I changed it into the following one:


sed 's/^ *|//' > ~/text.txt <<-EOF

| Load time-out reached and nothing to resume.
| $(date +%T) - Transmission-daemon exiting.
EOF

This need the tabs are placed in front of the second EOF.

[Janis Papanagnou]

On 29.08.2020 01:53, Kaz Kylheku wrote:
> On 2020-08-28, Hongyi Zhao <hongy...@gmail.com> wrote:
>> On Saturday, August 29, 2020 at 1:47:37 AM UTC+8, Lew Pitcher wrote:
>>> Agreed, I took a simpler approach, and just discard all indentation. A more
>>> sophisticated sed might be
>>> sed 's/^ *|//' >> ~/Desktop/text.txt << EOF
>>> | Load time-out reached and nothing to resume.
>>> | $(date +%T) - Transmission-daemon exiting.
>>> EOF
>>
>> I don't this method is graceful.

It's much better (with less issues; see below) than other proposals
I've seen; in my opinion it is very elegant, and by adding a visible
structure it supports creating maintainable code.

> The method is very graceful. It makes plainly readable what
> you are generating. There is no guess-work about which part of the
> whitespace is stripped and which is part of the output.
>
> It allows the first line to have leading whitespace, like the | Load
> line in the example.
>
> Unobtrusive explicitness is often better than implicitness.

I agree with that and want to also emphasize that the original awk code
has (yet unaddressed) issues; the awk code will work on characters and
character counts, with all consequences on subsequent lines if lines are
shorter and/or if TABs are also used.[*] - Lew's pipe-based suggestion
does not have these issues.

Also the OP's other complaint (quoted from Lew's reply)

>> It will need to type a leading | for each line.

is not really substantial. The whole here document and the indentation
is typically typed or at least formatted; if you (Hongyi) mind typing
one more character per line then you may have an inherent problem with
the choice of your job. A hint; if you have really long here documents
then use an editor's feature to insert that pipe character (along with
the indentation). These few seconds additional "work" will make life
for folks who have to work with your scripts (including yourself) much
easier.

Janis

[*] The ksh built-in does not has there issues, just BTW; it accepts
blanks and tabs and also doesn't cut payload data on shorter lines.

[gerg]

Hongyi Zhao <hongy...@gmail.com> wrote:
>
>Once more, the problem come to my mind: indent heredoc to match code block's
>indentation.
>

[suggestion from a web forum deleted]

>
>Any good ideas for this problem?
>

My idea is simple: Don't indent, and explain with a comment


previous line of code
previous line of code

# Note: the below here doc is not indented
command <<EOHD
Contents of the here doc
EOHD

subsequent line of code
subsequent line of code


The unindented here doc may be rather ugly, but IMO it's far
easier for less experienced people to understand and maintain
the group of unindented lines than to work with the techniques
I've seen for handling (removing) the indention.

-Greg
--
::::::::::::: Greg Andrews ::::: ge...@panix.com :::::::::::::
I have a map of the United States that's actual size.
-- Steven Wright

[Jorgen Grahn]

On Tue, 2020-09-01, gerg wrote:
> Hongyi Zhao <hongy...@gmail.com> wrote:
>>
>>Once more, the problem come to my mind: indent heredoc to match code block's
>>indentation.
>>
> [suggestion from a web forum deleted]
>>
>>Any good ideas for this problem?
>>
>
> My idea is simple: Don't indent, and explain with a comment

...

>
> The unindented here doc may be rather ugly, but IMO it's far
> easier for less experienced people to understand and maintain
> the group of unindented lines than to work with the techniques
> I've seen for handling (removing) the indention.

Yes. And I can repeat what I added /last/ time he asked the same
question, in January:

I tend to move large, boring here-docs into a separate shell
function. Then use the standard output from the function. The
indentation is still ugly, but limited to that function.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Janis Papanagnou]

On 01.09.2020 09:16, gerg wrote:
> Hongyi Zhao <hongy...@gmail.com> wrote:
>>
>> Once more, the problem come to my mind: indent heredoc to match code block's
>> indentation.
>>
> [suggestion from a web forum deleted]
>>
>> Any good ideas for this problem?
>>
>
> My idea is simple: Don't indent, and explain with a comment

Q: How can I achieve in heredocs to keep 2-level indentation intact?
A: By not indenting.

You missed the point.

Janis

> [...]

[Janis Papanagnou]

Yes, there are arbitrary workarounds for that supposedly simple demand
that more or less alleviate the heredoc's inherent issue.

The interesting question, though, is about approaches how to handle it,
preferably in a good and maintainable way. There's a reason why folks
are asking that question and are seeking for more elegant solutions
than applying any more or less clumsy workaround (which can be useful,
don't get me wrong). And there's also a reason why Kornshell decided
to address that feature while others still proclaim how bad it is to
try to avoid a clearer heredoc code.

Janis

> /Jorgen
>

[gerg]

In article <ril91n$v4d$1...@news-1.m-online.net>,

Thanks Janis, but that's not the case. I knew HZ wanted ideas about indenting
but I wanted to suggest he think outside that particular box. Sometimes a request
for directions to a destination needs the answer, "You can't get there from here."
I, too, have struggled with maintaining indentation in a shell script here doc,
so I didn't make my suggestion lightly.

If you'll forgive me for being a bit flippant, he did ask for *good* ideas for
the problem...

[Janis Papanagnou]

On 09.09.2020 07:19, gerg wrote:
> In article <ril91n$v4d$1...@news-1.m-online.net>,
> Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>> On 01.09.2020 09:16, gerg wrote:
>>> Hongyi Zhao <hongy...@gmail.com> wrote:
>>>>
>>>> Once more, the problem come to my mind: indent heredoc to match code block's
>>>> indentation.
>>>>
>>> [suggestion from a web forum deleted]
>>>>
>>>> Any good ideas for this problem?
>>>>
>>>
>>> My idea is simple: Don't indent, and explain with a comment
>>
>> Q: How can I achieve in heredocs to keep 2-level indentation intact?
>> A: By not indenting.
>>
>> You missed the point.
>>
>
> Thanks Janis, but that's not the case. I knew HZ wanted ideas about indenting
> but I wanted to suggest he think outside that particular box.

Okay. You ignored the point.

> Sometimes a request
> for directions to a destination needs the answer, "You can't get there from here."
> I, too, have struggled with maintaining indentation in a shell script here doc,
> so I didn't make my suggestion lightly.
>
> If you'll forgive me for being a bit flippant, he did ask for *good* ideas for
> the problem...

I don't see how ignoring the question would be a "good idea".

From own experience I understand the intention for the OP's demand very well.
Luckily I even have a simple and very good solution provided by my shell.
The existance of the ksh-construct <<# is also an indication that such a
demand, as the OP's has it, is widely asked for.

Janis

> -Greg
>

[Anssi Saari]

Janis Papanagnou <janis_pa...@hotmail.com> writes:

> Luckily I even have a simple and very good solution provided by my shell.
> The existance of the ksh-construct <<# is also an indication that such a
> demand, as the OP's has it, is widely asked for.

Interesting, I never heard of that ksh feature before. zsh and mksh have
<<- which strips leading tabs from here documents but not spaces.

[Janis Papanagnou]

That's the standard heredoc syntaxes. The <<# I've seen only in ksh and
it is a non-standard extension. I'm sure that feature is not widely known,
so I am not surprised that you haven't heard about it. I also don't think
that older ksh versions had it - at least I haven't notice it in ksh88 or
earlier versions of ksh93 (but I might have missed it, too). Even in the
man page (at least on my system) it's not explicitly listed by syntax but
only described in the text passage where heredocs ( << and <<- ) are
described:

<<[-]word The shell input is read up to a line that is the same as
word after any quoting has been removed, or to an end-of-
file. No parameter substitution, command substitution,
arithmetic substitution or file name generation is per‐
formed on word. The resulting document, called a here-
document, becomes the standard input. If any character
of word is quoted, then no interpretation is placed upon
the characters of the document; otherwise, parameter
expansion, command substitution, and arithmetic substitu‐
tion occur, \new-line is ignored, and \ must be used to
quote the characters \, $, `. If - is appended to <<,
then all leading tabs are stripped from word and from the
document. If # is appended to <<, then leading spaces
and tabs will be stripped off the first line of the docu‐
ment and up to an equivalent indentation will be stripped
from the remaining lines and from word. A tab stop is
assumed to occur at every 8 columns for the purposes of
determining the indentation.

So one has to thoroughly read the docs or stumble across it by accident.

Janis

[Jorgen Grahn]

On Wed, 2020-09-09, Janis Papanagnou wrote:
...

> That's the standard heredoc syntaxes. The <<# I've seen only in ksh and
> it is a non-standard extension. I'm sure that feature is not widely known,
> so I am not surprised that you haven't heard about it. I also don't think
> that older ksh versions had it - at least I haven't notice it in ksh88 or
> earlier versions of ksh93 (but I might have missed it, too). Even in the
> man page (at least on my system) it's not explicitly listed by syntax but
> only described in the text passage where heredocs ( << and <<- ) are
> described:
>
> <<[-]word The shell input is read up to a line that is the same as
> word after any quoting has been removed, or to an end-of-
> file. No parameter substitution, command substitution,

...

> quote the characters \, $, `. If - is appended to <<,
> then all leading tabs are stripped from word and from the
> document. If # is appended to <<, then leading spaces
> and tabs will be stripped off the first line of the docu‐

...

>
> So one has to thoroughly read the docs or stumble across it by accident.

Ugh. If I were you, I'd file a bug report on the documentation, and
maybe include a patch. Implementing something and then hiding the
documentation is so wasteful.

[Janis Papanagnou]

On 09.09.2020 22:24, Jorgen Grahn wrote:
> On Wed, 2020-09-09, Janis Papanagnou wrote:
> ...

>> [...]

>> So one has to thoroughly read the docs or stumble across it by accident.
>
> Ugh. If I were you, I'd file a bug report on the documentation, and
> maybe include a patch. Implementing something and then hiding the
> documentation is so wasteful.

Is Kornshell still supported and development continued after David Korn
and Glen Fowler have been fired (and went to Google?) a couple years ago?

But what sort of patch are you expecting? A man page documentation patch?
(Being a non-native Englisch speaker I certainly don't qualify for that.)

Janis

[Jorgen Grahn]

On Wed, 2020-09-09, Janis Papanagnou wrote:
> On 09.09.2020 22:24, Jorgen Grahn wrote:
>> On Wed, 2020-09-09, Janis Papanagnou wrote:
>> ...
>>> [...]
>>> So one has to thoroughly read the docs or stumble across it by accident.
>>
>> Ugh. If I were you, I'd file a bug report on the documentation, and
>> maybe include a patch. Implementing something and then hiding the
>> documentation is so wasteful.
>
> Is Kornshell still supported and development continued after David Korn
> and Glen Fowler have been fired (and went to Google?) a couple years ago?

I have no idea, I don't use ksh. I guess if I wanted to change
something, and since I use Debian, I'd send the report to them and let
the Debian feed it upstream.

> But what sort of patch are you expecting? A man page documentation patch?

Yes; you pointed out that the man page was misleading.

> (Being a non-native Englisch speaker I certainly don't qualify for that.)

Your english seems good, and anyway you don't have to be a poet
to write a man page. Clear communication is enough.

[Anssi Saari]

Janis Papanagnou <janis_pa...@hotmail.com> writes:

> On 09.09.2020 22:24, Jorgen Grahn wrote:
>> On Wed, 2020-09-09, Janis Papanagnou wrote:
>> ...
>>> [...]
>>> So one has to thoroughly read the docs or stumble across it by accident.
>>
>> Ugh. If I were you, I'd file a bug report on the documentation, and
>> maybe include a patch. Implementing something and then hiding the
>> documentation is so wasteful.
>
> Is Kornshell still supported and development continued after David Korn
> and Glen Fowler have been fired (and went to Google?) a couple years ago?

There has been some activity on Github even this year so someone is
doing something. There's even a release of ksh2020 but there's a note
saying that version is not supported or maintained. Weird.

So it's probably possible to get a documentation patch in but for
example Debian packages the stable branch (ksh93u) which probably won't
accept patches. Debian might though.

[jo...@schily.net]

In article <sm0r1ra...@lakka.kapsi.fi>, Anssi Saari <a...@sci.fi> wrote:
>Janis Papanagnou <janis_pa...@hotmail.com> writes:

>> Is Kornshell still supported and development continued after David Korn
>> and Glen Fowler have been fired (and went to Google?) a couple years ago?

The fact that they have been fired is not the reason for the frozen code.
David is no longer working for Google. If you are interested, the reason
is retrievable in the net...

>There has been some activity on Github even this year so someone is
>doing something. There's even a release of ksh2020 but there's a note
>saying that version is not supported or maintained. Weird.

The github repo set up by AT&T is not recommended as a source for ksh93
anymore, unless you just like to pull the version from December 2012.

The current changes there are from Redhat people that do not understand the
code and destroyed a lot. The portability is e.g. missing...

There is a new attempt at https://github.com/ksh93/ksh that looks much better.

>So it's probably possible to get a documentation patch in but for
>example Debian packages the stable branch (ksh93u) which probably won't
>accept patches. Debian might though.

Debian does not make development....
--
EMail:jo...@schily.net (home) Jörg Schilling D-13353 Berlin
joerg.s...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
URL: http://cdrecord.org/private/ http://sourceforge.net/projects/schilytools/files/