\scantokens after \endinput

[jfbu]

Hi,

I raised this on the TeXLive list but it is probably wrong list
for such topic. So cross-posting here.

I have encountered a phenomenon with \scantokens executed after \endinput:

---
% execute this file testscan.tex with etex on command line
\def\test{%
\begingroup
\endlinechar-1\everyeof{\noexpand}%
\edef\foo{\scantokens{}}%
\endgroup
}%
\test % ok
% \test\endinput % would be ok
\endinput\test % not ok
---

executing "etex testscan" (or pdftex, xetex, luatex, eptex) gives this:

Runaway definition?
->
! File ended while scanning definition of \foo.
<inserted text>
}
\test ...of {\noexpand }\edef \foo {\scantokens {}
}\endgroup
l.12 \endinput\test
% not ok^^M
? X

I would be curious if some expert could elucidate.

It seems the \noexpand which is usual trick to
be able to use \scantokens inside an \edef
fails here,

and this is caused by the \scantokens being executed
after the \endinput token was already encountered.

As a side note, one may execute some \input otherfile.tex
like this in mainfile.tex : \endinput \input otherfile.tex\relax
but only the first line of otherfile.tex will then be included

I am wondering if that has some relation with the
\scantokens issue above. Perhaps some "end of something"
is encountered earlier than where would be the actual
pseudo EOF which \noexpand helps into silencing.

But I have not read the e-TeX extended TeX sources,
perhaps someone here has.

Best,

Jean-François

[Ulrich Diez]

Jean-François Burnol wrote:

[...]

> I have encountered a phenomenon with \scantokens executed after \endinput:
>
> ---
> % execute this file testscan.tex with etex on command line
> \def\test{%
> \begingroup
> \endlinechar-1\everyeof{\noexpand}%
> \edef\foo{\scantokens{}}%
> \endgroup
> }%
> \test % ok
> % \test\endinput % would be ok
> \endinput\test % not ok
> ---
>
> executing "etex testscan" (or pdftex, xetex, luatex, eptex) gives this:
>
> Runaway definition?
> ->
> ! File ended while scanning definition of \foo.
> <inserted text>
> }
> \test ...of {\noexpand }\edef \foo {\scantokens {}
> }\endgroup
> l.12 \endinput\test
> % not ok^^M
> ? X
>
> I would be curious if some expert could elucidate.

[...]

I don't know if I can be called an expert, but the man who wrote
the TeXbook surely is. ;-)

Chapter 20: Definitions (also called macros) of the TeXbook
has a very large double-dangerous-bend paragraph which
begins with

| Now let’s consider the control sequences that are expanded whenever
| expansion has not been inhibited. Such control sequences fall into
| several classes:

About \endinput you find:

| - \endinput. The expansion is null. The next time TeX gets to the end
| of an \input line, it will stop reading from the file containing that line.

I *think* expecting *some* error-message does correspond to the
TeXbook's description of what behavior of TeX is triggered by
\endinput -- the reason for this expectation is:

\test expands to an \edef whose <definition text> contains \scantokens.

With \endinput\test the next time TeX gets to the end of an
\input line is "inside the \edef" while processing \scantokens which
emulates unexpanded writing tokens to external text file and then
processing that external text file via \input outgoing from current
catcode-régime .

Due to the previously encountered \endinput \scantokens' emulating
of processing that external text file via \input is stopped at the end
of the very first line of that "scantokens-file", i.e., at a moment in time
before the \edef-assignment is complete, and also at a moment in time
where the end of the "scantokens-file" is not reached yet and therefore
\everyeof is not carried out.

This in turn implies that TeX doesn't look at the "scantokens-file"
any more but at the previous input-file while an \edef-assignment
is in progress. Changing to previous input-file without having
\everyeof doing its\noexpand-trick while an \edef-assignment is
in progress yields the runaway-definition-error.


I *think* the error-message encountered by you is misleading:

You find the phrase "> ! File ended while scanning definition of \foo. "

This is not correct. The file did not end while scanning definition of \foo. "

Correct would be: "Reading the file was ceased after reading its first
line due to pending \endinput while scanning definition of \foo. "



Sincerely

Ulrich