Nice space token...
GL
I didn't know that:
\documentclass [a4paper]{minimal}
\begin{document}\makeatletter
{\loggingall
\let\test \@sptoken \relax
\meaning\test % \@sptoken or \relax ?
}
\end{document}\endinput
Sorry for the *big* noise...
Martin Scharrer
One space token is optional after the first token of \let.
This seems also be the case if the space token is an explicit one.
I fell over this once in my code.
It is saver to write \let\test= \@sptoken (note the space after '=').
I got told by Joseph Wright that LaTeX3 uses a special definition of
\let which adds the '=<space>' automatically
to avoid exactly such things.
Martin
Ulrich D i e z
> Hello,
>
> I didn't know that:
>
>
> \documentclass [a4paper]{minimal}
> \begin{document}\makeatletter
>
> {\loggingall
> \let\test \@sptoken \relax
>
> \meaning\test % \@sptoken or \relax ?
>
> }
> \end{document}\endinput
The pieces of the puzzle are provided in TeXBook Chapter 24:
Summary of Vertical mode:
| We shall define TeX's parts of speech by using a modified form of
| the grammatical notation that was introduced about 1960 by John
| Backus and Peter Naur for the definition of computer languages.
[...]
| <optional spaces> ----> <empty> | <space token><optional spaces>
[...]
| The quantity <empty> stands for "nothing," i.e., for no tokens at
| all;
[...]
| The quantity <space token>, which was used in the syntax of
| <optional spaces> above, stands for an explicit or an implicit
| space. In other words, it denotes either a character token of
| category 10, or a control sequence or active character whose
| current meaning has been made equal to such a token by \let or
| \futurelet.
[...]
| <one optional space> ----> <space token>|<empty>
[...]
| <equals> ----> <optional spaces>|<optional spaces>=_{12}
[...]
| <let assignment> ----> \futurelet[...]
| |\let<control sequence><equals><one optional space><token>
That means:
In case the \let-assignment contains an equal sign of
catcode 12 (other), any <space token> between \let and the
equal-sign will be discarded. One space-token traling the
equal-sign will be discarded.
In case the \let-assignment does not contain an equal sign of
catcode 12 (other), any <space token> trailing the \let-token
will be discarded.
<space token> can be an implicit space-token like \@sptoken as well.
( Be aware that according to what can be read in the TeXBook
about the quantity <space token>, a character-token of catcode
10 but with character-code differing from 32 (-> funny spaces)
is also a <space token>. You can get such tokens by
\lowercasing or \uppercasing character 32 after changing
\lccode32 or \uccode32 and such tokens might get discarded as
well. )
When needing to make sure to fetch exactly the next token (e.g. in
\let-\afterassignment-loops), provide both the catcode-12-=-character-
token and a single space-token trailing that catcode-12-=-character-
token.
% minimal example for TeX and LaTeX:
%
\catcode`\@=11
\def\@sptoken#1{#1}%
\@sptoken{\let\@sptoken= } % <- keep this comment-char!!!!!
\message{\string\@sptoken: \meaning\@sptoken}
\immediate\read -1 to \test
\let\test\@sptoken\@sptoken\@sptoken\@sptoken\@sptoken\@sptoken\relax
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\let\test\@sptoken\@sptoken\@sptoken\@sptoken\@sptoken=\@sptoken\relax
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\let\test\@sptoken\@sptoken\@sptoken\@sptoken=\@sptoken\@sptoken\relax
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\let\test=\@sptoken\@sptoken\relax
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\lccode32=65
\lowercase{\message{Funny-Space: }}
\immediate\read -1 to \test
\def\spaceinsert#1{\let\test=#1\relax}%
\lowercase{\spaceinsert{ }}
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\def\spaceinsert#1{\let\test=#1#1\relax}%
\lowercase{\spaceinsert{ }}
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\def\spaceinsert#1{\let\test#1#1=#1\relax}%
\lowercase{\spaceinsert{ }}
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\def\spaceinsert#1{\let\test#1#1=#1#1\relax}%
\lowercase{\spaceinsert{ }}
\message{\string\test: \meaning\test}
\immediate\read -1 to \test
\csname stop\endcsname\bye
Ulrich
GL
> GL wrote:
>
> ( Be aware that according to what can be read in the TeXBook
> about the quantity<space token>, a character-token of catcode
> 10 but with character-code differing from 32 (-> funny spaces)
> is also a<space token>. You can get such tokens by
> \lowercasing or \uppercasing character 32 after changing
> \lccode32 or \uccode32 and such tokens might get discarded as
> well. )
Yes but I wonder of what use could be those "funny spaces" apart to
serve as delimiters: certainly a good delimiter because those funny
spaces don't behave like the space token.
While with \scantokens you finally get a real space:
{\lccode32=44 \lowercase{\gdef\funnyspaceA{ }} }
\scantokens{{\catcode44=10\gdef\funnyspaceB{,}}\ignorespaces}
\ifx \funnyspaceA\funnyspaceB yes\else no \fi
\def\test\: {gobble space}
"\expandafter\@gobble\funnyspaceA A"
"\expandafter\@gobble\funnyspaceB A"
\expandafter\test\expandafter\:\funnyspaceB
%\expandafter\test\expandafter\:\funnyspaceA % <doen't match it's
% definition>
Regards.
GL
> GL wrote:
> \catcode`\@=11
> \def\@sptoken#1{#1}%
> \@sptoken{\let\@sptoken= } %<- keep this comment-char!!!!!
> \message{\string\@sptoken: \meaning\@sptoken}
In TeX this is simpler:
\afterassignment\romannumeral\futurelet\@sptoken0
Dan Luecking
In _real_ LaTeX it is even simpler :
\@firstofone{\let\@sptoken= } %
The first and last lines you quote are not part of
the code. Plus the first definition of \@sptoken
reinvents \@firstofone already present in LaTeX.
Actually LaTeX already contains \@sptoken so the
code required to define it in LaTeX is even simpler:
nothing. The LaTeX kernel defines \@sptoken by
\def\:{\let\@sptoken= } \: %
Your example code was explicitly stated to be for
both TeX and LaTeX. It is unclear to me whether the
romannumeral trick is somehow "better". I would look
at it, say "cute trick", and go on to real coding.
Dan
To reply by email, change LookInSig to luecking
Ulrich D i e z
> Yes but I wonder of what use could be those "funny spaces" apart to
> serve as delimiters: certainly a good delimiter because those funny
> spaces don't behave like the space token.
I seldom use them. Unlike ordinary space-character-tokens (catcode10,
character-code 32) they are not discarded when it comes to reading
undelimited macro-arguments:
\def\insertarg#1#2{#2#1#1#1#1#1#1#1}
\def\test#1#2{1ST_ARG:{#1} 2ND_ARG:{#2} NOT ARG ANY MORE.}
\message{\insertarg{ }{\test{FIRST}} {PROBABLY_SECOND}}
\immediate\read-1to\ItchyAndScratchy
\uccode32=65
\uppercase{\message{\insertarg{ }{\test{FIRST}} {PROBABLY_SECOND}}}
\immediate\read-1to\ItchyAndScratchy
\csname stop\endcsname\bye
You can \lowercase spaces/make spaces "funny" for replacing
them by other characters in some situations, but that's an "edge-
case-scenario" and I don't think its used in practice:
\begingroup
\lccode32=65
% ... you'd probably disable any other lccodes...
\lowercase{\scantokens{\def\spacesreplaced{ b b b }}}
\show\spacesreplaced
\lowercase{\scantokens{\endgroup\def\spacesreplaced{ }}}
\show\spacesreplaced
\message{Expansion of \string\spacesreplaced\space is
\expandafter\ifx\spacesreplaced Aequal to
\else different from \fi A.}
\immediate\read-1to\spacesreplaced
\csname stop\endcsname\bye
> While with \scantokens you finally get a real space:
Yes. \scantokens is like writing tokens unexpanded to file
and reading them back from file and tokenizing them under
the catcode-regime valid at the time of reading them back.
The "reading-back"-part is related to TeX' reading-apparatus
while in TeXbook - Chapter 8: The Characters You Type, you
can read:
| If TeX sees a character of category 10 (space), the action depends on
| the current state. If TeX is in state N or S, the character is simply
| passed by, and TeX remains in the current state. Otherwise TeX is in
| state M; the character is converted to a token of category 10 whose
| character code is 32 and TeX enters state S. The character code in a
| space token is always 32.
Ulrich
GL
> GL wrote:
>
>> Yes but I wonder of what use could be those "funny spaces" apart to
>> serve as delimiters: certainly a good delimiter because those funny
>> spaces don't behave like the space token.
>
> I seldom use them. Unlike ordinary space-character-tokens (catcode10,
> character-code 32) they are not discarded when it comes to reading
> undelimited macro-arguments:
>
> \def\insertarg#1#2{#2#1#1#1#1#1#1#1}
> \def\test#1#2{1ST_ARG:{#1} 2ND_ARG:{#2} NOT ARG ANY MORE.}
> \message{\insertarg{ }{\test{FIRST}} {PROBABLY_SECOND}}
> \immediate\read-1to\ItchyAndScratchy
> \uccode32=65
> \uppercase{\message{\insertarg{ }{\test{FIRST}} {PROBABLY_SECOND}}}
> \immediate\read-1to\ItchyAndScratchy
> \csname stop\endcsname\bye
>
> You can \lowercase spaces/make spaces "funny" for replacing
> them by other characters in some situations, but that's an "edge-
> case-scenario" and I don't think its used in practice:
Yes good ! I imagined that but I didn't manage to do it.
> \begingroup
> \lccode32=65
> % ... you'd probably disable any other lccodes...
Or first activate b then use \textcase from D Carlisle
(a good package): \textcase works like etoolbox \protecting
Then macro b active : -> b (funny) + protect until next b
> \lowercase{\scantokens{\def\spacesreplaced{ b b b }}}
\scantokens is useless here !
> \show\spacesreplaced
> \lowercase{\scantokens{\endgroup\def\spacesreplaced{ }}}
\scantokens is useless here !
> \show\spacesreplaced
> \message{Expansion of \string\spacesreplaced\space is
> \expandafter\ifx\spacesreplaced Aequal to
> \else different from \fi A.}
Obviously you didn't change the catcodes !
> \immediate\read-1to\spacesreplaced
> \csname stop\endcsname\bye
Here it is:
\def\defspacesreplaced
#1{\catcode32=11\def\spacesreplaced{#1}\catcode32=10\ignorespaces}
\begingroup
\lccode32=65
\lowercase{\endgroup\scantokens{\defspacesreplaced{ }}}
\message{Expansion of \string\spacesreplaced\space is
\expandafter\ifx\spacesreplaced Aequal to
\else different from \fi A.}
\immediate\read-1to\spacesreplaced
>> While with \scantokens you finally get a real space:
>
> Yes. \scantokens is like writing tokens unexpanded to file
> and reading them back from file and tokenizing them under
> the catcode-regime valid at the time of reading them back.
Not really ! \scantokens adds spaces. Noone takes care about that.
I've worked as a slave to be able to put Verbatim and listings
inside tabu with \scantokens. Noone takes care that
\end{Verbatim} \end{lstlisting} etc.
Becomes \end {Verbation} \end {lstlisting}
with a space that has the catcode of the current regime (that is
active, most of the time)
> | If TeX sees a character of category 10 (space), the action depends on
> | the current state. If TeX is in state N or S, the character is simply
> | passed by, and TeX remains in the current state. Otherwise TeX is in
> | state M; the character is converted to a token of category 10 whose
> | character code is 32 and TeX enters state S. The character code in a
> | space token is always 32.
Yes and this makes easier replacing a character by a space with
scantokens: no need of \lowercase which lowers everything else !
This is good for argument parsing.
Like in tabu: commas are like spaces... thank to \scantokens.
The "bastard" issue of \tracingnested is canceled out by LuaTeX.
I don't know if it's really an improvement.
Respecting \tracingnested makes an fine analyse of the code absolutely
necessary: what is in \scantokens what is outside.
Try:
\scantokens{\halign{..#&#\cr one & two \cr}}% # => ## \let\Sharp=#
\halign{#&#\cr \scantokens{one & two\cr}}
\halign{# & #\cr \scantokens{one}&\scantokens{two}\cr}
=> well easy this one !
just \scantokens {\ifnum0=`}\fi does not work: it's not \vadjust!
Not so easy to get out with that...
> Ulrich
GL
> On Mon, 28 Mar 2011 19:09:23 +0200, GL<goua...@gmail.com> wrote:
>
>> Le 28/03/2011 18:10, Ulrich D i e z a écrit :
>>> GL wrote:
>>> %
>>> \catcode`\@=11
>>> \def\@sptoken#1{#1}%
>>> \@sptoken{\let\@sptoken= } %<- keep this comment-char!!!!!
>>> \message{\string\@sptoken: \meaning\@sptoken}
>> Wouh !! This is LaTeX ...
>>
>> In TeX this is simpler:
>> \afterassignment\romannumeral\futurelet\@sptoken0
>
> In _real_ LaTeX it is even simpler :
> \@firstofone{\let\@sptoken= } %
Bah !!! Awful ! @ letter, braces, percent char !
> Actually LaTeX already contains \@sptoken so the
> code required to define it in LaTeX is even simpler:
> nothing. The LaTeX kernel defines \@sptoken by
> \def\:{\let\@sptoken= } \: %
Which redefines \:
In TeX :
\afterassignment\let\futurelet\spacetoken\: \:
whatever \: is
> Your example code was explicitly stated to be for
> both TeX and LaTeX. It is unclear to me whether the
> romannumeral trick is somehow "better". I would look
> at it, say "cute trick", and go on to real coding.
This is real coding. But in general you load latex
and don't need to define a space token...
And this is not a *trick*. This is probably the more
concise expression for a space token.
Only primitives. No braces.
\noalign {\romannumeral-`}\@testopt ...
Ulrich D i e z
I don't get the point.
I think \scantokens is not useless above.
It re-tokenizes the funny space/catcode-10-A as catcode-11(letter)-A:
E.g.:
\lccode32=65
\message{With \string\scantokens:}
\lowercase{\scantokens{\def\spacesreplaced{ }}}
\message{%
\ifcat A\spacesreplaced catcode-equal \else catcode-different\fi
}
\immediate\read-1to\spacesreplaced
\message{Without \string\scantokens:}
\lowercase{\def\spacesreplaced{ }}
\message{%
\ifcat A\spacesreplaced catcode-equal \else catcode-different\fi
}
\immediate\read-1to\spacesreplaced
\csname stop\endcsname\bye
> >> While with \scantokens you finally get a real space:
> >
> > Yes. \scantokens is like writing tokens unexpanded to file
> > and reading them back from file and tokenizing them under
> > the catcode-regime valid at the time of reading them back.
>
> Not really ! \scantokens adds spaces. Noone takes care about that.
This happens with ordinary \write also and I take care about that.
Besides this, catcode-6-character-tokens will be duplicated when
writing them.
If you write a control-word-token unexpanded (not "stringified") to
an external file, an additional space (ASCII-32) will be written
behind that token.
I think "writing control-word-tokens unexpanded" is implemented
this way because when reading control-word-tokens back from
external-file, TeX will enter state S after tokenizing them and
thus discard trailing spaces...
\newwrite\mywrite
\immediate\openout\mywrite test.txt
% some unexpanded-writing:
\immediate\write\mywrite{\noexpand\TeX\noexpand\TeX\noexpand\TeX#}
\csname stop\endcsname\bye
In the resulting file test.txt you find space-characters behind the "\TeX"-phrases
and the # is duplicated.
Exactly the same adding of spaces trailing control-words and duplicating
of catcode-6-character-tokens happens when \scantokens exexcutes its'
"writing-part" which in turn affects the result of executing its' "reading-back-
part".
> I've worked as a slave to be able to put Verbatim and listings
> inside tabu with \scantokens. Noone takes care that
> \end{Verbatim} \end{lstlisting} etc.
>
> Becomes \end {Verbation} \end {lstlisting}
>
> with a space that has the catcode of the current regime (that is
> active, most of the time)
The problem is \scantokens's writing-part.
At the time when \scantokens executes its' writing-part, control-words
get written unexpanded and the routine for unexpanded-writing adds
spaces behind them which affects how things get re-tokenized when
the reading-back-part begins...
Probably a routine would be useful which expandably and
recursively stringifies token-wise before passing the argument
to \scantokens because after "stringification" you won't have
catcode-6-tokens and control-words any more.
Of the cuff I can only come up with a bunch of code which seems
to work - but very very slow:
Ulrich
In case you are interested, please make sure to view this
posting in original format.
%% Stringification of token-lists
%%
%% Copyright (C) 2011 by Ulrich Diez (eu_an...@web.de)
%%
%% This work may be distributed and/or modified under the
%% conditions of the LaTeX Project Public Licence (LPPL), either
%% version 1.3 of this license or (at your option) any later
%% version. (The latest version of this license is in:
%% http://www.latex-project.org/lppl.txt
%% and version 1.3 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.)
%% The author of this work is Ulrich Diez.
%% This work has the LPPL maintenance status 'not maintained'.
%% Usage of any/every component of this work is at your own risk.
%% There is no warranty - neither for probably included
%% documentation nor for any other part/component of this work.
%% If something breaks, you usually may keep the pieces.
%%
\errorcontextlines=10000
\documentclass[twoside, a4paper]{article}
\makeatletter
%%===============================================================
%%---------------------------------------------------------------
%% Stopper for \romannumeral-Expansion:
%%...............................................................
\newcommand\@rmstop{0 }
%%---------------------------------------------------------------
%% Pass first argument as parameter to second argument:
%%...............................................................
\newcommand\PassFirstToSecond[2]{#2{#1}}
%%---------------------------------------------------------------
%% Check if argument is blank / is null:
%%...............................................................
\newcommand\@ifnull[3]{%
\romannumeral\iffalse{\fi\expandafter\@secondoftwo\expandafter
{\expandafter{\string#1}\expandafter\@secondoftwo\string}%
\expandafter\@firstoftwo\expandafter{\iffalse}\fi0 #3}{0 #2}%
}
\newcommand\@ifblank[1]{%
\romannumeral\expandafter\expandafter\expandafter\@gobble
\expandafter\@ifnull\expandafter{\@gobble#1.}%
}
%%---------------------------------------------------------------
%% Check if brace-balanced argument starts with a bgroup-char:
%%...............................................................
\newcommand\@ifbrace[3]{%
\romannumeral\expandafter\@secondoftwo\expandafter{\expandafter
{\string#1.}\expandafter\@firstoftwo\expandafter{\iffalse}\fi
0 #2}{0 #3}%
}
%%---------------------------------------------------------------
%% Check if brace-balanced argument starts with a space-char:
%%...............................................................
\newcommand\@ifLeadSpace[3]{%
\romannumeral\csname
@\@ifnull{#1}{second}{\iffalse{\fi\@@ifLeadSpace.#1 }%
{}}oftwo\endcsname{0 #2}{0 #3}%
}
\@ifdefinable\@@ifLeadSpace{\long\def\@@ifLeadSpace#1 {%
\expandafter\@ifnull\expandafter{\@secondoftwo#1{}}{first}{second}%
\expandafter\@secondoftwo\expandafter{\iffalse}\fi
}}
%%---------------------------------------------------------------
%% \rembraces{N}{<Argument>} -> In case <Argument> consists of
%% stuff that is surrounded by at
%% least N pairs of braces, N
%% pairs of braces will be removed.
%% Otherwise all surrounding braces
%% will be removed.
%% \rembraces{}{<Argument>} -> While <Argument> consists of stuff
%% that is surrounded by braces, bra-
%% ces will be removed from <Argument>
%%...............................................................
\newcommand\rembraces[1]{%
\romannumeral\@ifnull{#1}{\@@rembraces}{%
\expandafter\@rembraces\expandafter{\romannumeral#1000}%
}%
}
\newcommand\@rembraces[2]{%
\@ifnull{#1}{\@rmstop#2}{%
\expandafter\@ifnull\expandafter{\@gobble#2}{%
\@ifbrace{#2}{%
\expandafter\PassFirstToSecond
\expandafter{\@firstofone#2}%
{\expandafter\@rembraces\expandafter{\@gobble#1}}%
}{%
\@rmstop#2%
}%
}{%
\@rmstop#2%
}%
}%
}
\newcommand\@@rembraces[1]{%
\expandafter\@ifnull\expandafter{\@gobble#1}{%
\@ifbrace{#1}{%
\expandafter\@@rembraces\expandafter{\@firstofone#1}%
}{\@rmstop#1}%
}{%
\@rmstop#1%
}%
}
%%---------------------------------------------------------------
%% \expandtimes{N}{<Argument>} -> N expandafter-chains will hit
%% on the place of <Argument> and
%% the result will be spit out
%% without the braces surrounding
%% <Argument>.
%% \expandtimes{}{<Argument>} -> expandafter-chains will hit on the
%% place of <Argument> until something
%% occurs on this place whose first
%% token is not expandable and the
%% result will be spit out without
%% the braces surrounding <Argument>.
%%...............................................................
\newcommand\expandtimes[2]{%
\romannumeral\@expandtimes{2}%
\expandafter\@firstofone\expandafter{%
\romannumeral
\@ifnull{#1}{%
\expandafter\expandafter\expandafter\@rmstop
\expandafter\@firstofone
\expandafter{\romannumeral\@@expandtimes{#2}}%
}{%
\expandafter\expandafter\expandafter\@rmstop
\expandafter\@firstofone
\expandafter{\romannumeral
\@expandtimes{#1}#2}%
}%
}%
}
%
% \romannumeral\@expandtimes{<number>}<Trailing Stuff>
% -> <Trailing Stuff> will be hit by <number> \expandafter-chains.
% !!!!!Be aware that \@expandtimes just hits the <Trailing Stuff>
% which means it has no effect in case <Trailing Stuff>
% is something that is nested into braces - braces are not
% expandable.
%
\newcommand\@expandtimes[1]{%
\expandafter\@@@expandtimes
\expandafter{\romannumeral#1000}{}%
}
\newcommand\@@@expandtimes[2]{%
\@ifnull{#1}{\@rmstop}{%
\expandafter\@ifnull\expandafter{\@gobble#1}{%
\expandafter\@rmstop#2\romannumeral\expandafter\@rmstop
}{%
\expandafter\@@@expandtimes\expandafter{%
\@gobble#1%
}{%
#2\romannumeral\expandafter\expandafter
\expandafter\@rmstop
}%
}%
}%
}
%
% \romannumeral\@@expandtimes{<Stuff>}
% -> \expandafter-chains will hit <Stuff>
% until the first thing of <Stuff> a non-exandable token.
% !!!!!Be aware that \@@expandtimes hits the <Stuff> which is
% _inside_ the braces!!!!!
%
\newcommand\@@expandtimes[1]{%
\@ifnull{#1}{\@rmstop#1}{%
\@ifbrace{#1}{%
\@rmstop#1%
}{%
\@ifLeadSpace{#1}{%
\@rmstop#1%
}{%
\expandafter\expandafter
\expandafter\@ifexpandable
\@TrimAllButFirstLoopUndelimited{#1\SeLDom}{%
\expandafter\@@expandtimes\expandafter{#1}%
}{\@rmstop#1}%
}%
}%
}%
}
\newcommand\@ifexpandable[3]{%
\romannumeral\@ifbrace{#1}\@secondoftwo{%
\@ifblank{#1}%
\@secondoftwo
{%
\expandafter\ifx\noexpand#1#1%
\expandafter\@secondoftwo
\else
\expandafter\@firstoftwo
\fi
}%
}{0 #2}{0 #3}%
}
%%---------------------------------------------------------------
%% \expandandargs{<token-sequence>}{%
%% {{<expansion-depth 1>}{<argument-1>}{<brace-removal-depth 1>}}%
%% {{<expansion-depth 2>}{<argument-2>}{<brace-removal-depth 2>}}%
%% ...
%5 {{<expansion-depth N>}{<argument-N>}{<brace-removal-depth N>}}%
%% }%
%%
%% recursively iterates a list of argument-triplets held in its
%% second argument, passing the <expansion-depth K>-level-expan-
%% sion of <argument-K> as K-th argument to <token-sequence>
%% after stripping at most <brace-removal-depth K> brace-levels
%% from that <expansion-depth K>-level-expansion of <argument-K>.
%%
%% In case you leave the <expansion-depth K>-part empty, expan-
%% sion takes place until the first resulting token is not ex-
%% pandable.
%% In case you leave the <brace-removal-depth-K>-part empty,
%% brace-removal should take place until there are no more
%% brace-levels surrounding the entire argument.
%%...............................................................
\newcommand\expandargs[2]{%
\romannumeral\@expandtimes{2}%
\expandafter\@firstofone
\expandafter{%
\romannumeral\@expandargs{#2}{#1}}%
}
\newcommand\@expandargs[2]{%
\@ifnull{#1}{\@rmstop#2}{%
\expandafter\PassFirstToSecond
\romannumeral\@expandtimes{3}\expandafter\@@expandargs
\romannumeral\@expandtimes{3}%
\expandafter\expandafter\expandafter\@firstofone
\@TrimAllButFirstLoopUndelimited{#1\SeLDom}{#2}%
{\expandafter\@expandargs\expandafter{\@gobble#1}}%
}%
}
\newcommand\@@expandargs[4]{%
\romannumeral\expandafter\@rmstop
\expandafter
{\romannumeral\@expandtimes{2}%
\expandafter\PassFirstToSecond
\expandafter{%
\romannumeral\@expandtimes{4}\expandafter\PassFirstToSecond
\expandafter{%
\romannumeral\@expandtimes{2}%
\expandtimes{#1}{#2}}{\rembraces{#3}}}{#4}}%
}
%%---------------------------------------------------------------
%% Get the first element of an undelimited-arguments-list:
%% Pass the <argument-list> with the token \SeLDom attached
%% as argument to \@TrimAllButFirstLoopUndelimited,
%% \@TrimAllButFirstLoopUndelimited{<argument-list>\SeLDom}
%% , and you will obtain
%% - either, in case the list s not empty,
%% the first argument of that list, with a surrounding pair
%% of catcode-1/2-characters replaced by a surrounding
%% pair of catcode 1/2-braces
%% - or, in case the list is empty,
%% an "empty argument" formed by a pair of catcode 1/2-braces.
%% The token \SeLDom may occur in the <argument-list> as well.
%% In that case gobbling anything but the first argument will
%% just take more iterations.
%%...............................................................
\newcommand*\@gobblesp[1]{\def\@gobblesp#1{}}\@gobblesp{ }
\@ifdefinable\@getfirsteltUndelimited{%
\long\def\@getfirsteltUndelimited#1#2\SeLDom{{#1}}%
}
\newcommand\@TrimAllButFirstLoopUndelimited[1]{%
\romannumeral
\expandafter\expandafter\expandafter\@ifnull
\expandafter\expandafter\expandafter{%
\expandafter\@gobble\@getfirsteltUndelimited#1}{%
\expandafter\expandafter\expandafter\@rmstop
\expandafter\@firstofone\expandafter{%
\@getfirsteltUndelimited#1}%
}{%
\expandafter\expandafter\expandafter\@gobble
\expandafter\@TrimAllButFirstLoopUndelimited
\expandafter{\@getfirsteltUndelimited#1}%
}%
}
%%---------------------------------------------------------------
%% \stringify{<command for formatting each stringified token of
%% the token-list>}%
%% {<amount of expansion-steps to apply to command
%% for formatting>}%
%5 {brace-balanced token-list}
%%...............................................................
\newcommand\stringify[3]{%
\romannumeral\@stringify{#3}{#1}{#2}{0 }%
}
\begingroup
\newcommand\@stringify[1]{%
\endgroup
\newcommand\@stringify[3]{%
\romannumeral
\@ifnull{##1}{\expandafter\@rmstop\@firstofone}{%
\@ifLeadSpace{##1}{%
\@stringify@add{#1}{{{##2}}{{ }}{##3}}%
{\expandargs{\expandafter\@gobble\@stringify}%
{{{1}{\@gobblesp##1}{0}}}{##2}{##3}}%
}{%
\@ifbrace{##1}{%
\expandargs{\@stringify@add}{%
{{4}{\expandargs{\@TrimAllButFirstLoopUndelimited}{{{3}{%
\expandafter\@stringify@getopenbrace
\string##1\SeLDom}{0}}}}{1}}%
}%
{{{##2}}{{<ELEMENT>}}{##3}}%
{%
\expandargs{\@stringify@add}{%
{{4}%
{%
\expandargs{\@stringify}%
{%
{%
{2}%
{%
\@TrimAllButFirstLoopUndelimited
{##1\SeLDom}%
}%
{1}%
}%
}%
{##2}{##3}{}%
}%
{0}}%
}%
{{{}}{{}}{##3}}%
{%
\expandargs{\@stringify@add}%
{{{2}{\@stringify@getclosebrace{##1}}{1}}}%
{{{##2}}{{<ELEMENT>}}{##3}}%
{\expandargs{\expandafter\@gobble\@stringify}%
{{{1}{\@gobble##1}{0}}}{##2}{##3}}%
}%
}%
}{%
%%%%%%%%%%%%
% In order to get total-expansion of the token-list that is to be
% stringified, enable the following commented lines:
%%%%%%%%%%%%
% \expandargs{\@ifexpandable}%
% {%
% {{2}%
% {\@TrimAllButFirstLoopUndelimited{##1\SeLDom}}%
% {1}}%
% }%
% {\expandargs{\expandafter\@gobble\@stringify}%
% {{{}{##1}{0}}}{##2}{##3}}{%
%%%%%%%%%%%%
\expandargs{\@stringify@add}%
{%
{{2}%
{\@TrimAllButFirstLoopUndelimited{##1\SeLDom}}%
{1}%
}%
}%
{{{##2}}{{<ELEMENT>}}{##3}}%
{\expandargs{\expandafter\@gobble\@stringify}%
{{{1}{\@gobble##1}{0}}}{##2}{##3}%
}%
%%%%%%%%%%%%
% }%
%%%%%%%%%%%%
}%
}%
}%
}%
}%
\@makeother\ %
\@stringify{ }%
\newcommand\@stringify@getopenbrace[1]{%
\expandafter#1\expandafter{\iffalse}\fi
}
\newcommand\@stringify@getclosebrace[1]{%
\romannumeral
\expandargs{\@ifnull}%
{{{2}{\@TrimAllButFirstLoopUndelimited{#1\SeLDom}}{1}}}{%
\@expandtimes{4}%
\expandargs{\@TrimAllButFirstLoopUndelimited}%
{%
{%
{3}%
{\expandafter\expandafter\expandafter\string
\expandafter\@gobble\string#1\SeLDom}%
{0}%
}%
}%
}{%
\expandafter\@ifLeadSpace\expandafter{\@firstofone#1}{%
\expandargs{\expandafter\@gobble\@stringify@getclosebrace}%
{%
{%
{1}%
{\expandafter{\romannumeral\@expandtimes{4}\iffalse}\fi
\expandafter\expandafter\expandafter\@gobblesp
\expandafter\@gobble\string#1}%
{0}%
}%
}%
}{%
\expandargs{\expandafter\@gobble\@stringify@getclosebrace}%
{%
{%
{1}%
{\expandafter{\romannumeral\@expandtimes{3}\iffalse}\fi
\expandafter\@gobbletwo\string#1}%
{0}%
}%
}%
}%
}%
}%
\newcommand\@stringify@add[2]{%
\expandargs{\@ifblank}%
{{{2}{\expandafter\@firstoftwo\@secondoftwo#2}{0}}}{%
\expandargs{\@ifnull}%
{{{2}{\expandafter\@firstoftwo\@secondoftwo#2}{0}}}{%
%<action for brace-nested list>%
\@stringify@@add{#1}%
}{%
%<action for space-token>
\expandargs{\@stringify@@add}{{{4}{%
\expandargs{\expandtimes{\expandafter\@secondoftwo
\@firstoftwo#2}}{%
{{2}{\expandafter\@firstoftwo\@firstoftwo#2{#1}}{0}}%
}%
}{0}}}%
}%
}{%
%<action for ordinary token>
\expandargs{\@stringify@@add}{{{4}{%
\expandargs{\expandtimes{\expandafter\@secondoftwo
\@firstoftwo#2}}{%
{{4}{\expandargs{\expandafter\@firstoftwo\@firstoftwo#2}%
{{{1}{\string#1}{0}}}%
}{0}}%
}%
}{0}}}%
}%
}%
\newcommand\@stringify@@add[3]{#2{#3#1}}%
%%===============================================================
\begin{document}
\parindent\z@
\parskip\baselineskip
\ttfamily\selectfont
\def\one{\two1 }
\def\two{\three2 }
\def\three{\four3 }
\def\four{\five4 }
\def\five{\six5 }
\def\six{6 }
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{0}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{1}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{2}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{3}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{4}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{5}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{6}{\one}|
|\expandafter\string
\romannumeral\expandafter\expandafter
\expandafter \@rmstop
\expandtimes{}{\one}|
\expandafter\def\expandafter\test\expandafter{%
\romannumeral\@expandtimes{2}%
\expandargs{BLA}{%
{{0}{\one}{0}}%
{{1}{\one}{0}}%
{{2}{\one}{0}}%
{{3}{\one}{0}}%
{{4}{\one}{0}}%
{{5}{\one}{0}}%
{{6}{\one}{0}}%
{{}{\one}{0}}%
}%
}%
|\meaning\test|
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
\expandtimes{}{\one}}%
\meaning\test
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
\expandtimes{6}{\one}}%
\meaning\test
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
\expandtimes{7}{\one}}%
\meaning\test
\expandafter\def \expandafter\test
\expandafter{%
\romannumeral\@expandtimes{0}\one}%
\meaning\test
\expandafter\def \expandafter\test
\expandafter{%
\romannumeral\@expandtimes{1}\one}%
\meaning\test
\expandafter\def \expandafter\test
\expandafter{%
\romannumeral\@expandtimes{}\one}%
\meaning\test
\expandafter\def \expandafter\test
\expandafter{%
\romannumeral\@@expandtimes{ \one}}%
\meaning\test
\expandafter\def \expandafter\test
\expandafter{%
\romannumeral\@@expandtimes{\one}}%
\meaning\test
\def\one{\two}
\def\two{\three}
\def\three{\four}
\def\four{\five}
\def\five{\six}
\def\six{{{6}}}
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\test
\expandafter\expandafter\expandafter{%
\expandargs{<token-sequence>}{%
{{0}{\one}{0}}%
{{1}{\one}{0}}%
{{3}{\one}{0}}%
{{5}{\one}{0}}%
{{6}{\one}{2}}%
{{2}{\one}{2}}%
{{}{\one}{}}%
{{}{\one}{0}}%
{{}{\one}{1}}%
}%
}%
|\meaning\test|
\def\tokenformat#1{\fbox{#1}\allowbreak\thinspace}
\catcode`\W=13 \expandafter\let\expandafterW\expandafter=\stringW
\catcode`\Y=1
\catcode`\Z=2
\def\ELT{\ALT}
\def\ALT{\ULT}
\def\ULT#1{#1}
\def\test#1{%
|\stringify{\tokenformat}%
{2}%
{%
\TeX Test bla { } {blaYbl\relax
W12##&$ u#1#1#1u} \ #1 \% bliZ ## \bgroup%
}|%
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\ELTS
\expandafter\expandafter\expandafter{%
\stringify{\ELT}%
{2}%
{%
\TeX Test bla { } {blaYbl\relax
W12##&$ u#1#1#1u} \ #1 \% bliZ ## \bgroup%
}%
}%
|\meaning\ELTS|%
}
\test{ }%
\begingroup
\lccode`\ =`\K %
\lowercase{\test{ }}%
\endgroup
\end{document}
Dan
> Le 28/03/2011 23:16, Dan Luecking a crit :
>
> > Actually LaTeX already contains \@sptoken so the
> > code required to define it in LaTeX is even simpler:
> > nothing. The LaTeX kernel defines \@sptoken by
> > \def\:{\let\@sptoken= } \: %
>
> Which redefines \:
No. It defines \:, which is later redefined in the definition of
@xifnch and finally redefined again to its user-level meaning
(a medmuskip).
> > It is unclear to me whether the
> > romannumeral trick is somehow "better". I would look
> > at it, say "cute trick", and go on to real coding.
>
> This is real coding.
Its one of a dozen ways to get a space token, cuter than most.
Real coding would be actually doing something useful with it.
> And this is not a *trick*.
Of course it is a trick. It uses \romannumeral in an unexpected way.
That's pretty much the definition of a "trick" (as in Knuth's dirty
tricks).
> This is probably the more
> concise expression for a space token.
> Only primitives. No braces.
Can you back that up? Is it faster? More efficient? Consume
less memory? What's the big deal with using only primitives?
It produces a non-primitive token after all is done. And what is
wrong with braces?
>
> \noalign {\romannumeral-`}\@testopt ...
has nothing to do with space tokens as far as I can tell.
Also contains a nonprimitive and braces. What's your point?
Dan
GL
Yes I forgot the current catcode regime and your example was not clear
for it used \scantokens twice and the message did't saw A = A.
\def\defspacesreplaced
#1{\catcode32=11\def\spacesreplaced{#1}\catcode32=10\ignorespaces}
\begingroup
\lccode32=65
\lowercase{\endgroup\scantokens{\defspacesreplaced{ }}}
\message{Expansion of \string\spacesreplaced\space is
\expandafter\ifx\spacesreplaced Aequal to
\else different from \fi A.}
\immediate\read-1to\spacesreplaced
is a bit more "tricky" in a sense: (32|10) is first replaced by
(65|10) then scantoken reads (65|10) and interpret it as 32 but
at this time 32 has catcode 11 and finally prints (65|11)...
> \lccode32=65
> \message{With \string\scantokens:}
> \lowercase{\scantokens{\def\spacesreplaced{ }}}
> \message{%
> \ifcat A\spacesreplaced catcode-equal \else catcode-different\fi
> }
> \immediate\read-1to\spacesreplaced
>
>> I've worked as a slave to be able to put Verbatim and listings
>> inside tabu with \scantokens. Noone takes care that
>> \end{Verbatim} \end{lstlisting} etc.
>>
>> Becomes \end {Verbation} \end {lstlisting}
>>
>> with a space that has the catcode of the current regime (that is
>> active, most of the time)
>
> The problem is \scantokens's writing-part.
> At the time when \scantokens executes its' writing-part, control-words
> get written unexpanded and the routine for unexpanded-writing adds
> spaces behind them which affects how things get re-tokenized when
> the reading-back-part begins...
>
> Probably a routine would be useful which expandably and
> recursively stringifies token-wise before passing the argument
> to \scantokens because after "stringification" you won't have
> catcode-6-tokens and control-words any more.
Not usefull at all: think about the optimisation.
Thanks for your remark about the "writing" part.
Yours sincerely.
GL
> On Mar 28, 5:03 pm, GL<gouail...@gmail.com> wrote:
>> Le 28/03/2011 23:16, Dan Luecking a crit :
>>
>>> Actually LaTeX already contains \@sptoken so the
>>> code required to define it in LaTeX is even simpler:
>>> nothing. The LaTeX kernel defines \@sptoken by
>>> \def\:{\let\@sptoken= } \: %
>>
>> Which redefines \:
>
> No. It defines \:, which is later redefined in the definition of
> @xifnch and finally redefined again to its user-level meaning
> (a medmuskip).
>
Robin Fairbairns
huh? you said one thing, and dan pointed out you weren't strictly
right.
note that latex 2e is full of such space-saving diddles, since people
still running latex on 286-based machines were horrified to find that 2e
was noticeably larger than 2.09. (the diddles were introduced between
the first release in june 1994 and the second release in december 1994,
iirc.)
--
Robin Fairbairns, Cambridge
my address is @cl.cam.ac.uk, regardless of the header. sorry about that.
GL
> GL<goua...@gmail.com> writes:
>
>> Le 29/03/2011 07:17, Dan a écrit :
>>> On Mar 28, 5:03 pm, GL<gouail...@gmail.com> wrote:
>>>> Le 28/03/2011 23:16, Dan Luecking a crit :
>>>>
>>>>> Actually LaTeX already contains \@sptoken so the
>>>>> code required to define it in LaTeX is even simpler:
>>>>> nothing. The LaTeX kernel defines \@sptoken by
>>>>> \def\:{\let\@sptoken= } \: %
>>>>
>>>> Which redefines \:
>>>
>>> No. It defines \:, which is later redefined in the definition of
>>> @xifnch and finally redefined again to its user-level meaning
>>> (a medmuskip).
>>
>> Yes and so what ?
>
> huh? you said one thing, and dan pointed out you weren't strictly
> right.
>
> note that latex 2e is full of such space-saving diddles, since people
> still running latex on 286-based machines were horrified to find that 2e
> was noticeably larger than 2.09. (the diddles were introduced between
> the first release in june 1994 and the second release in december 1994,
> iirc.)
Humm ! One century ago then ;-)
Ulrich D i e z
news:in651o$mfn$2...@news.albasani.net (code-improvements)
and
news:in6oc9$aad$1...@news.albasani.net (submission/transfer-errors). ]
GL wrote:
Okay - I spent 15+2+2 minutes on replacing \expandargs
and \expandtimes (which rely on slow \romannumeral-
expansion) by nested calls to \PassFirstToSecond
and \expandafter-chains which makes more code
but nonetheless seems to run faster.
\stringify{<command for formatting each stringified token of
the token-list>}%
{<amount of expansion-steps to apply to command
for formatting>}%
{<brace-balanced token-list>}
recursively does the following with each token of the
<brace-balanced token-list>:
- apply \string
- wrap the result of applying \string in braces and pass it as
argument to the <command for formatting each stringified token
of the token-list> which in turn gets expanded <amount of
expansion-steps to apply to command for formatting>-times.
In case the argument for the <amount of expansion-steps to
apply to command for formatting> is empty, <command for
formatting each stringified token of the token-list> will
be expanded until either the first token of the expansion-
result is not an expandable token or the expansion-result is
"emptiness".
The macro is expandable and doesn't require any TeX-
extension like eTeX or LuaTeX.
The result is delivered within two expansion-steps.
You can e.g., try:
\scantokens\expandafter\expandafter\expandafter{%
\stringify{\@firstofone}{1}{<Token List>}%
}%
and you won't get #-dublicated and you won't get spaces behind
control-sequences. (But the \string-result depends on the value
of \escapechar.) Charcode-32-Spaces will be of catcode12 (other)
as well.
If needed, you can transform them into "normal" space-tokens by
applying \detokenize on the result of \stringify.
I still feel somewhat brain-sick about the whole thing.
In case you are interested anyway, please make sure to
view this posting in original format.
Ulrich
%% {<brace-balanced token-list>}
%%
%% recursively does the following with each token of the
%% <brace-balanced token-list>:
%% - apply \string
%% - wrap the result of applying \string in braces and pass it as
%% argument to the <command for formatting each stringified token
%% of the token-list> which in turn gets expanded <amount of
%% expansion-steps to apply to command for formatting>-times.
%% In case the argument for the <amount of expansion-steps to
%% apply to command for formatting> is empty, <command for
%% formatting each stringified token of the token-list> will
%% be expanded until either the first token of the expansion-
%% result is not an expandable token or the expansion-result is
%% "emptiness".
%%
%% The macro is expandable and doesn't require any TeX-
%% extension like eTeX or LuaTeX.
%% The result is delivered within two expansion-steps.
%%...............................................................
\newcommand\stringify[3]{%
\romannumeral\@stringify{#3}{#1}{#2}{0 }%
}
\begingroup
\newcommand\@stringify[1]{%
\endgroup
\newcommand\@stringify[3]{%
\romannumeral
\@ifnull{##1}{\expandafter\@rmstop\@firstofone}{%
\@ifLeadSpace{##1}{%
\@stringify@add{#1}{##2}{ }{##3}%
{%
\expandafter\PassFirstToSecond\expandafter{\@gobblesp##1}%
{\expandafter\@gobble\@stringify}%
{##2}{##3}%
}%
}{%
\@ifbrace{##1}{%
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter
\@stringify@add
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter
\@TrimAllButFirstLoopUndelimited
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter
{%
\expandafter\@stringify@getopenbrace\string##1\SeLDom
}%
{##2}{<ELEMENT>}{##3}%
{%
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\@stringify@add
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
{%
\expandafter\expandafter\expandafter
\@stringify\@TrimAllButFirstLoopUndelimited{##1\SeLDom}%
{##2}{##3}{}%
}%
{}{}{##3}%
{%
\expandafter\expandafter\expandafter
\@stringify@add\@stringify@getclosebrace{##1}%
{##2}{<ELEMENT>}{##3}{%
\expandafter\expandafter\expandafter\@gobble
\expandafter\@stringify\expandafter{\@gobble##1}%
{##2}{##3}%
}%
}%
}%
}{%
\expandafter\expandafter\expandafter
\@stringify@add\@TrimAllButFirstLoopUndelimited{##1\SeLDom}%
{##2}{<ELEMENT>}{##3}{%
\expandafter\expandafter\expandafter\@gobble
\expandafter\@stringify\expandafter{\@gobble##1}%
{##2}{##3}%
}%
}%
}%
}%
}%
}%
\@makeother\ %
\@stringify{ }%
\newcommand\@stringify@getopenbrace[1]{%
\expandafter#1\expandafter{\iffalse}\fi
}
\newcommand\@stringify@getclosebrace[1]{%
\romannumeral
\expandafter\expandafter\expandafter\PassFirstToSecond
\@TrimAllButFirstLoopUndelimited{#1\SeLDom}{\@ifnull}{%
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\@rmstop
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter
\@TrimAllButFirstLoopUndelimited
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter{%
\expandafter\expandafter\expandafter\string
\expandafter\@gobble\string#1\SeLDom
}%
}{%
\expandafter\@ifLeadSpace\expandafter{\@firstofone#1}{%
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter
\PassFirstToSecond
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter
{%
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter
{\iffalse}\fi
\expandafter\expandafter\expandafter\@gobblesp
\expandafter\@gobble\string#1%
}{\expandafter\@gobble\@stringify@getclosebrace}%
}{%
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter
\PassFirstToSecond
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter
{%
\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter{\iffalse}\fi
\expandafter\@gobbletwo\string#1%
}{\expandafter\@gobble\@stringify@getclosebrace}%
}%
}%
}%
\newcommand\@stringify@add[4]{%
\@ifblank{#3}%
{%
\@ifnull{#3}{%
%<action for brace-nested list>
\@stringify@@add{#1}%
}{%
%<action for space-token>
\expandafter\expandafter\expandafter
\@stringify@@add
\expandafter\expandafter\expandafter
{%
\expandtimes{#4}{#2{#1}}%
}%
}%
}{%
%<action for ordinary token>
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter
\@stringify@@add
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter\expandafter\expandafter\expandafter\expandafter\expandafter
\expandafter
{%
\expandafter\expandafter\expandafter
\PassFirstToSecond
\expandafter\expandafter\expandafter
{%
\expandafter\PassFirstToSecond\expandafter{\string#1}{#2}%
}{%
\expandtimes{#4}%
}%
}%
}%
}%
\newcommand\@stringify@@add[3]{#2{#3#1}}%
%%===============================================================
\begin{document}
\parindent\z@
\parskip\baselineskip
\ttfamily\selectfont
\catcode`\W=13 \defW{W recursion}
\catcode`\Y=1
\catcode`\Z=2
\def\TOPLEVEL{\FIRSTLEVEL}
\def\FIRSTLEVEL{\SECONDLEVEL}
\def\SECONDLEVEL#1{THIRDLEVEL{#1} }
\def\TOKENFORMAT#1{\fbox{#1}\allowbreak\thinspace}
\def\test#1{%
|\stringify{\TOKENFORMAT}%
{}%
{%
\TeX Test bla { } {blaYbl\relax
W12##&$ u#1#1#1u} \ #1 \% bliZ ## \bgroup%
}|%
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\SOMESTRING
\expandafter\expandafter\expandafter{%
\stringify{\TOPLEVEL}%
{0}%
{%
\TeX Test bla { } {blaYbl\relax
W12##&$ u#1#1#1u} \ #1 \% bliZ ## \bgroup%
}%
}%
|\meaning\SOMESTRING|%
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\SOMESTRING
\expandafter\expandafter\expandafter{%
\stringify{\TOPLEVEL}%
{2}%
{%
\TeX Test bla { } {blaYbl\relax
W12##&$ u#1#1#1u} \ #1 \% bliZ ## \bgroup%
}%
}%
|\meaning\SOMESTRING|%
\expandafter\expandafter\expandafter\def
\expandafter\expandafter\expandafter\SOMESTRING
\expandafter\expandafter\expandafter{%
\stringify{\TOPLEVEL}%
{}%
{%
\TeX Test bla { } {blaYbl\relax
W12##&$ u#1#1#1u} \ #1 \% bliZ ## \bgroup%
}%
}%
|\meaning\SOMESTRING|%
GL
> [ This posting replaces my postings
> news:in651o$mfn$2...@news.albasani.net (code-improvements)
> and
> news:in6oc9$aad$1...@news.albasani.net (submission/transfer-errors). ]
>
> GL wrote:
>
>> Le 29/03/2011 03:56, Ulrich D i e z a écrit :
>
>>> Probably a routine would be useful which expandably and
>>> recursively stringifies token-wise before passing the argument
>>> to \scantokens because after "stringification" you won't have
>>> catcode-6-tokens and control-words any more.
>>
>> Not usefull at all: think about the optimisation.
>
> Okay - I spent 15+2+2 minutes on replacing \expandargs
> and \expandtimes (which rely on slow \romannumeral-
> expansion) by nested calls to \PassFirstToSecond
> and \expandafter-chains which makes more code
> but nonetheless seems to run faster.
Yes I will try to look at it. It seems really interesting.
I don't like much your "vanilla" implementation of \ifnull, because it's
not easy to follow the .log, but I know it's yours and that you like
those tricks ;-)
Nevertheless, the long list of \expandafters would require further
optimisation in a "production release"...
Thanks.
Ulrich D i e z
> I don't like much your "vanilla" implementation of \ifnull, because it's
> not easy to follow the .log, but I know it's yours and that you like
> those tricks ;-)
I use \@ifnull a lot, but I don't deserve the credits.
Some time ago I fiddled something together based
on Robert R Schneck's \ifempty-macro
[ news:3eef1...@corp.newsgroups.com
Newsgroups: comp.text.tex,
Subject:
Re: \ifempty solution (was Macro puzzle: maximally general \ifempty)
Message-ID: <3eef1...@corp.newsgroups.com>
Date: 17 Jun 2003 08:42:50 -0500 ].
Actually the thing provided by me was an awful bunch
of unnecessary \expandafter and some discussion
took place. I think it was Ahmed Musa who came up
with the \ifnull-variant which I use nowadays.
Ulrich
Bruno Le Floch
comments below. I can't test it because of a broken \pdfelapsedtime in
TeXLive2009 (some day, I'll upgrade).
% ======== "\@expandtimes"
% % Ulrich Diez's specification
% % \romannumeral\@expandtimes{<number>}<Trailing Stuff>
% % -> <Trailing Stuff> will be hit by <number> \expandafter-chains.
%
%
% Bruno: \romannumeral#1000 produces #1 times "m". We put
% each one in a \csname construction: this way, the end
% simply makes another control sequence, "\@expandtimes@e".
% This avoids an expensive "\@ifnull" test.
%
% Then all the \romannumeral are stopped by \@rm@multistop,
% which stops one \romannumeral and replicates itself.
%
\newcommand\@expandtimes[1]{%
\expandafter\@rm@stop@multistop\romannumeral
\expandafter\@@@expandtimes\romannumeral#1000e\@rm@multistop%
}
\newcommand\@@@expandtimes[1]{\csname @expandtimes@#1\endcsname}
\newcommand\@expandtimes@m{\romannumeral\@@@expandtimes}
\newcommand\@expandtimes@e{}
\newcommand\@rm@multistop{0 \expandafter\@rm@multistop}
\newcommand\@rm@stop@multistop[2]{0 }