mathcode vs. mathchardef
Will Robertson
This goes back to some discussion by Ross about using \let rather
than \mathchardef to define control sequences that corresponded to
unicode characters. It seems sensible to define the control sequences
in terms of the raw input character so that only one set of
definitions needs to be kept up-to-date.
That is, better to say something like
(I hope these literal \Theta characters resolve in the emails you
receive.)
\XeTeXmathcode`\Θ = "0 "1 `\Θ
\let\itTheta = Θ
rather than
\XeTeXmathcode`\Θ = "0 "1 `\Θ
\XeTeXmathchardef\itTheta = "0 "1 `\Θ
However, some issues were raised with the former technique.
How about
\def\itTheta{Θ}
instead of using \let?
We don't need to worry about making \itTheta robust, because it is
"the same" as the literal character (well, I think it should be!).
And you can use \expandafter to get it into a form for checking
character codes, or whatever. (Or doesn't this work for arbitrary
macros? I don't know the scope of the commands you're trying to
write, Ross.)
Thoughts?
Will
Ross Moore
On 28/01/2007, at 2:33 PM, Will Robertson wrote:
>
> Hello,
>
> This goes back to some discussion by Ross about using \let rather
> than \mathchardef to define control sequences that corresponded to
> unicode characters. It seems sensible to define the control sequences
> in terms of the raw input character so that only one set of
> definitions needs to be kept up-to-date.
Yes; I've come to this conclusion too, and was going to compose
an email about it. You've beaten me to the key-board.
The problem with using \mathchardef (read the \XeTeXmathchardef
variant) is that it includes the expanded value of the math-family.
If that family needs to be changed then, as you say, there
are 2 things to change:
the \mathcode of the character,
and the \mathchardef definition.
By having macros expand into the character, one needs only change
the \mathcode entry, which will then apply whether the source
contains either a macro or a raw character.
>
> That is, better to say something like
> (I hope these literal \Theta characters resolve in the emails you
> receive.)
They do! (but not italiced)
>
> \XeTeXmathcode`\Θ = "0 "1 `\Θ
> \let\itTheta = Θ
>
> rather than
>
> \XeTeXmathcode`\Θ = "0 "1 `\Θ
> \XeTeXmathchardef\itTheta = "0 "1 `\Θ
>
> However, some issues were raised with the former technique.
> How about
> \def\itTheta{Θ}
> instead of using \let?
>
> We don't need to worry about making \itTheta robust, because it is
> "the same" as the literal character (well, I think it should be!).
Yes, it is automatically robust; however doing it this way allows
no possibility to check that you are in math-mode, so that you
are doing something sensible with the math-character.
(Chris will be at your neck about this. :-)
There is another problem too, which is better solved differently.
> And you can use \expandafter to get it into a form for checking
> character codes, or whatever. (Or doesn't this work for arbitrary
> macros? I don't know the scope of the commands you're trying to
> write, Ross.)
Here's an outline of what I now think is best:
1. if the macro-name is undefined (as most will be), then
\DeclareTextCommandDefault{\itTheta}{\non-math-warning
{\itTheta}Θ}
2. declare the (extended) \mathcode for the character (using
a symbolic name for the math-family):
\mathcode`\Θ = "0 \mathgkfam `\Θ
3. define a macro for use in math-only; e.g.
\def\math@itTheta{Θ}
In meta-macros, one can use \char"???? with hex-codes, instead
of the raw character or \scantokens{^^^^....} constructions.
3. define a robust switch that tests for the mode --- essentially:
\DeclareRobustCommand{\itTheta}{%
\ifmmode\expandafter\math@itTheta
\else
\?-cmd\itTheta\?\itTheta %%<<<--- hooks into the text default
\fi }
This latter coding needs tricky \csname ... constructions
to get it working correctly.
The point is that there *needs* to be a textual version of the macro,
so that it can appear in non-typesetting contexts --- such as
Bookmark strings, which are currently handled by a PU encoding.
Similarly if TeX is to be used to write out other codings;
e.g. for HTML, RSS, XML, etc. which require a different string
representation of the mathematical concept, then this is best
achieved using a special encoding, analogous to PD1 and PU .
>
> Thoughts?
Chris is correct in that a math character has no business
being typeset within a non-math context, at least not without
some kind of complaint being raised -- such as a warning message.
However, TeX has to do more than just typeset these days.
So I'm declaring a concept of "non-typesetting context".
Examples are:
writing out auxiliary files for the .aux ToC, Index, etc.
-- for which "robustness" is the often sufficient;
preparing input for Bookmark strings: requires PU encoding;
processing for other text-representations:
XML, character entities, parametrised entities, etc.
For these contexts you don't want non-math warnings, because
we know we aren't trying to typeset math, and you don't want
to have to make thousands of redefinitions every time TeX
enters such a processing phase.
I think the above outline, of the macros required for each
math-character, is the most efficient way to cope with everything.
Comments and further discussion is almost certainly needed
to get the details correct.
>
> Will
Thanks for bringing this issue to a head.
Cheers,
Ross
------------------------------------------------------------------------
Ross Moore ro...@maths.mq.edu.au
Mathematics Department office: E7A-419
Macquarie University tel: +61 +2 9850 8955
Sydney, Australia 2109 fax: +61 +2 9850 8114
------------------------------------------------------------------------
Will Robertson
On 1/29/07, Ross Moore <ro...@ics.mq.edu.au> wrote:
>
> Yes; I've come to this conclusion too, and was going to compose
> an email about it. You've beaten me to the key-board.
Great! I might be able to check in a new version of unicode-math
incorporating these ideas soon.
> > (I hope these literal \Theta characters resolve in the emails you
> > receive.)
>
> They do! (but not italiced)
Oops. Copy/paste error. I used the text version of the character, not
maths, by mistake.
> Yes, it is automatically robust; however doing it this way allows
> no possibility to check that you are in math-mode, so that you
> are doing something sensible with the math-character.
> (Chris will be at your neck about this. :-)
Ah, indeed; got to keep Chris happy.
> Here's an outline of what I now think is best:
>
> 1. if the macro-name is undefined (as most will be), then
> \DeclareTextCommandDefault{\itTheta}{\non-math-warning
> {\itTheta}Θ}
>
> 2. declare the (extended) \mathcode for the character (using
> a symbolic name for the math-family):
> \mathcode`\Θ = "0 \mathgkfam `\Θ
\mathgkfam will be expanded here anyway -- oh, but you're referring to
the fact that I was previously redeclaring all my mathcodes instead of
just changing the font. Yes, quite.
> 3. define a macro for use in math-only; e.g.
> \def\math@itTheta{Θ}
>
> In meta-macros, one can use \char"???? with hex-codes, instead
> of the raw character or \scantokens{^^^^....} constructions.
Is there a problem using the raw character these days? I've steered
clear of it in the source to unicode-math, nonetheless.
> 3. define a robust switch that tests for the mode --- essentially:
> \DeclareRobustCommand{\itTheta}{%
> \ifmmode\expandafter\math@itTheta
> \else
> \?-cmd\itTheta\?\itTheta %%<<<--- hooks into the text default
> \fi }
Hmmm. source2e comes to the rescue -- I haven't seen this \?-cmd
construction before. I see...yes, this looks to do the trick nicely.
> Comments and further discussion is almost certainly needed
> to get the details correct.
This is a great start. Will keep you posted on how this works out.
Will
Chris Rowley
chris
On Jan 29, 3:20 am, Ross Moore <r...@ics.mq.edu.au> wrote:
> Hi Will,
>
> On 28/01/2007, at 2:33 PM, Will Robertson wrote:
>
>
>
> > Hello,
>
> > This goes back to some discussion by Ross about using \let rather
> > than \mathchardef to define control sequences that corresponded to
> > unicode characters. It seems sensible to define the control sequences
> > in terms of the raw input character so that only one set of
> > definitions needs to be kept up-to-date.Yes; I've come to this conclusion too, and was going to compose
> an email about it. You've beaten me to the key-board.
>
> The problem with using \mathchardef (read the \XeTeXmathchardef
> variant) is that it includes the expanded value of the math-family.
> If that family needs to be changed then, as you say, there
> are 2 things to change:
> the \mathcode of the character,
> and the \mathchardef definition.
> By having macros expand into the character, one needs only change
> the \mathcode entry, which will then apply whether the source
> contains either a macro or a raw character.
>
>
>
> > That is, better to say something like
> > (I hope these literal \Theta characters resolve in the emails you
> > receive.) They do! (but not italiced)
>
>
>
>
>
> > \XeTeXmathcode`\Θ = "0 "1 `\Θ
> > \let\itTheta = Θ
>
> > rather than
>
> > \XeTeXmathcode`\Θ = "0 "1 `\Θ
> > \XeTeXmathchardef\itTheta = "0 "1 `\Θ
>
> > However, some issues were raised with the former technique.
> > How about
> > \def\itTheta{Θ}
> > instead of using \let?
>
> > We don't need to worry about making \itTheta robust, because it is
> > "the same" as the literal character (well, I think it should be!).Yes, it is automatically robust; however doing it this way allows
> no possibility to check that you are in math-mode, so that you
> are doing something sensible with the math-character.
> (Chris will be at your neck about this. :-)
>
> There is another problem too, which is better solved differently.
>
> > And you can use \expandafter to get it into a form for checking
> > character codes, or whatever. (Or doesn't this work for arbitrary
> > macros? I don't know the scope of the commands you're trying to
> > write, Ross.)Here's an outline of what I now think is best:
> > Thoughts?Chris is correct in that a math character has no business
> being typeset within a non-math context, at least not without
> some kind of complaint being raised -- such as a warning message.
>
> However, TeX has to do more than just typeset these days.
> So I'm declaring a concept of "non-typesetting context".
>
> Examples are:
> writing out auxiliary files for the .aux ToC, Index, etc.
> -- for which "robustness" is the often sufficient;
> preparing input for Bookmark strings: requires PU encoding;
> processing for other text-representations:
> XML, character entities, parametrised entities, etc.
>
> For these contexts you don't want non-math warnings, because
> we know we aren't trying to typeset math, and you don't want
> to have to make thousands of redefinitions every time TeX
> enters such a processing phase.
>
> I think the above outline, of the macros required for each
> math-character, is the most efficient way to cope with everything.
>
> Comments and further discussion is almost certainly needed
> to get the details correct.
>
>
>
> > WillThanks for bringing this issue to a head.
>
> Cheers,
>
> Ross
>
> ------------------------------------------------------------------------
> Ross Moore r...@maths.mq.edu.au
Chris Rowley
that is the fundamental point.
As in Word2007, the set of \names... that are just ascii input for
Unicode Chars (or UC strings) should be just that and only that.
1. These are best handled by the editor (as in Word20007, where it is
customisable but a lot of defaults based on Barbara's list are set-
up).
2. If they must be input to XeTeX then simnply \def them to expand
immediately (well, with a bit of luck) to Unicode.
3. They are not LICR's.
4. (almost) All Unicode strings are LICRs.
5. Use of unimath(-for-XeTeX) is not compatible with use of 8-bit TeX
or of standard LaTeX.
6. If you want XeTeX to write out ascii files then put a filter inside
XeTeX (but I am not sure it is a worthwhile activity).
PS: I do not think there si anything basically new above; I am simply
clarifying what Ross was expressing.
Chris Rowley
>
> > Yes, it is automatically robust; however doing it this way allows
> > no possibility to check that you are in math-mode, so that you
> > are doing something sensible with the math-character.
> > (Chris will be at your neck about this. :-)Ah, indeed; got to keep Chris happy.
>
Agreed: so no names should be roibust now.
> > Here's an outline of what I now think is best:
>
> > 1. if the macro-name is undefined (as most will be), then
> > \DeclareTextCommandDefault{\itTheta}{\non-math-warning
> > {\itTheta}Θ}
Not needed.
>
> > 2. declare the (extended) \mathcode for the character (using
> > a symbolic name for the math-family):
> > \mathcode`\Θ = "0 \mathgkfam `\Θ\mathgkfam will be expanded here anyway -- oh, but you're referring to
> the fact that I was previously redeclaring all my mathcodes instead of
> just changing the font. Yes, quite.
OK.
>
> > 3. define a macro for use in math-only; e.g.
> > \def\math@itTheta{Θ}
>
defined how?
No longer need math/text distincton as everything is jsut always
expended to the same UC string.
> > In meta-macros, one can use \char"???? with hex-codes, instead
> > of the raw character or \scantokens{^^^^....} constructions.Is there a problem using the raw character these days? I've steered
> clear of it in the source to unicode-math, nonetheless.
>
I do not follow.
> > 3. define a robust switch that tests for the mode --- essentially:
> > \DeclareRobustCommand{\itTheta}{%
> > \ifmmode\expandafter\math@itTheta
> > \else
> > \?-cmd\itTheta\?\itTheta %%<<<--- hooks into the text default
> > \fi }Hmmm. source2e comes to the rescue -- I haven't seen this \?-cmd
No longer needed.
Do not need anything like this in text mode with XeTeX, use the
Unicode character.
Chris Rowley
> > We don't need to worry about making \itTheta robust, because it is
> > "the same" as the literal character (well, I think it should be!).
> However, TeX has to do more than just typeset these days.
> So I'm declaring a concept of "non-typesetting context".
Yes, but XeTeX does not, so let's keep it that way.
If you really want XeTeX to do all this other stuff then rewrite XeTeX
or produce some other package. this has little todo with typesetting
maths so let's get only that sorted in unimath.
Or it may be better to start work on OpenOffice2007 instead.
Other stuff (yeuch!!:_):
>
> Examples are:
> writing out auxiliary files for the .aux ToC, Index, etc.
> -- for which "robustness" is the often sufficient;
> preparing input for Bookmark strings: requires PU encoding;
> processing for other text-representations:
> XML, character entities, parametrised entities, etc.
>
> For these contexts you don't want non-math warnings, because
> we know we aren't trying to typeset math, and you don't want
> to have to make thousands of redefinitions every time TeX
> enters such a processing phase.
>
chris
Will Robertson
I don't suppose you were sent the long post through email and there's
a copy somewhere in an outbox? I'm going to need some clarification
below.
On 31/01/2007, at 4:25 , Chris Rowley wrote:
> As in Word2007, the set of \names... that are just ascii input for
> Unicode Chars (or UC strings) should be just that and only that.
>
> 1. These are best handled by the editor (as in Word20007, where it is
> customisable but a lot of defaults based on Barbara's list are set-
> up).
>
> 2. If they must be input to XeTeX then simnply \def them to expand
> immediately (well, with a bit of luck) to Unicode.
"If they must"?
Why this sudden desire to be like Word2007?
Using macros is often more convenient than unicode characters,
especially for material being migrated from TeX. True, an editor can
be set up to do the transformations to get to unicode for us.
> 5. Use of unimath(-for-XeTeX) is not compatible with use of 8-bit TeX
> or of standard LaTeX.
In what sense? Enough is the same that maths should transpose without
problem.
Okay, maybe I should say that maths in LaTeX should work as maths in
XeLaTeX, but not the other way around *if unicode input* is used.
>>> Here's an outline of what I now think is best:
>>
>>> 1. if the macro-name is undefined (as most will be), then
>>> \DeclareTextCommandDefault{\itTheta}{\non-math-warning
>>> {\itTheta}Θ}
>
> Not needed.
Weren't you arguing just recently, Chris, that we want to make sure
that we typeset maths as maths and not in any other context? I
suppose the thing is that this will only give you the warning for
\ittheta and not for the literal unicode char.
>
>>> 3. define a macro for use in math-only; e.g.
>>> \def\math@itTheta{Θ}
>
> defined how?
Like this: \def\math@itTheta{Θ}
Or with ^^^^ff1234
Or with \char"FF1234, apparently
I'm not sure what you're asking?
****
So, for once and for all:
We *do* want macros and literal unicode chars to have exactly the
same behaviour, and the easiest way to do that is to have ...\def
\upTheta{Θ}... This doesn't need to be robust, because \upTheta
doesn't hold any information over Θ. The literal unicode char is the
"LICR".
We *don't* want to make maths characters active and perform sanity
checks on themselves (or transformations, if needed, for bookmark
strings et al.) before they insert themselves into maths.
****
There are some things in my package that distort this simplicity. To
accommodate different maths styles, ascii and text greek can have
different \mathcodes in different documents. They are still robust
(right?) because they only resolve to \upGreek or \itGreek during
typesetting.
Similarly, \theta is defined as either \uptheta or \ittheta due to
the same reasoning. Perhaps *that* definition should be robust just
to be safe?
...\DeclareRobustCommand\theta{ittheta}...
Maths alphabets must be robust because they locally change \mathcodes
(or even fonts, if \mathswashbuckle is supported). \mathit{\}
Will
Chris Rowley
> > As in Word2007, the set of \names... that are just ascii input for
> > Unicode Chars (or UC strings) should be just that and only that.
>
> > 1. These are best handled by the editor (as in Word20007, where it is
> > customisable but a lot of defaults based on Barbara's list are set-
> > up).
>
> > 2. If they must be input to XeTeX then simnply \def them to expand
> > immediately (well, with a bit of luck) to Unicode.
>
> "If they must"?
> Why this sudden desire to be like Word2007?
Well, Word2007 is trying hard to be LaTeX-like ... so I joined the
dance:-).
> Using macros is often more convenient than unicode characters,
> especially for material being migrated from TeX. True, an editor can
> be set up to do the transformations to get to unicode for us.
Maybe, but should that be true for XeTeX? Certainly one principle aim
was to get rid of the need for `names for characters' in .tex files.
>
> > 5. Use of unimath(-for-XeTeX) is not compatible with use of 8-bit TeX
> > or of standard LaTeX.
>
> In what sense? Enough is the same that maths should transpose without
> problem.
> Okay, maybe I should say that maths in LaTeX should work as maths in
> XeLaTeX, but not the other way around *if unicode input* is used.
And I am saying that unicode input should be used: and so are you if
you stop names for chabeing LICRs in maths.
Chris Rowley
... to continue.
> >>> Here's an outline of what I now think is best:
>
> >>> 1. if the macro-name is undefined (as most will be), then
> >>> \DeclareTextCommandDefault{\itTheta}{\non-math-warning
> >>> {\itTheta}Θ}
>
> > Not needed.
>
> Weren't you arguing just recently, Chris, that we want to make sure
> that we typeset maths as maths and not in any other context?
Yes, but I now agree with Ross that any Unicode string has to be able
to appear anywhere, even if it makes no sense. Text is text is text
is Unicode strings.
> I suppose the thing is that this will only give you the warning for
> \ittheta and not for the literal unicode char.
But if \itTheta is merely an input shorthand for a Unicode character
then it can appear anyhwere when using XeTeX. Thic character will
produce a `default output' when used with most fonts, but The XeTeX
Way is, I was told, tosimply ignore such deficiencies of fonts.
>
>
>
> >>> 3. define a macro for use in math-only; e.g.
> >>> \def\math@itTheta{Θ}
>
> > defined how?
>
> Like this: \def\math@itTheta{Θ}
> Or with ^^^^ff1234
> Or with \char"FF1234, apparently
>
> I'm not sure what you're asking?
Sorry, I meant `where is this used', not `defined'.
I think I folow the idea now.
>
> ****
>
> So, for once and for all:
>
> We *do* want macros and literal unicode chars to have exactly the
> same behaviour, and the easiest way to do that is to have ...\def
> \upTheta{Θ}... This doesn't need to be robust, because \upTheta
> doesn't hold any information over Θ. The literal unicode char is the
> "LICR".
Agreed.
And since this is all these names are then:
a: encourage or even rejig XeTeX editors to make the transformation
b: use the same names as the Word2007 defaults
> ****
>
> There are some things in my package that distort this simplicity. To
> accommodate different maths styles, ascii and text greek can have
> different \mathcodes in different documents. They are still robust
> (right?) because they only resolve to \upGreek or \itGreek during
> typesetting.
What `resolves'? The Unicode strings?
still make them simply a Unicode character.
>
> Similarly, \theta is defined as either \uptheta or \ittheta due to
> the same reasoning. Perhaps *that* definition should be robust just
> to be safe?
> ...\DeclareRobustCommand\theta{ittheta}...
Yes, it should if it is vital to support such ideas.
These are abstract math characters that probably need there own
Unicode slots.
Who decides which names have this property?
>
> Maths alphabets must be robust because they locally change \mathcodes
> (or even fonts, if \mathswashbuckle is supported). \mathit{\}
Yes, these commands are mark-up tags, not character names.
chris
Chris Rowley
> Why this sudden desire to be like Word2007?
It is sudden because I just found it ona amachine and discovered that
the suits had allowed all this secretly developed stuff for maths into
a product.
This all started about 15 years ago when a physicist and long-time
LaTeX user went to Redmond to head the devlopment team for Word2007, a
completely new code base with lots of Knuthian ideas built-in.
So it is Word2007s desire to become like us, not the other way round,
that forces us out in the real world to work with it.
chris