ascii input, classic \mathcode and interoperability
Chris Rowley
it does):
input of unicode strings in math mode
use of XeTeX's features:
large fonts and a larger number of families;
font mappings attached to a font
So where/how does support for ascii input work?
Must all fonts used be large?
Must all fonts used via XeTeX's extended math commands be large?
Will XeTeX ever look up a classic \mathcode?
Should there be an internal form (like LICR's) that is interoperable
with classic TeX?
I am not suggesting what the answers should be but I think we should be
clear about what we are doing (at least for ASCII input).
Will Robertson
I've been vaguely thinking about the last couple of days; here are my
thoughts.
> input of unicode strings in math mode
Yes, but I'm less interested in that for now (it's nice, but not
necessarily convenient).
> use of XeTeX's features:
>
> large fonts and a larger number of families;
>
> font mappings attached to a font
The former: definitely (although more families are just icing on the
cake; we could get by with less in an ideal world).
The latter: they certainly make things easier, but it's not clear that
it's the only/best solution to the problems we're facing.
> So where/how does support for ascii input work?
Everywhere. I've been thinking, and I realised I've got everything
backwards. I should be trying to emulate LaTeX first with its
\mathbf-type macros that switch the \fam; on top of that, changes can
be made. By this I mean I should be defining
\def\bfR{\mathbf{R}}
and not
\def\bf#1{\csname bf#1\endcsname}
where
\mathchardef\bfR= ... whatever ...
> Must all fonts used be large?
By "large" do you mean "implement a mostly complete subset of
mathematical unicode"?
That is, do you *not* mean "fonts don't follow unicode for their glyph
slots"?
For the former: definitely not; support already exists partially for
this in unicode-math.
For the latter: I hope not! That will be a nightmare.
> Must all fonts used via XeTeX's extended math commands be large?
I'm not sure what you mean by this.
> Will XeTeX ever look up a classic \mathcode?
I think this relates back to my answer above "That will be a
nightmare".
If it does, unicode-math should have nothing to do with it.
> Should there be an internal form (like LICR's) that is interoperable
> with classic TeX?
I think not, unless pdfTeX evolves in a significantly different
direction than XeTeX and yet we want to try and make them as compatible
as possible. I mean, you could develop newmath further (that 8-bit math
encoding) and organise some sort of interoperability, but it'd be
thankless work, I think.
I might have misunderstood this point; what situations were you
thinking of?
--
Will
Chris Rowley
Will
>
> > input of unicode strings in math mode
>
> Yes, but I'm less interested in that for now (it's nice, but not
> necessarily convenient).
>
Oh, I am surprised by that!!
> > use of XeTeX's features:
> >
> > large fonts and a larger number of families;
> >
> > font mappings attached to a font
>
> The former: definitely
Oh dear, that was not meant as a choice: you need both!!
> (although more families are just icing on the
> cake; we could get by with less in an ideal world).
Maybe but that is not necessarily sensible. I think that using
families well is precisely what t is all about, not changing mathcodes
all the time.
> The latter: they certainly make things easier, but it's not clear that
> it's the only/best solution to the problems we're facing.
I think it may well be the key. I shall clarify this next week.
>
> > So where/how does support for ascii input work?
>
> Everywhere. I've been thinking, and I realised I've got everything
> backwards. I should be trying to emulate LaTeX first
Yes: the more I think about it the more i think that the LaTeX idea of
an imutable LICR is even more important for math characters than for
the variable text encodings for which it was invented.
Then the glyphs are simply slected by choosing a font and font-mapping
(the latter being needed because some `modern fonts' can only be
accessed through Unicode stings, it matters not a jot to anyone but
that font what strings are used.
> > Must all fonts used be large?
>
> By "large" do you mean "implement a mostly complete subset of
> mathematical unicode"?
> That is, do you *not* mean "fonts don't follow unicode for their glyph
> slots"?
>
I mean not 8-bit (but probably something like whta you say also.
> For the former: definitely not; support already exists partially for
> this in unicode-math.
Good, Ok: but what about the 8-bit fonts?
> For the latter: I hope not! That will be a nightmare.
>
Not at all: use font mapings for this, it's the only way (and someone
is going to produce screwed up font encodings sometime, bound to.
> > Must all fonts used via XeTeX's extended math commands be large?
>
> I'm not sure what you mean by this.
>
Need to ask Jonathan.
> > Will XeTeX ever look up a classic \mathcode?
>
> I think this relates back to my answer above "That will be a
> nightmare".
> If it does, unicode-math should have nothing to do with it.
>
But what about compatibility hen, and using glyphs that don't exist in
My Big Font?
> > Should there be an internal form (like LICR's) that is interoperable
> > with classic TeX?
>
> I think not, unless pdfTeX evolves in a significantly different
> direction than XeTeX and yet we want to try and make them as compatible
> as possible. I mean, you could develop newmath further (that 8-bit math
> encoding) and organise some sort of interoperability, but it'd be
> thankless work, I think.
>
That was not my pint:L anything at the LaTeX level must be indpependent
of XeTeX and its names.
> I might have misunderstood this point; what situations were you
> thinking of?
Needing to use classic mth fonts (becuase people like the look of them,
if nothing else) that will remain for sometime in 8-bit form only.
And, especially for Ross and me, all math coding (at least all 7-bit
stuff) should be common to all TeX's and to the many other applications
that now also use it.
chris
>
> --
> Will
Will Robertson
>
> > > input of unicode strings in math mode
> >
> > Yes, but I'm less interested in that for now (it's nice, but not
> > necessarily convenient).
>
> Oh, I am surprised by that!!
Well, if someone gets around to defining an automatic mapping between
many of the ascii maths symbols to their unicode characters (a reverse
unicode-math in the editor, in a way) I'll generally be happy to use
it.
But sometimes I like to leave the actual representation undecided,
choosing which symbols to use at a later date (for example, whether to
use \triangleq or \eqdef). So in this case actually using the
codepoint in the source would be a bad idea.
(Even more so things like \vect{B}, which sometimes I like bold, and
other times with an arrow accent.)
> > > use of XeTeX's features:
> > >
> > > large fonts and a larger number of families;
> > >
> > > font mappings attached to a font
> >
> > The former: definitely
>
> Oh dear, that was not meant as a choice: you need both!!
:)
Well, I think it's correct at this stage to say that *WE* need both.
If it weren't for backwards compatibility concerns, I wouldn't have a
problem with ditching \mathbf in favour of the direct char names
\bfA--\bfz, etc.
> > (although more families are just icing on the
> > cake; we could get by with less in an ideal world).
>
> Maybe but that is not necessarily sensible. I think that using
> families well is precisely what t is all about, not changing mathcodes
> all the time.
Yes, I agree. But my comment above still stands.
> Yes: the more I think about it the more i think that the LaTeX idea of
> an imutable LICR is even more important for math characters than for
> the variable text encodings for which it was invented.
>
> Then the glyphs are simply slected by choosing a font and font-mapping
> (the latter being needed because some `modern fonts' can only be
> accessed through Unicode stings, it matters not a jot to anyone but
> that font what strings are used.
With this in mind, how to do you propose handling \mathbf and friends?
Something l daren't think about coding up?
\mathbf{A+B}
-> \mathbf{A}\mathbf{+}\mathbf{B}
-> \bfA+\bfB
> Good, Ok: but what about the 8-bit fonts?
>
> > For the latter: I hope not! That will be a nightmare.
>
> Not at all: use font mapings for this, it's the only way (and someone
> is going to produce screwed up font encodings sometime, bound to.
Oh, I hadn't thought about that. This will mean a different font
encoding for every 7-bit maths font, but no matter.
> But what about compatibility hen, and using glyphs that don't exist in
> My Big Font?
Jonathan's philosophy, which I don't disagree with, has been to
largely disregard non-unicode fonts and to say "well, it's time to
move on". I'm hoping that the TeXGyre project will produce some nice
tools to easily create unicode (maths) fonts such that this doesn't
become too much of a problem.
People will always be free not to use unicode-math and continue to use
their 8-bit Euler, Fourier, and Lucida maths fonts.
I do acknowledge that we might be able to do more to accomodate those
fonts, however. That comes back to what unicode-math actually is
designed to do.
> And, especially for Ross and me, all math coding (at least all 7-bit
> stuff) should be common to all TeX's and to the many other applications
> that now also use it.
Let's get unicode-math working before I start thinking about
unicode-compatibility-math :) But yes, I see no reason that the
*interface* of unicode-maths should be dependent in any way on either
XeTeX or the fact that it's using unicode fonts.
Will
P.S. I will be away over the next two weeks walking around New
Zealand. As much as I would like to stay home and work on this ... :)
Ross Moore
On 01/12/2006, at 5:54 AM, Chris Rowley wrote:
>
>
> Will
>
>
>>
>>> input of unicode strings in math mode
>>
>> Yes, but I'm less interested in that for now (it's nice, but not
>> necessarily convenient).
>>
>
> Oh, I am surprised by that!!
Me too.
In my opinion, support for direct unicode (UTF-8 and UTF-16) input
of mathematics is vital.
It will become the primary means whereby people input mathematics
into word-processing documents --- not just TeX-based ones ---
by copy/paste/edit using other documents as the initial source.
This is similar to what has happened over the past 20+ years with
normal text input, once it became common to have multiple active
windows, and a Clipboard system to facilitate the transfer.
Prior to this it was still possible, using vi or emacs (say),
to do this by saving to temporary files and importing those.
The difference for us now is that it will be (and to some extent
already is!) possible to copy from PDF and Word documents,
and paste into the source for XeLaTeX. Of course some editing
is still required to declare the math-mode, and to declare the
need for vertical positioning of super/sub-scripted material.
>
>>> use of XeTeX's features:
>>>
>>> large fonts and a larger number of families;
>>>
>>> font mappings attached to a font
>>
>> The former: definitely
>
> Oh dear, that was not meant as a choice: you need both!!
Yes.
>
>> (although more families are just icing on the
>> cake; we could get by with less in an ideal world).
>
> Maybe but that is not necessarily sensible. I think that using
> families well is precisely what t is all about, not changing mathcodes
> all the time.
Let's see if we are thinking the same way about this.
The various "families" consisting of different type-faces:
italic, roman, bold-faced, sans-serif, etc.
may be required on the basis of their visual effect only,
not on the kind of meaning attached to the symbol within the
particular context.
e.g., use of bold-face or sans-serif when creating a poster,
simply because it is more readable when viewed from a distance.
Suppose I have a document that was setup as a poster, using a
bold-face presentation of what is really just ordinary mathematics.
I wish to copy the maths, and reuse it in a different context;
perhaps even a report on this poster, or in a new paper inspired
by that work.
Ordinary math in the poster uses a bold-math-X which might have
been done with the character U+1D47F (𝑿) ,
instead of U+1D44B (𝑋) set in a bold-faced font.
In my new context, of an abstract or review say, the character
need not --- rather, *should not* --- appear in bold face.
One way to achieve this is when copy/pasting into a (La)TeX
document, the raw character is made active, having been \let to
\mitX or \mbfitX (or whatever names).
This allows macro-programming to determine within the new context
which symbol will actually be shown in the resulting PDF.
An alternative to making the characters active is to have new
math-families, with the ability to easily specify fonts and
font-mappings for these families.
Thus something input as U+1D47F results in U+1D44B (with
appropriate spacing) directly as a result of the font-mapping.
The latter is perhaps the easiest way, when there are
a lot of symbols that need to be treated similarly.
This still allows making some particular symbols active,
when there are deviations from the general pattern.
>> The latter: they certainly make things easier, but it's not clear
>> that
>> it's the only/best solution to the problems we're facing.
>
> I think it may well be the key. I shall clarify this next week.
It's been more than a week.
Does the above help in this, or just muddy the waters?
>
>>
>>> So where/how does support for ascii input work?
>>
>> Everywhere. I've been thinking, and I realised I've got everything
>> backwards. I should be trying to emulate LaTeX first
>
> Yes: the more I think about it the more i think that the LaTeX idea of
> an imutable LICR is even more important for math characters than for
> the variable text encodings for which it was invented.
Yes; a concept needs an associated name.
On the other hand, I can see that variations in the font-face
used to present mathematics can cause a problem here:
-- does the different face mean a different meaning ?
-- or is it just being used as a presentational device ?
Some weeks back I suggested that all the Plane-1 math characters
be made \active . If instead there were alternative math-families
which give flexible control over their output, then that would
probably be sufficient.
> Then the glyphs are simply slected by choosing a font and font-mapping
> (the latter being needed because some `modern fonts' can only be
> accessed through Unicode stings, it matters not a jot to anyone but
> that font what strings are used.
Sounds fair enough to me.
>
>
>>> Must all fonts used be large?
>>
>> By "large" do you mean "implement a mostly complete subset of
>> mathematical unicode"?
>> That is, do you *not* mean "fonts don't follow unicode for their
>> glyph
>> slots"?
>>
>
> I mean not 8-bit (but probably something like whta you say also.
>
>> For the former: definitely not; support already exists partially for
>> this in unicode-math.
>
> Good, Ok: but what about the 8-bit fonts?
>
>> For the latter: I hope not! That will be a nightmare.
>>
>
> Not at all: use font mapings for this, it's the only way (and someone
> is going to produce screwed up font encodings sometime, bound to.
Of course.
TeX needs to be sufficiently flexible that it can take arbitrarily
coded input and produce properly coded output.
I see my work on CMAP resources, for the /ToUnicode property of a
font dictionary within PDF documents, as contributing to this idea.
That is, the font may be AMSa or AMSb, but when you copy/paste from
PDF the result ought to be either Unicode, or (La)TeX macro names.
>
>>> Must all fonts used via XeTeX's extended math commands be large?
>>
>> I'm not sure what you mean by this.
>>
>
> Need to ask Jonathan.
I'd guess this means something like:
"will it work already with AMSa and AMSb" ?
>
>>> Will XeTeX ever look up a classic \mathcode?
>>
>> I think this relates back to my answer above "That will be a
>> nightmare".
>> If it does, unicode-math should have nothing to do with it.
>
> But what about compatibility hen, and using glyphs that don't exist in
> My Big Font?
If there's no \mathcode then how does it know what spacing to use,
for relations, operators and fences, etc. ?
>
>>> Should there be an internal form (like LICR's) that is interoperable
>>> with classic TeX?
>>
>> I think not, unless pdfTeX evolves in a significantly different
>> direction than XeTeX and yet we want to try and make them as
>> compatible
>> as possible. I mean, you could develop newmath further (that 8-bit
>> math
>> encoding) and organise some sort of interoperability, but it'd be
>> thankless work, I think.
>>
>
> That was not my pint:L anything at the LaTeX level must be
> indpependent
> of XeTeX and its names.
Agreed.
>
>> I might have misunderstood this point; what situations were you
>> thinking of?
>
> Needing to use classic mth fonts (becuase people like the look of
> them,
> if nothing else) that will remain for sometime in 8-bit form only.
e.g., AMSa and AMSb .
>
> And, especially for Ross and me, all math coding (at least all 7-bit
> stuff) should be common to all TeX's and to the many other
> applications
> that now also use it.
Yep. ( and also those that abuse it! :-)
>
>
> chris
>>
>> --
>> Will
Cheers,
Ross
------------------------------------------------------------------------
Ross Moore ro...@maths.mq.edu.au
Mathematics Department office: E7A-419
Macquarie University tel: +61 +2 9850 8955
Sydney, Australia 2109 fax: +61 +2 9850 8114
------------------------------------------------------------------------
Will Robertson
>>>> use of XeTeX's features:
>>>>
>>>> large fonts and a larger number of families;
>>>>
>>>> font mappings attached to a font
>>>
>>> The former: definitely
>>
>> Oh dear, that was not meant as a choice: you need both!!
>
> Yes.
Hmmm. I don't necessarily want to exclude luatex/pdftex from the
discussion, which will support the former of those choices but not
the latter.
> Ordinary math in the poster uses a bold-math-X which might have
> been done with the character U+1D47F (𝑿) ,
> instead of U+1D44B (𝑋) set in a bold-faced font.
This would be wrong, in my opinion.
It should be cases like this that we wish to avoid.
Let me just clarify: you want to provide for situations when users
input the character for "bold math italic X" and yet receive "math
italic X" instead? That seems madness to me. If the character "means"
$X$, then it should definitely use use the character U+1D44B in a
bold font.
We need, then, to make it necessarily obvious to the user the
difference between bold for semantics and bold for other purposes.
LaTeX does this well at present with \mathversion, but less so with
\boldsymbol and \mathbf (how often must users ask about \mathbf and
uppercase greek letters?).
***
Now that I'm back from holiday and may have some time over Christmas
to implement some of these thoughts, I hope to actually get somewhere
with a stable release or two, and a proper roadmap to solidify my
direction.
Thanks for all of your time with this,
Will
Ross Moore
On 15/12/2006, at 1:14 PM, Will Robertson wrote:
>>> Oh dear, that was not meant as a choice: you need both!!
>>
>> Yes.
>
> Hmmm. I don't necessarily want to exclude luatex/pdftex from the
> discussion, which will support the former of those choices but not
> the latter.
At present this may be so; there's no reason why that cannot
change in future. There is much work going on here.
We need to devise techniques that are *flexible*.
>> Ordinary math in the poster uses a bold-math-X which might have
>> been done with the character U+1D47F (𝑿) ,
>> instead of U+1D44B (𝑋) set in a bold-faced font.
>
> This would be wrong, in my opinion.
I agree that it is wrong, at least in theory. But that does not
mean that it will not occur in practice.
> It should be cases like this that we wish to avoid.
>
> Let me just clarify: you want to provide for situations when users
> input the character for "bold math italic X" and yet receive "math
> italic X" instead? That seems madness to me. If the character "means"
> $X$, then it should definitely use use the character U+1D44B in a
> bold font.
People often say are write things that they do not actually mean.
Language is flexible enough that the context gives sufficient hints
and the right meaning is actually inferred by the listener.
When people submit papers and abstracts to journals and conferences
there will be many errors in what they submit. The editor should
not need to go back to individual authors to ask for clarification.
Instead the system should be designed with sufficient flexibility
that such errors can be identified and fixed easily, preferably
without too much editing of the supplied source.
Thus my example was designed to be interpreted as:
"people are going to get it wrong, but in understandable ways.
We need to have adequate mechanisms to be able to program
automatic fixes."
Recognition of such practicalities should be used to help us
make the design choices that best broaden the scope of how
the new system can be used.
>
> We need, then, to make it necessarily obvious to the user the
> difference between bold for semantics and bold for other purposes.
Sure -- within the documentation!
But such limitations should not be "programmed-in".
> LaTeX does this well at present with \mathversion, but less so with
> \boldsymbol and \mathbf (how often must users ask about \mathbf and
> uppercase greek letters?).
> Now that I'm back from holiday and may have some time over Christmas
> to implement some of these thoughts, I hope to actually get somewhere
> with a stable release or two, and a proper roadmap to solidify my
> direction.
I think my thrust is that:
a. we need more math-families than TeX's original 8.
(The TeXBook uses terminology math-"class" rather than
math-"family". But the concept seems to be the same.)
b. face-families, such as \mathbb , \mathrm, etc.
are all currently Class 0 (ordinary).
These are candidates for extra families, so are
really sub-families of `Ordinary'.
c. such extra families are easily mis-used, so the
programming to handle them needs extra flexibility,
similar to what is available in text-mode.
> Thanks for all of your time with this,
>
> Will
Now that classes and exams are over, I'll have more time too.
Cheers,
Ross
BTW, do any of you have students, or know of any, who are interested
in doing a PhD in this kind of work ?
My university may well be able to offer a scholarship to come here,
or to work in collaboration with another institution.
Will Robertson
> Recognition of such practicalities should be used to help us
> make the design choices that best broaden the scope of how
> the new system can be used.
Okay, sure. I have no problem with this angle, then.
> BTW, do any of you have students, or know of any, who are interested
> in doing a PhD in this kind of work ?
> My university may well be able to offer a scholarship to come here,
> or to work in collaboration with another institution.
Yikes. I'd do a *2nd* PhD if I could.
Possibly :)
But my first won't be over for a little while...
Will
Jonathan Kew
> I think my thrust is that:
>
> a. we need more math-families than TeX's original 8.
> (The TeXBook uses terminology math-"class" rather than
> math-"family". But the concept seems to be the same.)
Ross, I think you're in danger of confusing us, or at least me. Knuth
uses *both* terms "class" and "family", for two separate concepts
within the program. (See p. 154.) Let's not mix or swap these terms.
There are precisely 8 "classes" defined in TeX:
0 Ordinary
1 Large operator
2 Binary operation
3 Relation
4 Opening
5 Closing
6 Punctuation
7 Variable family
where the class is used by TeX to control things like spacing within
the formula. I have not seen any compelling argument for changing the
number of classes (although as I've been largely off-line for a
while, I'm not claiming to have read and understood everything that's
been posted!).
(Any such change would presumably involve a comprehensive review of
the math layout algorithms.)
Then there are "math families", of which TeX supports 16, but the
number is essentially arbitrary. (Knuth's choice was dictated by data
structure sizes, and he didn't actually use all of them in any macro
package I've seen. I'm sure there are LaTeX packages that use more
than Plain TeX did, though.)
These families are used for two quite distinct purposes:
(a) As a way to access more than 256 symbols, in a world of 8-bit
character codes and font addressing. Thus, he separated math symbols
into three main types: alphabetic (further divided into upright and
italic), small symbols, and large/extensible symbols, and assigned
these to families 0, 1, 2 and 3 respectively. In a world of Unicode
and large fonts, this division is unnecessary.
(b) As a way to easily access stylistic variants (whether for
stylistic or semantic reasons!) of alphabetic symbols via the same
input characters, by changing a "current family" parameter. This
mechanism remains useful, I believe, and provides a clean and easy
way to implement things like \mathbb, \mathcal, etc.
Please don't use "family" to refer to Knuth's "class" of math
characters.... I, at least, am likely to end up mightily confused.
> BTW, do any of you have students, or know of any, who are interested
> in doing a PhD in this kind of work ?
> My university may well be able to offer a scholarship to come here,
> or to work in collaboration with another institution.
Really? That's kind of tempting..... :)
JK
Ross Moore
On 16/12/2006, at 4:37 AM, Jonathan Kew wrote:
>
> On 15 Dec 2006, at 3:05 am, Ross Moore wrote:
>
>> I think my thrust is that:
>>
>> a. we need more math-families than TeX's original 8.
>> (The TeXBook uses terminology math-"class" rather than
>> math-"family". But the concept seems to be the same.)
>
> Ross, I think you're in danger of confusing us, or at least me. Knuth
> uses *both* terms "class" and "family", for two separate concepts
> within the program. (See p. 154.) Let's not mix or swap these terms.
You are absolutely correct.
I was trying to interpret some of the remarks that Chris and Will
have made, which weren't at all clear -- at least not to me.
There's a need, I think, for us to be clear on concepts and to use
a consistent terminology. In trying to be clear, I guess instead
I've just swirled up the mess differently. At least it is forcing
us to face the issue. :-)
>
> There are precisely 8 "classes" defined in TeX:
>
> 0 Ordinary
> 1 Large operator
> 2 Binary operation
> 3 Relation
> 4 Opening
> 5 Closing
> 6 Punctuation
> 7 Variable family
>
> where the class is used by TeX to control things like spacing within
> the formula. I have not seen any compelling argument for changing the
> number of classes (although as I've been largely off-line for a
> while, I'm not claiming to have read and understood everything that's
> been posted!).
OK. I agree.
Indeed I see no reason to change anything about the structure of
TeX's model for handling mathematics, apart from extending the range
of values that some parameters may adopt.
>
> (Any such change would presumably involve a comprehensive review of
> the math layout algorithms.)
Yes. Hopefully the only changes can be limited to extending the
range of values that need to be supported.
>
> Then there are "math families", of which TeX supports 16, but the
> number is essentially arbitrary. (Knuth's choice was dictated by data
> structure sizes, and he didn't actually use all of them in any macro
> package I've seen. I'm sure there are LaTeX packages that use more
> than Plain TeX did, though.)
LaTeX seems to be open-ended on this.
But I'd guess that it cannot go beyond TeX's maximum of 16 without
causing trouble somewhere.
Yes, here it is: (from latex.ltx :)
\def\new@mathgroup{\alloc@8\mathgroup\chardef\sixt@@n}
\let\mathgroup\fam
\let\newfam\new@mathgroup
That use of \sixt@@n determines that no more than 16 usages
of \fam may occur.
For our purposes (e.g., with XeLaTeX), this should be expanded
to 256 or greater.
>
> These families are used for two quite distinct purposes:
>
> (a) As a way to access more than 256 symbols, in a world of 8-bit
> character codes and font addressing. Thus, he separated math symbols
> into three main types: alphabetic (further divided into upright and
> italic), small symbols, and large/extensible symbols, and assigned
> these to families 0, 1, 2 and 3 respectively. In a world of Unicode
> and large fonts, this division is unnecessary.
Unnecessary in principle only.
In practice I think it remains both desirable and quite practical.
For instance, with MS's Cambria font, there are in fact 4 physical
fonts.
Most of the math-symbols are in cambria.ttf (Regular)
with styled families (+ some symbols) in:
cambriab.ttf (Bold) cambriai.ttf (Italic) cambriaz.ttf (Bold Italic)
For Windows Vista, there is an extra file Cambria.TCC which contains
extra tables that affect how (and when) the other fonts are used.
There is discussion of this here:
http://www.microsoft.com/OpenType/OTSpec/otff.htm
>
> (b) As a way to easily access stylistic variants (whether for
> stylistic or semantic reasons!) of alphabetic symbols via the same
> input characters, by changing a "current family" parameter. This
> mechanism remains useful, I believe, and provides a clean and easy
> way to implement things like \mathbb, \mathcal, etc.
Absolutely.
Also, it is needed when a single font doesn't contain all of the
necessary glyphs, even if it's a STIX font. e.g. the Unicode specs
on Mathematics support states: (http://www.unicode.org/reports/tr25/)
The new repertoire enables the display of virtually all standard
mathematical symbols.
Nevertheless no collection of mathematical symbols can ever be
considered complete;
mathematicians and other scientists are continually inventing new
mathematical symbols
and the plan is to add them as they become widely accepted in the
scientific communities.
Adding such new characters by a small font containing just the new ones
will probably be a requirement, if these symbols are to be used before
a revised STIX font has been released.
Also, Cambria has quite a few symbols that do not have Unicode points
(e.g. complete alphabets of super/sub-scripted letters).
>
> Please don't use "family" to refer to Knuth's "class" of math
> characters.... I, at least, am likely to end up mightily confused.
Sure. Sorry about that.
With XeTeX and font-mappings, there is a rich new aspect to math-
families.
The same font can be used for different families, but with different
character mappings.
This is how I envisage \mathbb etc. being implemented.
Here the ordinary letters A, B, ..., Z can be mapped to Plane-1
characters, in Code2001 font say.
But there may be exceptions, such as \mathbb{P}, and C,H,Q,R,N,Z.
For these, there would be a different family using Code2000
and a character map taking these to Plane-0 characters.
This latter family need only be used to support 7x2=14 characters
(the plain and styled versions) within a XeTeX document.
With such mechanisms handling special cases, the need for lots
of families is apparent.
For this kind of thing to work, the call to use the \mathbb
family may have to cause the \mathcode of certain characters
to be adjusted, so that the correct family is used.
However, to implement and test this in its full generality,
requires direct support for Plane-1 characters.
Jonathan, can the \char and ^^^^ notation be extended to allow
5 Hex digits; e.g. ^^^^^1d54b for \mathbb{T} ?
>> BTW, do any of you have students, or know of any, who are interested
>> in doing a PhD in this kind of work ?
>> My university may well be able to offer a scholarship to come here,
>> or to work in collaboration with another institution.
>
> Really? That's kind of tempting..... :)
Yes, this is serious --- but time-limited.
That is, I need to propose that a scholarship be made available
in a specific area. The university needs to decide that this is a
suitable topic in which to offer a scholarship, which then will
be advertised. The uni wants to attract good research candidates,
so with careful crafting and a particular candidate in mind, the
scholarship can be created and offered.
COTUTEL arrangements with other universities also can be crafted.
>
> JK
Cheers,
Ross
Will Robertson
> Jonathan, can the \char and ^^^^ notation be extended to allow
> 5 Hex digits; e.g. ^^^^^1d54b for \mathbb{T} ?
I've wondered about this too. It would require (I believe) XeTeX to
use 32-bit values internally where it would currently use 16.
If this change is unfeasible, Ross, what do you think of implementing
something like how utf8 is currently handled in LaTeX; that is,
making the first 16-bit character of all higher plane chars active
and sniffing ahead with futurelet?
Cheers,
Will
Jonathan Kew
>
> On 21/12/2006, at 16:41 , Ross Moore wrote:
>
>> Jonathan, can the \char and ^^^^ notation be extended to allow
>> 5 Hex digits; e.g. ^^^^^1d54b for \mathbb{T} ?
\char already accepts the full Unicode range, up to \char"10FFFF.
^^^^ currently accepts only 4 hex digits, and therefore is limited to
Plane 0. You can give two UTF-16 code units (a surrogate pair) to
generate a supplementary-plane character this way; thus,
^^^^d835^^^^dd4b would print \mathbb{T} in Cambria Math, Code2001,
etc. But obviously that's not a satisfactory way forward.
It shouldn't be difficult to extend ^^^^ for 5- and 6-digit Unicode
values. The replacement of ^^ sequences with the corresponding
character codes happens right at the input stage; I can easily
generate the surrogate pairs at that time, so that you can use the
Unicode values directly in the input. (You can also use literal
characters like 𝕋 in the text, whether the input is UTF-8 or
UTF-16; the input code converts the encoding forms as needed.)
>
> I've wondered about this too. It would require (I believe) XeTeX
> to use 32-bit values internally where it would currently use 16.
No, I'd still use UTF-16 as the internal representation, because that
interfaces most readily with the ATSUI and ICU libraries I'm using.
The code will just have to be made aware of surrogate pairs (in more
places than currently), and treat them as a unit where necessary.
The main limitation at present is that the \catcode, \lccode,
\uccode, \sfcode, \mathcode and \delcode tables are all limited to
64K entries, and are indexed by the UTF-16 code units. So although
you can use characters like 𝕋, you don't have full control of their
TeX behavior. So what I need to do is extend these to the full
"10FFFF range, and ensure that UTF-16 surrogate pairs are also
handled in situations like \catcode`𝕋 = ..., \lowercase{𝕋...},
etc.
I hope to take a serious look at this stuff once the TL2006 release
is out of the way.
> If this change is unfeasible, Ross, what do you think of
> implementing something like how utf8 is currently handled in
> LaTeX; that is, making the first 16-bit character of all higher
> plane chars active and sniffing ahead with futurelet?
Ugh.... let's not go there. The idea of xetex is to handle this stuff
*right*! :)
JK
Ross Moore
On 22/12/2006, at 1:02 AM, Jonathan Kew wrote:
>
> \char already accepts the full Unicode range, up to \char"10FFFF.
>
> ^^^^ currently accepts only 4 hex digits, and therefore is limited
> to Plane 0. You can give two UTF-16 code units (a surrogate pair)
> to generate a supplementary-plane character this way; thus,
> ^^^^d835^^^^dd4b would print \mathbb{T} in Cambria Math, Code2001,
> etc. But obviously that's not a satisfactory way forward.
Yes; I aware that surrogate pairs work this way, to produce characters
for output. But, as I'm sure you realise, I want to be able to also
manipulate the \catcode and \mathcode and \delcode, for use with maths.
>
> It shouldn't be difficult to extend ^^^^ for 5- and 6-digit
> Unicode values. The replacement of ^^ sequences with the
> corresponding character codes happens right at the input stage; I
> can easily generate the surrogate pairs at that time, so that you
> can use the Unicode values directly in the input. (You can also
> use literal characters like in the text, whether the input is
> UTF-8 or UTF-16; the input code converts the encoding forms as
> needed.)
Right on. As the codes are input, we need to be able to get a token
and declare its *code properties...
> The main limitation at present is that the \catcode, \lccode,
> \uccode, \sfcode, \mathcode and \delcode tables are all limited to
> 64K entries, and are indexed by the UTF-16 code units.
... such as these.
> So although you can use characters like , you don't have full
> control of their TeX behavior. So what I need to do is extend
> these to the full "10FFFF range, and ensure that UTF-16 surrogate
> pairs are also handled in situations like \catcode` = ...,
> \lowercase{ ...}, etc.
>
> I hope to take a serious look at this stuff once the TL2006
> release is out of the way.
Fine. Any ideas on when that'll be achieved?
In the meantime, here's another problem that exists with XeTeX 0.995 .
A. Try processing this source code (with comments):
\documentclass{article}
\usepackage[cm-default]{fontspec} % Will's latest version
%\setromanfont[Mapping=tex-text]{Lucida Grande}
%\setromanfont[Mapping=tex-text]{Code2001}
\begin{document}
$^^^^d835^^^^dd4b$
^^^^d835^^^^dd4b
\char"1D54B
\end{document}
The PDF is blank, as it should be, with the .log saying:
This is XeTeXk, Version 3.141592-2.2-0.995 (Web2C 7.5.5)
(format=xelatex 2006.11.13) 22 DEC 2006 15:58
...
Missing character: There is no ?| ? in font cmr10!
Missing character: There is no ?~K in font cmr10!
Missing character: There is no ?| ? in font cmr10!
Missing character: There is no ?~K in font cmr10!
Missing character: There is no character number "1D54B in font cmr10!
B. Next, uncomment the line for Code2001 and re-process.
Now the PDF shows the latter 2 instances of \mathbb{T} ,
but the 1st produces a missing-character symbol, and .log message:
Missing character: There is no ?~K in font Code2001/ICU:mapping=tex-
text,!
That is, the surrogate pair has not worked within math-mode.
Maybe not surprising, as no \mathcode has been set, yet it still
accepted Code2001 as the font to use.
C. Finally, comment-out Code2001 and uncomment for Lucida Grande.
Then re-process --- disaster: bus error !!!!
I'm not going to speculate on what may be going wrong here.
I know that Lucida Grande doesn't support anything in Plane-1,
but surely the resulting error shouldn't cause such a crash.
I'm using a PowerBook Titanium, running ...
[GlenLivet:] rossmoor% uname -a
Darwin GlenLivet.local 8.8.0 Darwin Kernel Version 8.8.0: Fri Sep 8
17:18:57 PDT 2006; root:xnu-792.12.6.obj~1/RELEASE_PPC Power
Macintosh powerpc
Jonathan Kew
>>
>> So although you can use characters like , you don't have full
>> control of their TeX behavior. So what I need to do is extend
>> these to the full "10FFFF range, and ensure that UTF-16
>> surrogate pairs are also handled in situations like \catcode`
>> = ..., \lowercase{ ...}, etc.
>>
>> I hope to take a serious look at this stuff once the TL2006
>> release is out of the way.
>
> Fine. Any ideas on when that'll be achieved?
Well, TL2006 is supposed to be finished Real Soon Now... after all,
there's not much of 2006 left! So I hope I'll be able to look at this
fairly early in the new year.
> In the meantime, here's another problem that exists with XeTeX 0.995 .
> .....
The behavior with Code2001 is pretty much as expected, given that in
math mode, the characters are not actually rendered as such, they're
used to look up mathcode values -- and as already noted, that process
doesn't (yet) support supplementary-plane values. So it looks up each
UTF16 code unit separately.
Anyhow, thanks for the report. The basic answer is "that isn't
supported yet", but of course it still shouldn't crash! I wonder if
ATSUI gets upset by lone surrogates.
JK
Ross Moore
On 23/12/2006, at 10:51 AM, Jonathan Kew wrote:
>
> On 22 Dec 2006, at 5:10 am, Ross Moore wrote:
>>>
>>> So although you can use characters like , you don't have full
>>> control of their TeX behavior. So what I need to do is extend
>>> these to the full "10FFFF range, and ensure that UTF-16 surrogate
>>> pairs are also handled in situations like \catcode` = ...,
>>> \lowercase{ ...}, etc.
>>>
>>> I hope to take a serious look at this stuff once the TL2006
>>> release is out of the way.
>>
>> Fine. Any ideas on when that'll be achieved?
>
> Well, TL2006 is supposed to be finished Real Soon Now... after all,
> there's not much of 2006 left! So I hope I'll be able to look at this
> fairly early in the new year.
Great.
I'm on leave for most of January.
It'd be great to have implemented full math support,
at least with XeTeX, during that time.
Later (March-->July) I'll be on sabbatical in Europe.
It'd be great to be able to show off full Unicode math-support
at BachoTeX.
>
>> In the meantime, here's another problem that exists with XeTeX 0.995 .
>> .....
>
> The behavior with Code2001 is pretty much as expected, given that in
> math mode, the characters are not actually rendered as such, they're
> used to look up mathcode values -- and as already noted, that process
> doesn't (yet) support supplementary-plane values. So it looks up each
> UTF16 code unit separately.
OK. This is what I was guessing would happen.
But I canot find (using FontBook or the Characters palette), within
Code2001,
the glyph that gets displayed for ^^^^d835 --- that I previously
described
as a "missing character" symbol.
>
> Anyhow, thanks for the report. The basic answer is "that isn't
> supported yet",
sure.
> but of course it still shouldn't crash! I wonder if ATSUI gets upset
> by lone surrogates.
Does it not return an error value, for clean handling?
Otherwise it'd be Apple's bug, yes?
>
> JK
>
Cheers,
Ross
Jonathan Kew
>> Well, TL2006 is supposed to be finished Real Soon Now... after
>> all, there's not much of 2006 left! So I hope I'll be able to
>> look at this fairly early in the new year.
>
> Great.
> I'm on leave for most of January.
> It'd be great to have implemented full math support,
> at least with XeTeX, during that time.
It'll depend how many "real" jobs are clamouring for my attention....
>
> Later (March-->July) I'll be on sabbatical in Europe.
> It'd be great to be able to show off full Unicode math-support
> at BachoTeX.
Indeed it would -- I'd love to have something fairly complete by
then. I'm intending to go to BachoTeX again; hope to see you there!
> OK. This is what I was guessing would happen.
> But I canot find (using FontBook or the Characters palette), within
> Code2001,
> the glyph that gets displayed for ^^^^d835 --- that I previously
> described
> as a "missing character" symbol.
I assume you'd either get glyph 0 from Code2001 (probably a box), or
a fallback glyph from the LastResort font (a box with a symbol
inside, suggestive of the Unicode range, and with info in the border
if you magnify it enough).
>>
>> but of course it still shouldn't crash! I wonder if ATSUI gets
>> upset by lone surrogates.
>
> Does it not return an error value, for clean handling?
> Otherwise it'd be Apple's bug, yes?
Don't know yet, I'll need to take a look.
JK