Unicode math language and LaTeX input
Chris Rowley
rapid senility!
Without wishing to imply that it the best syntax, there are good
reasons (not listed here) to support and even privilege a syntax for
LaTeX-related Unicode maths that shares the following property with
Office2007's linear input methods:
Convert everything (not really, but the maximum that is sensible in
the TeX context) into plain Unicode character tokens __as soon as
possible__! (In Office, the things that start with a \ are all simply
input abbreviations for single Unicode crharacters but that is too far
fro LaTeX and friends).
One medium-term reason for trying to set up an ad hoc standard for
maths entry is that there is now a lot of momentum from useful people
_with resources_ to produce open source systems in which users can
believe that they are using LaTeX (and they may even have a well-
hidden TeX engine (or the even quicker jsmath engine) formatting the
maths instantaneously for them on the fly, and thus appearing directly
in their editor window, like Word2007 and TeXmacs (I think). Even
more importantly, what is actually being produced is content loaded to
the gunnels with information that supports cut-and-paste to a very
wide range of applications (including, and here I get really silly in
my speculations, a version of Star Office etc that do fomatting of a
large range of material sensibly).
Happy criticising, chris
PS:until Morten agrees with me, this message has nothing whatsoever to
do with The LaTeX3 Project.
Will Robertson
> Convert everything ... into plain Unicode character tokens __as
> soon as
> possible__!
[...]
> PS: until Morten agrees with me, this message has nothing
> whatsoever to
> do with The LaTeX3 Project.
I can't disagree with this at all.
In unicode-math (at the moment) all of the macros to symbols expand
to their unicode glyph counterparts. Most of the unicode characters
are defined in terms of \XeTeXmathcode or whatever but a few are made
active to enable \nolimits behaviour.
Now, the thing with Morten's latest project, breqn, is that ALL maths
characters are made active in order to be able to insert the correct
penalties in the right places. Integrating that with unicode-math is
a high "later on" priority for me. (It strikes me that a new feature
of XeTeX may allow this to happen without making all those characters
active, but we'll have to see if LuaTeX supports character classes or
equivalent before re-writing breqn for them I'd say.)
W
Ross Moore
Hi guys.
Sorry to have been silent for so long.
After a sabbatical in Europe, I've had a very heavy
teaching semester. But that is now coming to an end,
and it's time to get back into this new math-support
stuff.
Sorry if my comments below sound a bit rusty.
On 03/11/2007, at 3:23 AM, Will Robertson wrote:
> On 02/11/2007, at 20:02 , Chris Rowley wrote:
>
>> Convert everything ... into plain Unicode character tokens __as
>> soon as
>> possible__!
.... Hmm. since multiple fonts may be involved in the output, I'm not
so sure that this is best.
Certainly convert to something standardised, as soon as possible.
However, that may not be a single character token.
> [...]
>> PS: until Morten agrees with me, this message has nothing
>> whatsoever to
>> do with The LaTeX3 Project.
>
> I can't disagree with this at all.
> In unicode-math (at the moment) all of the macros to symbols expand
> to their unicode glyph counterparts. Most of the unicode characters
> are defined in terms of \XeTeXmathcode or whatever but a few are
> made active to enable \nolimits behaviour.
>
> Now, the thing with Morten's latest project, breqn, is that ALL
> maths characters are made active in order to be able to insert the
> correct penalties in the right places.
This can be done using \mathcode <character-code> = "8000 , yes?
This way the character is non-active in text-mode, but active in math.
It seems to me that this is going to be a vital trick, to be able to
use different fonts for the output of different kinds of symbols.
Of course it means that you still need to have delimiters for the
switches into and out of math-mode.
So I'm thinking along the lines of:
1. The standardised form of a mathematical symbol is a macro-name.
2. Input can be either a unicode character or macroname.
3. Input of a unicode character can have different results,
according to whether it is done in math-mode or not.
This is controlled by packages and macro definitions.
e.g. I may input using unicode characters, which I can see in
my editor using Code2000, say.
But I want my mathematics typeset with Stix fonts.
Also, I may be writing a document describing how to
typeset mathematics, so need to explain the input method.
Thus I need a verbatim-like representation of the input,
which will be typeset in a non-Stix font (e.g. Code2000).
So for the non-Stix I'll use text-mode with \font = Code2000
but for the proper maths I'll use $...$ resulting in
\font = ...Stix-whatever... appropriate to each character.
Supposing that I have a single font which does mathematics
"correctly" and "completely" --- will such a thing ever exist?
Then the result from input in text-mode and math-mode way well
be identical --- this would be the best of all possible worlds.
But reality is different.
It seems to me that having the standard representation of
a math-symbol to be the unicode character itself would
only work in this ideal situation.
Rather restrictive and currently impractical, yes?
(But quite appropriate for the Face-book situation.)
This leaves open the question of what should happen when the
macroname for a math-character is input in non-math mode.
Should it
a. give an error/warning message (as TeX does now); or
b. switch to math-mode silently
(dangerous --- where will math-mode finish ?); or
c. behave like \ensuremath for this character;
d. do something else ????
> Integrating that with unicode-math is a high "later on" priority
> for me. (It strikes me that a new feature of XeTeX may allow this
> to happen without making all those characters active, but we'll
> have to see if LuaTeX supports character classes or equivalent
> before re-writing breqn for them I'd say.)
>
> W
Consider the round-trip:
my input source, in Unicode (readable with Code2000, say)
---> PDF via XeTeX (or LuaTeX)
---> copy/paste into a (UTF8 or UTF16) text-file
To recover my original source, I still have to insert
the math-delimiters $....$ to differentiate what is
actual mathematics from the descriptive parts about
how to typeset with math-symbols.
Pity. But there would have to be extra invisible markup
within the PDF (but which can be recovered with copying)
in order to avoid this.
Cheers,
Ross
------------------------------------------------------------------------
Ross Moore ro...@maths.mq.edu.au
Mathematics Department office: E7A-419
Macquarie University tel: +61 +2 9850 8955
Sydney, Australia 2109 fax: +61 +2 9850 8114
------------------------------------------------------------------------
Will Robertson
Indeed, I haven't thought about unicode maths for several months.
On 07/11/2007, at 6:12 , Ross Moore wrote:
>>> Convert everything ... into plain Unicode character tokens __as
>>> soon as possible__!
>
> .... Hmm. since multiple fonts may be involved in the output, I'm
> not
> so sure that this is best.
Don't forget that \mathcode's can assign different fonts to different
symbols. unicode-math.sty currently uses the "macros expand to
unicode" and it supports multiple fonts used for different unicode
ranges.
>> Now, the thing with Morten's latest project, breqn, is that ALL
>> maths characters are made active in order to be able to insert the
>> correct penalties in the right places.
>
> This can be done using \mathcode <character-code> = "8000 , yes?
>
> This way the character is non-active in text-mode, but active in math.
Yes, but I'm not looking at it yet.
Perhaps I should.
> This leaves open the question of what should happen when the
> macroname for a math-character is input in non-math mode.
> Should it
> a. give an error/warning message (as TeX does now); or
> b. switch to math-mode silently
> (dangerous --- where will math-mode finish ?); or
> c. behave like \ensuremath for this character;
> d. do something else ????
As it currently stands a maths character in text will typeset it in
the current font if possible but otherwise do a "missing glyph"
warning. I don't think there's a good solution to this problem,
besides maybe disallowing it completely.
In the same vein, if you're typesetting Russian, should you be not
allowed to insert some Japanese text without any markup? A new
feature of XeTeX allows invisible markup between "script switching"
that can be used for this purpose, and this could also be used for
maths.
I hope LuaTeX adopts this feature; it's very useful. (Ask me again
about it later if you're interested.)
Will
Ross Moore
On 07/11/2007, at 8:09 AM, Will Robertson wrote:
> Hi Ross,
>
> Indeed, I haven't thought about unicode maths for several months.
>
> On 07/11/2007, at 6:12 , Ross Moore wrote:
>
>>>> Convert everything ... into plain Unicode character tokens __as
>>>> soon as possible__!
>>
>> .... Hmm. since multiple fonts may be involved in the output,
>> I'm not
>> so sure that this is best.
>
> Don't forget that \mathcode's can assign different fonts to
> different symbols.
... via the math-family, right?
Yes, of course; see I'm a bit rusty.
Setting the math-code is generally a very good approach.
But it only makes sense when you are already in math-mode,
and it forces appropriate spacing, etc.
In effect it assumes that you are doing math, not just
talking/writing about the symbol and/or how it is to be used;
let's call this "notation" mode, say.
(Note that MathML recognises notational as different to
functional form, for a mathematical expression.)
OK, if it requires a macro or environment to declare that
we are in notation-mode, as it should do, then this can change
how the math-symbols are handled while in this mode.
(This is like what \verb currently does, but needs to be
extended to cover many more characters.)
> unicode-math.sty currently uses the "macros expand to unicode" and
> it supports multiple fonts used for different unicode ranges.
I'm reading this now; looks pretty comprehensive.
Some comments:
The 4 styles are a good thing: iso/tex/french/literal .
Why is [iso] the default ? not [tex] ?
It's surprising that [french] is so different to [iso] !!!
Is the purpose of [literal] to preserve the difference between
italic and upright Latin/Greek, when the input source contains
either (or both) style?
Whereas any "non-literal" option would mean to view them
as having identical meaning, so results in identical output?
What do the constants 25 and 26 mean here?
(Should there be macros expanding to these two character strings?
This'd be infinitesimally more efficient than always using the
strings.)
You must have spent some time researching 'partial' and 'Nabla' !
I'm sure I've seen some versions that are more slanted than others,
but it's doubtful that the authors made a conscious choice about
whether to use an italic face or not.
Is it intentional to have non-ascii characters in this error message?
\PackageError{fontspec/unicode-math}
{The ‘#1’ font feature can only be used for maths fonts}
{The feature you tried to use can only be in commands
like \protect\setmathfont}%
The final line, installing for "Cambria Math" is surely
just a temporary default.
Clearly there'll need to be more options for other font-sets.
What about special support for:
i \imath imaginary i
e \emath base of natural logs
d differential-d
The latter "should" use an upright face, though most
authors ignore this typographic nicety.
Note that systems like Mathematica have special characters
for each of these symbols.
>
>>> Now, the thing with Morten's latest project, breqn, is that ALL
>>> maths characters are made active in order to be able to insert the
>>> correct penalties in the right places.
>>
>> This can be done using \mathcode <character-code> = "8000 , yes?
>>
>> This way the character is non-active in text-mode, but active in
>> math.
>
> Yes, but I'm not looking at it yet.
> Perhaps I should.
>
>> This leaves open the question of what should happen when the
>> macroname for a math-character is input in non-math mode.
>> Should it
>> a. give an error/warning message (as TeX does now); or
>> b. switch to math-mode silently
>> (dangerous --- where will math-mode finish ?); or
>> c. behave like \ensuremath for this character;
>> d. do something else ????
>
> As it currently stands a maths character in text will typeset it in
> the current font if possible but otherwise do a "missing glyph"
> warning. I don't think there's a good solution to this problem,
> besides maybe disallowing it completely.
I'm against disallowing it.
Warning messages are good, but jobs should not stop
unless there is a compelling reason to do so.
I want to be able to copy/paste large chunks of source and
look at a (partially) typeset result as quickly as possible.
Let the .log tell me that there are minor problems, that I
can then choose to fix at my own leisure, assigning priority
to those that occur most frequently, say.
>
> In the same vein, if you're typesetting Russian, should you be not
> allowed to insert some Japanese text without any markup? A new
> feature of XeTeX allows invisible markup between "script switching"
> that can be used for this purpose, and this could also be used for
> maths.
That sounds useful.
>
> I hope LuaTeX adopts this feature; it's very useful. (Ask me again
> about it later if you're interested.)
Sure.
Some examples of this in practice would be nice to see.
>
> Will
Will Robertson
Keep in mind that unicode-math.sty is very "alpha"-quality software
and isn't ready to be made public :) but it does contain some of my
thinking. It desperately needs a re-visit but I probably won't get
much time before December.
On 11/7/07, Ross Moore <ro...@ics.mq.edu.au> wrote:
>
> OK, if it requires a macro or environment to declare that
> we are in notation-mode, as it should do, then this can change
> how the math-symbols are handled while in this mode.
Yep; for now let's just consider characters inside maths mode.
> The 4 styles are a good thing: iso/tex/french/literal .
> Why is [iso] the default ? not [tex] ?
I didn't look into it at the time, but whichever output is more common
in the publishing world today should be chosen as the default. If that
is indeed TeX, that's fine too.
> It's surprising that [french] is so different to [iso] !!!
:)
> Is the purpose of [literal] to preserve the difference between
> italic and upright Latin/Greek, when the input source contains
> either (or both) style?
Indeed, you got it in one.
> Whereas any "non-literal" option would mean to view them
> as having identical meaning, so results in identical output?
Yes. After some thought this seemed to be the best compromise.
> What do the constants 25 and 26 mean here?
That's the length of the alphabets; it's a shorthand so I can set the
codes for an entire alphabet (or just a range of adjacent characters)
at once.
> (Should there be macros expanding to these two character strings?
> This'd be infinitesimally more efficient than always using the
> strings.)
The code is terribly, terribly, inefficient and needs a lot of work in
general :)
> You must have spent some time researching 'partial' and 'Nabla' !
> I'm sure I've seen some versions that are more slanted than others,
> but it's doubtful that the authors made a conscious choice about
> whether to use an italic face or not.
Not really any research, actually, just logic (so I might be wrong).
It came about simply from the different definitions in unicode
(perhaps an area where the different shapes AREN'T supposed to be
represented by different unicode code points) and how I would think
they should be used.
> Is it intentional to have non-ascii characters in this error message?
Probably, since XeTeX doesn't mind, but I can understand your
reluctance. I'm not too fussed either way.
> The final line, installing for "Cambria Math" is surely
> just a temporary default.
> Clearly there'll need to be more options for other font-sets.
Yes; I was hoping the default could be the STIX fonts and have them
distributed alongside the package, but that's not an option at the
moment.
> What about special support for:
>
> i \imath imaginary i
> e \emath base of natural logs
> d differential-d
Have you seen the cool package? Besides the name and some of the
implementation, I really really like it. It defines stuff like this
and a whole lot more, and I'd consider adding a few such things to
unicode-math. (Especially if they have their own codepoints, which I'm
not sure of at this stage -- even with the Mathematica fonts.)
Will
Apostolos Syropoulos
>
> > The final line, installing for "Cambria Math" is surely
> > just a temporary default.
> > Clearly there'll need to be more options for other font-sets.
>
> Yes; I was hoping the default could be the STIX fonts and have them
> distributed alongside the package, but that's not an option at the
> moment.
I have just downloaded the STIX fonts and I have noticed that they do not
include a MATH table. FontForge shows that they include a "bare-bones" table
something that is verified by otfinfo that reports that each font
contains a MATH table
that is 242 bytes long. Note that the corresponding MATH table of
CambriaMath is
25662 bytes and that of Asana-Math is 9980 bytes.
A.S.
--
Apostolos Syropoulos
366, 28th October Str.
GR-671 00 Xanthi, GREECE
Web-page at http://obelix.ee.duth.gr/~apostolo
Blog at http://asyropoulos.wordpress.com/
Jonathan Kew
> It strikes me that a new feature of XeTeX may allow this to happen
> without making all those characters active
You're referring to character classes and inter-character token
insertion, right? I don't think that'll help you here, as that
feature operates only in the context of building a horizontal list
(e.g., for a paragraph), not a math list. Math is very different from
text; I haven't tried to think about whether it could be applied there.
JK
Will Robertson
It was only a stray thought :) If you used character classes would
you need a math mode? Well, no doubt I'm simplifying, but it's
interesting to consider...
Cheers,
W
Ross Moore
On 09/11/2007, at 11:58 PM, Will Robertson wrote:
> On 09/11/2007, at 8:33 , Jonathan Kew wrote:
>
>>> It strikes me that a new feature of XeTeX may allow this to happen
>>> without making all those characters active
>>
>> You're referring to character classes and inter-character token
>> insertion, right? I don't think that'll help you here, as that
>> feature operates only in the context of building a horizontal list
>> (e.g., for a paragraph), not a math list. Math is very different from
>> text; I haven't tried to think about whether it could be applied
>> there.
>
> It was only a stray thought :) If you used character classes would
> you need a math mode?
If there is no math-mode then the \mathcode settings, and subsequent
consequences,
would never be invoked ...
> Well, no doubt I'm simplifying, but it's interesting to consider...
... so you would need to transfer all this math stuff into the new
character
classes that you define for math symbols --- unless you could rely on
the font
doing it correctly by itself.
But then you would not be able to easily use mathematical characters
by themself,
without using math-spacing conventions. Maybe that's OK; I'm really
not sure.
Doing away with math-mode switches is a logical nightmare, that requires
complicated parsing of whole sentences to get the correct meaning.
Consider a phrase like:
a \sin\theta term
Is this: $a\sin\theta$ term ?
or is it: a $\sin\theta$ term ?
You need to know what was coming before it to make the correct choice.
It should be quite easy to construct examples which require knowledge
of the actual mathematical meaning embodied in earlier sentences or
formulae, to get it right. This is surely well beyond what we want
from a typesetting program.
>
> Cheers,
> W
Will Robertson
> Doing away with math-mode switches is a logical nightmare
Yes, it was only a silly idea of mine.
If you ever do consider creating hooks for inserting code between
maths characters of different math classes, Jonathan & Taco, have a
chat with Morten to see how breqn has been implemented. It'd be nice
to not have to make all of the maths characters math-active in order
to implement the algorithm.
Cheers,
Will